A development program for junior faculty submitting National Institutes of Health grant...

Introduction

The Department of Medicine at Baylor College of Medicine has over 100 active faculty researchers in 20 Sections and Programs. The Department's Pre-Award Research Office is a separate area of departmental administration providing, among other things, information to new

faculty and staff on pre-award services, guidance in identifying funding opportunities, budgetary and administrative review, communication on research issues, and educational sessions. Common sense and good grantsmanship would ensure that all grants submitted by junior faculty were reviewed carefully by the section/program chiefs, mentors, and collaborative faculty. However, this was not being consistently done, partially due to vacancies in the section chief positions. Therefore, in an effort to improve the NIH "hit" rate, in July 1998 Dr. Andrew I. Schafer (1), Chair, and Dr. Glenn Cunningham, Vice Chair for Research, introduced a departmental-wide, mandatory program to the section/program chiefs at their monthly executive faculty meeting; the program was initiated later that year. Dubbed the "junior faculty review process," the program involved an internal and external review of new and competing NIH research (R01only) and career (K) proposals submitted by instructors and assistant professors, regardless of their age or prior/current holding of an NIH grant. This process was intended to supplement, not supplant, the section/program review. The junior faculty were still expected to have mentors and section/program chiefs review their proposals.

Guidelines

The guidelines were as follows: two months before the NIH standard submission deadlines, a good draft of the application was to be submitted to our office. Because of time considerations, junior faculty could apply to RFAs and RFPs without going through the internal review. Competing applications would also go through the full departmental review process. Two reviews would be performed. One would be a scientific review, with the applicant choosing a senior investigator outside his or her section/program, since the senior investigators within the section/program should already be reading the proposals. The reviewer could be anyone, giving applicants the opportunity to contact someone well known in their field. Pre-Award Research Office personnel would contact the reviewer and explain our purpose and process. The other review would be a readability review, involving a more senior member of the department whom we had chosen, again not in the applicant's section/program. Both reviewers should return the marked-up proposal within two weeks from receipt; local reviewers were encouraged to meet with the junior faculty. Since many of the junior faculty did not speak English as a first language, if any of the reviewers felt that the proposal needed extensive English editing, they were to notify us and we would provide an editor.

The reviewers would be paid $150 for the review (subsequently increased to $200). In the event the reviewer's suggestions resulted in an application's being extensively revised, the reviewer could request to see the proposal after the changes had been made. We would then pay an additional $100 for the second review.

Applicants were encouraged to meet with a biostatistician on our staff if they did not have the requisite expertise. The staff biostatisticians were salaried; therefore any investigator could use their services without charge in the application stage. Two years into the process, after the biostatistical group disbanded, our office began paying a biostatistician to review the proposals.

Implementation

We sent a letter and a page explaining the procedures to every instructor and assistant professor in the department and the administrative assistants of each section/program. A former section chief interested in mentoring agreed to be the readability reviewer for several proposals each cycle. We made a list of all associate professors and professors in the department who were involved in research, including those with secondary appointments in our department, since they would be potential readability reviewers, and sent a letter to all of the potential readability reviewers explaining the program and enlisting their support.

When proposals involved in the process arrived at our office, we immediately sent them (either by hand-delivery or courier service) to all reviewers with a letter explaining why we had implemented the process; an instruction sheet tailored to whether they were doing the readability or scientific review; and for those outside of the institution, a contractor form to sign for payment.

Information regarding the applications was stored in a Microsoft Access database. Items included 1) faculty information--the targeted submission date, faculty name, identification number, and section/program; 2) reviewers' information--name, date we first contacted them, date the proposal was sent to them, date it was received back, how much they were paid for the review, and how payment was made (salary supplement, check, or unrestricted account); and 3) NIH information--proposal title, and eventually, proposal number, priority score and percentile.

Initial Roadblocks

The issue of whether the department could refuse to allow submission of a proposal that did not comply with the review process was discussed with the Office of Research (OOR) at the College. Since all proposals submitted by the College must have a Department Chair (or designee) approval, the OOR agreed that they would not sign a proposal which the Department had not approved. The Department has not had to withhold a proposal from submission. On the few occasions where a proposal was submitted without the formal review process, there was some legitimate reason.

Many junior faculty were not happy with the new policy and made statements such as "It's mildly irritating when one realizes that certain senior faculty will continue to wait to the last minute," and "I have received 12 years of NIH funding so please explain why someone needs to hold my hand while I apply for my next one?" But the Chairman had decided that all junior faculty would participate.

Ongoing Issues

Problems often arose in finding readability reviewers. Although the section/program chiefs had initially endorsed the program, not all were enthusiastic about being a reviewer. Excuses ranged from "Use me as a last hope," to "I send my own proposals to someone else to read," to submitting their own proposal that cycle, or being a reviewer for an NIH Study Section with many proposals to review. Our resident senior reviewer tremendously eased this burden; however, in cycles where many junior faculty were submitting proposals, we were often reduced to begging for reviewers. After a shipping error sent a proposal to the wrong continent, we changed the rules so that the scientific reviewer chosen by the junior faculty member had to be located in North America.

Although the instructions are available on the department website and we send the link to all junior faculty three times a year when we remind the faculty of upcoming deadlines, some still misinterpret. We can attribute some of these misinterpretations to the English-as-a second-language investigators; the rest are assumed to result from problems adhering to instructions.

Several sections believed that they could handle the review as well within the section. These were well-organized groups, with strong section/program leadership and biostatisticians. Therefore, within a few years the reviews for these groups were returned to them. The department has tracked their proposal success rate, although so far the number of applications is insufficient to provide statistical significance.

A year-and-a-half after the program began, we recognized that a biostatistical review could not remain optional, since the applicants were not taking advantage of the service available to them. Many, especially bench scientists, believed that their proposal did not need this input. We then required that the proposals also be submitted to our biostatisticians, along with the scientific and readability reviewers. If the biostatisticians agreed that the proposal contained nothing for them to review, the applicant was so notified. In many cases, however, the biostatisticians uncovered issues that should have been addressed. Our current biostatistician has sent applicants such tactful comments as, "Although disagreeing about the need for biostatistics in this particular proposal, an argument exists for at least a minimum attention to such matters"; and "Your proposal may not succeed or fail because of absence of biostatistical content; however, it should normally be included and would make the proposal easier to read and understand." (J.I. Thornby, personal communication, January 13, 2003)

Since the program started, just over $45,000 has been paid to the reviewers. We estimate that approximately 30% of a staff member's time is spent on the junior faculty review process, including sending reminder announcements every cycle, lining up the readability reviewers, distributing the grant applications to all reviewers, answering numerous questions from the junior faculty, and processing payments to the reviewers. The two most vexing aspects of our process recur every review cycle: the litany of reasons by the junior faculty of why the proposal can't be turned in on the review deadline and the senior faculty's reluctance to being readability reviewers.

Assessment

Four years after the start of the process, we sent a questionnaire to the junior faculty asking them to rate the reviewers. The names of all of their reviewers were listed, including the scientific ones. Even though the junior faculty members had chosen the scientific reviewers, their assessment was equally important, since someone in the future might choose the same reviewer. If a faculty member had submitted multiple applications, a separate page was used for each application. A numerical scale, using 1=excellent to 4=poor, was used for the following criteria: turnaround time of the reviewer's critique; quality of critique provided by the reviewer; quality of interaction with reviewer; overall effectiveness of the reviewer; and overall effectiveness of the review process. Additionally, the junior faculty were asked if the reviewer should be used again. Space was provided for comments and suggestions. We entered the scores into a spreadsheet and tabulated a cumulative score for each reviewer. Any reviewer rated poorly by more than one junior faculty member was deleted from the list of available reviewers.

We also sent a survey to the senior readability reviewers explaining that candid assessment would help us refine the process. We used the same rating scale for these criteria: was there enough time to review the proposal; did they think their review helped improve the quality of the proposal; if the proposal was reviewed a second time, were recommendations sufficiently incorporated; if they met with the applicant, was the interaction beneficial; was our payment to them fair; and overall effectiveness. And again, we provided space for comments and suggestions.

By receiving the NIH Summary Statements from the junior faculty, we have an opportunity to address areas that need improvement. No summary statements indicated that the proposal was poorly written (except for one very contradictory summary statement which said, "the application is poorly written" on one page and then "highly polished and well-written application" on the next page), so we think that the readability of the proposals has improved. The summary statements from the K series have been especially valuable. Comments regarding letters of recommendation not showing enough institutional commitment or not stating a plan for growing independence will prove helpful in future submissions as we review those sections more carefully.

Funding Results

During the period studied (through 2002), seven proposals were evaluated by our reviewers as not developed enough to be submitted to NIH. Of the remaining 75 initial applications submitted to NIH, 16 initial applications were funded. Of those, eight were K awards (67% of all the initial K proposals submitted), reflecting a higher success rate during this period than the national average (NIH, n.d.). The R01 rates, however, were not stellar. One can rationalize this to result from external funding factors, but regardless of how readable a proposal is, nothing will compensate for science that is not stellar.

Originally we did not include resubmissions in our review, assuming that applicants would address the concerns of the NIH Study Section and render the review unnecessary. Several of our junior faculty had, however, voluntarily requested that their NIH resubmissions undergo our review process. Three years into our review process, after reviewing our funding data, the most clear-cut results turned out to be in the category of resubmissions. Of the eight resubmitted applications that went through the departmental review process, 50% were funded, higher than the national average (NIH, 2003), with 25% unscored. Of the 11 resubmitted applications that did not go through the departmental review process, only 18% were funded, with 36% unscored. Therefore, we decided that all junior faculty resubmissions would go through an abbreviated review process. Unless the NIH Summary Statement provided evidence to the contrary (such as needing more statistical input), only the original readability reviewer would review the proposal again. Using the Summary Statement as a guideline, the reviewer would determine whether the applicant responded appropriately to the Study Section's concerns.

Conclusion

When much of the material contained in this article was presented at a recent Executive Faculty meeting, the section/program chiefs strongly favored retaining the program. However, due to recent budget and staff-reductions, we are assessing where our resources can make the most impact. Our approach may need to be refined and better targeted. Most particularly, we are looking at focusing on K applicants and the areas of specialized help--including biostatistics and English editing for those whose first language is not English.

An important payoff of the process is the collaboration that many of the junior faculty have established with their reviewers. Additionally, the junior faculty are submitting a better product, albeit not always funded. By requiring them to think more critically as they go through the reviews, we are contributing to ensuring that the junior faculty become better proposal writers. "Hit" rates, the original motivator, cannot be the only measure of the value of a process.

(1) currently Chair of the Department of Internal Medicine, University of Pennsylvania School of Medicine, Philadelphia, PA

References

National Institutes of Health Office of Extramural Research. (n.d.). Success Rates for Selected K Activities, Fiscal Years 1990-2002. Retrieved November 20, 2003 from http://grants.nih.gov/training/data/ K_awards/sld012.htm

National Institutes of Health. (2003, September 4). Success Rates for R01, R29 and R37 Applications Amendment Status, FY1990-2002. Extramural Trends, slide 12. Retrieved December 8, 2003 from http://grants2.nih.gov/grants/award/ trends/airi2003.ppt

Author's Note: The author gratefully acknowledges the contributions of Donald Marcus, M.D, and Sharon Shaw of the Department of Medicine, Baylor College of Medicine, and Joey Barr. Contact: Barbara Gordin, Department of Medicine, Baylor College of Medicine, 6550 Fannin #1423, Houston,TX 77030. Ph: (713) 793-8328. Email: bgordin@bcm.tmc.edu.

Barbara Gordin

Baylor College of Medicine