CHARACTERISTICS OF RESEARCH ACTIVITIES SUPPORTED BY ICS

NIH uses 245 activity codes to characterize the funding mechanisms it employs to fund diverse activities. The committee focused on a select number of activity codes to characterize NIH- and IC-funded autoimmune disease research activities. The committee used RePORTER activity code data to assess autoimmune disease research activities by extramural research (R and P grants); cooperative agreements (U grants); and fellowship, training, and training center activities (F, K, and T grants); and intramural research activities (Z grants) (Table 6-3). Figure 6-3 provides NIH autoimmune disease funding for R, U, and P grants for FY 2008–2020 and the average funding for these grants over that period.

TABLE 6-3

Committee-Reviewed NIH Grant Types, by Activity Code.

FIGURE 6-3

NIH autoimmune disease funding by research (R), cooperative agreement (U), and research program project and center (P) grants, FY 2008–2020 (in millions). NOTE: FY, fiscal year. SOURCE: NIH, 2021f.

Research project (R) grant funding for autoimmune disease research was highest in 2020 and lowest in 2008 and during 2013–2016. Funding for cooperative agreements (U) grants was greater than funding for research program project and center grants (P) during FY 2008–2020. Funding for cooperative agreements increased from 2008 to 2017, dropped in 2018, and began to rise again in 2019 and 2020. Funding for research programs project and centers was slightly higher in 2009–2011 and remained constant after 2011.

Training grants contribute to building the autoimmune disease research workforce and to advancing research. Table 6-4⁵ shows NIH funding for fellowships (F), research career programs (K), and institutional training programs (T). Research career program grants had higher levels of funding throughout the entire period.

TABLE 6-4

Autoimmune Disease Spending: Fellowship, Research Career, and Training Program Grant Funding per FY 2008–2020 and Average over the Period.

The committee examined the proportion of autoimmune disease spending used for training to total autoimmune disease spending and compared it to other Research, Condition, and Disease Categories (RCDC) spending categories (Table 6-5). The comparison groups are chronic diseases and are more common in women or have a similar prevalence between men and women (cardiovascular).

TABLE 6-5

Training Investment for Selected RCDC Spending Categories, FY 2020.

IC FUNDING OF GRANTS

To explore ICs’ use of grants to further autoimmune disease research, the committee examined IC funding for research (R), cooperative agreement (U), program project/center (P) and training (F, K, T) grants. Figure 6-4 and Figure 6-5 present funding of R grants categorized by ICs with the largest research budgets and smaller research budgets.

FIGURE 6-4

IC funding for research (R) grants by large ICs for FY 2020 and average FY 2008–2020 (in millions). NOTES: Large Institutes have FY 2020 actual total obligations greater than $1 billion. FY, fiscal year; IC, Institute or Center; NIAID, National (more...)

FIGURE 6-5

IC funding for research (R) grants by small ICs for FY 2020 and average FY 2008–2020 (in millions). NOTES: Small Institutes have FY 2020 actual total obligations less than $1 billion. While the Office of the Director (OD) is not an institute, (more...)

For the purposes of this report, ICs are arranged in order by budget level and have been separated into two categories: large ICs and small ICs. Large ICs have FY 2020 actual total obligations greater than $1 billion, and small ICs have actual total obligations of less than $1 billion. Among the larger ICs, NIDDK and NIAID had the highest level of funding for R grants in FY 2020. NIDDK had the highest level of average autoimmune disease research funding compared with the other ICs. Among the smaller ICs, NIAMS had a significantly higher level of funding in 2020 and average of funding for 2008–2020 for R grants to support autoimmune disease research compared to other small ICs. The OD spent an average of $10 million on autoimmune disease research grants, which was comparable to the average funding spent by other smaller ICs.

U grants support research activities that NIH considers to be complex and of high-priority research areas. Unlike independent investigator R grants, U grants require greater involvement from NIH program or scientific staff. Figures 6-6 and 6-7 show the level of funding for autoimmune disease U grants for large and small ICs.

FIGURE 6-6

IC autoimmune disease funding for cooperative agreement (U) grants for large ICs, FY 2020 and average FY 2008–2020 (in millions). NOTES: Large Institutes have FY 2020 actual total obligations greater than $1 billion. FY, fiscal year; IC, Institute (more...)

FIGURE 6-7

IC autoimmune disease funding for cooperative agreement (U) grants for small ICs, FY 2020 and average FY 2008–2020 (in millions). NOTES: Small Institutes have FY 2020 actual total obligations less than $1 billion. While the Office of the Director (more...)

Among the larger ICs (Figure 6-6), NIDDK had a significantly higher level of funding for U grants for autoimmune disease research activities in FY 2020 and over the FY 2008–2020 period. NIDDK funding levels for U grants over the period and in FY 2020 were more than double that of NIAID, the IC with the next level of U grant funding. Other large Institute funding of U grants was less than $6 million in 2020. In FY 2020, NIAMS, among the smaller ICs (Figure 6-7) had the highest level of funding for U grants ($9 million). During the FY 2008–2020 period, NHGRI had the highest average U grant funding level ($7 million).

P grants support large, multidisciplinary, and long-term research on autoimmune disease. Figure 6-8 shows P grant funding levels of at least $1 million by IC. NIAID and NIAMS had the highest funding levels for P grants and were similar, although NIAID has a larger overall research budget than NIAMS.

FIGURE 6-8

IC autoimmune disease funding for program project/center (P) grants by ICs, FY 2020 and average FY 2008–2020 (in millions). NOTES: Institutes with program/center (P) grant spending of less than $1 million in 2020 or over the 2008–2020 (more...)

ICs can play an important role in supporting the research workforce in their area of focus. Funding can help build the research workforce through fellowships (F), research career programs (K), and institutional training programs (T) grants, in addition to traditional research grants. Table 6-6 shows IC funding of training grants that support the autoimmune disease research workforce. In FY 2020, NIDDK, NIAID, and NHLBI had the highest level of funding for F grants ($2.0 million, $814,000, and $804,000 respectively). NIDDK also had the highest average funding of F grants over the FY 2008–2020 period ($1.6 million).

TABLE 6-6

Training Grant Funding for Autoimmune Disease Research by IC and FY.

Funding for K grants was higher than funding for F or T grants. In FY 2020, NIDDK had the highest level of funding of K grants ($11.3 million), followed by NIAMS ($8.2 million). Over the FY 2008–2020 period, NIDDK ($9.8 million) and NIAMS ($7.9 million) had the highest level of average funding for K grants compared to other ICs. The committee notes that OD funding for K grants was $1.1 million in FY 2020, which is more than double than the average funding over the FY 2008–2020 period.

ICs other than NIAMS relied less on the T grant mechanism to support training of autoimmune disease researchers. In FY 2020, NIAMS funded $3.9 million in T grants with an average funding level of $4.2 million over the FY 2008–2020 period. NIAID, NIDDK, and OD used this mechanism at a significantly lower funding level.

The committee reviewed NIH spending on autoimmune disease research by type of grant and type of research according to three categories: intramural, investigator-initiated, and solicited. As noted earlier, intramural research funding (Z) is for internal NIH investigator research activities, and R grant funding is the most common independent research grant mechanism. ICs use solicited research funding opportunities to support priority research areas. NIH communicates with the research community about these grant opportunities through a notice of special interest, program announcement, request for application, request for proposal, and other mechanisms. Figure 6-9⁶ shows NIH funding for autoimmune disease research for the average over the study period and FY 2020. Of the three types of grants, investigator-initiated research grants received greater than 70 percent of the funding, however, funding for each type of research in FY 2020 is higher than the average over the study period. Table 6-7 shows funding of intramural, investigator-initiated, and solicited grants by ICs.

FIGURE 6-9

NIH funding for autoimmune disease research for the average FY 2008–2020 and FY 2020. NOTES: Autoimmune disease research categories were determined using Funding Opportunity Announcement (FOA) and Combined Total Cost of Funded Autoimmune Disease (more...)

TABLE 6-7

Intramural, Investigator-Initiated, and Solicited Autoimmune Disease Research by IC, FY 2020 and Average FY 2008–2020.

Funding for intramural research was highest for NIAID and NIAMS (about $34 million each) in FY 2020 (Table 6-7).⁷ NIAID and NIDDK had the highest level of funding for investigator-initiated research ($187 million and $185 million respectively), followed by NIAMS ($86 million). NIDDK had the highest level of average FY 2008–2020 funding ($127 million per year). Of the ICs reviewed, NIDDK also had the highest level of funding for solicited research grants. NIDDK solicited research funding was 2.7 times higher than NIAID funding for this grant type.

Intramural research activities represent 17.3 percent of NIH funding for autoimmune disease research in FY 2020. A total of 50 intramural principal investigators conduct autoimmune disease research, which may include basic, translational, and clinical studies. However, the committee notes that there is a relative lack of intramural principal investigators who focus on epidemiology and autoimmune diseases. The committee searched the NIH Intramural Research Program database⁸ to identify principal investigators who conduct autoimmune disease research and principal investigators who engage in epidemiology-focused research (N=121). There are three intramural principal investigators who conduct research that overlaps both of these areas. This number is significantly lower than the number of principal investigators in the NCI intramural research program whose research focuses on both cancer and epidemiology (N=69).

RESEARCH GRANTS BY AUTOIMMUNE DISEASE

NIH introduced the RCDC in 2009 to categorize funded research. RCDC categories include a research area, specific condition, or disease area, but the categories are not exhaustive or mutually exclusive (NIH, 2021a). Funded research as reported in NIH RePORTER may have multiple RCDC disease spending categories depending on the scientific and research focus of the project. Furthermore, diseases or conditions that do not have a specific RCDC spending category (diabetes mellitus type 1, antiphospholipid syndrome, Sjögren’s disease, celiac disease, autoimmune thyroid disease, primary biliary cholangitis) must be searched for using “key terms,” including the name of the condition and all iterations of it (e.g., names roughly equivalent to antiphospholipid syndrome include Hughes’ syndrome, lupus anticoagulant, anticardiolipin, etc.).

The committee was interested in examining trends in the number of grants for autoimmune diseases of specific interest identified in Chapter 3. These diseases include inflammatory bowel disease (IBD), multiple sclerosis, type 1 diabetes, rheumatoid arthritis, systemic lupus erythematous (SLE), Sjögren’s disease, psoriasis, celiac disease, autoimmune thyroid disease, antiphospholipid syndrome (APS), and primary biliary cholangitis (PBC). Hashimoto’s disease and Grave’s disease were combined into a single “autoimmune thyroid disease” category. In addition, grants that were not associated with the committee’s specific disease categories or were not disease specific were labeled as “other autoimmune disease” throughout the analysis. RePORTER data includes funding information for each year of a grant (e.g., a 3-year R01 grant is identified as three increments of funding); the committee aggregated funding increments with the same grant number to represent one grant. In total, the committee identified and aggregated 28,148 funding increments into 8,470 autoimmune disease grants.⁹ Figure 6-10 shows number of grants by disease category for FY 2008–2020.

FIGURE 6-10

Number of grants by disease category, FY 2008–2020 (N=8,470). NOTES: Percentages in the figure sum to more than 100 percent because grants can be associated with more than one autoimmune disease. FY, fiscal year. SOURCE: NIH, 2021f.

Of the autoimmune diseases the committee examined, the four diseases with the greatest number of funded grants over the FY 2008–2020 period were other autoimmune diseases (31.0 percent), IBD (17.6 percent), multiple sclerosis (16.4 percent) and type 1 diabetes (13.2 percent). Figure 6-11 shows the total number of grants by disease category funded by year, between 2008 and 2020. The number of IBD and type 1 diabetes grants increased gradually over time; multiple sclerosis grants declined after 2009. The number of other specific disease grants fluctuated over the period.

FIGURE 6-11

Number of autoimmune disease grants by disease, FY 2008–2020 (N=8,470). NOTES: Percentages in the figure sum to more than 100 percent because grants can be associated with more than one autoimmune disease. FY, fiscal year. SOURCE: NIH, 2021f.

Autoimmune disease grants involved 98 distinct activity codes. Figure 6-12 shows the top 10 activity codes. Among these, R01 (35.9 percent) and R21 (12.4 percent) accounted for the largest number of grants. R01 grants are the standard investigator-initiated grants, and R21 grants are research exploratory/developmental grants meant to encourage the development of new research activities in specific program areas.

FIGURE 6-12

Top 10 activity codes for autoimmune disease research (N=8,470). NOTE: R01 are the standard investigator-initiated grants, R21 are research exploratory or developmental grants for new research activities in categorical program areas, R43 are small business (more...)

For R01 grants, other autoimmune disease (light purple) multiple sclerosis (dark purple), and IBD (yellow) were the most funded grants in FY 2020 (Figure 6-13). The number of grants increased for about half of the disease categories after 2016. IBD grants increased steadily over the period. Sjögren’s disease, psoriasis, celiac disease, autoimmune thyroid disease, APS, and PBC had fewer than 25 grants per year.

FIGURE 6-13

Number of R01 grants by autoimmune disease, FY 2008–2020 (N=3,042). NOTE: FY, fiscal year. SOURCE: NIH, 2021f.

Among R01 research grants, non-competing continuation research grants (R01-5) were the most funded (32.5 percent) followed by new R01-1 grants (22.3 percent). Figure 6-14 shows the number of non-competing R01 grants by disease and Figure 6-15 shows new R01 grants by disease for FY 2008–2020. Non-competing continuation grants for all disease categories decreased in 2016 as a result of an administrative accounting change (Figure 6-15). There were no clear patterns in the trends in the number of new R01 grants funded by disease categories except for the increase in IBD grants over the study period.

FIGURE 6-14

Number of non-competing continuation R01 grants by autoimmune disease, FY 2008–2020 (N=2,751). NOTES: In FY 2014 and 2015, non-competing awards were issued as extension awards to support NIH’s transition to Payment Management System subaccounts. (more...)

FIGURE 6-15

New R01 grants by autoimmune disease, FY 2008–2020 (N=1,892). NOTE: FY, fiscal year. SOURCE: NIH, 2021f.

The committee also examined the ICs designated as the funding IC for autoimmune disease grants (Figure 6-16). Among the top 10 funding ICs, NIDDK, NIAID, and NIAMS funded the largest number of grants.

FIGURE 6-16

Top 10 funding ICs by grants, FY 2008–2020 (N=8,470). NOTES: Percentages do not sum to 100 percent because the figure includes only the top 10 funding ICs and does not include grants without a funding IC listed in NIH RePORTER. FY, fiscal year; (more...)

Figures 6-17 through 6-19 show trends in autoimmune diseases by the top three funding ICs (NIDDK, NIAID, and NIAMS). Figure 6-17 shows that among NIDDK-funded grants, the disease categories with the greatest number of grants were IBD (yellow), type 1 diabetes (black), and other autoimmune disease (light purple). Figure 6-18 shows NIAID-funded grants by disease. The top three disease categories with the most grants were other autoimmune diseases (light purple), SLE (dark blue), and multiple sclerosis (dark purple). While NIAID-funded IBD grants (yellow) seemed to increase steadily since 2011, SLE and multiple sclerosis grants decreased in 2015 but have since increased. Figure 6-19 shows NIAMS-funded grants by disease. The top three disease categories with the most grants were rheumatoid arthritis (orange), other autoimmune diseases (light purple), and SLE (dark blue).

FIGURE 6-17

NIDDK (funding IC) grants by disease, FY 2008–2020 (N=2,185). NOTES: This graph represents the number of grants with NIDDK funding. NIDDK was not necessarily the sole funder of each grant. FY, fiscal year; IC, Institute or Center; NIDDK, National (more...)

FIGURE 6-19

NIAMS (funding IC) grants by disease, FY 2008–2020 (N=1,150). NOTES: This graph represents the number of grants with NIAMS funding. NIAMS was not necessarily the sole funder of each grant. FY, fiscal year; IC, Institute or Center; NIAMS, National (more...)

FIGURE 6-18

NIAID (funding IC) grants by disease, FY 2008–2020 (N=1,671). NOTES: This graph represents the number of grants with NIAID funding. NIAID was not necessarily the sole funder of each grant. FY, fiscal year; IC, Institute or Center; NIAID, National (more...)

IC Collaboration on Research Funding Opportunities

ICs can take steps among themselves to coordinate their research interests and to leverage their mutual expertise and resources.¹⁰ One measure of collaboration is the degree to which ICs engage in joint funding of specific autoimmune diseases research activities. To assess the level of collaboration among ICs, the committee examined co-funded autoimmune disease research grants for FY 2008–2020. To accomplish this, the committee used NIH RePORTER data to identify administrative ICs funding autoimmune disease research grants and corresponding funding ICs that contributed to the total cost of grants during this period.

All 14 ICs collaborated with other ICs¹¹ to co-fund autoimmune disease research activities during the period. Among the large ICs, NIAID (4.23), NHLBI (2.70), and NIDDK (2.62) had the largest average number of joint IC funding collaborations (Table 6-8).¹² These ICs jointly funded research grants with one to six other ICs in a given year and collaborated with eight to nine different ICs over the period. NIDDK collaborated consistently with at least one IC each year and with up to six ICs during a given year. Among the smaller ICs, NIAMS (3.4) had the highest average number of joint IC funding collaborations followed by NIDCR (1.10) and NCATS (1.00). NIAMS collaborated with nine different ICs over the period. NIDCR and NCATS collaborated with four and three different ICs (respectively) during the period. Most collaborations were on P, R, and U grants.

TABLE 6-8

IC Collaboration, FY 2008–2020.

RESEARCH FOCUS OF FUNDED GRANTS

The statement of task directed the committee to evaluate the NIH research portfolio with particular attention to issues such as risk factors, diagnostic tools, barriers to diagnoses, treatments, and prospects for cures. Given the enormity of the task, which would have required that the committee review the 8,470 abstracts associated with the autoimmune disease grants funded from 2008 to 2020, the committee found three ways to approach the task in a more efficient and manageable way. The first approach used research topic modeling, a type of statistical modeling to identify popular “topics” that appeared in the abstracts. In the second approach, committee members utilized RCDC spending categories and RePORTER data to determine which particular spending categories were co-occurring with the autoimmune disease spending category. In the third approach, the committee reviewed a sample of research grant abstracts to identify the primary focus of the research.

The committee, with the aid of a consultant, used latent dirichlet allocation (LDA) statistical modeling to identify research topics in the set of 8,470 NIH research abstracts related to autoimmune disease grants funded between 2008 and 2020. Topic modeling using LDA is a method of fitting a pre-determined number of topics to a set of texts. The LDA algorithm repeatedly selects and assigns words in each document to a topic and assesses the “fit” of the word to the topic by looking at what words are frequently found together across the texts (Blei et al., 2003). In the past, researchers have applied LDA modeling to scientific abstracts to analyze funding patterns and trends in research (Park et al., 2016; Porturas and Taylor, 2021). Based on the coherence score of words¹³ that tended to co-occur together in the dataset, 30 topics were determined to be the optimal number of topics that could be used to fit the data (words) in the abstracts (Bittermann and Fischer, 2018; Park et al., 2016; Porturas and Taylor, 2021). See Appendix H for more detailed information.

One of the outputs of the LDA model is a matrix that shows the proportion of topics assigned to each abstract. For example, 30 percent of the words in an abstract might belong to topic 1, and 70 percent of words in an abstract might belong to topic 2. The topic with the highest proportion within each abstract was assigned to the abstract. Topics were combined based on similarity of words within topics. Box 6-1 shows the final list of 23 topics that fit the data the best.

Box 6-1

LDA-Derived Topics Based on 8,470Autoimmune Disease Grants.

Figures 6-20 through 6-23 show the frequency of these topic assignments from 2008 to 2020. This graphic is a depiction of the popularity of the research topics in the grants over the period. Trends over time show that the research topics related to IBD, treatment/therapy, lung disease, diagnostic [tools], and imaging have trended upward in contrast to animal model, genetics, and pathogenesis. Compared to the other disease focused topics, type 1 diabetes was prevalent among grants over time (Figure 6-20). Cancer, rheumatoid arthritis, cardiovascular, and multiple sclerosis remained consistent over the time period. In Figure 6-21, immune response (innate immunity) may be disproportionately high because of its relevance to various aspects of autoimmune disease research as well as a lack of a concrete definition for innate immunity. Between 2008 and 2018, the mention of treatment/therapy in autoimmune disease grant abstracts drastically increased, but then decreased sharply (Figure 6-22). However, mention of disease progression, diagnostic, and imaging generally increased during the latter part of the decade. In Figure 6-23, centers/core project funding was variable throughout the time period, and there has been a consistent decrease in the mention of training being present in autoimmune disease abstracts, especially between 2015 and 2020.

FIGURE 6-20

LDA-generated topics for NIH autoimmune disease grants, FY 2008–2020: Disease focused topics. NOTE: IBD, inflammatory bowel disease; LDA, latent dirichlet allocation; SLE, systemic lupus erythematosus. SOURCE: NIH, 2021f.

FIGURE 6-23

LDA-generated topics for NIH autoimmune disease grants, FY 2008–2020: Administrative topics. NOTE: FY, fiscal year; LDA, latent dirichlet allocation. SOURCE: NIH, 2021f.

FIGURE 6-21

LDA-generated topics for NIH autoimmune disease grants, FY 2008–2020: Immune response related topics. NOTE: FY, fiscal year; LDA, latent dirichlet allocation. SOURCE: NIH, 2021f.

FIGURE 6-22

LDA-generated topics for NIH autoimmune disease grants, FY 2008–2020: Clinical topics. NOTE: FY, fiscal year; LDA, latent dirichlet allocation. SOURCE: NIH, 2021f.

One of the downsides of using LDA for topic modeling is that the topics must be discovered “latently” within the texts and cannot be inputted into the model algorithm. In other words, this method cannot be used to actively search for specific research topics.

RCDC Spending Categories

The second approach the committee used to evaluate the NIH research portfolio was to determine which other NIH RCDC spending categories co-occur with the autoimmune disease spending category. This can provide insight into themes within NIH’s autoimmune disease research portfolio. The definitions of the categories include all aspects of the topic, including basic, pre-clinical, clinical, biomedical, health services, behavioral, and social research, as defined by NIH scientific experts (NIH, 2020a). However, the definitions are not publicized. Table 6-9 shows the RCDC spending categories that the committee selected for analysis based on the statement of task, concepts, or conditions known to be related to autoimmune disease, and affected populations. Out of the 25 RCDC spending categories the committee analyzed, Genetics (32.1 percent), Biotechnology (23.4 percent), Prevention (22.9 percent), Pediatric (15.9 percent), and Women’s Health (12.9 percent) were the top five spending categories co-occurring with the autoimmune disease spending category in a total of 8,470 grants.

TABLE 6-9

Percent of Autoimmune Disease Grants Co-occurring with other RCDC Spending Categories (N=8,470).

PICO Portal

PICO Portal is an online tool used to review and manually categorize scientific literature. Given the large variation in the number of grants per disease, the committee decided to take random, weighted samples of the selected diseases as follows: a 10 percent sample of diseases with greater than 1,000 grants (“other autoimmune disease,” IBD, multiple sclerosis, type 1 diabetes, rheumatoid arthritis, and SLE), a 50 percent sample of diseases between 100 and 999 grants (Sjögren’s, psoriasis, and celiac disease), and a 100 percent sample of diseases with fewer than 99 grants APS (autoimmune thyroid disease and PBC). This resulted in a total of 1,148 grants. Grants were then deduplicated (because a grant could have been assigned to more than one disease), resulting in a total of 1,127 grants for the committee to manually categorize. 1,127 is approximately 13 percent of the total number of grants (N=8,470).

Research characteristics were chosen to evaluate the portfolio and identify the focus of the research. These characteristics encompass various concepts in the statement of task, co-morbid diseases or complications of autoimmune disease, and type and method of research. Examining more than one autoimmune disease aimed to capture grants exploring the co-occurrence of autoimmune diseases. Within PICO Portal, the committee used the tagging feature to categorize the abstracts with one or more of the following research characteristics: animal model, cancer, cardiovascular, depression, diagnostic, environment/infectious agents, epidemiology, genetics, nursing/behavioral health services, pathophysiology/mechanisms, pediatrics, primary prevention, treatment/therapy, in vivo human study, in vivo animal study, in vitro study, and more than one autoimmune disease. The committee predefined specific definitions of the research characteristics in order to tag abstracts objectively.

Figures 6-24 and 6-25 show the number and percentage of abstracts tagged with the research characteristics. More than half (59.1 percent) of the 1,127 abstracts were tagged as having a focus on pathophysiology/mechanisms, followed by genetics (25.3 percent), treatment/therapy (23.2 percent), and animal model (23.2 percent). “Animal model” refers to the development of a new in vivo animal study for use to study autoimmune disease, whereas “in vivo animal study” refers to the use of an already existing animal model. Approximately 22 percent of the abstracts mentioned or focused on more than one autoimmune disease. In addition, the committee tagged abstracts based on the type of study researchers intended to conduct with their grant funding (Figure 6-25). Almost half of the abstracts were identified as being an in vivo animal study (48.4 percent), followed by in vitro (40.8 percent), and in vivo human study (29.3 percent).

FIGURE 6-24

Research characteristics identified within autoimmune disease grant abstracts (N=1,127). NOTE: Percentages in the figure sum to more than 100 percent because grants could be tagged with more than one research characteristic. SOURCE: NIH, 2021f.

FIGURE 6-25

Type of studies identified within autoimmune disease grant abstracts (N=1,127). NOTE: Percentages in the figure sum to more than 100 percent because grants could be tagged with more than one research characteristic. SOURCE: NIH, 2021f.

It is important to note the discrepancy between the RCDC spending category “prevention” and “primary prevention” as a predefined research characteristic tag in PICO Portal. The committee defined primary prevention, found in 1.3 percent of abstracts, as “research that focuses on preventing an autoimmune disease from occurring in an individual,” and is only one of the three levels of prevention. In contrast, almost one-third of autoimmune disease grants included the RCDC prevention category, suggesting that RCDC may use a different definition that includes other levels of prevention. Alternatively, depression as both an RCDC category and a PICO Portal research characteristic tag both occurred infrequently in autoimmune disease grants (0.5 percent). This points to a potential opportunity to explore depression as a disease that coincides with autoimmune disease.

Study Section Analysis

Investigator-initiated research (R01) grants are the most common research grants funded (Figure 6-9). As noted in Chapter 5, study sections review applications for these grants, with study section peer reviewers responsible for assigning priority scores based on scientific merit. Peer reviewers draw upon their expertise in investigating line(s) of research comparable to those under review to assign scores that distinguish the very best applications from the weaker ones.

The committee used the NIH Center for Scientific Review (CSR) study section search tool to analyze study sections that review R01 grant applications.¹⁴ As many as 100 different study sections may be involved in reviewing unsolicited investigator-initiated research grant applications focusing on the select autoimmune diseases the committee chose to examine. Given that such a large number of diverse study sections can potentially review autoimmune disease grant applications, CSR should ensure that individual study section rosters include the appropriate expertise, including expertise for reviewing animal and human studies, to provide the proper review of these applications.

Table 6-10¹⁵ provides a snapshot of data generated by the RePORTER Matchmaker tool. The table shows study sections—the names of which appear in the far left column—that reviewed funded grants focusing on the autoimmune diseases of interest to the committee; the table reflects a 2-year period.^16,17 The table identifies those study sections that reviewed the most funded grants for each disease of focus according to a bar graph RePORTER automatically generates to provide this information; the number of grants are indicated in the corresponding cells. One to three study sections reviewed the majority of funded grants for the two years, suggesting that this more limited number of study sections has the correct expertise for review of autoimmune disease grant applications. The committee notes that a more valuable analysis would have been to identify, for a given period, the grant applications that were reviewed for the select diseases regardless of whether they were approved. This would have provided a better baseline for analyzing patterns of review by the study sections. However this information was not available. The most common study sections responsible for the grant applications in this analysis were Gastrointestinal Mucosal Pathobiology; Hypersensitivity, Autoimmune and Immune-Mediated Diseases; Arthritis, Connective Tissue and Skin; Oral, Dental and Craniofacial Science; Cellular and Molecular Immunology B; Clinical Neuroimmunology and Brain Tumor; Innate Immunity and Inflammation; Cellular and Molecular Biology of Glia; Clinical and Integrative Diabetes and Obesity; Cellular Aspects of Obesity and Diabetes; and Hepatobiliary Pathophysiology.

TABLE 6-10

Chartered Study Sections That Reviewed Funded Grants (FY 2019–2020) According to Select Autoimmune Diseases.

NIH AUTOIMMUNE DISEASE RESEARCH: INDICATORS OF ACCOMPLISHMENTS

The statement of task directed the committee to include NIH accomplishments in its report. The committee addressed this task by conducting a high-level review of a number of activities and measures that reflect progress in research (clinical trials and patents), and indicators of scientific performance, quality, and influence on the research field (bibliometric indicators). The committee describes the results of its examination below.

Clinical Trials

Results of biomedical research on autoimmune disease can yield opportunities to improve the prevention, detection, and treatment of these diseases. Clinical trials, also known as interventional studies, translate basic science knowledge for use in clinical care (NLM, 2021a). Clinical trials focus on developing, testing, and implementing diagnostics and therapeutics across a wide range of diseases and conditions to evaluate the effects of the interventions on safety and health-related outcomes (NLM, 2021a). The committee examined clinical trial activity by autoimmune disease as one measure of research portfolio advancement.

The committee obtained clinical trial data from clinicaltrials.gov¹⁸ and linked clinical trials to the grants associated with the committee’s autoimmune diseases of interest. Clinical trials were de-duplicated using their Clinical Trial ID and included based on their relevance to the committee’s specific diseases of interest and autoimmune disease as a broader category (“other autoimmune disease”). Ultimately, the committee identified 353 autoimmune disease clinical trials between filing year 2008 and 2020. The “other autoimmune disease” category represents approximately 27 percent of the clinical trials. The remaining 73 percent were associated with one or more autoimmune disease.

Figure 6-26¹⁹ shows the number of clinical trials by disease from filing years 2008–2020. The top five autoimmune disease categories with the most clinical trials over the study period were type 1 diabetes (38.7 percent), other autoimmune disease (26.6 percent), rheumatoid arthritis (10.3 percent), multiple sclerosis (5.8 percent), and IBD (5.5 percent). Each of the remaining autoimmune diseases accounted for less than 5 percent of clinical trials over this period. Clinical trials were fewer for the remaining autoimmune diseases.

FIGURE 6-26

Clinical trials by autoimmune disease, 2008–2020 (N=353). NOTE: Percentages in the figure sum to more than 100 percent because clinical trials can be associated with more than one autoimmune disease. SOURCES: NIH, 2021c, 2021f.

Out of the 353 clinical trials pertaining to autoimmune disease, 20 are phase 4 clinical trials. A phase 4 clinical trial is a “phase of research to describe clinical trials occurring after the Food and Drug Administration (FDA) has approved a drug for marketing. They include postmarket requirement and commitment studies that are required of or agreed to by the study sponsor, and gather additional information about a drug’s safety, efficacy, or optimal use”(NLM, 2021a). Table 6-11 lists select phase 4 clinical trials.

TABLE 6-11

Phase 4 Clinical Trials, Autoimmune Disease Prevention or Treatment.

Publications

The research community often looks to publication citations and other bibliometric measures as indicators of scientific performance, quality, and influence on a research field. The committee selected a number of these measures to examine the performance of the autoimmune disease research portfolio from 2008–2020. These indicators include the number of publications associated with funded grants, relative citation ratio, and journal impact factor.

The committee conducted a bibliometric analysis of publications in PubMed associated with autoimmune disease-associated grant numbers in an effort to characterize scientific productivity and impact. Figure 6-27²⁰ shows publications by disease category per year. Of the autoimmune diseases examined, the four diseases associated with the greatest number of publications over the 2008–2020 period were other autoimmune disease (27.7 percent), IBD (22.5 percent), type 1 diabetes (18.2 percent), and multiple sclerosis (16.1 percent).

FIGURE 6-27

Publications by autoimmune disease, 2008–2020 (N=159,416). NOTE: Percentages in the figure sum to more than 100 percent because publications can be associated with more than one autoimmune disease. SOURCES: Clarivate, 2021; NIH, 2015, 2021f; NLM, (more...)

Figure 6-28²¹ shows the trends in autoimmune disease publications. Between publication year 2008 and 2020, other autoimmune disease publications (light purple) generally trended upward and had the most publications overall with an average of 3,398 per year. Publications associated with IBD (yellow) had the greatest number of publications among the committee’s diseases of interest but appear to have peaked in 2014. Publications associated with type 1 diabetes (black), multiple sclerosis (dark purple), and SLE (light blue) have also trended downward.

FIGURE 6-28

Number of autoimmune disease publications by disease and year, 2008–2020 (N=159,416). SOURCES: Clarivate, 2021; NIH, 2015, 2021f; NLM, 2021b.

The Relative Citation Ratio (RCR) is a metric developed within the NIH Office of Portfolio Analysis (OPA). RCR “represents the field- and time-normalized citation rate” to the expected citation rate based upon the article’s co-citation network of NIH-funded publications (Hutchins et al., 2016; NIH, 2021d). A publication with an RCR of 1 means that it has the same influence as the average article. The RCR metric can be used to determine the allocation of research funding and the extent to which NIH awardees “maintain high or low levels of influence on their respective fields of research” (Aksnes et al., 2019; Hutchins et al., 2016). RCR is often used under the assumption that citations are a proxy for quality; however, articles can be cited for a variety of reasons and RCR does not take the specific context of a citation into account (NIH Library, 2020). This counters the argument that RCR, alone, is a measure of quality.

Table 6-12²² shows the publications with RCRs by disease. IBD publications had the highest average RCR of 3.029 and are therefore three times as influential as the average article according to this metric. Although PBC publications had the lowest average RCR of 1.894, they are still approximately twice as influential as the average article.

TABLE 6-12

Publications with RCRs by Autoimmune Disease, 2008–2020 (N=152,286).

Journal impact factor (JIF) is “a measure of the frequency with which the average article in a journal has been cited in a particular year and it is used to measure the importance or rank of a journal by calculating the times its articles are cited” (University of Illinois Chicago Library, 2016). The scientific impact of an individual article is unrelated to the impact factor of a journal (The University of Texas MD Anderson Cancer Center, 2021), and JIF is a measure of journal quality exclusively. In the context of evaluating NIH’s autoimmune disease portfolio, the committee focused on publications resulting from NIH-funded autoimmune disease research and the quality of the journals in which they are published.

Figure 6-29²³ shows average JIF for the 10 journals with the highest JIF for all projects. Of the 159,416 autoimmune disease-related publications, 63 percent (101,199 of 159,416) had JIF data. The size of the bubble represents the number of publications in the journal, while the location of the bubble on the y-axis shows the JIF relative to each other. Although Nature (dark gray) published the most articles between 2008 and 2020, Chemical Reviews (brown) has the highest JIF (43.9). In most fields, the impact factor of 10 or greater is considered an excellent score while 3 is flagged as good and the average score is less than 1 (SCI Journal, 2021).

FIGURE 6-29

Average JIF for the 10 journals with the highest JIF for all autoimmune disease publications, 2008–2020. NOTES: Because of the vast differences in the size of citation pools for general journals (large citation pools) versus very specialized journals (more...)

Patents

A patent grants a property right to the inventor and prohibits others from offering, selling, or importing that invention (USPTO, 2018). Patents granted as a result of NIH-funded autoimmune disease research aim to encourage and protect biomedical innovation and may serve as a proxy for knowledge generation and translational value in the field (Kalutkiewicz and Ehman, 2014). The committee examined patents and patent applications by autoimmune diseases.

The committee obtained patent data linked to the grants associated with the committee’s autoimmune disease of interest from the United States Patent and Trademark Office (USPTO) at uspto.gov. A total of 4,113 patent applications and patents were associated with NIH grants funding research in autoimmune diseases, of which 60 percent (2,470 of 4,113) were patent applications and 40 percent (1,643 of 4,113) were granted applications (patents). All were included in the analysis. A total of 30 percent of the patents and patent applications were associated with only one disease category. The remaining 70 percent were associated with more than one disease category.

Figure 6-30²⁴ shows the number of patent applications by disease for filing years 2008–2020. The top three autoimmune disease categories with the most patent applications were other autoimmune disease (30.3 percent), multiple sclerosis (22.8 percent), and IBD (19.8 percent). Fewer patent applications were filed for the remaining autoimmune diseases.

FIGURE 6-30

Patent applications by disease, 2008–2020 (N=2,470). NOTE: Percentages in the figure sum to more than 100 percent because patents can be associated with more than one autoimmune disease. SOURCES: NIH, 2021f; USPTO, 2018.

Figure 6-31²⁵ shows the granted patents by disease for filing years 2008–2020. The top three disease categories with the most patents were other autoimmune disease (29.1 percent), multiple sclerosis (22.7 percent), and IBD (18.4 percent).

FIGURE 6-31

Patents by disease, 2008–2020 (N=1,643). NOTE: Percentages in the figure sum to more than 100 percent because patents can be associated with more than one autoimmune disease. SOURCES: NIH, 2021f; USPTO, 2018.

The ultimate outcome of clinical and biomedical research is an observable impact on the health and wellbeing of individuals. This downstream effect is a result of fundamental scientific knowledge “applied to develop actionable interventions that are implemented to enhance health, lengthen life, and reduce illness and disability” (NIH, 2014). Publications disseminated throughout the scientific community often report the outcomes of clinical and biomedical research, and the number of publications resulting from autoimmune disease-related research is astounding, as seen by the thousands of relevant autoimmune-related publications produced between 2008 and 2020.

The committee was unable to perform a comprehensive assessment of research accomplishments and successes and therefore was unable to assess clinical and biomedical achievements as a measure of knowledge in the field beyond quantifying publications, patents, and clinical trials. Another way to assess achievements is to determine how research translates into clinical care and treatment of individuals, but this too is an immense undertaking. Instead, the committee provides a number of illustrative research findings associated with the research portfolio that have the potential to improve the health and healthcare of individuals living with autoimmune diseases in Table 6-13.²⁶

TABLE 6-13

Autoimmune Disease Research Highlights.

SUMMARY

In reviewing NIH’s research portfolio on autoimmune diseases, the committee employed a general framework that considered the inputs (spending), activities (research conducted), and outputs (clinical trials, patents, and other indicators) of NIH’s investment in autoimmune disease research. Spending on autoimmune diseases as a percentage of overall NIH obligations has remained at only 2.6 percent between 2013 and 2020. This is in marked contrast to increases seen in overall NIH obligations during the same time period. In addition, the distribution of this funding for autoimmune diseases indicates that certain ICs dominate overall spending on research and training and the types of research activities funded. NIH may need to reconsider the current strong focus on investigator-initiated research in order to target areas of high priority and effectively address the long-term, complex, and heterogeneous nature of autoimmune diseases.

Regarding the activities conducted under funded autoimmune disease research grants, the multiple methods the committee used yielded useful information at different levels of specificity. However, metrics are not available for evaluating the overall research portfolio for autoimmune diseases.

The outputs of NIH’s investment in autoimmune disease research can be measured in a variety of ways, including clinical trials (the translation of research knowledge into interventions), publications, and the number of patents associated with autoimmune disease research grants (an important indicator of innovation). There is a greater percentage of clinical trials from filing years 2008–2020 for such autoimmune diseases as type 1 diabetes, SLE, IBD, and rheumatoid arthritis, with a much smaller percentage seen for other autoimmune diseases. In general, the knowledge associated with existing research grants has been successfully disseminated to the research community through publications in well-regarded scientific journals. In terms of patent applications, filings occur primarily for some autoimmune diseases with far fewer filings for others. In general, findings related to research outputs suggest the need for a nuanced and in-depth examination of why progress via clinical trials and patent filings does not seem to have been made for certain autoimmune diseases.

Given the findings of Chapter 6, the committee concludes that the absence of an overarching strategic plan with concrete metrics makes it difficult to assess the distribution of funding by IC or disease. The committee also concludes that the relatively small percentage of NIH funding devoted to autoimmune disease research and how it is distributed could be a significant factor in hampering scientific progress on diseases that affect more than 23 million people.

REFERENCES

• Aksnes DW, Langfeldt L, Wouters P. Citations, citation indicators, and research quality: An overview of basic concepts and theories. SAGE Open. 2019;9(1):2158244019829575.
• Bittermann A, Fischer A. How to identify hot topics in psychology using topic modeling. Zeitschrift für Psychologie. 2018;226(1):3–13.
• Blei DM, Ng AY, Jordan MI. Latent dirichlet allocation. Journal of Machine Learning Research. 2003;3:993–1022.
• Breton MD, Kanapka LG, Beck RW, Ekhlaspour L, Forlenza GP, Cengiz E, Schoelwer M, Ruedy KJ, Jost E, Carria L, Emory E, Hsu LJ, Oliveri M, Kollman CC, Dokken BB, Weinzimer SA, DeBoer MD, Buckingham BA, Chernavvsky D, Wadwa RP., for the iDCL Trial Research Group. A randomized trial of closed-loop control in children with type 1 diabetes. The New England Journal of Medicine. 2020;383(9):836–845. https://doi​.org/10.1056/nejmoa2004736 . [PMC free article: PMC7920146] [PubMed: 32846062]
• Clarivate. Web of science. 2021. [December 14, 2021]. https://clarivate​.com​/webofsciencegroup/solutions​/webof-science/
• Cristian J. Topic modeling LDA using textminer and tidytext. 2020. [December 14, 2021]. https://rpubs​.com/jojoecp/643113 .
• Ford JA, Liu X, Chu SH, Lu B, Cho MH, Silverman EK, Costenbader KH, Camargo CA Jr, Sparks JA. Asthma, chronic obstructive pulmonary disease, and subsequent risk for incident rheumatoid arthritis among women: A prospective cohort study. Arthritis & Rheumatology. 2020;72(5):704–713. [PMC free article: PMC7188599] [PubMed: 32129572]
• Goldmuntz E. Paper read at NASEM Committee for the Assessment of NIH Research on Autoimmune Diseases: Meeting 1. 2020. [November 18, 2020]. Charge to the committee. Webinar.
• Hall JC, Baer AN, Shah AA, Criswell LA, Shiboski CH, Rosen A, CasciolaRosen L. Molecular subsetting of interferon pathways in Sjögren’s syndrome. Arthritis & Rheumatology. 2015;67(9):2437–2446. [PMC free article: PMC4551661] [PubMed: 25988820]
• Herold KC, Bundy BN, Long SA, Bluestone JA, DiMeglio LA, Dufort MJ, Gitelman SE, Gottlieb PA, Krischer JP, Linsley PS, Marks JB, Moore W, Moran A, Rodriguez H, Russell WE, Schatz D, Skyler JS, Tsalikian E, Wherrett DK, Ziegler AG, Greenbaum CJ, Type G. An anti-CD3 antibody, teplizumab, in relatives at risk for type 1 diabetes. The New England Journal of Medicine. 1 Diabetes TrialNet Study. 2019;381(7):603–613. [PMC free article: PMC6776880] [PubMed: 31180194]
• Hodge D. Paper read at NASEM Committee for the Assessment of NIH Research on Autoimmune Diseases: Meeting 2 (Continuation). 2021. [February 1, 2021]. NIH peer review, a CSR perspective. Webinar.
• Hong S, Banchereau R, Maslow BL, Guerra MM, Cardenas J, Baisch J, Branch DW, Porter TF, Sawitzke A, Laskin CA, Buyon JP, Merrill J, Sammaritano LR, Petri M, Gatewood E, Cepika AM, Ohouo M, Obermoser G, Anguiano E, Kim TW, Nulsen J, Nehar-Belaid D, Blankenship D, Turner J, Banchereau J, Salmon JE, Pascual V. Longitudinal profiling of human blood transcriptome in healthy and lupus pregnancy. Journal of Experimental Medicine. 2019;216(5):1154–1169. [PMC free article: PMC6504211] [PubMed: 30962246]
• Hutchins BI, Yuan X, Anderson JM, Santangelo GM. Relative citation ratio (RCR): A new metric that uses citation rates to measure influence at the article level. PLOS Biology. 2016;14(9) e1002541. [PMC free article: PMC5012559] [PubMed: 27599104]
• Kalutkiewicz MJ, Ehman RL. Patents as proxies: NIH hubs of innovation. Nature Biotechnology. 2014;32(6):536–537. [March 3, 2022]; https://www​.acadrad.org​/wp-content/uploads​/2016/06/Patents-as-Proxies-article​.pdf . [PubMed: 24911494]
• Martin JC, Chang C, Boschetti G, Ungaro R, Giri M, Grout JA, Gettler K, Chuang LS, Nayar S, Greenstein AJ, Dubinsky M, Walker L, Leader A, Fine JS, Whitehurst CE, Mbow ML, Kugathasan S, Denson LA, Hyams JS, Friedman JR, Desai PT, Ko HM, Laface I, Akturk G, Schadt EE, Salmon H, Gnjatic S, Rahman AH, Merad M, Cho JH, Kenigsberg E. Single-cell analysis of Crohn’s disease lesions identifies a pathogenic cellular module associated with resistance to anti-tnf therapy. Cell. 2019;178(6):1493–1508. e1420. [PMC free article: PMC7060942] [PubMed: 31474370]
• McNamara J. Paper read at NASEM Committee for the Assessment of NIH Research on Autoimmune Diseases: Meeting 2 (Continuation). 2021. [February 1, 2021]. Autoimmune disease research supported by extramural NIAID. Webinar.
• Muraro PA, Pasquini M, Atkins HL, Bowen JD, Farge D, Fassas A, Freedman MS, Georges GE, Gualandi F, Hamerschlak N, Havrdova E, Kimiskidis VK, Kozak T, Mancardi GL, Massacesi L, Moraes DA, Nash RA, Pavletic S, Ouyang J, Rovira M, Saiz A, Simoes B, Trneny M, Zhu L, Badoglio M, Zhong X, Sormani MP, Saccardi R., for the Multiple Sclerosis–Autologous Hematopoietic Stem Cell Transplantation (MS-AHSCT) Long-term Outcomes Study Group. Long-term outcomes after autologous hematopoietic stem cell transplantation for multiple sclerosis. JAMA Neurology. 2017;74(4):459–469. https://doi​.org/10.1001/jamaneurol​.2016.5867 . [PMC free article: PMC5744858] [PubMed: 28241268]
• NIH (National Institutes of Health). SPIRES. 2015. [December 14, 2021]. https://dpcpsi​.nih.gov​/sites/default/files​/opa/document/Posters​/Tools/15%20Jordan​%20SPIRES-OPA-Symposium-Poster-Final-07212015.pdf .
• NIH. Predicting response to standardized pediatric colitis therapy (PROTECT). 2018. [December 16, 2021]. https://clinicaltrials.gov/ct2/show/NCT01536535?id=NCT01536535&draw=2&rank=1 .
• NIH. Types of grant programs. 2019. [September 3, 2021]. https://grants​.nih.gov​/grants/funding/funding_program.htm .
• NIH. Estimates of funding for various research, condition, and disease categories (RCDC). 2020a. [April 12, 2021]. https://report​.nih.gov​/funding/categorical-spending#/
• NIH. RHYTHM (formerly Escape II Myocardium) (RHYTHM). 2020b. [December 16]. https://clinicaltrials.gov/ct2/show/NCT01548768 .
• NIH. About RCDC. 2021a. [July 03, 2021]. https://report​.nih.gov​/funding/categorical-spending/rcdc .
• NIH. Actual total obligations by institute and center: FY 2000–FY 2020. 2021b. [September 2, 2021]. https:​//officeofbudget​.od.nih.gov/pdfs/FY21​/spending-hist/Actual​%20Obligations%20By​%20IC%20FY%202000​%20-%20FY%202020%20(V).pdf .
• NIH. Clinicaltrials.GovClinicaltrials.Gov is a database of privately and publicly funded clinical studies conducted around the world. 2021c. [May 8, 2021]. https:​//clinicaltrials.gov/
• NIH. iCite: Influence module. 2021d. [December 7, 2021]. https://icite​.od.nih​.gov/user_guide?page_id=ug_infl .
• NIH. New onset type 1 diabetes: Role of exenatide. 2021e. [December 16, 2021]. https://clinicaltrials.gov/ct2/show/study/NCT01269034?id=NCT01269034&draw=2&rank=1 .
• NIH. NIH reporter database. 2021f.
• NIH Grants & Funding. Activity codes search results. 2021. [December 7, 2021]. https://grants​.nih.gov​/grants/funding/ac_search_results.htm .
• NIH Library. Week 11: Citation impact. 2020. [December 7, 2021]. https://www​.youtube.com​/watch?v=bxv0tzmbwxg .
• NLM (National Library of Medicine). Glossary of common site terms. 2021a. [December 14, 2021]. https:​//clinicaltrials​.gov/ct2/about-studies/glossary .
• NLM. Pubmed.GovPubmed.Gov. 2021b. [December 14, 2021]. https://pubmed​.ncbi.nlm.nih.gov/
• Park J, Blume-Kohout M, Krestel R, Nalisnick E, Smyth P. Analyzing NIH funding patterns over time with statistical text analysis. Palo Alto, CA: Association for the Advancement of Artificial Intelligence; 2016.
• Porturas T, Taylor RA. Forty years of emergency medicine research: Uncovering research themes and trends through topic modeling. American Journal of Emergency Medicine. 2021;45:213–220. [PubMed: 33059985]
• SCI Journal. What’s a good impact factor & why it matters (with 2018/2019 impact factor ranking in 27 categories). 2021. [December 7, 2021]. https://www​.scijournal​.org/articles/good-impact-factor .
• Spencer EA, Davis SM, Mack DR, Boyle BM, Griffiths AM, LeLeiko NS, Sauer CG, Keljo DJ, Markowitz JF, Baker SS, Rosh JR, Baldassano RN, Oliva-Hemker M, Pfefferkorn MD, Otley AR, Heyman MB, Noe JD, Patel AS, Rufo PA, Group PS, Alison Marquis M, Walters TD, Collins MH, Kugathasan S, Denson LA, Hyams JS, Dubinsky MC. Serologic reactivity reflects clinical expression of ulcerative colitis in children. Inflammatory Bowel Diseases. 2018;24(6):1335–1343. [PMC free article: PMC6093192] [PubMed: 29718391]
• The University of Texas MD Anderson Cancer Center. Q. What is considered a good impact factor? 2021. [December 7, 2021]. https://mdanderson​.libanswers​.com/faq/26159 .
• University of Illinois Chicago Library. Measuring your impact: Impact factor, citation analysis, and other metrics: Journal impact factor (if). 2016. [December 7, 2021]. https:​//researchguides.uic.edu/if/impact .
• USPTO (United States Patent and Trademark Office). General information concerning patents. 2018. [December 7, 2021]. https://www​.uspto.gov​/patents/basics/general-information-patents .