CHEM 184/284 (Chemical Literature) - Huber - Winter 2025: Lecture 9: Citations, Part III - Evaluating Using Citations - Impact Factors and Journal Citation Reports

Citation Analysis and Impact Factors

• Indexing of cited references has permitted not only their use as access points to locate references, but also allowed for citation analysis. See the Eugene Garfield essays linked in Part I of this lecture for more a more thorough discussion of various aspects of citation analysis.
• Of particular importance to researchers is the notion that the more highly cited a given article, or author, or journal or institution is, the greater its impact on scholarly research, and so the more "important" it is. The data in the Web of Science databases has been analyzed in a variety of ways, and used to produce several products based on citation analysis:
  • Essential Science Indicators analyzes highly cited papers to identify hot areas of research in 22 fields of research. This tool is especially oriented toward administrators in industry and academia who want to identify potential recruits, new research areas to explore or chart what their competitors are researching. UCSB does not currently subscribe to this database.
  • Journal Citation Reports (see below) uses citation data to rank scholarly journals in the sciences and social sciences.
• The most commonly used measure of journal ranking by citation is the impact factor. Generally speaking, it is the ratio of the number of citations of a given journal to the number of articles published in that journal. The higher the number, the greater the "impact" of the journal on its field. More specifically, impact factors are usually measured for a given year. Thus, for the year 2013, the impact factor is calculated as

 Impact Factor (2019) = (# of cites in 2014 of articles published in 2017 and 2018) / (# of articles published in 2017 and 2018)

• Impact factors are most often calculated for individual journal titles, but may also be calculated for groups of journals, such as all journals in a given subject category.
• Other factors calculated for Journal Citation Reports include:
  • 5-Year Impact Factor - Same as impact factor, but calculated using five years of data, rather than two years of data. Helpful for journals where the pool of articles and cites is small, or to smooth out unusual "bumps" in the data.
  • Immediacy Index -- The number of cites in the same year as publication divided by the number of articles published.
  • Cited Half-Life -- For a given year, half of the citations to that journal are for articles published after X years.
  • Citing Half-Life -- For a given year, half of the articles cited in articles from that journal were published after X years.
  • Eigenfactor™ Score - The Eigenfactor was created by Carl T. Bergstrom and colleagues, this measure of journal importance is based on the number of times articles from the journal published in the past five years have been cited in the JCR year, but it also considers which journals have contributed these citations so that highly cited journals will influence the network more than lesser cited journals. References from one article in a journal to another article from the same journal are removed, so that Eigenfactor Scores are not influenced by journal self-citation.
  • Article Influence Score - Another Bergstrom creation, it is calculated by dividing a journal's Eigenfactor Score by the number of articles in the journal, normalized as a fraction of all articles in all publications. This measure is roughly analogous to the 5-Year Journal Impact Factor in that it is a ratio of a journal's citation influence to the size of the journal's article contribution over a period of five years. The mean Article Influence Score is 1.00. A score greater than 1.00 indicates that each article in the journal has above-average influence. A score less than 1.00 indicates that each article in the journal has below-average influence.
• Important considerations in using impact factors:
  • Impact factors vary dramatically depending on the subject field. Comparing impact factors from journals in two different fields is like comparing apples and oranges. Some fields cite new articles much more heavily than older ones; some cite older articles equaly heavily.  Since impact factors are calculated for the most recent two years of citations, journals in the latter fields will tend to have lower impact factors.
  • Review articles tend to be more highly cited than original research papers. So, journals which publishe only review articles (e.g., Accounts of Chemical Research, Chemical Reviews) tend to have disproportionately high impact factors compared to other journals in their subject category.
  • Since industrial research tends to be reflected in the patent literature more than in the journal literature, articles and journals dealing with applied research tend to be less cited in journals than basic research, and so generate lower impact factors.
  • Over time, impact factors tend to rise, due to the growth in overall scholarly research. So, a trend of increasing impact factors for a given journal is only significant when compared to others in its field.
  • Remember that journal title changes, splitting and recombining of journal titles, and newly introduced journals may lead to anomalies.
  • Remember, too, that sometimes an article may be highly cited because of the number of article published to refute it. Really awful research can sometimes be as highly cited as really good research.
• Uses of impact factors
  • Librarians use impact factors to help make decisions on journal purchases and cancellations.
  • Researchers use them to help determine where to submit their research for publication.
  • Grant-issuing agencies, tenure and promotion committees and agencies issuing work permits ("green cards") use them to evaluate the quality of a researcher's work, assuming that a publication in a high-impact journal is more important than one in a low-impact journal.
  • Note that this kind of use of citation analysis is tricky (should self-citation count? do authors and editors try to manipulate citation numbers to their advantage?)

Alternatives to JCR Impact Factors:

• SCImago Journal Rank (SJR)  https://www.scimagojr.com/ - SJR is weighted by the prestige of a journal Subject field, quality, and reputation of the journal have a direct effect on the value of a citation. SJR assigns relative scores to all of the sources in a citation network. Its methodology is inspired by the Google PageRank algorithm, in that not all citations are equal. A source transfers its own 'prestige', or status, to another source through the act of citing it. A citation from a source with a relatively high SJR is worth more than a citation from a source with a lower SJR.  A source's prestige for a particular year is shared equally over all the citations it makes in that year; this is important because it corrects for the fact that typical citation counts vary widely between subject fields. The SJR of a source in a field with a high likelihood of citing is shared over a lot of citations, so each citation is worth relatively little. The SJR of a source in a field with a low likelihood of citing is shared over few citations, so each citation is worth relatively much. The result is to even out the differences in citation practice between subject fields and facilitate direct comparisons of sources.SJR emphasizes those sources that are used by prestigious titles. SJR allows the user to rank their own customized set of sources, regardless of their subject fields.  SJR values are available in the Analytics section of Scopus, based on their data.
• Source Normalized Impact per Paper (SNIP) - SNIP essentially divides the impact factor for a journal by the average impact factor for its field.  The idea is to allow one to compare impact across different fields.  Scopus Analytics https://www.scopus.com/source/eval.url (requires an account with Elsevier) reports SNIP values based on their data.  This calculation can get tricky depending on what subject field(s) a journal may belong to.
• h-index -- This is a tool to rank the work of individual researchers. First proposed by J. E. Hirsch in "An index to quantify an individual's scientific research output", PNAS, 102, 16569-16572 (Nov. 15, 2005), it defines the h-index as the number of papers "h", which each have at least "h" citations in the literature. So, if I have published 600 papers, and the 110 most cited papers have at least 110 citations each, my h-index would be 110. If, on the other hand, I publish 600 papers and only 5 have more than 5 citations, even though those five may have thousands of citations, my h-index would be 5. This index tends to favor researchers who consistent publish large numbers of highly-cited papers, relative to authors who are, perhaps, equally well-regarded but less prolific. Note that Web of Science now can quickly generate h-indices based on their citation data for any set of publications - see Lecture 9.1. Google Scholar Metrics   https://scholar.google.com/citations?view_op=top_venues&hl=en has a list of h-indices, grouped by subject area, calculated for journals using citation data from Google Scholar.
• Usage counts -- This measure, increasing in popularity as journal usage shifts more and more into electronic journals, counts the number of times an article is downloaded. This tends to reduce the advantage of basic research over applied research in citation counts, but just because an article was downloaded, was it actually read? Was it considered important by the readers? No way to tell!
• Social Web metrics - Some researchers have begun to measure how often a researcher or an article is referred to in blogs or social networking sites like Facebook and Twitter, or how often the scientist's own blog, tweets, etc. are mentioned in social media.  The term "altmetrics" has been coined to describe such social metrics. See "Scholars Seek Better Ways to Track Impact Online" by Jennifer Howard in the Jan. 29, 2012 issue of Chronicle of Higher Education. Some journals are now incorporating altmetric tools, such as PLOS One  - display any article and click on the "Metrics" tab for an example.  See also "Research impact: Altmetrics make their mark" by Roberta Kwok, originally published in Nature in 2013 See Lecture 17 notes for a more in-depth discussion of altmertics..
• See the article by Sophie L. Rovner, "The Import of Impact", Chemical & Engineering News, 86, 39-42 (May 19, 2008) for a good introductory discussion of the topic.

© 2024 Charles F. Huber

This work by Charles F. Huber is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.
Based on a work at guides.library.ucsb.edu

Screenshots of Journal Citation Reports are copyright © 20120 by Clarivate Analytics and are used under fair use for educational purposes only.