Award Abstract # 1735194
Echo Chambers in Science? The Impact of Academic Recommender Systems on the Dissemination of Scientific Knowledge

NSF Org: SMA
SBE Office of Multidisciplinary Activities
Recipient: UNIVERSITY OF WASHINGTON
Initial Amendment Date: June 13, 2017
Latest Amendment Date: August 28, 2018
Award Number: 1735194
Award Instrument: Continuing Grant
Program Manager: Joshua Trapani
SMA
 SBE Office of Multidisciplinary Activities
SBE
 Directorate for Social, Behavioral and Economic Sciences
Start Date: August 1, 2017
End Date: July 31, 2020 (Estimated)
Total Intended Award Amount: $234,128.00
Total Awarded Amount to Date: $293,372.00
Funds Obligated to Date: FY 2017 = $203,224.00
FY 2018 = $90,148.00
History of Investigator:
  • Katherine Stovel (Principal Investigator)
    stovel@u.washington.edu
  • Jevin West (Co-Principal Investigator)
Recipient Sponsored Research Office: University of Washington
4333 BROOKLYN AVE NE
SEATTLE
WA  US  98195-1016
(206)543-4043
Sponsor Congressional District: 07
Primary Place of Performance: University of Washington
Seattle
WA  US  98195-4320
Primary Place of Performance
Congressional District:
07
Unique Entity Identifier (UEI): HD1WMN6945W6
Parent UEI:
NSF Program(s): Cross-Directorate Activities,
SciSIP-Sci of Sci Innov Policy
Primary Program Source: 01001617DB NSF RESEARCH & RELATED ACTIVIT
01001718DB NSF RESEARCH & RELATED ACTIVIT

01001819DB NSF RESEARCH & RELATED ACTIVIT
Program Reference Code(s): 1397, 7626
Program Element Code(s): 139700, 762600
Award Agency Code: 4900
Fund Agency Code: 4900
Assistance Listing Number(s): 47.075

ABSTRACT

Digitization of the scientific literauture has occurred in two phases: first, scientific papers began to be digitized, and second, comprehensive search engines such as Google Scholar, JSTOR and Web of Science emerged that allow scholars to quickly search for and locate published research. Most scientists now access the literature through online search engines and digital libraries, and rare is the scientist who walks to the library and peruses the journal shelves for new papers. These new algorithmic search engines thus provide scholars with a new lens into the published scientific literature, and this project aims to better understand the implications of these technological changes on the practice of scientific discovery and information dissemination. The project investigates whether these new tools are increasing access to a wider range of prior literature and thereby democratizing science, or instead concentrating scholars' gaze onto an ever smaller set of "star" papers. If the new tools are truly making the widest range of scientific literature more accessible, it certainly has implications for the prospects for scientific discovery, since scientists are able to consider and engage with all relevant prior work, a critical ingredient for high quality scientific activity. On the other hand, evidence from other contexts suggests that the availability of massive amounts of information puts new pressure on searching and filtering processes. If this occurs in science it could mean that scientists increasingly rely upon more concentrated subset of papers that appear at the top of search results, thereby creating an echo chamber in science with unintended effects on scientific careers and potentially negative downstream impacts on scientific innovation.

This research addresses these critical questions primarily via statistical analyses of comprehensive citation data from JSTOR and the Web of Science. The research focuses on the extent of citation concentration within and across disciplines; and the role of journals in an article-based search environment. Citation patterns have changed dramatically in the wake of the digital transition, becoming both more concentrated and more vulnerable to cumulative advantage processes. The population-level analyses are augmented by observational data describing how scientists actually interact with information technologies and academic recommender systems in the course of their scientific practice. This combination of methods links processes at the individual and population levels over time, and emphasizes the downstream impacts of information retrieval and citation on scientific innovation and career trajectories. A critical policy related implication is that online recommender systems can bias search results and thus the visibility of specific scientific findings. This is relevant to the next generation of scientists using these search environments and those who evaluate scientists and their work.

PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH

Note:  When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external site maintained by the publisher. Some full text articles may not yet be available without a charge during the embargo (administrative interval).

Some links on this page may take you to non-federal websites. Their policies may differ from this site.

Hope, Tom and Portenoy, J and Vasan, K and Borchardt, J and Horvitz, Eric and Weld, DS and Hearst, MA and West, Jevin. "SciSight: Combining faceted navigation and research group detection for COVID-19 exploratory scientific search" Proceedings of the 2020 EMNLP (Systems Demonstrations), Association for Computational Linguistics , 2020 https://doi.org/10.1101/2020.05.23.112284 Citation Details
Kim, Lanu and Adoph, Christopher and West, Jevin and Stovel, Katherine "The Influence of Changing Marginals on Measures of Inequality in Scholarly Citations: Evidence of Bias and a Resampling Correction" Sociological Science , v.7 , 2020 https://doi.org/10.15195/v7.a13 Citation Details
Kim, Lanu and Portenoy, Jason H. and West, Jevin D. and Stovel, Katherine W. "Scientific journals still matter in the era of academic search engines and preprint archives" Journal of the Association for Information Science and Technology , v.71 , 2019 https://doi.org/10.1002/asi.24326 Citation Details
Larsen, Kai R. and Hovorka, Dirk S. and Dennis, Alan R. and West, Jevin D. ""Understanding the Elephant: The Discourse Approach to Boundary Identification and Corpus Construction for Theory Review Articles"" Journal of the Association for Information Systems , 2019 https://doi.org/10.17705/1jais.00556 Citation Details
Portenoy, Jason and West, Jevin D. "Constructing and evaluating automated literature review systems" Scientometrics , v.125 , 2020 https://doi.org/10.1007/s11192-020-03490-w Citation Details

PROJECT OUTCOMES REPORT

Disclaimer

This Project Outcomes Report for the General Public is displayed verbatim as submitted by the Principal Investigator (PI) for this award. Any opinions, findings, and conclusions or recommendations expressed in this Report are those of the PI and do not necessarily reflect the views of the National Science Foundation; NSF has not approved or endorsed its content.

The core question motivating our research is, what impact are new web-­based academic recommender systems having on how scientists engage with prior scientific research? Our aim was to assess the extent to which these tools have enhanced scholars' access to (and citation of) a wide range of potentially relevant prior work ­­ thereby democratizing science, or if they are concentrating scholars’ gaze onto an ever smaller set of ”star” papers.  If the new tools are making scientific literature more accessible, it will have implications for the prospects for scientific discovery, since it will signal that scientists are drawing on all relevant prior work, a critical ingredient for high quality scientific activity. On the other hand, evidence from other contexts suggests that the availability of massive amounts of information puts new pressure on searching and filtering processes. This could mean that scientists are increasingly reading and citing a more concentrated subset of papers (i.e. those that appear at the top of search results), and are therefore at risk of being trapped in an unproductive scientific echo chamber.  Answering these questions requires the development of new methods and data sources, including new methods for comparing inequalities over time or across contexts.  Using these methods, we find minimal change in citation concentration even as new search technologies become dominant.

Additional work addresses the continued significance of scientific journals. Journals play a critical role in the scientific process because they evaluate the quality of incoming papers and offer an organizing filter for search. However, the role of journals has been called into question because new preprint archives and academic search engines make it easier to find articles independent of the journals that publish them. Research on this issue is complicated by the deeply confounded relationship between article quality and journal reputation. We present an innovative proxy for individual article quality that is divorced from the journal's reputation or impact factor: the number of citations to preprints posted on arXiv.org. Using this measure to study three subfields of physics that were early adopters of arXiv, we show that prior estimates of the effect of journal reputation on an individual article's impact (measured by citations) are likely inflated. While we find that higher-quality preprints in these subfields are now less likely to be published in journals compared to prior years, we find little systematic evidence that the role of journal reputation on article performance has declined. 

 


Last Modified: 01/29/2021
Modified by: Jevin West

Please report errors in award information by writing to: awardsearch@nsf.gov.

Print this page

Back to Top of page