AI-Powered Scholar Labs: Revolutionizing Research Discovery

Google has introduced a new AI-driven search tool called Scholar Labs that is designed to provide detailed answers to research questions. However, its demonstration has raised questions about the credibility of scientific studies. Scientists are now debating how much they will trust a tool that uses AI to analyze the relationships between words to identify good research, rather than relying on traditional metrics of a study’s popularity within the scientific community.

This new search tool uses AI to analyze a user’s query and identify the main topics and relationships within it. Currently, it is only available to a limited set of logged-in users. In a demo video, Scholar Labs was shown answering a question about brain-computer interfaces (BCIs), which piqued the interest of a PhD holder in BCIs.

The tool pulled up a review paper on BCI research published in 2024 in a journal called Applied Sciences. Scholar Labs provides explanations for why the results matched the query, highlighting that the paper discusses research on a noninvasive signal called electroencephalogram and surveys leading algorithms in the field.

One notable feature of Scholar Labs is that it does not include filters based on common metrics used to assess the quality of studies. Metrics like the number of times a study has been cited by other studies and the impact factor of the journal it was published in are often used to gauge a study’s popularity and reputation. However, Scholar Labs does not consider these metrics when ranking search results.

The original Google Scholar allows users to rank studies by relevancy and lists the number of citations for each result. In contrast, Scholar Labs aims to prioritize the most useful papers for a user’s research quest by weighing factors such as the full text of each document, where it was published, who wrote it, and how often and recently it has been cited in other scholarly literature.

While metrics like citation count and impact factor are commonly used to assess a paper’s quality, they are considered by some to be coarse assessments. Associate professor of neurology at Vanderbilt University Medical Center, Matthew Schrag, believes that these metrics speak more about the social context of a paper rather than its actual quality. Despite this, many researchers, including Professor James Smoliga of Tufts University, still rely on highly cited papers as indicators of trustworthiness, even though this may not always reflect the true quality of the study.

In a comparison, the Scholar Labs demo query about BCI research for stroke patients was repeated in PubMed, a leading repository of biomedical and health research. Unlike Scholar Labs, PubMed allows users to filter search results based on factors such as time, article type, and peer-review. This demonstrates the different approaches taken by different search tools in surfacing relevant research.

Scholar Labs users will have the ability to request recent papers in their query and specify a time period for their search. The tool uses the full text of research papers to find results that match the user’s query, providing a more comprehensive search experience.

Google describes Scholar Labs as a new direction for them, aiming to incorporate user feedback in the future. The tool is currently available on a waitlist basis. Matthew Schrag believes that AI-powered search tools like Scholar Labs have a place in the scientific ecosystem, potentially casting a wider net to surface papers that may have been overlooked and providing additional context about a paper’s popularity.

Ultimately, scientists are responsible for determining the impact and quality of scientific research. While AI tools can assist in surfacing relevant papers, it is essential for researchers to engage with the literature and make informed judgments about the quality of studies. Schrag emphasizes the importance of researchers being the final arbiters of what is considered high-quality science, rather than letting algorithms make that decision.