Beyond citations – measuring the societal impact of research
We’ve explored open access publishing a lot here at CiteAb over recent months, and time and again the impact factor of papers and the measurement of impact using citations alone in a digital world have been key topics for discussion.
We were interested to learn about Altmetric, and the move towards measuring how scholarly articles are shared online. We have recently spoken to Jean Liu, data curator for Altmetric, about how the service will allow researchers greater insight into the societal impact of their work.
Hi Jean. Can you tell us a bit about what Altmetric aims to achieve?
Our main goal is to collect and showcase the online activity around scholarly literature. Through our data, we aim to complement traditional bibliometrics and help authors find early indicators of research impact for their papers. Moreover, we hope to be able to showcase non-traditional forms of impact as well, which might have otherwise been missed if one were only tracking citations.
Altmetric have a number of products, can you give an overview of them?
All of Altmetric’s products are focused around the conversations that surround research. Whenever a scholarly paper is mentioned on news sites, blogs, social media, or other places on the web, Altmetric collects this as attention. Our data include the number of times an article has been mentioned in particular sources (‘article-level metrics’), as well as the actual conversations that make up the numbers.
The Altmetric Explorer is our flagship product. It’s a web application that lets users browse and filter through the attention paid to all of the papers that we track.
The Altmetric badges are visualisations that show how much and what kind of attention a particular paper has received. Scholarly publishers often embed the badges into their journal article pages in order to showcase the buzz that their papers are receiving.
For article pages that don’t have an embedded Altmetric badge, we offer a free browser add-on called the Altmetric Bookmarklet, which instantly retrieves the article-level metrics from our database and displays them on a sidebar.
The Altmetric API is the most flexible in terms of data, and provides the raw metrics and mentions, which developers can then pull into custom applications, journal platforms, and more.
There must be challenges to dealing with such a large amount of data?
Aside from the technical challenges – determining what sources to track and how to capture them – the real difficulty lies in interpreting all of the data we collect. For example, what does the Altmetric score of a particular paper mean in that specific case? We need to set appropriate benchmarks for comparison, as what is considered a high score for one journal may not actually be a high score for another.
At the moment, we put information about context by comparing the Altmetric score for a particular paper to the scores of: a) all articles in the same journal, b) all articles of the same age, c) all articles of the same age in the same journal, and d) all articles in our database. In the future, what we want to do is display contextual information according to individual disciplines. We’re working on this capability, although it is proving to be technically challenging.
Ultimately, we always emphasise that the score is not an indicator of quality, but rather that it is just one possible measure of attention. It’s important to audit the metrics by examining the actual conversations.
You won the Elsevier prize for apps, how did that boost Altmetric’s growth?
The prize money from the Elsevier Apps for Science competition in 2011 enabled us to build the Altmetric Explorer, our flagship product, which was released in February of last year.
What groups of people can use Altmetric and in what ways?
Altmetric can be used by many different groups of people, but we generally cater to three groups: researchers, scholarly publishers, and institutions. Researchers who author scholarly papers are interested in finding out who’s talking about their work and what’s being said, so publishers and institutions can provide Altmetric’s tools as an author service.
Moreover, publishers (particularly editors, marketers, and press officers) and institutions (librarians, repository managers, communications officers, researcher managers, and so on) can use Altmetric to track and analyse the uptake, visibility, and impact of their organisations’ papers in society.
How have you commercialised Altmetric (what service do users pay for)?
Altmetric has been commercialised through our services for publishers. Depending on the use case, scholarly publishers sometimes purchase licenses to display Altmetric badges and data on their journal platforms; they may also purchase licenses for the Altmetric API. Additionally, editors, press officers, and marketers may purchase accounts to use the Altmetric Explorer, which has tools for data exploration and analysis.
How can Altmetric data contribute to academic reporting, such as REF impact reports?
Depending on the article, qualitative Altmetric data might be useful to refer to when building an impact case study to inform research assessment. Since Altmetric aggregates links to mentions of papers, the listed mentions may be able to serve as key pieces of evidence for impact case studies. For instance, if you were to search through mentions of a paper in the mainstream news, you may be able to find evidence of critical review in the media, and subsequently refer to relevant examples in your reporting.
Why should researchers and institutions care about altmetrics as well as citations?
Good question! I think that altmetrics showcase different usages of research, as compared to citations, which come purely from academia. I believe that in many cases, altmetrics can capture evidence of societal impact, which wouldn’t be obvious by looking at citations alone.
Researchers who are seeking evidence of public engagement should look into altmetrics to see the online discussions that take place both inside and outside of academia. Altmetrics also tend to appear much earlier than citations do – tweets and news reports about papers usually appear within days of publication, and blog posts show up a week or so after publication. Contrast that with citations, which take months or years to accrue.
Have you tried measuring the tweets, blog posts or social shares linked to your research before – or is it something you’d like to try? Do you agree that measures other than citations are useful indicators of research impact? We’d love to know your thoughts – post here or tweet us @CiteAb and let us know!
– the CiteAb team
