The societal impact of research funding in the health domain

What is Data4Impact

In this post we will describe our work in the project Data4Impact that addresses key challenges of the EU Call CO-CREATION-08–2016–2017 on the impact of research and innovation in policy making. The focus of our project is on the health domain.

Qualia’s partners in the project: PPMI (Lithuania), Athena RC (Greece), Fraunhofer (Germany), CNR (Italy), University of Boras (Sweden).

The project has started on November 1, 2017 and will end on October 31, 2019.

The measurement of research impact helps funders to direct the allocation of research resources, to maximize research benefit, and to minimize research waste. The project has 2 main objectives:

1. To develop a set of indicators for assessing the performance of EU and national R&I systems.

2. To gather data and to develop methodologies that will help the measurement of the indicators on health-related challenges.

Data4Impact collects data from a broad range of data sources, covers all the key stages of the R&I lifecycle in the health domain and measures a set of indicators that answer key business questions such as: Do we fund relevant research? What is the impact of the research that we have funded? Which topics should we fund next? Should we enter areas where few others invest? Should we fund rare topics? How does our funding interact with the researchers and organizations perspective?

To the best of our knowledge, Data4Impact is the first research project to track the results and impact of R&I activities after the end of research funding. It is also the first project which attempts to establish links between EU research activities and health innovations and products which are currently on the market.

Methodology

A wide range of approaches to measure the impact of R&I (Research and Innovation) have been developed and used over the last thirty years, each of them relying on different assumptions and measuring impact in different ways. The various methodological frameworks focus on:

- the academic, societal, economic, and cultural impact using narrative and quantitative metrics

- the interaction process between stakeholders and researchers

- the importance of partnerships between researchers and policy makers

- linking processes between research and impact

- the ability of the health technology to influence the efficiency of healthcare

A detailed description of the various frameworks are presented in [1] and [2].

While there is a great variety of conceptual contributions in the literature regarding the societal impact of R&I, comparable empirical evidence, as well as concrete indicators are scarce. Impact measurement is context dependent, which means that is it difficult to agree on indicators that produce comparable results. R&I activities are currently the most prominent proxy to R&I impact.

Data4Impact’s analytical model is divided into four analytical stages: the input, the throughput, the output stage and the impact stage. The model is called AMOSIA (A Model Of Societal Impact Assessment) and its detailed description can be found in the project deliverables D2.1 and D2.2 here. The following dense image depicts the conceptual framework of Data4Impact.

Image 1: Data4Impact conceptual framework

Input: the main input indicator is the amount of funding at program/project level.

Throughput: final reports, scientific publications and patent applications, are the main indicators for a successful R&I activity and the knowledge created.

Output: two dimensions, academic and economic. The academic output reflects the generated content within the scientific community, mostly citation-based metrics. The economic output utilizes the knowledge produced by developing new products, prototypes, technical solutions, or processes: trademarks, certified medical devices.

Impact: academic, economic and social media impact. The academic impact describes a demonstrable contribution of R&I made to academia, such as a change in the research agenda, a special focus on research aspects, the development of new methods and a change of the educational content. The economic impact describes the exploitation of new markets, production efficiency and cost, competitiveness. Finally, the social media impact is measured in the on-line discussions in news, blogs, fora and twitter.

Indicators

The next table presents an overview of the set of indicators that we have developed during the project.

Table 1: Data4Impact indicators

Social Media Impact

We will now focus on the social media indicators that we compute in order to measure the societal impact of health related topics. Topic modelling is a core component of the Data4Impact processing pipeline. We compute 500 health topics from over 5M medical publications and 2K EU funded projects (FP7 & H2020). The topics are linked to funders, organizations, authors, schools, countries, etc. The topic models help us to identify active areas of research, to understand what is actually produced, to discover communities, to identify emerging research areas, and to identify gaps and new challenges.

From the topic models we form the social media search queries that we apply in order to search on news, blogs, fora and twitter from all over the world. We show a subset of the search queries in the following table, where the first column indicates the number of the topic, the second column the name of the topic and the third column the search query that corresponds to the topic.

Table 2: social media search queries

We retrieve ~36M search results (mentions). We calculate two social media indicators: [a] number of mentions (buzz), which is the number of articles, blogposts, fora posts and tweets that we retrieve, [b] link sharing (engagement), which is the % of articles about each topic that have at least 5 shares on Facebook.

Examples

In this section, we will show indicative examples of the insights that we gain from the social domain. We start with an example of each indicator, calculated over the period from January 13 to February 07, 2019. Image 2 shows the buzz and image 3 the engagement metrics.

Image 2: buzz indicator

Image 3: engagement indicator

In the next example, we focus on the topic of cardiovascular diseases. This is a large and well-funded topic: In the EU FP7 & H2020 framework programs, over 25 projects on cardiovascular diseases were funded, with a total budget that exceeded 80M€. We have used the set of keywords corresponding to the topic of cardiovascular diseases in order to collect news articles, blogposts, fora posts and tweets of the topic. Next, we have clustered the texts into the main aspects of discussion around the cardiovascular diseases, as shown in the Image 4 below. The percentages in parenthesis indicate the share-of-voice of each aspect of discussion within the topic of cardiovascular diseases. Diabetes and obesity are the most mentioned sub-topics in the texts that discuss the cardiovascular diseases.

Image 4: aspects of the discussions on the cardiovascular diseases

In our last example, we cluster together news articles that discuss the same story, in various outlets from all over the world, under the topic of stem cells, as shown in the following image:

Image 5: news topics on stem cells

The number at the beginning of each line is the number of articles that constitute each cluster. The title of the cluster is selected among all the titles of the articles, as the most representative according to a clustering centroid metric. The articles were published on February 2019.

The CRISPR/Cas9 technology used to correct defects in genes (38 mentions) was never funded by the EU. Same for the retinas that were grown in a Petri dish (also 38 mentions) and could potentially help the researchers to form therapies for eye diseases such as color blindness or macular generation. In this example we have started with the topic model of the stem cells, we have created the corresponding social media search queries, fetched social media articles and posts, clustered the retrieved texts into topics and examined whether the discussed topics were previously funded. Thus, we can continuously monitor trending and interesting topics in order to decide whether to fund them in future funding frameworks and calls.

Next steps

We are constantly refining and enriching our models, and also working on a demonstrator that will be alive until the end of August 2019. We will organize a workshop on September 2, 2019, in Rome, as a part of the 17th International Conference on Scientometrics and Informetrics (ISSI2019). In this hands-on, interactive workshop, we aim to receive feedback on the chosen methodology, the coverage and latency/timeliness of the developed indicators, to maximise the relevance for all stakeholders involved (particularly for funding agencies and policy makers). We would appreciate the opportunity to collaborate with you at our workshop, so send me a message if you will be able to attend.

Citations

[1] Cruz Rivera S, Kyte DG, Aiyegbusi OL, Keeley TJ, Calvert MJ (2017). Assessing the impact of healthcare research: A systematic review of methodological frameworks. PLoS Med 14(8): e1002370.

[2] Bastow, S., Dunleavy, P., & Tinkler, J. (2014). The impact of the social sciences: How academics and their research make a difference. Los Angeles: SAGE.