New quality and quantity indices in science (NewQIS): the study protocol of an international project
© Groneberg-Kloft et al; licensee BioMed Central Ltd. 2009
Received: 22 May 2009
Accepted: 26 June 2009
Published: 26 June 2009
Benchmarking systems are important features for the implementation of efficacy in basic and applied sciences. These systems are urgently needed for many fields of science since there is an imbalance present between funding policies and research evaluation. Here, a new approach is presented with an international study project that uses visualisation techniques for benchmarking processes. The project is entitled New Quality and Quantity Indices in Science (NewQIS). The juxtaposition of classical scientometric tools and novel visualisation techniques can be used to assess quality and quantity in science. In specific, the tools can be used to assess quality and quantity of research activity for distinct areas of science, for single institutions, for countries, for single time periods, or for single scientists. Also, NewQIS may be used to compare different fields, institutions, countries, or scientists for their scientific output. Thus, decision making for funding allocation can be made more transparent. Since governmental bodies that supervise funding policies and allocation processes are often not equipped with an in depth expertise in this area, special attention is given to data visualisation techniques that allow to visualize mapping of research activity and quality.
Progress in science is crucial for the economic development of most countries. This advance is commonly directly related to a country's intramural and extramural governmental and non-governmental funding policy. Funding sources are limited. Currently, the economic crisis leads to a further reduction of the sources and both health care systems and research funding are endangered [1, 2]. Therefore, numerous research projects and grant proposals may not be financed . This leads to an enormous potential of arising conflicts. The scientific community discusses these issues in great detail and there is an increasing interest in the underlying processes that lead to the allocation of funding budgets at the international, national and subnational levels [4–6].
Hypothesis and objectives
We hypothesize that without the use of scientometric techniques, there will be a growing discontent among scientists for funding allocation policies. In this respect, the use of specific benchmarking systems could be of help to implement transparency within funding allocation processes.
While single scientometric and bibliometric methods are known for many areas to dissect research activities of faculties or single scientists, the use of these techniques is often hampered. Reviewing the existing policies in Europe  or other general statements [8–11], it becomes clear that we need an improvement in this area. Therefore, the present work describes aims to establish an approach to visualize research quantity and quality indices.
The methods used for the international NewQIS studies are based on classic data bases such as the PubMed, Scopus or the Web of Science. Within these data bases, biomedical research output can be categorized with the numbers of published entries as an index marker for quantity of output. Quantities can be analysed with regard to different main characteristics: 1) specific fields of science, specific organs, diseases or other phenomena 2) countries 3) publication dates 4) authors 5) affiliations depending on the focus of the study. The data can then be transferred to visualisation techniques such as density equalizing calculations.
Additional file 1: Time-space distribution of published items related to respiratory medicine using density-equalizing mapping and one-year steps. The film sequence visualizes the total increase in respiratory medicine publications. The color coding encodes the total number of published items per country over the time. Data from  BMC Health Serv Res. 2009; 9: 16. Published online 2009 January 27. doi: 10.1186/1472-6963-9-16. Copyright © 2009 Groneberg-Kloft et al; licensee BioMed Central Ltd. (AVI 3 MB)
It is obvious that the policy for the allocation of research grants could benefit from an increase in 1) objectivity and 2) transparency. This is due to the fact that non-transparent and questionable allocation policies have contributed to weakening the reputation of basic and applied research all over the globe. There are numerous studies on research evaluation and policy in countries such as the USA or Australia which have described the existence and nature of this problem in their countries but there is still a lack of approaches that encompass both valid assessment tools and the ability to visualize results. We here establish an international project that uses scientometric tools together with visualising techniques. The NewQIS studies can be efficiently used to dissect scientific progress in closer detail. Here, they may be used to categorize research progress in all fields of science. They may also help for decision making in specific research grant calls. In this case, future NewQIS studies can incorporate parameters such as citation analysis or authorship and institution network analysis. This can be used specifically to evaluate research proposals on a supranational, national or infranational level.
As a conclusion, the use of the presently described approach to assess research quantity and quality may be used to 1) establish an objective basis for the evaluation of science 2) provide a visualizing platform for non-specialists or non-scientists who coordinate funding and funding policy at governmental or non-governmental levels.
We thank D. Groneberg for helpful discussions.
Source of support: Charité Faculty funding
- Catalano R: Health, medical care, and economic crisis. N Engl J Med 2009, 360: 749–751. 10.1056/NEJMp0809122PubMedView ArticleGoogle Scholar
- Furlow B: Financial crisis threatens US health reform. Lancet Oncol 2008, 9: 1028–1029. 10.1016/S1473-3099(08)70234-2PubMedView ArticleGoogle Scholar
- Giles J: Research grants: the nightmare before funding. Nature 2005, 437: 308–311. 10.1038/437308aPubMedView ArticleGoogle Scholar
- Smith PC: Resource allocation and purchasing in the health sector: the English experience. Bull World Health Organ 2008, 86: 884–888. 10.2471/BLT.07.049528PubMed CentralPubMedView ArticleGoogle Scholar
- Ruger JP: Health, capability, and justice: toward a new paradigm of health ethics, policy and law. Cornell J Law Public Policy 2006, 15: 403–482.PubMedGoogle Scholar
- Abelson J, Giacomini M, Lehoux P, Gauvin FP: Bringing 'the public' into health technology assessment and coverage policy decisions: from principles to practice. Health Policy 2007, 82: 37–50. 10.1016/j.healthpol.2006.07.009PubMedView ArticleGoogle Scholar
- Anderson A: Funding in Europe: how the big three cope. Science 1991, 254: 1118. 10.1126/science.1957164PubMedView ArticleGoogle Scholar
- Beauchamp TL: Ethical issues in funding and monitoring university research. Bus Prof Ethics J 1992, 11: 5–16.PubMedView ArticleGoogle Scholar
- Jellinek MS, Levine RJ: IRBs and pharmaceutical company funding of research. IRB 1982, 4: 9–10. 10.2307/3564273PubMedView ArticleGoogle Scholar
- Gronbjerg KA: How nonprofit human service organizations manage their funding sources: key findings and policy implications. Nonprofit Manag Leadersh 1991, 2: 159–175. 10.1002/nml.4130020206PubMedView ArticleGoogle Scholar
- Geisow MJ: Public funding of research and development: a broad picture. Trends Biotechnol 1991, 9: 76–77. 10.1016/0167-7799(91)90026-EPubMedView ArticleGoogle Scholar
- Groneberg-Kloft B, Scutaru C, Kreiter C, Kolzow S, Fischer A, Quarcoo D: Institutional operating figures in basic and applied sciences: Scientometric analysis of quantitative output benchmarking. Health Res Policy Syst 2008, 6: 6. 10.1186/1478-4505-6-6PubMed CentralPubMedView ArticleGoogle Scholar
- Gastner MT, Newman ME: Diffusion-based method for producing density-equalizing maps. Proc Natl Acad Sci USA 2004, 101: 7499–7504. 10.1073/pnas.0400280101PubMed CentralPubMedView ArticleGoogle Scholar
- Borger JA, Neye N, Scutaru C, Kreiter C, Puk C, Fischer TC, Groneberg-Kloft B: Models of asthma: density-equalizing mapping and output benchmarking. J Occup Med Toxicol 2008,3(Suppl 1):S7. 10.1186/1745-6673-3-S1-S7PubMed CentralPubMedView ArticleGoogle Scholar
- Groneberg-Kloft B, Scutaru C, Fischer A, Welte T, Kreiter C, Quarcoo D: Analysis of research output parameters: Density equalizing mapping and citation trends. BMC Health Serv Res 2009, 9: 16. 10.1186/1472-6963-9-16PubMed CentralPubMedView ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.