Journal of Occupational Medicine and Toxicology BioMed Central Study protocol New quality and quantity indices in science (NewQIS): the study

Benchmarking systems are important features for the implementation of efficacy in basic and applied sciences. These systems are urgently needed for many fields of science since there is an imbalance present between funding policies and research evaluation. Here, a new approach is presented with an international study project that uses visualisation techniques for benchmarking processes. The project is entitled New Quality and Quantity Indices in Science (NewQIS). The juxtaposition of classical scientometric tools and novel visualisation techniques can be used to assess quality and quantity in science. In specific, the tools can be used to assess quality and quantity of research activity for distinct areas of science, for single institutions, for countries, for single time periods, or for single scientists. Also, NewQIS may be used to compare different fields, institutions, countries, or scientists for their scientific output. Thus, decision making for funding allocation can be made more transparent. Since governmental bodies that supervise funding policies and allocation processes are often not equipped with an in depth expertise in this area, special attention is given to data visualisation techniques that allow to visualize mapping of research activity and quality.


Introduction
Progress in science is crucial for the economic development of most countries. This advance is commonly directly related to a country's intramural and extramural governmental and non-governmental funding policy. Funding sources are limited. Currently, the economic crisis leads to a further reduction of the sources and both health care systems and research funding are endangered [1,2]. Therefore, numerous research projects and grant proposals may not be financed [3]. This leads to an enor-mous potential of arising conflicts. The scientific community discusses these issues in great detail and there is an increasing interest in the underlying processes that lead to the allocation of funding budgets at the international, national and subnational levels [4][5][6].

Hypothesis and objectives
We hypothesize that without the use of scientometric techniques, there will be a growing discontent among scientists for funding allocation policies. In this respect, the use of specific benchmarking systems could be of help to implement transparency within funding allocation processes.
While single scientometric and bibliometric methods are known for many areas to dissect research activities of faculties or single scientists, the use of these techniques is often hampered. Reviewing the existing policies in Europe [7] or other general statements [8][9][10][11], it becomes clear that we need an improvement in this area. Therefore, the present work describes aims to establish an approach to visualize research quantity and quality indices.
This approach should be usable to assess quality and quantity of research activity i.e. for 1) distinct areas of science, for 2) single institutions, for 3) single countries, for 4) single time periods, or for 5) single scientists (Fig. 1). In addition to this evaluation, NewQIS studies may also be used for comparative issues: In this respect, different fields of science, different institutions, different countries, or different scientists or different time periods can be compared in terms of scientific output. Thus, decision making for funding allocation can also be made more transparent. Our hypothesis is that this approach is a useful tool for all kind of decision making concerning the allocation of funding. The approach can be termed "New Quality and Quantity Indices in Science (NewQQIS)". Arising from this approach, future NewQIS -studies may be conducted to provide a sound basis for the analysis and evaluation of research activity in distinct field of biomedicine.

Methods
The methods used for the international NewQIS studies are based on classic data bases such as the PubMed, Scopus or the Web of Science. Within these data bases, biomedical research output can be categorized with the numbers of published entries as an index marker for quantity of output. Quantities can be analysed with regard to different main characteristics: 1) specific fields of sci- The NewQIS study protocol can be used to assess research output for 1) distinct areas of science, for 2) institutions, for 3) countries, or for 5) scientists Figure 1 The NewQIS study protocol can be used to assess research output for 1) distinct areas of science, for 2) institutions, for 3) countries, or for 5) scientists.

International differences in focus of research
ence, specific organs, diseases or other phenomena 2) countries 3) publication dates 4) authors 5) affiliations depending on the focus of the study. The data can then be transferred to visualisation techniques such as density equalizing calculations.

Study example
Three recently published studies may serve as prototype examples for the NewQIS approach and were used to establish the study protocols: Using the two large databases Scopus and Web of Science biomedical research output was categorized with the numbers of published entries as an index marker for quantity of output [12]. The quantities were analyzed with regard to three main characteristics: 1) organs 2) countries 3) publication dates.
Density-equalizing mapping was used in this study for the visualization of data. In this respect, the territories were resized according to a particular variable, i.e. the number of published items. For the re-sizing procedure the area of each country was scaled in proportion to its total number of published items regarding the organs heart, brain, liver, lung and skin. The specific calculations were based on Gastner and Newman's algorithm [13]. In total, 5,527,558 published items were analysed in this study [12]. Using this approach a dichotomy was shown to be present between Western countries such as the US, UK or Germany and Asian countries such as Japan, China or South Korea concerning research focuses (Fig. 2). This was the first large scale analysis of global research activity and output over the last 50 years and the approach was used to establish the NewQIS protocol.
As a second example, the recently published data by Borger et al can be used [14]. Here, different animal models of asthma were analysed using in part the presently proposed NewQIS techniques. Density-equalizing algorithms were used and data was retrieved from the Thomson Institute for Scientific Information database Web of Science. During the period from 1900 to 2006 a number of 3489 filed items were found to be connected to animal models of asthma, the first being published in the year 1968 [14]. These studies were published by 52 countries. The US, Japan and the UK were the most productive countries, participating in 55.8% of all published items (Fig.  3). When analyzing the average citation per item as an indicator for research quality Switzerland ranked first (30.54/item) and New Zealand ranked second for countries with more than 10 published studies [14]. The 10 most productive journals included 4 journals with a main focus allergy and immunology and 4 journals with a main focus on the respiratory system. Two journals had a focus on pharmacology or pharmacy. In all assigned subject categories examined for a relation to animal models of asthma, the field of immunology ranked first. As a last step the numbers of published items were categorized with regard to specific animal species. Here it was found that mice were the preferred species followed by guinea pigs [14]. In summary it was concluded that the use of animal models of asthma is restricted to a relatively small number of countries with major differences in subsets of the analysis. The differences can be attributed to variations in the research focus as assessed by subject category analysis.
In a third example the visualisation procedures were extended to video analysis in order to be able to assess the evolution over the time [15]. In this study, the neighbouring fields of cardiovascular and respiratory medicine served as models for diverging patterns of health research. Density equalizing mapping procedures were used in A. Density-equalizing map illustrating the number of publica-tions in each particular country combination with video analysis. In this study specific areas of major research activity were identified for European countries and in general large differences were found [15]. In this respect, the spatial distribution of published items for cardiac and cardiovascular systems differed in comparison to the distribution for the respiratory system. Large countries dominated the overall number of published items. In order to evaluate the kinetics of publication activity the total publication output of the countries in five years intervals was analysed [15]. A continuing rise of publication numbers was found with a tendency to increased progression after the year 1997. The increase of publications was visualized for the field respiratory medicine using video density-equalizing mapping in one-year steps (see additional file 1).

Discussion
It is obvious that the policy for the allocation of research grants could benefit from an increase in 1) objectivity and 2) transparency. This is due to the fact that non-transparent and questionable allocation policies have contributed to weakening the reputation of basic and applied research all over the globe. There are numerous studies on research evaluation and policy in countries such as the USA or Australia which have described the existence and nature of this problem in their countries but there is still a lack of approaches that encompass both valid assessment tools and the ability to visualize results. We here establish an international project that uses scientometric tools together with visualising techniques. The NewQIS studies can be efficiently used to dissect scientific progress in closer detail. Here, they may be used to categorize research progress in all fields of science. They may also help for decision making in specific research grant calls. In this case, future NewQIS studies can incorporate parameters such as citation analysis or authorship and institution network analysis. This can be used specifically to evaluate research proposals on a supranational, national or infranational level.

Conclusion
As a conclusion, the use of the presently described approach to assess research quantity and quality may be used to 1) establish an objective basis for the evaluation of science 2) provide a visualizing platform for non-specialists or non-scientists who coordinate funding and funding policy at governmental or non-governmental levels.