The role of non-parametric and parametric methods in benchmarking research performance

Anar Abdikadirova ,Lyazzat Sembiyeva ,Ceslovas Christauskas ,Zharaskhan Temirkhanov

doi:10.24294/jipd9333

Home - JIPD - Volume 8 - Issue 16

Submit to JIPD

Journal Browser

Volume | Year

Issue

• Current lssue

View All

News and Announcements

View All

The role of non-parametric and parametric methods in benchmarking research performance

Anar Abdikadirova

Lyazzat Sembiyeva

Ceslovas Christauskas

Zharaskhan Temirkhanov

Journal of Infrastructure Policy and Development 2024, 8(16), 9333; https://doi.org/10.24294/jipd9333

Submitted：26 Sept 2024

Accepted：30 Oct 2024

Published：18 Dec 2024

Download PDF(595.59KB)

XML

Abstract

This study examines the effectiveness of Kazakhstan’s grant funding system in supporting research institutions and universities, focusing on the relationship between funding levels, expert evaluations, and research outputs. We analyzed 317 projects awarded grants in 2021, using parametric methods to assess publication outcomes in Scopus and Web of Science databases. Descriptive statistics for 1606 grants awarded between 2021 and 2023 provide additional insights into the broader funding landscape. The results highlight key correlations between funding, evaluation scores, and journal publication percentiles, with a notable negative correlation observed between international and national expert evaluations in specific scientific fields. A productivity analysis at the organizational level was conducted using non-parametric methods to evaluate institutional efficiency in converting funding into research output. Data were manually collected from the National Center of Science and Technology Evaluation and supplemented with publication data from Scopus and Web of Science, using unique grant numbers and principal investigators’ profiles. This comprehensive analysis contributes to the development of an analytical framework for improving research funding policies in Kazakhstan.

Keywords

References

Abramo, G., D’Angelo, C. A., & Di Costa, F. (2011). National research assessment exercises: a comparison of peer review and bibliometrics rankings. Scientometrics, 89(3), 929-941. https://doi.org/10.1007/s11192-011-0459-x

Agasisti, T., & Johnes, G. (2009). Beyond frontiers: Comparing the efficiency of higher education decision-making units across more than one country. Education Economics, 17(1), 59–79. https://doi.org/10.1080/00036840701721463

Banker, R.D., Charnes, A., & Cooper, W.W. (1984). Some models for estimating technical and scale inefficiencies in data envelopment analysis. Management Science, 30(9), 1078–1092. https://dx.doi.org/10.1287/mnsc.30.9.1078

Bol, T., De Vaan, M., & van de Rijt, A. (2018). The Matthew effect in science funding. Proceedings of the National Academy of Sciences, 115(19), 4887-4890. https://doi.org/10.1073/pnas.1719557115

Damaševičius, R.; Zailskaitė-Jakštė, L. The Impact of a National Crisis on Research Collaborations: A Scientometric Analysis of Ukrainian Authors 2019–2022. Publications 2023, 11, 42. https://doi.org/10.3390/publications11030042

Demarest, B., Freeman, G., & Sugimoto, C. R. (2014). The reviewer in the mirror: examining gendered and ethnicized notions of reciprocity in peer review. Scientometrics, 101, 717-735. https://doi.org/10.1007/s11192-014-1354-z

Ginther, D. K., Schaffer, W. T., Schnell, J., Masimore, B., Liu, F., Haak, L. L., & Kington, R. (2011). Race, ethnicity, and NIH research awards. Science, 333(6045), 1015-1019. https://doi.org/10.1126/science.1196783

Gross, K., & Bergstrom, C. T. (2019). Contest models highlight inherent inefficiencies of scientific funding competitions. PLoS biology, 17(1), e3000065. https://doi.org/10.1371/journal.pbio.3000065

Herbert, D. L., Coveney, J., Clarke, P., Graves, N., & Barnett, A. G. (2014). The impact of funding deadlines on personal workloads, stress and family relationships: a qualitative study of Australian researchers. BMJ open, 4(3), e004462. https://doi.org/10.1136/bmjopen-2013-004462

Kuzhabekova, A., Lee, J. International Faculty Contribution to Local Research Capacity Building: A View from Publication Data. High Educ Policy 31, 423–446 (2018). https://doi.org/10.1057/s41307-017-0067-3

Langfeldt, L., Bloch, C. W., & Sivertsen, G. (2015). Options and limitations in measuring the impact of research grants: Evidence from Denmark and Norway. Research Evaluation, 24(3), 256–270. https://doi.org/10.1093/reseval/rvv012

Lescrauwaet, L., Wagner, H., Yoon, C., & Shukla, S. (2022). Adaptive legal frameworks and economic dynamics in emerging tech-nologies: Navigating the intersection for responsible innovation. Law and Economics, 16(3), 202-220. https://doi.org/10.35335/laweco.v16i3.61

Linton, J. D. (2016). Improving the Peer review process: Capturing more information and enabling high-risk/high-return research. Research Policy, 45(9), 1936-1938. https://doi.org/10.1016/j.respol.2016.07.004

MacRoberts, M.H., MacRoberts, B.R. Problems of citation analysis. Scientometrics 36, 435–444 (1996). https://doi.org/10.1007/BF02129604

Materia, V. C., Pascucci, S., & Kolympiris, C. (2015). Understanding the selection processes of public research projects in agriculture: The role of scientific merit. Food Policy, 56, 87-99. https://doi.org/10.1016/j.foodpol.2015.08.003

Narbaev, T.; Amirbekova, D. Research Productivity in Emerging Economies: Empirical Evidence from Kazakhstan. Publications 2021, 9, 51. https://doi.org/10.3390/publications9040051

National Center of Science and Technology Evaluation (NCTSE, 2020), Competitive documentation for the Grant Funding 2021-2023, Available online: https://www.ncste.kz/assets/GNTE/kd-gf-2021-2023-rus.-07-08-2020-11-19-49.doc (accessed on 28 June 2024).

Nazarko, J., & Šaparauskas, J. (2014). Application of DEA method in efficiency evaluation of public higher education institutions. Technological and Economic Development of Economy, 20 (1), 25–44. https://doi.org/10.3846/20294913.2014.837116

Neufeld, J. (2016). Determining efects of individual research grants on publication output and impact: The case of the Emmy Noether Programme (German Research Foundation). Research Evaluation, 25(1), 50–61. https://doi.org/10.1093/reseval/rvv029

Official Foreign Exchange Rates on average for the period. The National Bank of Kazakhstan https://nationalbank.kz/file/download/61479

Official Foreign Exchange Rates on average for the period. The National Bank of Kazakhstan https://nationalbank.kz/file/download/72826

Official Foreign Exchange Rates on average for the period. The National Bank of Kazakhstan https://nationalbank.kz/file/download/86404

Ohniwa RL, Takeyasu K, Hibino A (2023) The effectiveness of Japanese public funding to generate emerging topics in life science and medicine. PLoS ONE 18(8): e0290077. https://doi.org/10.1371/journal.pone.0290077

Pearson, G. S. (2023). The ongoing importance of peer review. Journal of the American Psychiatric Nurses Association, 29(6), 445-446. https://doi.org/10.1177/10783903231205311

Pendlebury, D.A. The use and misuse of journal metrics and other citation indicators. Arch. Immunol. Ther. Exp. 57, 1–11 (2009). https://doi.org/10.1007/s00005-009-0008-y

Pölönen, J., Auranen, O. Research performance and scholarly communication profile of competitive research funding: the case of Academy of Finland. Scientometrics 127, 7415–7433 (2022). https://doi.org/10.1007/s11192-022-04385-8

Roumbanis, L. (2019). Peer review or lottery? A critical analysis of two different forms of decision-making mechanisms for allocation of research grants. Science, Technology, & Human Values, 44(6), 994-1019. https://doi.org/10.1177/0162243918822744

Shaw, J. (2023). Peer review in funding-by-lottery: A systematic overview and expansion. Research Evaluation, 32(1), 86-100. https://doi.org/10.1093/reseval/rvac022

Suleymenov, E.Z., Ponomareva, N.I., Dzhumabekov, A.K. et al. An assessment of the contributions of Kazakhstan and other CIS countries to global science: the Scopus database. Sci. Tech.Inf. Proc. 38, 159–165 (2011). https://doi.org/10.3103/S0147688211030051

Sun, D., Yu, B., & Ma, J. (2023). Research on the impact of digital empowerment on China’s human capital accumulation and human capital gap between urban and rural areas. Sustainability, 15(6), 5458. https://doi.org/10.3390/su15065458

Thanassoulis, E., Kortelainen, M., Johnes, G., & Johnes, J. (2011). Costs and efficiency of higher education institutions in England: A DEA analysis. Journal of the Operational Research Society, 62(7), 1282–1297. https://doi.org/10.1057/jors.2010.68

The Law of the Republic of Kazakhstan dated 18 February 2011 № 407-IV “On Science” https://adilet.zan.kz/eng/docs/Z1100000407

The Regulation on National Scientific Councils approved by the Government of Kazakhstan No. 519 dated May 16, 2011 https://adilet.zan.kz/rus/docs/P1100000519

The Rules for basic, grant, and program-targeted funding of scientific and/or scientific-technical activities approved by the Government of the Republic of Kazakhstan No. 575 dated May 25, 2011 https://adilet.zan.kz/eng/docs/P1100000575

The Rules for the Organization and Conduct of State Scientific and Technical Expertise approved by the Government of the Republic of Kazakhstan No. 891 dated August 1, 2011 https://adilet.zan.kz/eng/docs/P1100000891

Van Der Lee, R., Ellemers, N., Fiske, S.T., 2015. Gender contributes to personal research funding success in The Netherlands. Proceedings of the National Academy of Sciences of the United States of America 112, 12349-12353. https://doi.org/10.1073/pnas.1510159112

Wolszczak-Derlacz, J., & Parteka, A. (2011). Efficiency of European public higher education institutions: A two-stage multicountry approach. Scientometrics, 89(3), 887. https://doi.org/10.1007/s11192-011-0484-9

Previous
article in this issue

Next
article in this issue