Assessing the impact of small-research grants supported by WHO in the Eastern Mediterranean Region 2010–2018

751 Research article EMHJ – Vol. 28 No. 10 – 2022 Assessing the impact of small-research grants supported by WHO in the Eastern Mediterranean Region 2010–2018 Bahareh Yazdizadeh,1 Ahmed Mandil,2 Sima Nikooee1 and Arash Rashidian2 1Knowledge Utilization Research Center, Tehran University of Medical Sciences, Tehran, Islamic Republic of Iran. 2World Health Organization Office for the Eastern Mediterranean, Cairo, Egypt. (Correspondence to: Ahmed Mandil: mandila@ who.int) Introduction Since the 1940s and the inception of WHO, 3 of its core functions have been dedicated to health research. These include shaping the research agenda and stimulating the generation, translation and dissemination of valuable knowledge; articulating ethical and evidence- based policy options; and providing technical support, catalysing change, and building sustainable institutional capacity. Over the decades, several publications related to health research were produced by WHO and these could be accessed at: http://www.emro.who.int/rpc/ publications/. The World Health Assembly (WHA) in its resolution WHA63.21 (WHO’s role and responsibilities in health research) called on Member States to establish governance mechanisms for research for health; to ensure rigorous application of good research norms and standards, including protection for human subjects involved in research; and to promote open dialogue between policymakers and researchers on national health needs, capacities, and constraints. One of the activities to achieve WHO objectives is the provision of funds for local evidence. In the Eastern Mediterranean Region (EMR), WHO has been providing targeted funding for health research on priority areas of public health for decades. Applications are being received through open calls for proposals via different grant schemes, including: Research in Priority Areas in Public Health (RPPH), Tropical Disease Research – Small Grant Scheme (TDR-SGS), and occasionally improving Programme Implementation through Embedded Research (iPIER). Since 2020, new small grant schemes have been initiated focusing on migration health research, International Health Regulations (IHRs), and COVID-19- related priority research (1). These grant schemes are very popular, and every biennium, WHO receives about 10 times more proposals than can be supported in response to each call. These grant schemes are very important for producing local evidence, which has been emphasized in recent reports (2-4). EMR has implemented some interventions to empower countries to use the local evidence, such as asking principal investigators to share Abstract Background: For decades, WHO has been providing targeted funding for health research on priority areas of public health in the Eastern Mediterranean Region through different grant schemes. Aims: This paper investigated the impact of WHO/EMRO’s funding schemes and factors facilitating or hindering such impact. Methods: We assessed the impact of health research funded by WHO/EMRO during 2010–2018 from the health, economic, decision-making, and knowledge translation perspectives, emphasizing accountability and analysis, using the Payback framework, mixed-method approach (quantitative, qualitative), and triangulation. Results: Principal investigators of 45 (45.9%) out of the 98 funded projects responded to the questionnaire. Almost all (88.0%) the 45 projects reported developing at least one decision-making document. Less than half reported producing peer-reviewed documents and conducting target group empowerment, while 24.0% said they secured research funds from other organizations. For 23 projects (51.0%), research results could have had a direct impact on health and on economy, and 25 (56.0%) projects conducted at least one active knowledge translation activity. Using multiple logistic regression, there was no significant association between the country of research and impact on decision-making and implementation of result if health or economic impact was expected. Conclusion: To strengthen the impact of research, WHO/EMRO should embark on a series of interventions to guide and empower countries in the use of research results. Discrepancies between health research systems in the Eastern Mediterranean Region and differences in individual and organizational capacities in the different countries require targeted interventions. Keywords: Research, small grant, Eastern Mediterranean, WHO/EMRO, health systems Citation: Yazdizadeh B; Mandil A; Nikooee S; Rashidian A. Assessing the impact of small-research grants supported by WHO in the Eastern Mediterranean Region in 2010–2018. East Mediterr Health J. 2022;28(10):751–757. https://doi.org/10.26719/emhj.22.072 Received: 02/12/21; accepted: 14/08/22 Copyright © Authors 2022; licensee World Health Organization. EMHJ is an open access journal. This paper is available under the Creative Commons Attribution Non-Commercial ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo). 752 Research article EMHJ – Vol. 28 No. 10 – 2022 a “manuscript for consideration for publication”, instead of the final grant report, thus encouraging principal investigators to publish and share outcomes of their research with the scientific community. To improve these funding schemes, it is important to investigate their impact in the region and understand factors that facilitate or hinder their performance. Generally, the impact assessment is conducted to facilitate advocacy, accountability, fund allocation, and analysis (5). As funding agencies are often at the core of strategies to improve “implementation to impact”(6), this assessment is very important for EMR to tailor it’s research policymaking and processes. Hence, this study aimed to investigate the impact of WHO/EMRO’s funding schemes and understand factors facilitating or hindering such impact as a first step towards refining grant programmes and tailoring capacity-building activities to countries context. While studies have been carried out to map health research institutions and assess evidence to policy and practices and health research outputs in the EMR (9–11,18– 20), we are not aware of any regional studies assessing the impact of WHO-funded research. Methods Health research impact assessment can be both top-down (ecologic studies) and bottom-up (case studies) (6). In top- down studies, the variables are collected and analysed in aggregated levels, such as evaluating the relationship between the expenditure on research and the impact of that research on reducing mortality burden attributed to specific diseases. In bottom-up studies, a set of research studies is selected, examining their impact over time via intense data collection (desk review and interview), where attribution of impact to specific research is more possible (although a big challenge in the top-down method). The payback framework for health research impact assessment was developed in 1996 for conducting case study approach (7–9). It introduces the impact of health research in 5 categories: knowledge advancement, capacity-building, informing decision-making, as well as impact on health, society and economy. In 2009, detailed indicators for each of the areas covered by the payback framework were suggested by the Canadian Academy of Health Sciences, helping in the measurement of impact (10). Based on this framework and related indicators, several studies have been conducted to evaluate the impact of research in different countries, including Australia, Hong-Kong SAR China, and United Kingdom (11–14). In this study we chose the case study approach using the payback framework as a base and mixed- methods approach (triangulation design with validating quantitative data) for data collection (12). The data collection tool included quantitative and qualitative questions, where narrative responses were used to validate quantitative responses. For example, in one question we asked, “Have the research outcomes been utilized in policymaking outside the health system (directly or indirectly)?”, and we asked principal investigator, “If your answer is yes, please explain how”. The study targeted principal investigators of health research projects funded by WHO/EMRO during the period 2010 to 2018, under 3 grant schemes, i.e. iPIER (14 projects), TDR-SGS” (36 projects), and “RPPH” (48 projects). This period was selected to allow for sufficient time from the completion of the studies, as most research impact requires time to materialize. To assess the impact of funded health research in the EMR, an updated version of a data collection tool used in previous studies (7,8) and had been designed according to the payback framework, was used. The tool included sections enquiring about project objectives; expected impact; impact domains, including “knowledge advancement; capacity-building achieved by conducting research (empowering target group during research, using the outcome of research to define the following projects, to facilitate securing of research funds from other organizations); impact on decision-making (producing systematic reviews, clinical practice guidelines, public health guidelines, health technology assessment, policy briefs, legislation, health policymaking); health and economic impact (using the outcome of research to change some decisions)”; knowledge translation activities; and stakeholder engagement. For each question, we asked about the relevance of the research to making the impact, firstly, and if the response was “yes”, we asked principal investigators to continue. These two last variables were chosen because they increase the probability of changing practice (13). For each claimed impact, we requested attachment of related documents to the online questionnaire. An online version of the questionnaire was prepared, using SurveyMonkey platform and the link to the online tool was sent to all the principal investigators. Two reminders were sent at regular intervals to researchers who did not respond. To verify responses to the quantitative questions, one of the researchers examined the narrative responses and the attached documents, then “perceived invalid responses to quantitative questions” were discussed with another research team member, to reach consensus. Descriptive and analytical statistical methods were used for analysis. For the field “advanced knowledge”, attached documents were peer-reviewed and bibliometric information (impact factor and number of citations) checked using Scopus database. The aggregation variables were defined as follows: Active knowledge translation interventions: including sending a summary or full report to potential users; providing findings to media reporters or participating in interviews; organizing a meeting with potential users; preparing or submitting outcomes in plain language for suitable audience; and taking necessary measures to commercialize the findings. 753 Research article EMHJ – Vol. 28 No. 10 – 2022 Impact on decision-making through decision- making document (IDMD): if research outcomes were used in one of the decision-making documents. Impact on health or the economy (IHE): if the response to one of the 8 questions related to the real use of research results and the impact on health or economy was “yes”. The relationship between some predictor variables and the two variables “IDMD” and “IHE” was investigated. Predictor variables were, the type of grant, the year of proposal approval, active knowledge translation intervention, and order-oriented research subject. First, bivariate analysis was performed by using chi- square test (and if necessary Fisher’s exact test), then, the variables that had significant relationship were entered into multivariable analysis using logistic regression method. Results The questionnaire was sent to the principal investigators of 98 funded studies, of whom 45 responded (response rate: 45.9%). To clarify the selection bias, we compared the response and non-response cases based on the three variables, i.e. type of grant, year of grant approval, and country. Only year of grant approval was found to be statistically different, the response rate was more in the new approved proposals. Respondents reported 10 peer-reviewed publications, 2 factsheets, 1 national roadmap, 1 national strategy document, and 1 preprint paper. For the 10 peer-reviewed publications, the mean citation was 2.83 (minimum 0 and maximum 8) and the mean impact factor of journals that published the works was 2.87 (minimum 2 and maximum 3.29) We observed more visible impact on capacity- building, as summarised in Table 1. Twenty-two research studies (out of 45) reported enhanced capacity (change in awareness, attitudes, skills) of the target groups as a result of the funded study. Fifteen research studies (out of 45) were conducted in response to expressed needs of the policymakers. Tables 2 and 3 present the results of other questions in this section. Despite being relevant, the results of 10 studies were not used in any policymaking. In 17 cases, despite being relevant, the results were not used in national policies and in 3 cases they were used in the local organizational policy. Generally, after omitting missing values, 36 studies had at least been relevant enough to use their results for one of three levels of policymaking. In 20 cases, they were used for decision-making in at least 1 level. Respondents for 23 projects (out of 45) reported that the results of their research could have had direct impact on health if the stakeholders used them, of which 13 research studies were reportedly used (56.5% of 23 projects). For 23 research studies (51%), research results could have had direct impact on economy if the stakeholders used them, of which 7 (30.4%) research results were reported to have been used. Generally, 29 projects (64.4%) could have had direct impact on health or economy; for 14 projects (48%) the results were reported to have been used. For 25 research studies (out of 45), at least one active knowledge translation activity was reported to have been performed; but for 12 projects (27%) no knowledge translation activity was done. Regarding stakeholder engagement, for 26 studies (57.8%), the identified stakeholders did not play a role in research question formulation and proposal writing. Using Chi-squared test, the relationship between IDMD and health or economic impact with predictor variables was examined. None of the variables was found to be associated with the impact on decision-making, however, in terms of the impact on health or economy, the variable related to conducting research in response to policymaker demand was found to be significantly linked (p-value: 0.002 (Table 4). All variables which their P-values were less than 0.1 were entered to the logistic regression (14) (Table 5). No significant association between the country of research and impact on decision-making and implementation of result if health or economic impact was expected. Discussion The objective of this study was to assess the impact of the research projects that were funded by WHO/EMRO from 2010 to 2018 to enable improvements in research policymaking and processes in this organization. There was only a modest potential impact in generating (and sharing) new knowledge, as only 10 research studies (about 1 in 5) reported publishing their results in peer- reviewed publications. The impact on capacity-building was more prominent, as half of research studies reported empowerment of their target groups. The main premise of the enquiry in our study was that condcuting participatory research and integrated collaboration (in which the target group is included in the research team) will empower target groups and increase the probability of using research results (15). A recent WHO study on Table 1 Capacity building in target groups, WHO/EMRO funded research, 2010–2018 Yes (%) No (%) Missing Empowerment of target group 22 (48.9) 21 (46.7) 2 Defining subsequent projects 19 (42.2) 12 (26.7) 14 Facilitating securing of research funds from other organizations 11 (24.4) 32 (71.1) 2 754 Research article EMHJ – Vol. 28 No. 10 – 2022 EMR health research institutions shows that training on this kind of research is rare in the region (16). It is therefore necessary to implement special capacity-building on how to conduct participatory research. One in 4 of the research teams reported that the WHO small grant helped them in receiving subsequent research fundings. This is important in several ways, as it helped gain the trust of relevant organizations (getting credit from providers or executive agencies), and the research results have led to new research questions (which are needed to solve health problems). Regarding impact on decision-making, reported documentation of research outcomes to inform decision- making was very low, usually assessed by the production of decision-making documents in respective countries (11). Less than half of the respondents reported publishing a peer-reviewed article. Further efforts are required to link research to policy and enhance evidence-informed policymaking. A little above half of the respondents (55%) reported using the research outcomes for policymaking. About two-thirds of respondents (64%) reported that their research outcomes had potential to make direct impact on health or economy, while about half (48%) Table 2 Use of research outcomes for decision-making documents, WHO/EMRO-funded research, 2010–2018 No. of relevant cases No. of projects which have been used Unclear Systematic reviews 18 0 3 Clinical practice guidelines 22 1 2 Public health guidelines 41 3 2 Health technology assessment 19 0 2 Educational content for healthcare facility visitors and/or the public 30 2 2 Educational content for professionals 36 1 5 Policy brief development 44 3 5 National/provincial/local legislation 44 1 6 Table 3 Using research outcomes for policy making, WHO/EMRO-funded research, 2010–2018 No. of relevant cases No. of projects which have been used (%) Missing Used for the Ministry of Health policymaking processes 43 14 (32.6) 8 Used for policymaking outside the health system 41 3 (07.3) 8 Used for policymaking within the organization 37 13 (35.1) 8 Table 4 Bivariate analysis of IDMD and health or economic impact with predictor variables Impact on decision- making P-value (χ2) Impact on health/ economy P-value (χ2) Type of grant Yes No 0.330 Yes No 0.125 iPIER 1 5 4 2 RPPH 5 13 5 13 SGS 2 19 5 16 Approval year 0.388 0.734 2014 2 11 5 8 2016 3 6 3 6 2018 3 20 6 17 Active KTE 0.714 0.065 Yes 1 5 4 10 No 7 32 2 29 Conducted upon demand of a specific order 0.444 0.003 Yes 3 12 10 5 No 3 20 4 19 755 Research article EMHJ – Vol. 28 No. 10 – 2022 reported that their research outcomes were used to support health and wellbeing. Regarding impact on decision-making and impact on health and organization, the relevance of research proposals to the impact observed was related to the funding agency’s vision, mission and criteria to accept the proposals, and an impact is looked at as a consequence of contextual factors within each country of implementation. Based on the result of this study, it is essential to change some criteria for accepting proposals to promote the use of evidence in decision-making in countries. On the factors influencing the probability of using research for decision-making, our findings demonstrate the importance of conducting research in response to decision-makers’ needs. One-third of the research studies included in this assessment were conducted in response to policymakers’ needs, and the results were more likely to be adopted for subsequent decisions. About two-thirds of respondents (58%) reported that the stakeholders did not play a role in the formulation of the research question(s) and the writing of the proposal. It therefore seems that it is necessary to improve the research priority-setting in countries and in EMR, and encourage stakeholders to get involved in shaping the research question. One major limitation of our study was that more than half of the study population (those receiving WHO/ EMRO small grants) did not respond to our online survey, in spite of repeated reminders. If non-response could be explained by recall difficulties, then the selection bias by the variable “time” would result in underestimating the impact. If non-response could be attributed to non- impact, then the selection bias has forced overestimation. Another limitation was that during the verification of responses, we found that respondents did not understand some questions properly. A third limitation is insufficient evidence required to compare the health research systems of countries in the region. To be able to assess research impact and its influencing factors and make appropriate recommendations, countries should be compared in terms of health system building blocks, relationship between the health research system and health system, and the situation of KTE components. The main strength of this study was the data collection tool, which included questions about the relevance of the research to making an impact, and the qualitative questions about claimed impact, which helped in verifying the quantitative responses. To strengthen implementation and the impact of research outcomes, we recommend the following actions: • An integrated knowledge translation approach should be observed when approving proposals for funding, i.e. principal investigators should ensure that stake- holders will actively participate in the research cycle from the beginning to the end, and that its outcomes will indeed be used by relevant health policymakers. • Research questions should be based on national, pro- vincial or public health priorities. Therefore, WHO/ EMRO’s calls for proposals, including the small grant schemes, should relate to national priorities, taking into account outcomes of national research priority setting exercises, like the ones conducted in Jordan and Pakistan (17,18). • To strengthen the use of research-generated evidence by health systems, we recommend that countries should regularly conduct research impact assessment and interventions to increase the skills of researchers to identify challenges of the health systems and im- prove their skills regarding the importance of active participation of stakeholders. • To identify factors affecting the implementation of research outcomes, in-depth studies of the research cycle (from the time of planning to the implementa- tion/non-implementation of outcomes) are needed (qualitatively and quantitatively). • It is important to review and compare studies that had impact on health and/or economy and/or deci- sion-making (using the decision-making documents). • We need to conduct situation analysis of the nation- al health research systems in each country, which would help identify gaps, challenges, and priorities and allow better use of resources and facilitate evi- dence-based decision-making. Table 5 Logistic regression for impact on health or economy, WHO/EMRO-funded research, 2010–2018 Independent variables B SE P-value Active KTE 0.568 1.081 0.601 Conducting by demand 0.122 0.818 0.010 Constant 2.953 0.947 0.253 Acknowledgement We thank the principal investigators of WHO-funded grants in the EMR during 2010–2018, who responded to our data collection tool, and hence supported the outcomes and recommendations of this study. We also acknowledge the contributions of Mr Ahmed Elarousy, Mr Hosam Younes, and Ms Fatma Abdel-Megeed for their assistance in developing/ testing the online data collection form. Funding: This study was supported by the Division of Science, Information & Dissemination, WHO/EMRO, Cairo, Egypt. Competing interests: This study was ethically cleared by the EM-Research Ethics Review Committee in June 2020. 756 Research article EMHJ – Vol. 28 No. 10 – 2022 Évaluation de l'impact des petites subventions à la recherche financées par l'OMS dans la Région de la Méditerranée orientale de 2010 à 2018 Résumé Contexte : Depuis des décennies, l'OMS fournit un financement ciblé pour la recherche en santé dans des domaines prioritaires de la santé publique dans la Région de la Méditerranée orientale à travers différents programmes de subventions. Objectifs : Le présent article examine l'impact des programmes de financement du Bureau régional de l'OMS pour la Méditerranée orientale et les facteurs qui facilitent ou entravent cet impact. Méthodes : Nous avons évalué l'impact de la recherche en santé financée par le Bureau régional de l'OMS pour la Méditerranée orientale entre 2010 et 2018 du point de vue de la santé, de l'économie, de la prise de décision et de l'application des connaissances, en mettant l'accent sur la responsabilisation et l'analyse, au moyen du cadre de récupération, de l'approche mixte (quantitative, qualitative) et de la triangulation. Résultats : Les chercheurs principaux de 45 (45,9 %) des 98 projets financés ont répondu au questionnaire. La plupart de ces chercheurs (88,0 %) ont indiqué avoir élaboré au moins un document de prise de décision. Moins de la moitié d'entre eux ont déclaré avoir produit des documents évalués par des pairs et avoir mené des actions d'autonomisation des groupes cibles, tandis que 24,0 % ont signalé avoir obtenu des fonds de recherche par le biais d'autres organisations. Les résultats de recherche de 23 projets (51,0 %) ont pu avoir un impact direct sur la santé et l'économie, et 25 projets (56,0 %) ont permis de mener au moins une activité d'application des connaissances. À l'aide d'une régression logistique multiple, aucune association significative n'a été trouvée entre le pays sur lequel porte la recherche et l'impact sur la prise de décision ainsi que sur la mise en œuvre du résultat lorsqu'un impact sur la santé ou l'économie était attendu. Conclusion : Pour renforcer l'impact de la recherche, le Bureau régional de l'OMS pour la Méditerranée orientale devrait entreprendre une série d'interventions visant à orienter et à autonomiser les pays dans l'utilisation des résultats des recherches. Les disparités entre les systèmes de recherche en santé de la Région de la Méditerranée orientale et les différences dans les capacités individuelles et organisationnelles des différents pays nécessitent des interventions ciblées. 2018و 2010 ْيَماع ينب ام ةدلما في طسوتلما قشر ميلقإ في ةمظنلما اهمعدت يتلا ةيرغصلا ةيثحبلا حَِنلما رثأ مييقت نايديشر شرأ ،يوكين مايس ، ليدنم دحمأ ،هدازدزي ةرابه ةصلالخا قشر ميلقإ في ةماعلا ةحصلل ةيولولأا تاذ تلااجلما في ةيحصلا ثوحبلل اًفَدهتسُم ًلًايوتم ،دوقع ىدم لىع ،ةيلماعلا ةحصلا ةمظنم تم َّدق :ةيفللخا .حَنِملل ةفلتمخ طَطُخ للًاخ نم طسوتلما ءاقترلاا لماوعو ،ةيلماعلا ةحصلا ةمظنلم عباتلا ،طسوتلما قشرل يميلقلإا بتكملل ةيليومتلا ططلخا رثأ صيقت لىا ةساردلا هذه تفده :فادهلأا .هقوعي ام وأ رثلأا اذبه ْيَماع ينب ام ةدلما للًاخ ،طسوتلما قشرل يميلقلإا بتكلما /ةيلماعلا ةحصلا ةمظنم اهتل َّوم يتلا ةيحصلا ثوحبلا يرثأتل ًماييقت انيرجأ :ثحبلا قرط جنهو دئاعلا راطإ مادختساب ،ليلحتلاو ةلءاسلما لىع ديكأتلا عم ،تامولعلما ةجمرتو رارقلا عنُصو داصتقلااو ةحصلا روظنم نم ،2018و 2010 .ثيلثتلاو ،)يعونو يمك( بيلاسلأا ددعتم ينعبرلأاو ةسملخا عيراشلما عيجم تدافأو .)%45.9( ًلاومم اًعوشرم 98 لصأ نم 45 نم نويسيئرلا نوثحابلا نايبتسلاا في كراش دقل :جئاتنلا ءارجإو ،نارقلأا ةعجارلم تعضخ تادنتسم عضوب ينكراشلما فصن نم لقأ دافأو .لقلأا لىع ةدحاو رارق عنُص ةقيثو تعضو انهأ )%88.0( اًبيرقت نأ نكملما نم ناك ،)%51.0( اًعوشرم 23 ةلاح فيو .ىرخأ تماظنم نم ليوتم لىع اولصح منهإ %24.0 لاق ينح في ،ةفَدهتسُلما ةعومجلما ينكتم تامولعلما ةجمرت ةطشنأ نم لقلأا لىع اًدحاو اًطاشن )%56.0( اًعوشرم 25 ذ َّفن مانيب ،داصتقلااو ةحصلا لىع شرابم يرثأت ثوحبلا جئاتنل نوكي ،جئاتنلا ذيفنتو رارقلا ذاتخا ةيلمع لىع عقاولا رثلأاو ثحبلا دلب ينب هب ُّدتعُي طابترا دوجو مدع َّينبت ،ددعتلما يتسجوللا رادحنلاا مادختسابو .ةلاعفلا .اًع َّقوتُم يداصتقلاا وأ يحصلا رثلأا ناك اذإ اهنيكتمو نادلبلا هيجوتل تلًاخدتلا نم ةلسلس ءدب طسوتلما قشرل ةيلماعلا ةحصلا ةمظنلم يميلقلإا بتكملل يغبني ،ثوحبلا رثأ زيزعتل :تاجاتنتسلاا ةيدرفلا تاردقلا في توافتلاو ،طسوتلما قشر ميلقإ في ةيحصلا ثوحبلا مُظُن ينب توافتلا نإف ،كلذ لىع ًةولًاعو .ثوحبلا جئاتن نم ةدافتسلاا نم .ةه َّجوُم تلًاخدت بَّلطتت ،نادلبلا فلًاتخاب ةيميظنتلاو 757 Research article EMHJ – Vol. 28 No. 10 – 2022 References 1. Information, Evidence and Research, Eastern Mediterranean Regional Office, World Health Organization , http://www.emro. who.int/about-who/regional-director/dr-arash-rashidian-director-information-evidence-and-research.html 2. The world health report 2013: research for universal health coverage. 3. Hanney SR, Kanya L, Pokhrel S, Jones TH, Boaz A. How to strengthen a health research system: WHO’s review, whose literature and who is providing leadership? Health Research Policy and Systems. 2020;18(1):72. 4. Alla K, Hall WD, Whiteford HA, Head BW, Meurk CS. How do we define the policy impact of public health research? A systemat- ic review. Health Res Policy Syst. 2017;15(1):84. 5. Guthrie S, Wamae W, Diepeveen S, Wooding S, Grant J. Developing a research evaluation framework: RAND Corporation; 2013. 6. Hanney SR, Ovseiko PV, Graham KER, Chorzempa H, Miciak M. A systems approach for optimizing implementation to impact: meeting report and proceedings of the 2019 In the Trenches: Implementation to Impact International Summit BMC Proceed- ings. 2020;14(6):10. 7. Yazdizadeh B, Majdzadeh R, Janani L, Mohtasham F, Nikooee S, Mousavi A, et al. An assessment of health research impact in Iran. Health Res Policy Syst. 2016;14(1):56. 8. Yazdizadeh B, Mohtasham F, Velayati A. Impact assessment of Iran’s health technology assessment programme. Health Res Policy Syst. 2018;16(1):15. 9. Tadmouri GO, Mandil A, Rashidian A. Biomedical and health research geography in the Eastern Mediterranean Region. East Mediterr Health J. 2019;25(10):728-43. 10. Rashidian A, Mandil A, Mahjour J. Improving evidence informed policy-making for health in the Eastern Mediterranean Region. East Mediterr Health J. 2018;23(12):793-4. 11. El-Jardali F, Mandil A, Jamal D, BouKarroum L, El-Feky S, Nour M, et al. Engagement of health research institutions in knowl- edge translation in the Eastern Mediterranean Region. East Mediterr Health J. 2018;24(7):672-9. 12. JW. C, Plano Clark. VL. Designing and Conducting Mixed Methods Research Second Edition: Sage publication; 2007. 13. LaRocca R, Yost J, Dobbins M, Ciliska D, Butt M. The effectiveness of knowledge translation strategies used in public health: a systematic review. BMC Public Health. 2012;12(1):751. 14. Jewell N. Statistics for Epidemiology. 1st Edition ed: Chapman and Hall/CRC; 2003. 15. Graham ID, McCutcheon C, Kothari A. Exploring the frontiers of research co-production: the Integrated Knowledge Translation Research Network concept papers. Health Res Policy Syst. 2019;17(1):88. 16. Mandil A, El-Jardali F, El-Feky S, Nour M, Al-Abbar M, Bou-Karroum L. Health research institutional mapping: an Eastern Medi- terranean Regional perspective. East Mediterr Health J. 2018;24(2):189-97. 17. Mandil A, Rashidian A, Hajjeh R. Health research prioritization: global and regional perspectives. East Mediterr Health J. 2020;26(3):254-6. 18. Mandil A, Motevalian SA, Altaf A, Obeidat R, Muthu N, Afzal S, et al. Research on health priorities: a regional agenda and its implementation in Jordan and Pakistan. East Mediterr Health J. 2021;27(4):319-20.