Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines Report of the joint national/international mission Democratic People’s Republic of Korea, 8–17 October 2016 SEA-Immun-111 Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines © World Health Organization 2017 Some rights reserved. This work is available under the Creative Commons Attribution-Non- Commercial-ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons. org/licenses/by-nc-sa/3.0/igo). Under the terms of this licence, you may copy, redistribute and adapt the work for non- commercial purposes, provided the work is appropriately cited, as indicated below. In any use of this work, there should be no suggestion that WHO endorses any specific organization, products or services. The use of the WHO logo is not permitted. If you adapt the work, then you must license your work under the same or equivalent Creative Commons licence. If you create a translation of this work, you should add the following disclaimer along with the sug- gested citation: “This translation was not created by the World Health Organization (WHO). WHO is not responsible for the content or accuracy of this translation. The original English edition shall be the binding and authentic edition”. Any mediation relating to disputes arising under the licence shall be conducted in accordance with the mediation rules of the World Intellectual Property Organization.. Suggested citation. Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines. New Delhi: World Health Organization, Regional Office for South-East Asia; 2017. Licence: CC BY-NC-SA 3.0 IGO. Cataloguing-in-Publication (CIP) data. CIP data are available at http://apps.who.int/iris. Sales, rights and licensing. To purchase WHO publications, see http://apps.who.int/boo- korders. To submit requests for commercial use and queries on rights and licensing, see http:// www.who.int/about/licensing. Third-party materials. If you wish to reuse material from this work that is attributed to a third party, such as tables, figures or images, it is your responsibility to determine whether permis- sion is needed for that reuse and to obtain permission from the copyright holder. The risk of claims resulting from infringement of any third-party-owned component in the work rests solely with the user. General disclaimers. The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of WHO concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted and dashed lines on maps represent approximate border lines for which there may not yet be full agreement. The mention of specific companies or of certain manufacturers’ products does not imply that they are endorsed or recommended by WHO in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters. All reasonable precautions have been taken by WHO to verify the information contained in this publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall WHO be liable for damages arising from its use. iiiPost-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines Contents Abbreviations .......................................................................................................v 1. Background ............................................................................................... 1 2. Objectives ................................................................................................. 3 3. Methodology .............................................................................................. 5 3.1 Stages of conducting the PIE ............................................................. 5 3.2 Preparatory stage ............................................................................... 5 3.3 Findings of the desk review ................................................................ 7 3.4 Implementation stage ...................................................................... 13 4. Findings and recommendations ................................................................. 18 4.1 Planning and management of the NIP for introduction of two new vaccines ............................................................................ 18 4.2 Immunization service delivery .......................................................... 21 4.3 Vaccine cold chain management and supply chain logistics .............. 24 4.4 Immunization data, coverage monitoring and surveillance ............... 27 4.5 Advocacy, social mobilization and communications ......................... 30 4.6 Assessment of special bodies/institutes at the national Level ............. 31 5. Key lessons learned from the current PIE to inform future vaccine introductions ............................................................................................ 34 6. Principal findings, conclusions and recommendations .............................. 37 6.1 Most remarkable achievements ....................................................... 37 6.2 Most critical issues ........................................................................... 37 6.3 Principal recommendations ............................................................ 38 Annex ............................................................................................................... 39
vPost-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines Abbreviations AEFI adverse events following immunization AFP acute flaccid paralysis BCG Bacillus Calmette–Guérin bOPV bivalent oral polio vaccine CES coverage evaluation survey CMW central medical warehouse CRS congenital rubella syndrome cVDPV circulating vaccine derived poliomyelitis virus DPR Korea Democratic Peoples’ Republic of Korea DQS data quality self-assessment EPI expanded programme of immunization EVM effective vaccine management Hib Haemophilus B HSCC health sector coordination committee ICC interagency coordination committee IEC information, education and communication ILR ice-lined refrigerators IPV inactivated polio vaccine IVD immunization and vaccine development JA joint appraisal JE Japanese encephalitis MR measles-rubella vi Report of the joint national/international mission NCL national control laboratory NIP national immunization programme NITAG national immunization technical advisory group NRA national regulatory authority OPV oral polio vaccine PCV pneumococcal conjugated vaccine PHC primary health care PIE post-introduction evaluation RV rotavirus vaccine SAGE strategic advisory group of experts on immunization SDD solar driven drive SEARO WHO South-East Asia Regional Office Td tetanus-diphtheria tOPV trivalent oral polio vaccine TOR terms of reference UNICEF United Nations Children’s Fund VPD vaccine preventable diseases WHO World Health Organization 1Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines 1 Background On 12 July 2012, the Ministry of Public Health of the Democratic People’s Republic of Korea (DPR Korea) successfully launched the nationwide introduction of the pentavalent (DTP-Hib-Hepatitis B) vaccine. The collaborative partnership between the Ministry of Public Health, United Nations Children’s Fund (UNICEF), World Health Organization (WHO) and Gavi, The Vaccine Alliance, made introduction of the vaccine possible. DPR Korea, by introducing the Haemophilus-b (Hib) antigen through the pentavalent vaccine (DTP-Hepatitis B-Hib), expected to further reduce the overall child morbidity and mortality by significantly reducing the disease burden of Hib-specific pneumonia and meningitis. In addition, it was anticipated that the new vaccine introduction would be an opportunity for DPR Korea to further enhance the capacity of its national immunization programme (NIP). The second new vaccine that DPR Korea introduced was the inactivated polio vaccine (IPV) in April 2015. DPR Korea has maintained poliomyelitis-free status since 1996. However, it is cognizant of the fact that there is a potential threat of wild poliovirus importation from endemic and currently infected countries as well as emergence of circulating vaccine-derived poliovirus (cVDPV) until all countries of the world are polio-free and cessation of oral polio vaccines (OPV) is complete. Hence, DPR Korea was fully aware of the importance of the November 2013 recommendations of the Global Strategic Advisory Group of Experts on Immunization (SAGE) to introduce one dose of IPV in the routine immunization by the end of 2014 and to replace the trivalent oral polio vaccine (tOPV) with bivalent oral polio vaccine (bOPV) in 2016 under the Poliomyelitis Eradication and Endgame Strategic Plan (2013–2018). Taking into consideration the importance of IPV introduction, the Ministry of Public Health with partners (WHO and UNICEF), departments in other ministries including the Ministry of Finance and Foreign Affairs, decided to introduce one dose of IPV in routine immunization. The decision was endorsed at the meeting of the interagency coordination committee (ICC) and the health sector coordinating 2 Report of the joint national/international mission committee (HSCC) on 11 March 2014. An IPV introduction taskforce was formed to operationalize the plan and to oversee its implementation. ICC and HSCC fully endorsed the IPV introduction plan on 17 April 2014. Gavi, the Vaccine Alliance, financially supported the IPV introduction. The vaccine was ceremoniously introduced in routine immunization on 8 April 2015. However, IPV administration has been discontinued since March 2016 due to the global shortage of IPV vaccines. WHO recommends a post-introduction evaluation (PIE) ideally 6 to 12 months after introduction of a new vaccine, to assess the impact of new vaccine introductions on different components of the immunization system in a country. Guided by this recommendation, the Ministry of Public Health planned to conduct a joint national/ international PIE following the introduction of IPV in April 2016. However, this was not materialized due to some logistic issues. Recognizing the importance of conducting the PIE, the Ministry of Public Health with its key partners (WHO and UNICEF) planned to conduct the PIE of the IPV in October 2016. Although a PIE was not conducted to evaluate the impact of the pentavalent (DTP-Hib-Hepatitis B ) vaccine following its introduction in 2012, the Ministry of Public Health felt that the proposed PIE for the IPV would be an opportunity to evaluate the impact of the introduction of the pentavalent (DTP-Hepatitis B-Hib) vaccine. Also taken into account is the fact that an evaluation performed in a period of more than 12 months post-introduction runs the risk of missing the objective findings related to the introduction due to recall bias. However late, the Ministry of Public Health and partners were of the opinion that findings of such a combined PIE enables improving components of the NIP for effective implementation of new vaccine introductions targeted for the future. For example, in the new comprehensive multiyear plan (cMYP) of immunization, measles-rubella (MR), Pneumococcal Conjugate vaccine (PCV) and Rotavirus vaccine (RV) will be introduced. The PIE was also viewed as an opportunity to provide some insight into other aspects of the immunization programme. This includes overall programme management, service delivery system, vaccine safety and quality, surveillance and communication, considering that a full-scale programme evaluation of the immunization programme has never been carried out in the past. 3Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines 2 Objectives The following were the objectives of the post-introduction evaluation: A. General objective § To evaluate the overall impact of the introduction of pentavalent (DTP-Hib- Hepatitis B) and IPV vaccines on the national immunization programme of DPR Korea. B. Specific objectives § To review introduction of pentavalent (DTP-Hib-Hepatitis B) and IPV vaccines in relation to – Planning for introduction of pentavalent (DTP-Hib-Hepatitis B) and IPV vaccines – Advocacy, social mobilization, communication and community acceptance – Immunization data management, coverage monitoring – Training and supervision – Health workers’ knowledge and quality of service delivery – Cold chain management, supply chain logistics and immunization waste management – Vaccine preventable diseases (VPD) and adverse events following immunization (AEFI) surveillance § To describe the programmatic strengths and explore areas for further improvement in the immunization programme, in general, and linked to the introduction of pentavalent (DTP-Hib-Hepatitis B) and IPV vaccines, in particular 4 Report of the joint national/international mission § To formulate recommendations to further strengthen the national programme of immunization based on the PIE § To describe key lessons learned from the introduction of pentavalent (DTP- Hib-Hepatitis B) and IPV vaccines with a view to improving future new vaccine introductions in the country. 5Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines 3 Methodology 3.1 Stages of conducting the PIE The PIE was conducted at national, provincial, county and Ri hospital/clinic levels. It consisted of two stages. Details of how the PIE was conducted at each stage are described below. 3.2 Preparatory stage Development of the methodology for the PIE: the WHO country office in close collaboration with the immunization and vaccine development (IVD) unit of the WHO South-East Asia Regional Office (SEARO) developed the methodology in alignment with WHO guidelines used to conduct a PIE. Adaptation of global, standard, post-introduction evaluation tools to the local context: the WHO country office worked closely with the IVD unit at SEARO, the Ministry of Public Health and UNICEF to adapt the global and standard PIE tools to suit the context of DPR Korea. Finalization of the methodology, PIE tools and approval of the Ministry of Public Health for the proposed PIE: The Ministry of Public Health and in-country partners (WHO and UINCEF), and other in-country stakeholders reviewed the methodology. WHO and UNICEF regional offices, and the focal point of PIEs at WHO headquarters then reviewed the draft for alignment with global guidelines and inclusion of required thematic areas. Once the methodology and the tools were finalized, the Ministry of Public Health provided the necessary approvals to conduct the PIE using the tools. Five different types of questionnaires were developed for administering at the central, provincial, county and Ri levels and also for mothers of vaccinated children. The desk review: A desk review of relevant documents, materials and data available at WHO and UNICEF country offices in DPR Korea was conducted. The documents reviewed are listed in Table 1. 6 Report of the joint national/international mission Table 1: List of relevant documentation pertaining to the current immunization system (for desk review) Category Reviewed materials/documents/data Documents related to immunization § Comprehensive multiyear plan (cMYP) 2011–2015 and 2016–2020 § Medium-term strategic plan (MTSP) 2011–2015 and 2016– 2020 § Gavi annual progress reports (APR) 2012–2014 § Joint appraisal (JA) reports 2015 and 2016 § National policy documents on adverse events following immunization (AEFI) surveillance; injection safety and waste disposal § National immunization guidelines § Immunization recording forms, records, supervisory checklists § Effective vaccine management (EVM) assessment report 2011, 2015 § Cold chain improvement plan with progress reports § Immunization coverage evaluation survey (CES) report 2008 § EPI bottleneck analysis workshop report 2013 § Report of Demographic Health Surveys, 2014 § WHO Country Cooperative Strategy, 2016 Materials related to the pentavalent (DTP-Hib- Hepatitis B) and IPV vaccine introductions § Gavi applications for pentavalent (DTP-Hib-Hepatitis B) and IPV vaccines § Vaccine introduction plan ( pentavalent and IPV ) § Training materials and summary of trainings conducted § Advocacy and information, education and communication (IEC) § Materials related to new vaccine introductions and EPI Data related to the EPI § Immunization coverage data § WHO UNICEF joint reporting form (JRF ) (2011–2015) § SEARO annual EPI reporting form (2011–2015) § Vaccine preventable disease surveillance data (measles-rubella, acute flaccid paralysis, AEFI) § Vaccine wastage rates at different administrative levels § Data related to vaccine supply and logistics Source : WHO and UNICEF country offices 7Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines 3.3 Findings of the desk review The review made it possible for a summary of information, recommendations made during several previous evaluations, and reviews related to the immunization system and the current progress made to date to be prepared. The summary of the desk review, which is given below, provided essential background information for the PIE. A. Background information § DPR Korea is committed to universal and free health care as per the constitution and public health law § Overall, the Ministry of Public Health is responsible for national health programmes § The system of health management is based on centralized planning models § There is no private sector for providing health services in the country § The immunization programme is managed nationally through the central Hygiene and Anti-Epidemic Institute § Vaccination units at county and Ri/Dong clinics provide front-line immunization services § Logistics are managed through the network of central, provincial and county medical warehouses. Figure 1: Functional organogram of the national EPI within the Ministry of Public Health, DPR Korea Central hygiene and an -epidemic sta on Provincial hygiene and an -epidemic sta on County hygiene and an -epidemic sta on Central Medical Ware house Ministry of Public Health Provincial Public Health Bureau County Public Health Department Vaccina on unit (Clinic) Provincial medical warehouse County medical warehouse Source : Ministry of Public Health, Democratic People’s Republic of Korea 8 Report of the joint national/international mission B. Immunization service delivery system § The service delivery system is supported by an extensive network of health facilities. There are 1694 county and Ri hospitals, and 6263 Ri clinics and polyclinics. § The facilities are run by a total of 228 731 trained health staff who are engaged in immunization service delivery. § DPR Korea has one of the highest ratios of health staff to population in the WHO South-East Asia Region. § The household doctor system is the cornerstone of the health service delivery in DPR Korea. One household doctor is responsible for 130 households, and there are approximately 50 000 household doctors in DPR Korea. Household doctors are responsible for provision of all primary health care (PHC) services, including immunization and disease surveillance, in their catchment areas or sections. § Immunization services are delivered through the fixed facility mode on set immunization days in each Ri; while in counties, immunization services are provided through more than 12 000 immunization posts including 7008 PHC units, 433 county level clinics/hospitals, and 130 central and provincial level hospitals and specialized hospitals. § Monthly immunization sessions are held at fixed sites on a fixed day (11th or 12th of each month) throughout the country simultaneously with additional 1–2 days for default tracking. Local-level microplanning practice for immunization service delivery does not exist. C. Immunization schedule and targets § Currently, the NIP provides the following vaccines: Bacillus Calmette–Guérin (BCG), Hepatitis B birth dose, OPV, IPV, pentavalent (DTP-Hib-Hepatitis B), and measles and tetanus-diphtheria (Td). § All vaccines used in the NIP are WHO prequalified and supplied through the UNICEF supply division. 9Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines § Target groups for immunization under the national immunization programme are: – Newborn (BCG, Hepatitis B birth dose) – Infants (other vaccines) – Pregnant women (Td) § The Pyongyang vaccine production unit reportedly produces some vaccines such as the pertussis vaccine, cholera vaccine and Japanese encephalitis (JE) vaccine. However, the vaccine production unit does not follow criteria of good manufacturing practices. These vaccines are neither produced in bulk nor WHO prequalified. They are not currently in use by the national EPI. Table 2: National immunization schedule in DPR Korea Vaccine 1st dose 2nd dose 3rd dose Target group BCG At birth Newborn Hepatitis B birth dose At birth Newborn <24 hours Pentavalent 6 weeks 10 weeks 14 weeks Infants OPV 6 weeks 10 weeks 14 weeks Infants Measles 9 months 15 months Infants IPV 14 weeks Infants Td 3rd month of pregnancy 4th month of pregnancy Pregnant women Source : Ministry of Public Health, Democratic People’s Republic of Korea D. Immunization coverage During the last 5 years, the reported coverage of immunization of all antigens has been very high, and national DPT3, OPV3, MCV2 and TT2 coverages are >95%. Moreover, equity targets were achieved during this time, as 100% of counties reported >90% of DPT3 coverage. The dropout rates are also very low in all counties; the dropout rate from BCG to measles in all counties (100%) was <5%. 10 Report of the joint national/international mission Figure 2: Immunization coverage of DTP 3 in DPR Korea from 2000–2015 100.00 90.00 80.00 70.00 60.00 50.00 20 00 20 01 20 02 20 03 20 04 20 05 20 06 20 07 20 08 20 09 20 10 20 11 20 12 20 13 20 14 20 15 Year C ov er ag e (% ) Source : Ministry of Public Health, Democratic People’s Republic of Korea Figure 2 on DPT3 coverage shows that the country had rapid improvement in DPT3 coverage. It went from a low level of about 60% in 2001–2002 to a consistently high coverage of above 90% from 2006 onwards. The picture is similar for all other antigens. The immunization coverage of all antigens in Figure 3 is a testimony to this. Figure 3: National immunization coverage of vaccines used in the NIP in DPR Korea in 2015 BCG HepB Birth Dose OPV3 IPV3 Penta3 MCV1 MCV2 Vaccine C ov er ag e (% ) 96.7 97.3 98.9 98.9 96 98.4 97.4 Source : WHO/UNICEF Joint Reporting Form 11Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines No CES had been conducted since 2008; currently, plans are under way to conduct a CES in 2017. However, the CES of 2008 found no significant difference between the administrative coverage and coverage estimates from the CES. This adds confidence to the available administrative coverage data. The CES in 2008 highlighted that immunization coverage in the five north-eastern provinces of the country was 3–5% below the national average. During the following years, additional efforts were made in these five provinces to intensify routine immunization based on the findings of the CES (2008) and in accordance with the recommendations of the expanded programme of immunization (EPI) bottleneck analysis workshop was conducted in 2013. Following these interventions, these five provinces have demonstrated significant progress by being now at par with other provinces of the country. E. Cold chain and vaccine management Following a nationwide cold chain assessment in October 2008, the cold chain replacement plan was developed and implemented in 2009–2010. Presently, the national cold store at the central medical warehouse (CMW) is equipped with walk- in cold rooms and walk-in refrigerators; all provincial cold stores have walk-in cold rooms. The county or district level cold stores are equipped with ice-lined refrigerators, deep freezers and in some places, solar driven drive (SDD) refrigerators. Some of the Ri hospitals are equipped with SDD refrigerators; and by 2017, all 1200 Ri hospitals throughout the country will be equipped with SDD refrigerators. Passive vaccine containers are in use at Ri and Dong clinics during immunization sessions. Temperature monitoring is carried out by freeze tags -2, which are available at all levels. For reporting and monitoring of immunization supplies and logistics, including vaccines, an electronic reporting system (e-LMIS) is used from provinces to the national level, while the paper-based reporting system operates from Ri/Dongs to counties/ districts and from counties to provinces. Vaccine transportation from the CMW to provincial warehouses is carried out quarterly using refrigerated and insulated vehicles; and from provincial warehouses to county warehouses, it is carried out monthly using insulated vehicles. From county warehouses to Ri/Dongs, vaccines are transported on the day or just before the day of vaccination in vaccine carriers/cold boxes with icepacks using bicycles and vans. 12 Report of the joint national/international mission F. Vaccine safety and waste management § Plans to develop the revised waste management guidance and new guidance for waste disposal are under way; it has been included as a planned activity under cMYP 2016–2020. § Auto-disable syringes are used for vaccination and also for reconstitution of BCG/measles vaccines in the NIP. § Sharp wastes are stored temporarily in safety boxes. Safety boxes that are filled to capacity are disposed either through incineration at facilities where incinerators are available or through pit burning and burial. G. EVM assessments § EVM assessments were conducted in 2011 and 2015. § EVM assessment of 2011 noted availability of adequate cold chain capacity for existing needs as well as for the introduction of pentavalent vaccine and IPV at all levels of cold chain facilities. § The areas needing improvement were underlined and based on these areas; a cold chain improvement plan was developed. § EVM assessment 2015 has shown substantial improvements since the assessment in 2011. The following key areas of improvement were noted: – The cold chain capacity improved at the Ri level and other supply chain levels. Since the last EVM assessment, 207 new refrigerators were procured; – Fridge tag 2 was introduced countrywide in 2014. This has improved quality of temperature monitoring at all levels; – Planned preventive maintenance of cold chain equipment at county medical warehouses was observed. H. Recommendations of the Gavi joint appraisal mission The Gavi Secretariat with Gavi Alliance partners conducted the joint appraisal (JA) mission to review the Gavi Alliance-supported programme in June 2016. The JA provided several recommendations for further strengthening of EPI in DPR Korea. Recommendations and their status of implementation are as follows: 13Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines § A more independent national immunization technical advisory group (NITAG): The NITAG composition is revised § NITAG would be better led by national independent academics/experts: The present chair is a professor from the Pyongyang Medical College § Further expansion of the cold chain (for RI hospitals and back-up for the central level): On track; significant progress has been achieved § Data quality self-assessment (DQS) and improvement plan: DQS guidelines were drafted. The assessment will commence soon followed by development of the data quality improvement plan § EPI CES was planned for late 2016: Due to the natural disaster in North Hamgyong province, the CES was postponed until April 2017 § Improving communication on minor AEFI: Data collection of minor AEFI has been initiated § Improving VPD and AEFI surveillance: The activity is still in progress § Gender-disaggregated reporting: Reporting forms were developed in July 2016. 3.4 Implementation stage Selection of sites for the review The Ministry of Public Health in close consultation with WHO and UNICEF conducted the selection of sites representing all levels of the health system for the review. The selection was aligned with WHO guidelines on selecting national and subnational level facilities for PIEs. At the national level, the PIE team met with members of NITAG, the national regulatory authority/national control laboratory and staff of the Ministry of Public Health and central warehouse. In addition to the national level, at the subnational level, five provinces, six counties and 14 health facilities were selected for the PIE. However, interviewers managed to interview two provincial EPI teams and five county EPI teams only. Fourteen immunization sessions were observed at 14 selected health facilities. In the process, 49 mothers/caregivers were also interviewed. At the provincial level, two medical warehouses were observed; at the county level, five medical warehouses were observed; and cold chain maintenance at the health 14 Report of the joint national/international mission facility level was observed. Provinces, counties and health facilities selected for the PIE are given in Table 3. Table 3: Provinces, counties and health facilities selected for the PIE in DPR Korea Province County Health facility Number of mothers/caregivers selected Nampo Cholima Wonjong RI polyclinic 5 Kochang RI clinic 3 Daebosan RI clinic 0 S. Pyongan Pyongwon Wolil RI clinic 4 Ryongi RI clinic 3 North Hwanghae Hwangju Reydong polyclinic 3 Soksan RI hospital 5 Kangwong Munchon Samha RI clinic 3 Namchang RI clinic 4 Anbyon Mopung RI hospital 3 Sapyong RI clinic 4 Pyongyang Rakrang Rakrang Dong clinic 3 Tongil Dong clinic 5 Chuongsong Dong clinic 4 Figure 4 shows the location of provinces covered by the joint national/international PIE team. Due to logistical reasons, the furthermost areas of the country could not be covered in the joint mission. However, due to the unique homogeneity of the health system and the NIP across the entire country, the team opines that the findings emanated from the provinces reviewed are likely to vary less with the other provinces in the country and therefore it is reasonable to infer findings of the PIE to the whole expanded programme of immunization. Teams used to conduct the PIE The PIE was conducted by three teams comprising both external and in-country evaluators. The team composition is given in Table 1 of the annexes. 15Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines Thematic areas covered by the evaluation Components of the EPI covered by the evaluation were as follows: § Planning and management of the NIP for introduction of pentavalent (DTP- Hib-Hepatitis B) and IPV vaccines § Advocacy, social mobilization, communication and community acceptance of the new vaccines (pentavalent and IPV vaccines) § Immunization data management, coverage monitoring and supportive supervision § Staff training for two new vaccines introduced (pentavalent and IPV vaccines) § Health workers’ knowledge on immunization in general and two new vaccines introduced § Quality of immunization service delivery Figure 4: Provinces and counties visited by PIE teams Source : Ministry of Public Health, DPR Korea 16 Report of the joint national/international mission § Management of storage of vaccines (cold chain maintenance) and other supplies and logistics § Immunization waste management in the context of two new vaccines introduced § AEFI surveillance with special focus on AEFI due to pentavalent and IPV vaccine § Vaccine preventable disease surveillance with special focus on diseases targeted by pentavalent and IPV vaccines Post-introduction evaluation tools WHO standardized, PIE tools that were adapted to the country setting were used in the PIE. These questionnaires aimed to assess the pentavalent (DTP-Hepatitis B-Hib) and IPV introduction at the (a) national level, (b) provincial level, (c) county level and (d) Ri hospital/Ri clinic level. In addition, another questionnaire was designed to ascertain awareness of the two new vaccines of mothers of children who received the pentavalent (DTP-Hepatitis B-Hib) and IPV vaccines at observed clinic sessions or in the community. Questionnaires assessing the pentavalent (DTP- Hepatitis B-Hib) and IPV at the national, provincial, county and Ri hospitals/clinics entailed an observational checklist to inspect storage areas of vaccines and other immunization supplies. The health facility questionnaire contained a section to assess the knowledge of health-care workers on pentavalent (DTP-Hepatitis B-Hib) and IPV. The other thematic areas included in the questionnaires were (1) documents and data required for the evaluation, (2) background information, (3) integrated planning, (4) pre-implementation planning and vaccine introduction, (5) training, (6) vaccine coverage, (7) vaccine wastage, (8) cold chain management, (9) vaccine management, (10) waste-management and injection safety, (11) supervision, (12) adverse events following immunization (AEFI), and (13) financial sustainability. The questionnaires had both open-ended and close-ended questions. Most of the questions were close ended. However, open-ended questions were also used as guided questions for in-depth interviews to receive additional information on attitudes and impressions regarding the new vaccines and service delivery. The quantitative analysis of the variables in questionnaires used in the PIE is attached in the annex as separate tables (Tables 2–11). 17Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines Data collection Data were collected through interviews, nonparticipatory observations, record and data reviews using standard questionnaires and checklists. At the national level, interviews were conducted with key informants from the Ministry of Public Health related to EPI, vaccine preventable disease surveillance, national regulatory authority, central medical warehouse, and EPI/supply chain logistics management information. Additionally, data were collected from the chairperson/members of the NITAG/ICC, representatives and staff involved in EPI at WHO and UNICEF country offices. In addition to interviews, records were reviewed and practices of dry and cold storage at the central medical warehouse were observed. At the provincial and county levels, interviews were conducted with the EPI focal point and other relevant officials (vaccine management, medical warehouse, and immunization data management). Practices at the medical warehouse were observed, and data/records were reviewed. At the health facility level (Ri hospital/Ri clinics), interviews were conducted with health worker/s who are responsible for immunization activities. Field visits were organized on 12–13 October, 2016 which was the time when immunization sessions were taking place. At all these levels, teams also reviewed existing guidelines, manuals, standard operating procedures, registers, reports and returns related to the immunization system, feed- forwards and feed-backs on immunization system components, immunization databases, etc. 18 Report of the joint national/international mission 4 Findings and recommendations 4.1 Planning and management of the NIP for introduction of two new vaccines A. General observations and strengths § DPR Korea is committed to universal and free health care as per the constitution and public health law. The Ministry of Public Health is responsible for overall national health programmes including the immunization programme. The system of health management is based on centralized planning models. All health services are provided through government health facilities, and there is no private sector for delivery of health services. § The country has a clear programmatic vision that is reflected in the comprehensive multiyear immunization plan for 2016–2020. It spells out the plan for future new vaccine introductions as well. Although the document is well drafted, it is still awaiting final approval. § There is good ownership of the programme and an appropriate national focus on immunization with sufficient resources and structures. Health staff were found to be committed and well trained, with a relatively low turnover. § Access to every family and every child through household doctors is good with well-demarcated catchment areas. This generates the opportunity to provide near universal access for all to benefit from all vaccines in the national schedule including the two new vaccines. Awareness regarding and acceptance of immunization in general and the two new vaccines in particular in the community is good. § Front-line immunization services are provided through health facilities at county and Ri/Dong clinics. Immunization services are provided on a “fixed- day, fixed-site” basis. There is no outreach approach to service delivery. Vaccination takes place using a “one day per month” schedule simultaneously 19Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines throughout the country with another 1-2 days being added for tracking and vaccinating dropouts. § Regular, ongoing, and monthly orientation of EPI staff at all levels is conducted. This opportunity has been well used to provide staff with refresher trainings on key immunization-related topics every month at all levels. This is an ideal platform for orientation of staff during future new vaccine introductions and also to orient new staff not exposed to cascade training on the two new vaccines introduced. § Vaccines and supply chain logistics are managed through the network of central, provincial and county medical warehouses. § New vaccine introduction was found to be well planned and implemented. Overall, the two new vaccine introductions were smooth. § Planned cascade trainings were conducted before the new vaccine introduction at all levels (from national to RI/Dong level). It was observed that in most of these trainings, there was at least one senior level official who was present from the higher administrative level. § All immunization documents and materials including immunization cards were updated to reflect the two new vaccines introduced. B. Challenges and areas needing strengthening § Long-term financial sustainability of the immunization programme is in jeopardy because of the high dependency on donors, especially in vaccine procurement and logistic supply. Nonavailability of funds is also hindering introduction of planned new vaccines, such as MR, PCV and Rotavirus vaccines. In DPR Korea, WHO and UNICEF are the main partners in terms of supporting all non-Gavi-supported vaccines, printing of materials, conducting trainings, supervision, cold chain upgrade, etc. The co-financing requirements for Gavi-supported vaccines have been met in all years except in 2015. Gavi health system strengthening (HSS) support has allowed the country to renew and strengthen its cold chain and immunization supply chain logistic infrastructure facilitating the introduction of two new vaccines. However, the evaluation of Gavi HSS1 has still not been completed and is pending. In this context, increased financial resources from the country are required to ensure sustainability of the immunization programme. 20 Report of the joint national/international mission § The immunization programme offers its service delivery only on one day in a month simultaneously all across the country. This seems to overwhelm the system in terms of transportation of vaccines to all Ris/Dongs on the same day. The one day per month approach all across the country also leaves little opportunity for programme officers to supervise the clinics during immunization sessions. There was limited bottom-up feedback loops, limited flexibility in reacting to unforeseen needs and events, especially in the context of using two new vaccines. § There seems to be a lot of scope in strengthening the intradepartmental programme review and supervision at all levels. Joint reviews of programme performance between the vaccination doctor, cold chain technician and surveillance focal point at the hygienic and anti-epidemic station (HAES), although important from the routine monitoring and also from the enhanced monitoring of new vaccine introduction point of view, were observed to be limited. § The only vaccine that was seen to be out of stock since April 2016 was the IPV vaccine. This is due to global vaccine supply shortage. In relation to the pentavalent vaccine, the new, updated immunization card showed entries with provision of four doses of the pentavalent vaccine. § Although cascade trainings were planned and conducted in the context of new vaccine introduction, the cascade training approach and materials were not well adapted to respective administrative levels. Training materials were limited in content, graphics and illustrations, and were without practical examples (e.g. monitoring chart). § The officials at different administrative levels were not very clear with the open vial policy concept for IPV. The scope of implementation of the open vial policy in the country is limited as vaccinations are delivered on a given day in a month followed by the next session exactly after a month (>28 days). However, it was observed that after the “fixed–day” session, vaccinators keep the vaccines for another 1–2 days for tracking dropouts. In such a situation, it becomes pertinent for health staff to understand the use of open vial policy. § The target set for control of Hepatitis B by the country for 2020 (HBs- Ag prevalence 2% in <5 year olds) is not in agreement with the WHO regional target (≤1%). At the time of conducting the PIE, the Hepatitis B seroprevalence in the country was not known. 21Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines C. A3. Recommendations (1) Update and ensure availability of national EPI guidelines at all levels. (2) Review immunization session schedule and consider different vaccination days for different Ris/Dongs within counties to improve vaccine transport and supportive supervision. (3) Establish regular meetings of programme review for vaccination doctors, cold chain technicians and HAES officers (quarterly at the central and provincial levels; once in two months at the county and Ri levels). (4) Plan and implement the survey to assess the seroprevalence of Hepatitis B and review the national target for Hepatitis B control. (5) Correct vaccination cards to contain three entries for the pentavalent vaccine. (6) Improve the training component, support establishment of a national immunization training centre. (7) Support development of audio-visual aids and more hands-on and innovative training approaches (e.g. through tele-medicine) for new vaccine introduction training programmes. (8) Conduct refresher trainings that focus on vaccination practices, including how best to structure immunization sessions and interpretation of immunization coverage monitoring charts. (9) Advocate for sufficient domestic resources by the Ministry of Public Health for vaccine procurement, Gavi co-financing and other EPI operations to help prepare for future new vaccine introductions, such as MR, PCV, RV, JE (in high-risk areas) in line with the comprehensive multiyear plan for 2016–2020. (10) Plan and conduct evaluation of cycle I of the Gavi HSS grant in DPR Korea. 4.2 Immunization service delivery A. General observations and strengths § There is a strong immunization system with high administrative coverage of all antigens, including the 3rd dose of pentavalent vaccine coverage (> 95%). 22 Report of the joint national/international mission The vaccine wastage rate was within the acceptable range at the national level (pentavalent - 3-5%; IPV - 20%; except for BCG - 75%). § Health workers plan and prepare for immunization sessions with appropriate administration of vaccines. Health workers administered the vaccines, including two new antigens, appropriately at 10 health facilities out of the 14 facilities observed. § Health-care workers have good knowledge of vaccines (including pentavalent and IPV) and vaccine delivery practice, and good maps of catchment areas are available at health facilities. § The household doctor system is the cornerstone of the health service delivery in DPR Korea. Household doctors provide PHC services to the community in their responsible catchment section. They have established a good follow-up and defaulter tracking system of children, which provides a strong foundation for advocacy, high and equitable immunization coverage, surveillance and response to VPD and AEFI. § In terms of waste management practices at health facilities, no discarded needles or syringes on the ground outside facilities were found. Once full, safety boxes from health facilities are sent back to the county for final disposal at the hospital. The two new vaccines have not influenced changes to the existing waste management practices. B. Challenges and areas needing strengthening § Some deviations from recommended safe vaccination procedures were observed, such as unsafe handling of vaccines at some sites, with injection safety guidance not always being followed; for example, swabbing of skin and even needles with cotton swabs. 23Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines § There was also recapping of needles observed before and after vaccination in many health facilities. This practice is a risk for health workers as they are potentially vulnerable to needle-stick injuries. Recapping of the needle was observed in five out of 14 health facilities. It was also observed that health workers prefilled syringes prior to vaccination and before preparing the baby. § Open safety boxes were found in some areas as shown in the picture below. § No child immunization registers were found in some of the clinics. § Immunization cards are kept at the clinic level and they are not given to mothers. Filling the cards during the session is not always immediate and this might lead to erroneous recording and reporting. § Some children may have been missed as evidenced by the difference in targets for different vaccines. § Health workers’ knowledge regarding interpretation of coverage monitoring charts was found to be inadequate. § Activity reports at the Ri/Dong level in areas visited and reports of supervisory visits were not recorded properly in logbooks by supervisors; also no follow- up on implementation of earlier recommendations were made. § There was no heating available in the Ri/Dong vaccination rooms to prevent the freezing of vaccines; this compromised the quality of vaccines and also mother/child comfort. 24 Report of the joint national/international mission C. Recommendations (1) Establish an immunization registration system with an appropriate list of names (immunization registration book at the Ri/Dong level) (2) Update vaccination cards immediately during vaccination sessions (3) Provide a copy of the vaccination card to the mother for her to have the vaccination records at home (4) Encourage and enhance supportive supervision with focus on planning, and arrangement of transportation facilities for supervision, using supervisory checklists, providing written supervisory feedback reports with the signature of the supervisor and appropriate follow-up of recommendations (5) Take steps to improve health-care workers’ understanding and interpretation of immunization coverage monitoring charts (6) Upgrade vaccination rooms with appropriate heating systems 4.3 Vaccine cold chain management and supply chain logistics A. General observations and strengths § The evaluation team agreed that there were no major cold chain issues detected during field visits for the PIE that required immediate action on the part of the national EPI staff. It was noted that in 2015, a full national cold chain inventory had been conducted and that DPR Korea was well equipped and had prepared sufficient volume capacity for all current vaccines, including the two new vaccines introduced. Local UNICEF personnel also confirmed that owing to the capacity planning process, the country can accommodate the additional vaccines scheduled to be introduced by 2020 (i.e. PCV, RV and JE). § The team was pleased to find that a system of back-up electric generators is in place to overcome the occasional problems in the electric grid encountered throughout the country. § Much of the highly praised success of the high-quality cold chain in DPR Korea is owed to its effective temperature monitoring, which is occurring at all levels of the country. Of note is the continuous computerized temperature 25Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines monitoring system found at the provincial level. However, this could be rendered even more beneficial with the exploitation of the SIM card function, which would allow for cold chain technicians to be informed of alarms by way of an SMS. The evaluation team neither encountered any freezing issues nor found any expired vaccines. All vaccine vial monitors inspected were in good, usable stages. The human resource capacity also appears to be appropriate to the needs of the programme. § The vaccine forecasting that occurs at the national level is effective and meets the demands of the country. DPR Korea is also commended for facilitating all customs clearing processes for incoming vaccines shipped by UNICEF. Apart from the expected unavailability of IPV, there were no reports of any stock-outs at any of the levels visited. Furthermore, no vials of tOPV were found anywhere, suggesting that the recently implemented switch from tOPV to bOPV was a successful operation. With the support of the Gavi Alliance through its health system strengthening funds, the vaccine distribution system is being strengthened through the provision of new vehicles at all levels. This is expected to help improve the low score attributed to the “distribution” component (E7) in the EVM assessment conducted in 2015. § Each PIE team that visited the field reported that vaccines were well stored in warehouses, adhering to proper storage principles and bundling of vaccines with injection supplies. However, at lower administrative levels, more caution must be exercised to avoid storing more freeze-sensitive vaccines at the bottom of ice-lined refrigerators (ILRs). Likewise, there were accounts of frozen ice packs being used during transportation of vaccines between the county and Ri level or during immunization sessions. This practice also presents a similar risk of freezing to storing freeze-sensitive vaccines at the bottom of ILR. It is therefore recommended that training efforts be reinforced with the concept of conditioned ice packs for transporting and storage of vaccines in vaccine carriers. EPI staff are also reminded that vaccine vials must be kept in the vaccine carrier during vaccination sessions, with the open vial kept in the accompanying foam pad. This practice was seldom seen well implemented during the vaccination activities observed during the PIE. Instead, vials were frequently found at ambient temperatures having been kept on tables in between vaccinations or in a refrigerator that required frequent opening. It has to be noted that frequent opening of SDD refrigerators is also undesirable, and such infractions could easily be avoided through effective and documented supervision, as well as improved training. 26 Report of the joint national/international mission § Most of the personnel interviewed did not understand properly how and why calculation of vaccine wastage rates is important. This observation in the PIE suggests that this is not commonly carried out and possibly not used in stock management and forecasting. It was also noted that remaining vaccine stocks at the Ri level are not returned to their respective county levels. Nor are they accounted for in documentation and general stock management. Both these measures could have been implemented in the cascade training designed on the eve of the two new vaccine introductions. Implementation of the above recommendations could help render vaccine utilization more efficient. B. Challenges and areas needing strengthening § All facilities and equipment appear to be well maintained, appropriately managed, and functioning well. However, it would be preferable if the dry storage at the CMW was eventually transferred from the tented structure used currently for storage to a more permanent storage site. Also highlighted is the need for ensuring the safe use and care of electric cables serving the respective equipment on which the cold chain relies, so as to avoid jeopardizing the continuous electric supply to this equipment. § Documentation of actions taken in support for maintaining cold chain, including preventive maintenance, was not adequate in most places. C. Recommendations (1) Consider moving dry storage at CMW to a permanent structure. (2) Design a standardized checklist for the cold chain technician for field visits, which allows documented feedback. (3) Install SDD refrigerators at all counties and Ri hospitals, as planned. (4) Improve training on vaccine wastage calculations and utility of tracking wastage. (5) Plan refresher training for vaccinator doctors and vaccine handlers focused on issues identified in the PIE. (6) Strengthen stock management, including appropriate handling of extra stock at all levels. 27Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines 4.4 Immunization data, coverage monitoring and surveillance A. General observations and strengths § Although in some facilities, records were not available, the difference between the dropout rate from DTP1 to DTP3 prior to pentavalent introduction and the same for pentavalent 1 to pentavalent 3 in the post-introduction year were negligible. § The evaluation team found that surveillance of vaccine preventable diseases (VPD) with global commitment, such as acute flaccid paralysis, measles and rubella are in place. They opined that there were sufficient numbers of staff for VPD surveillance in every facility and administrative levels visited. Furthermore, sentinel surveillance such as congenital rubella syndrome, was established in three sentinel hospitals in 2015. § The team also identified keeping population registries (mothers, children, etc.) with information on VPD and other diseases by household doctors (HHD) as the strength behind the country’s health system. § Regular reporting (including zero reporting) on acute flaccid paralysis (AFP) and adverse events following immunization (AEFI) sent to the higher levels of administration are considered important indicators of awareness on surveillance of diseases. They were found to be timely and complete. § With regard to an AEFI, there was an example of one serious case in 2014 being investigated and followed up without disruption to activities of the immunization programme. It demonstrated that the mechanism of managing an AEFI is in place. § To ensure data quality, a national data quality self-assessment guideline has been drafted. B. Challenges and areas needing strengthening § In the areas of immunization and surveillance, targeting children and denominators are important issues. There were different targets and denominators for different vaccines. BCG and Hepatitis B, which both target newborn children, had a target of children different from targets for other 28 Report of the joint national/international mission vaccines. There were different targets of children for other vaccines such as OPV, IPV, pentavalent vaccine, measles and Td. § With regard to reporting and recording procedures, records of data were not available at the point where data are generated once they are submitted to the next level. Also, there was no systematic collection of data that could be used for assessment of the impact of pentavalent vaccine in hospital settings (e.g. in-patient data on pneumonia and meningitis). Another challenge related to preparing for new vaccine introductions was the nonavailability of current operational research pertinent to new vaccines (already introduced or planned for the future). § Surveillance indicators: Although systems are in place, indicators such as the discarded non-measles, non-rubella rates and non-polio AFP rates were not meeting the expected targets levels. § Surveillance of some diseases targeted by vaccines, which either have already been introduced (Hib) or will be introduced in the future in the EPI (Pneumococcal and Japanese encephalitis vaccines), has not been established. These surveillance networks include invasive bacterial vaccine preventable disease surveillance targeting Hib, meningococcal and pneumococcal disease and acute encephalitic syndrome for Japanese encephalitis. § The PIE team suggested that there was insufficient connection between hospital-based surveillance activities and surveillance activities that are under the HAES. Training materials on VPD surveillance were not fully updated and the VPD surveillance system relies highly on passive surveillance with a poor understanding of the active surveillance concept. § AEFI surveillance: The AEFI case definition was outdated. The guideline requires updating along with refresher training. The AEFI surveillance system also needs to be more sensitive given that not one serious AEFI case was reported in 2015 or 2016. Mild AEFI was recorded only for the pentavalent vaccine and limited to local reactions and low-grade fevers. 29Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines C. Recommendations (1) Clarify denominator issues and establish uniform targets for all vaccines. (2) Finalize DQS guideline, conduct training and annual review meetings on data quality and prepare data quality improvement plans. (3) Keep records of data at points where data are generated (at all administrative levels). (4) Provide necessary feedback to informants of VPD cases. (5) Perform operational research for vaccine impact and disease burden assessments (pentavalent, PCV, RV vaccines and targeted diseases including CRS). (6) Conduct immunization coverage survey in 2017. AFP, MR, CRS, VPD surveillance: (7) Encourage culture of reporting of diseases/syndromes under surveillance. (8) Enhance MR and AFP case-based surveillance according to WHO regional surveillance standards. (9) Provide data from current CRS sentinel surveillance to WHO. (10) Initiate IBVPD and AES surveillance. (11) Consider conducting a nationwide VPD surveillance review. Adverse events following immunization (AEFI): (12) Improve sensitivity of AEFI surveillance and AEFI reporting. (13) Update AEFI surveillance guidelines and tools. (14) Perform AEFI training specifically focusing on case definition, case classification and management of AEFI. 30 Report of the joint national/international mission 4.5 Advocacy, social mobilization and communications A. General observations and strengths § The PIE mission viewed advocacy and communication efforts around the introduction of pentavalent and IPV as appropriate. There had been launching ceremonies with the presence of high ministerial and provincial dignitaries for both vaccine introductions, and the mission found evidence of good communication via radio, television, posters and banners. An additional positive feature with both vaccine introductions was the “push messages” sent through the national mobile phone services. These provided recipients with information about the benefits of the new vaccines as well as reminded parents of the need to have their children vaccinated. § Overall, the mission witnessed very good knowledge on and acceptance of all vaccines including the two new vaccines in the communities visited. Almost all (94%) mothers could correctly name the vaccine that their children had received at the observed immunization session; 86% of mothers knew when they would require returning for the subsequent vaccine dose; and 90% were aware that the IPV had been introduced recently in the country. Mothers stated that they had received this information from household doctors (94%) as well as radio (67%) and television messages (43%). None of the mothers had ever been turned away from an immunization session (e.g. due to stock-out of vaccines) and all of them stated that they were satisfied with the immunization services provided. B. Challenges and areas needing strengthening § The PIE mission saw the potential for reinforcing key messages on immunization during immunization sessions, as there were instances of insufficient communication with the mothers, missed opportunities for delivering health education messages related to immunization and other public health interventions. § The mission also viewed the design of some IEC materials, specifically the recent IPV poster, as slightly inappropriate. The large glass syringe and needle does not reflect practices of the 31Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines immunization programme of DPR Korea, which for a long period had been using AD syringes. It may actually look ‘scary’ rather than comforting to mothers and caretakers. C. Recommendations (1) The PIE mission recommends to further improve health-care providers’ skills in health communication during vaccination sessions and to provide them with specific communication materials (e.g. visual aids, handouts). This would enable them to provide state-of-the-art health education messages and information to attending mothers and families for immunization. (2) New communication and educational materials should be vetted by senior communication experts and pretested in community focus groups to ensure appropriate content, messaging and ‘appeal’. In view of the anticipated re- introduction of IPV in 2017/2018, it will be important to re-orient health- care workers and the community at the appropriate time. 4.6 Assessment of special bodies/institutes at the national Level A. National immunization technical advisory group (NITAG) The PIE mission held a meeting with members of the national immunization technical advisory group and witnessed remarkable improvements in NITAG functionality over the past year. NITAG had been established on the basis of a Ministry of Public Health decree, and was involved in all decisions related to the national immunization programme, including all new vaccine introductions. A NITAG Secretariat is now functional and terms of reference (TOR) for NITAG members are available. A total of 11 members have been appointed with a sufficiently diverse technical background and expertise, and external expertise is invited on a needs basis. There are at least two planned meetings per year with an advanced agenda setting and provision of meeting minutes. Additional ad-hoc meetings are convened as necessary. NITAG reports directly to the Ministry of Public Health. Five of six global JRF NITAG indicators of functionality are thus deemed to be met, namely the appropriate legislative basis, the terms of reference, a membership 32 Report of the joint national/international mission with sufficiently diverse areas of expertise, and an appropriate number of meetings held with agenda and background documents provided in advance. The sixth indicator, i.e. declarations of interest of NITAG members, has so far not been met as such declarations had not been required in view of the lack of a private health sector in the country. The PIE mission recommends further consolidating and strengthening NITAG, and calls on WHO to continue to provide close guidance and support. The mission recommends that NITAG review the issue of potential conflicts of interest and – if deemed appropriate – amend their TOR accordingly. NITAG should also include in its near-term agenda a session discussing recommendations for IPV re-introduction, once vaccine supply becomes available again. B. National regulatory authority (NRA) The NRA of DPR Korea was established on the basis of the “Regulation on Authorization of Biological Products” (Ministry of Public Health, DPR Korea 2005, revision 2013). This regulation provides the legal foundation for registration and licensing of drugs, medical products and vaccines, the review of their quality, safety and efficacy, the establishment and performance of a pharmacovigilance system, the conduct of specific studies on safety and efficacy and the recognition of decisions of other NRAs or international bodies. It also regulates the transactions of the state hygiene inspection board and the national AEFI Committee. The mission noted that at present, there are not any acceptable manufacturing practice-compliant production facilities in the country. Given the interest of DPR Korea in domestic production of vaccines and biologicals, an appraisal of the prospect of establishing good manufacturing practice- compliant production facilities should be undertaken. WHO is asked to review the possibility of providing necessary guidance and support for this assessment. C. National control laboratory (NCL) The NCL was established in 1981 for the control of the quality of biological products and vaccines. Given its present infrastructure, the PIE mission evaluates it as not being fully functional. 33Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines The mission therefore recommends performing an in-depth assessment of the NCL capacity and functionality with a review of both human resources as well as infrastructure capabilities. On the basis of recommendations arising from this assessment, the NCL would need to be upgraded and further strengthened. Human resource capacity-building for the NCL could benefit from international experience- sharing and on-the-job training. WHO is requested to provide the necessary support. 34 Report of the joint national/international mission 5 Key lessons learned from the current PIE to inform future vaccine introductions § Noteworthy collaborative partnerships with partners (WHO, UNICEF and Gavi, the Vaccine Alliance) and an intersectoral corporation with different government ministries (Ministry of Public Health, Ministry of Foreign Affairs and Ministry of Finance) and active involvement of ICC/HSCC were instrumental in successful rolling out of the two new vaccines in DPR Korea. § NITAG was pivotal in guiding the government in informed decision-making related to the new vaccines introduced. Further consolidation of NITAG with exposure to evolving new knowledge, methods and technology will immensely benefit future new vaccine introductions. § The concept of the pentavalent vaccine introduction taskforce and the IPV taskforce that convened to operationalize introduction plans and oversee implementation could serve as a model that could be replicated in future new vaccine introductions for effective implementation of these new vaccine introduction plans. § The government’s commitment to provide free and universal access to health care supported by the vision of the national EPI and strengths of the national health system such as the system of household doctors with clearly demarcated catchment areas, the committed, knowledgeable health staff, nationwide service delivery points and well-organized, functional vaccines and supply chain logistics system down to the grass- roots level were inalienable in achieving a high, equitable immunization coverage for new vaccines introduced. The favourable government policies, vision of the EPI and strength of the health system act as the backbone for future new vaccine introductions. § Although the new vaccine introduction was smooth and the overall service delivery was good, there were specific thematic areas where service delivery, 35Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines staff capacity enhancement, recording, reporting and ensuring quality of immunization data, disease/AEFI surveillance, etc., could have been further strengthened parallel to introduction of the two new vaccines. Lessons learned call for effectively using the new vaccine introduction process itself in the future to further improve specific components of the immunization system to delivering high-quality vaccination services to service users in addition to the very high coverage already achieved. § Advocacy and communications around the new vaccine introduction and innovative measures such as “push messages” via the local mobile network were helpful in enhanced knowledge of parents and demand generation for targeted new vaccines. While this could be a strong platform for advocacy and communication for future new vaccine introductions, the performance of demand generation could be further enhanced by greater involvement of communication experts in developing appropriate content and appeal of communication materials and pretesting these in community groups. § In collaboration with partners, organization of the EVM assessment, preparation and implementation of the cold chain improvement plan with sufficient time in advance to the new vaccine introduction and thus bridging the incremental cold chain capacity gaps generated not only due to the new vaccines introduced but also to the new vaccines earmarked for future introduction in the cMYP was exemplary. Learning from this experience, adopting a similar approach to improve the dry storage capacity will be beneficial for routine immunization and future new vaccine introductions. § Lessons learned from PIE call for planning for designing and integrated implementation of surveillance of diseases, which the introduced new vaccines targeted parallel to rolling out these vaccines. It will help in assessing the impact of the newly rolled-out vaccine. Improving collection of data through alternative data sources, such as routinely collected data such as hospitalized pneumonia, meningitis cases too could help vaccine impact assessment. Since country plans to introduce MR, JE, PCV and RV in the future as per the cMYP, designing strategies such as sentinel surveillance based on lessons learned from the PIE could be useful to generate evidence before introduction and to assess the impact of vaccine in the post-introduction of the above vaccines. 36 Report of the joint national/international mission § Currently, there is a high dependency of the national immunization programme on donors. This dependency is a stumbling block not only to introduce future new vaccines appropriate to the country, but also to sustain delivery of current vaccines in the EPI schedule. These include non-Gavi supported vaccines and immunization supplies and logistics. The current scenario offers the country the development of a comprehensive sustainable immunization plan with a realistic resource mobilization plan, including domestic resources based on the new cMYP, to drive the country towards the full ownership of the programme in the long run. 37Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines 6 Principal findings, conclusions and recommendations The detailed findings, conclusions and recommendations have been described in the previous sections concerning the different areas of the review. However, the team felt that a section on major findings, conclusions and recommendations could be useful so that the programme can concentrate on priority issues to further strengthen the already strong immunization programme in the country. 6.1 Most remarkable achievements § The strong immunization system (EPI) within the strong health system § Good acceptance of immunization by the community § Committed health staff § High vaccination coverage § Well-equipped and functional cold chain and effective supply chain logistic system § No vaccine stock-outs § Well implemented pentavalent and IPV vaccine introductions 6.2 Most critical issues § Sustainability of the overall national immunization programme is challenged due to existing gaps in resources needed for supporting the programme. § Vaccination sessions conducted only once a month simultaneously across the country exert enormous strain on ensuring vaccine quality, transportation of the vaccines, supervision and monitoring. 38 Report of the joint national/international mission § Varying targets and denominators for different vaccines, especially at the primary health-care level, do not conform to the national and international recommendations and give a skewed picture. § Child immunization registers, which have proven to be a good tool for programme supervision, were found to be not in use. § There was an insufficient documentation of programme activities. § There was a lack of systematic supportive supervision. § There was a hesitancy in reporting VPD and AEFI cases. § There was a lack of coordination between vaccination doctors, cold chain technicians and HAES surveillance focal points. § Safe injection practice is not consistently followed. § Inadequate functional status of the national control laboratory (NCL). 6.3 Principal recommendations (1) Revise the immunization session schedule. (2) Fix varying denominators for different vaccines. (3) Take measures to ensure financial sustainability of the national immunization programme. (4) Update the EPI Manual, recording and reporting forms including vaccination cards and registers. (5) Strengthen supportive supervision. (6) Strengthen AFP, MR, CRS and AEFI surveillance. (7) Establish joint review mechanisms at all levels (vaccination doctor, cold chain technician, surveillance focal point). (8) Redesign training contents and modalities. (9) Focus on safe injection practices and the documentation process. (10) Prepare a plan to strengthen the national control laboratory (NCL) after thorough assessment by an international expert from the relevant field. 39Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines Annex Table 1: List of evaluators of the post-introduction evaluation and the administrative levels/facilities surveyed, teams interviewed Team Team members Areas/facilities visited, teams interviewed Central team 1. Dr Carsten Mantel (WHO, Geneva) 2. Mr Yann Folly (Gavi Secretariat) 3. Dr Zobaidul Haque Khan (WHO, DPR Korea) 4. Dr Muhammad Tariq Iqbal (UNICEF, DPR Korea) 5. Dr Jang Ra Son (WHO, DPR Korea) Visited Pyongyang city 1. Met the national EPI team 2. Met the VPD surveillance team 3. Visited the central medical warehouse 4. Met the NITAG members 5. Met the ICC members 6. Visited the NRA Provincial team 1 1. Dr Selina Ahmed (WHO, Bangladesh) 2. Dr Asmaniar Saleh (WHO, Indonesia) 3. Anna-Lea Kahn (WHO, Geneva) 4. Dr Devashish Dutta (UNICEF, EAPRO) 5. Mr Kim Chol Nam (Ministry of Public Health, DPR Korea) 6. Dr Sin Un Suk (WHO, DPR Korea) Visited Kangwon province 1. Visited provincial medical warehouse 2. Met the provincial EPI team 3. Met EPI teams in two counties 4. Visited six Ri hospitals Provincial team 2 1. Dr Balwinder Singh Chawla (WHO, India) 2. Dr Sidik Utoro (WHO, Indonesia) 3. Dr Kim Jong OK (UNICEF, DPR Korea) 4. Dr Kim Chol Su (Ministry of Public Health, DPR Korea) Visited Nampo city 1. Met the Nampo EPI team 2. Visited 1 district 3. Visited 3 Ri hospitals in Nampo city Visited South Pyongan province 1. Visited 1 county 2. Visited 3 Ri hospitals Visited North Hwanghae provinces 1. Visited 1 county 2. Visited 3 Ri hospitals 40 Report of the joint national/international mission Table 2: Advocacy, social mobilization, communication and training activities performed at provincial, county and health facility levels Advocacy, social mobilization, communication, training Provinces (n=2) County (n=5) Health facilities (n=14) Penta IPV Penta IPV Penta IPV Conducting an official launch ceremony at the time of the vaccine introduction 0 0 01 01 – – Conducting the training before introduction 02 02 05 05 – – Conducting the training after introduction 0 0 0 0 – – Encountering problems in conducting training 0 0 0 0 – – Receiving health education materials for training Not relevant Not relevant Not relevant Not relevant 13 13 Community resistance to new vaccines Not relevant Not relevant Not relevant Not relevant 0 0 Staff participation in the training Not relevant Not relevant Not relevant Not relevant 13 14 Mean duration of the training Not relevant Not relevant Not relevant Not relevant 2 days 2 days Including immunization topics other than the vaccine of interest in the training Not relevant Not relevant Not relevant Not relevant 07 10 Availability of new vaccine vials( including dummies) for the training Not relevant Not relevant Not relevant Not relevant 14 14 Opportunities to practice skills in administration of new vaccines Not relevant Not relevant Not relevant Not relevant 14 14 41Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines Advocacy, social mobilization, communication, training Provinces (n=2) County (n=5) Health facilities (n=14) Penta IPV Penta IPV Penta IPV Materials available for training Pentavalent vaccine introduction guidelines Not relevant Not relevant Not relevant Not relevant 09 09 Posters Not relevant Not relevant Not relevant Not relevant 10 08 Record and reporting forms Not relevant Not relevant Not relevant Not relevant 07 06 Training materials Not relevant Not relevant Not relevant Not relevant 14 13 Others Not relevant Not relevant Not relevant Not relevant 0 02 Availability of training materials used for training at the facility Not relevant Not relevant Not relevant Not relevant 12 12 Availability of room for improving training in a future new vaccine introduction Not relevant Not relevant Not relevant Not relevant 09 Overall training provided was satisfactory Not relevant Not relevant Not relevant Not relevant 14 42 Report of the joint national/international mission Table 3: Reported coverage and dropout rates of 1st and 3rd doses of DTP and pentavalent vaccines in the year before (2011) and after introduction (2013) of the pentavalent vaccine and in the year before (2014) IPV introduction Level DTP1 -2011 DTP3- 2011 Dropout rate from DTP1-3 Penta1 2013 Penta3 2013 Dropout rate from Penta1-3 Pentavalent coverage 2014 Provincial level Kangwon NA NA NA 93.3% 93.0% 0.23% 92.6% South Pyongan 100% 100% 0 92.9% 92.3% 0.01% 92.6% County level Rakrang District, Pyongyang NA NA NA NA 98.8% NA 99% Munchon County 95.8% 94.2% 1.73% 93.2% 99.2% -6.4%% 99.1% Cholima District 94.9% 92.9% 2.0% 93.6% 93.2% 0.5% 93.4% Pyongwon County NA NA NA 93.7% 93.4% 0.3% 98.9% Hwang Ju County NA NA NA NA NA NA NA Rankang Dong Clinic – – – – – – 100% Tongil Dong Clinic – – – – – – 100% Chuong Song Dong Clinic – – – – – – 98.5% Samha Ri Hospital NA NA NA – – – NA Namchang Ri Clinic NA NA NA – – – NA Mopung Ri Hospital NA NA NA – – – NA 43Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines Level DTP1 -2011 DTP3- 2011 Dropout rate from DTP1-3 Penta1 2013 Penta3 2013 Dropout rate from Penta1-3 Pentavalent coverage 2014 Sapyong Ri Clinic 93.2% 93.2% 0 – – – 95.8 Wonjong RI Polyclinic 100% 100% 0 – – – 100% Kochang RI 100% 100% 0 – – – 100% Daebosan Ri NA NA NA – – – NA Wolil RI clinic NA NA NA – – – NA Ryong RI Hospital NA NA NA – – – NA Reydong Polyclinic 100% 89% 11% – – – 98.2% Soksan Hospital 100% 100% 0 – – – 100% 44 Report of the joint national/international mission Table 4: Planning immunization sessions, target children for immunization, immunization data reporting and coverage monitoring Provinces (n=2) County (n=5) Health facilities (n=14) Number of clinic sessions per month Not relevant Not relevant 01 day in each facility Availability of a map of the catchment area Not relevant Not relevant 13 Knowledge of the target children for vaccination Not relevant Not relevant 13 Mean (SD) number of targeted children in health facilities surveyed Not relevant Not relevant Mean=144 SD=106.8) Range: 47–432 Availability of immunization coverage monitoring charts Not relevant Not relevant 08 Up-to-date immunization coverage monitoring charts Not relevant Not relevant 07 Ability to explain the immunization coverage charts Not relevant Not relevant 01 Monthly immunization compilation reports submitted timely 02 05 Not relevant Monthly immunization compilation reports submitted timely 02 05 Not relevant Availability of a defaulter tracking system/method/strategy Not relevant Not relevant 14 Monthly reporting to the county/ district level Not relevant Not relevant 14 Availability of updated registers to include pentavalent vaccine and IPV Immunization cards Not relevant Not relevant 14 Tally sheets Not relevant Not relevant 13 Child registration books Not relevant Not relevant 07 Vaccine indent forma Not relevant Not relevant 07 45Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines Table 5: Vaccine management, vaccine wastage, immunization waste management and injection safety Vaccine management, vaccine wastage, immunization waste management and injection safety Provinces (n=2) County (n=5) Health facilities (n=14) Vaccine management Availability of operational guidelines for vaccine management 02 05 Not relevant Change of estimated needs with the introduction of two new vaccines 0 03 Not relevant Presence of vaccine stock-outs during October 2015–October 2016 02 0 Not relevant Availability of expired vaccines during October 2015–October 2016 0 0 Not relevant Availability of vaccines with VVM stage III or IV during October 2015–October 2016 0 0 0 Practice of bundling vaccines with vaccine supplies 02 05 Not relevant Facilities where bundling was verified by stock records 0 03 Not relevant Vaccine wastage, waste management and injection safety Facilities that calculated vaccine wastage rates 01 04 Vaccine wastage rates for pentavalent vaccine in 2015 6.7% 1%-3% Vaccine wastage rate for IPV in 2015 46.4% 15%-21.2% Number of facilities that follow national immunization waste management guidelines 02 05 Number of facilities that had to make changes to their waste disposal systems to introduce pentavalent vaccines 0 0 0 Number of facilities that had to make changes to waste disposal system to introduce IPV vaccines 0 0 0 Number of facilities that incinerated safety boxes Not relevant Not relevant 0 Number of facilities that pit-burned and buried safety boxes Not relevant Not relevant 0 Number of facilities that sent safety boxes to higher facilities for disposal Not relevant Not relevant 14 46 Report of the joint national/international mission Table 6: Monitoring and supervision activities at provincial, county and health facility levels Monitoring and supervision Provinces (n=2) County (n=5) Health facilities (n=14) Availability of a supervisory plan for immunization activities 02 02 Not relevant Frequency of supervision to the county/ Ri level Annual Generally 1 per month 5 per month in one facility Not relevant Facilities that had records of supervisory visits made to the counties/Ri level after introduction of pentavalent vaccines 02 (as a part of annual supervision) 05 (1–2 supervisions per month) Not relevant Facilities that had records of supervisory visits made to counties/Ri level after introduction of IPV 02 Annual 05 1–2 /month Not relevant Facilities that followed up with sites of inadequate performance and continuing problems 02 05 Not relevant Facilities that received visits from the higher level in the previous 12 months 02 05 14 Facilities that provided written supervisory feedback to counties/Ri levels 01 02 Not relevant The range of visits received from the higher level per month/year – 1–3 month 2–24/year Facilities that had the supervisory visit from the higher level documented 0 Both reported visits but were undocumented 0 10 Facilities that had a formal, written supervisory report of the visit from the higher level 0 0 10 47Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines Table 7: Cold chain management and observation of vaccine storage areas Cold chain management and observation of vaccine storage areas Provinces (n=2) County (n=5) Health facilities (n=14) Cold chain management Facilities that use freeze watch monitors during transportation 02 05 Not relevant Facilities that had freeze tags 01 04 Not relevant Facilities that had records of inadvertent freezing of vaccines 0 0 Not relevant Observation of vaccine storage area Facilities that had refrigerators which were all clean 02 05 Not relevant Facilities that had refrigerators which were all properly functional 02 05 Not relevant Facilities that had freeze tags inside cold rooms 0 02 Not relevant Facilities that had fridge tags inside cold rooms 02 05 Not relevant Facilities that had other temperature monitoring devices inside cold rooms 02 – Not relevant Facilities that had working freeze tags inside cold rooms Not relevant 02 Not relevant Facilities that had working fridge tags inside cold rooms 02 05 Not relevant Facilities that had functional, other temperature monitoring devices inside cold rooms 02 – Not relevant Facilities that had maintained storage temperature between +2-+8 C in refrigerators 02 05 Not relevant Facilities that had temperature monitoring charts for every freezer and refrigerator 02 05 Not relevant Facilities that had freeze tag data that corresponded with the e with figures marked in temperature monitoring charts Not relevant 05 Not relevant 48 Report of the joint national/international mission Cold chain management and observation of vaccine storage areas Provinces (n=2) County (n=5) Health facilities (n=14) Facilities that measured temperature twice daily 02 05 Not relevant Facilities that monitored and recorded temperature during weekends and public holidays 02 05 Not relevant Facilities that had stored vaccines using “first expiry- first out” method 02 05 Not relevant Facilities that had expired vaccines during the PIE 0 0 Not relevant Facilities that had vaccines in the freezers and refrigerators that had VVM in the usable stage 02 05 Not relevant Facilities that had spaces between vaccine boxes/trays to allow air circulation 01 04 Not relevant Table 8: Observations at immunization sessions Observations at the immunization sessions Health facilities (n=14) Facilities that reconstituted respective vaccines correctly 09 Facilities with proper storage and handling of vaccines during the immunization session 08 Facilities where appropriate administration techniques were observed 10 Facilities that used AD syringes for injecting vaccines 13 Facilities that recapped needles 05 Facilities where syringes and needles were properly disposed of in a safety box 12 Facilities that had vaccine vials with expired dates that should have been discarded 00 Facilities that had diluents kept in the cold chain prior to reconstitution and vaccine administration 10 Facilities that had emergency lifesaving medicines available to manage severe AEFI 11 Facilities that maintained a vaccine stock register 10 49Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines Table 9: Surveillance of AEFI and vaccine preventable diseases AEFI surveillance Provinces (n=2) County (n=5) Health facilities (n=14) Facilities that had a system for monitoring and reporting AEFIs for all vaccines 02 05 14 Facilities that had written protocols for AEFI monitoring and reporting 02 05 14 Facilities with availability of an AEFI registry Not relevant Not relevant 14 Facilities where AEFI reporting/investigation forms were available Not relevant Not relevant 14 Facilities that had completed AEFI investigation forms for the last 6 months Not relevant Not relevant 0 Number of AEFI cases reported during the last 3 years (2013–2016) Not available Not available 0 Facilities that had AEFI rates or sources to verify AEFI rate pre-pentavalent vaccine introduction Not available Not available 0 Facilities that had AEFI rates or sources to verify AEFI rate post-pentavalent vaccine introduction Not available Not available 0 Facilities that had AEFI rates or sources to verify AEFI rate pre-IPV introduction Not available Not available 0 Facilities that had AEFI rates or sources to verify AEFI rate post-IPV introduction Not available Not available 0 Number of AEFI due to pentavalent vaccine Not available Not available 0 Number of AEFI due to IPV Not available Not available 0 Number of AEFI due to other vaccines Not available Not available 0 Number of serious AEFI due to pentavalent vaccine Not available Not available 0 Number of serious AEFI due to IPV Not available Not available 0 Vaccine preventable disease surveillance Number of all VPD cases reported during the last 3 years 0 3 AFP cases ( 2014-2015) Not relevant Facilities that reported having problems with VPD surveillance 1 0 Not relevant 50 Report of the joint national/international mission Table 10: Health-care workers’ knowledge and practices Health-care workers’ knowledge and practices Health- care workers (n=14) Correct knowledge on immunization schedule for the pentavalent vaccine 14 (100%) Correct knowledge on immunization schedule for the IPV 14(100%) Correct knowledge on administration of pentavalent vaccine 14 (100%) Correct knowledge on administration of IPV 12 (86%) Correct knowledge on diseases prevented by the IPV 14 (100%) Correct knowledge on diseases prevented by the pentavalent vaccine 14 (100%) Provision of information on the name of the vaccine to parents 07(50%) Provision of information on diseases the two new vaccines protect against 08 (67%) Provision of information on benefits to the child and the family 09 (75%) Provision of information on vaccine schedules and when to return 07(50%) Provision of information on common side effects 07(50%) Provision of information on what they should do for side effects 07(50%) Provision of other health messages 01 (7%) Problems encountered in administration of two new vaccines 0 Problems encountered in administration of two vaccines simultaneously 0 Delivery of other health interventions by the immunization staff 07(50%) 51Post-Introduction Evaluation (PIE) of Pentavalent (DTP-Hib-Hepatitis B) and Inactivated Polio Vaccines Table 11: Findings of interviews with mothers Findings of interviews with mothers Mothers (n=49) Availability of the child immunization card with the mother 49(100%) Availability of a new card updated to include pentavalent vaccine and IPV 49(100%) Status of receiving age appropriate vaccines at the time of the interview 49(100%) Ability to name the vaccine/s received on the day of the interview 46(94%) Awareness on the recent introduction of the IPV to the national EPI schedule 44(90%) Sources of information on the newly introduced IPV vaccines 1. Household doctors 46(94%) 2. Radio 33 (67%) 3. Television 21(43%) 4. Newspaper 4(8%) 5. Health worker 3(6%) 6. Others 9(18%) Ability to name the due date for the next vaccination 42(86%) Experience of AEFI for earlier vaccines that the child received 8 (16.3%) Readiness to continue with future vaccinations if an AEFI appears after vaccination 37(76%) Experience of being turned away without vaccination during a clinic session 0 Satisfaction with the immunization services offered 49(100%)