Evaluation Approach, Purpose, Objectives and Use

The Cluster Evaluation will assess the progress made by the assessed interventions towards the achievement of the outcomes and goals set under the CO’s Strategic Note, analyze the results achieved and challenges encountered, adjust the implementation modalities as needed and recommend any modifications to the approaches for the projects’ components for the remaining implementation period of the SN. The evaluation will focus on two projects - “Building democratic, peaceful and gender equal society in Ukraine” and “Decentralisation and Law Enforcement Reforms: Transformative approaches to Gender Equality and Women’s Empowerment in Ukraine”.

Specifically, the objectives of this Cluster Evaluation are to:

• Analyse and review the Theory of Change and key assumptions made in light of the current context.
• Analyse the relevance of the UN Women’s approach in the implementation of the initiatives at the national, regional and local levels against the outcomes of the UN Women Strategic Note;
• Assess effectiveness of the projects’ intervention on the target regions and communities;
• Assess organizational efficiency and coordination mechanisms in progressing towards the achievement of the projects’ results as defined in the intervention;
• Analyse and harvest any outcomes towards transformative gender changes at i) individual level and ii) systemic level;
• Assess the intended sustainability of the results and the intervention in advancing gender equality in the regions and communities;
• Identify and document lessons learned, good practices and innovations, success stories and challenges to inform future work of UN Women in the areas of Governance and Participation; Elimination of Violence against Women and Girls (EVAWG); and Women, Peace and Security (WPS);
• Identify strategies for replication and up-scaling of the identified best practices of the interventions during the implementation of the remaining period of the Ukraine’s CO SN.

The findings of the Cluster Evaluation are expected to contribute to effective programming, organizational learning and accountability.

Targeted users of the evaluation are the management and staff of UN Women CO in Ukraine, Government of Norway and Government of Denmark (projects’ donors), the responsible parties, and the government counterparts at local and national levels, CSOs, and other UN agencies, donor community and development partners in Ukraine, and the project beneficiaries. The evaluation should also provide specific recommendations as to the priority areas that should be considered to inform future programming in this area in Ukraine. This would include interventions that require continued support, successful interventions for expansion, and recommendations on prioritizing interventions to maximize impact in the strategic areas to be evaluated. It should also define recommendations to improve project management and maximize ownership by partners. The information generated by the evaluation will moreover be used to engage policy makers and other stakeholders at local, national and regional levels in evidence-based policy dialogues and to advocate for gender-responsive strategies, Women, Peace and Security localization, prevention of Gender-Based Violence, and to inform national reforms with a particular focus on gender equality, human rights, and Women, Peace and Security at local, national and regional levels.

Evaluation Criteria and Key Evaluation Questions

The Cluster Evaluation will address the OECD-DAC evaluation criteria of relevance, coherence, effectiveness, efficiency, and sustainability. More specifically, the evaluation will address the following key questions under each of the criterion.

The evaluation will address the following questions (subject to further discussion with UN Women):

Relevance:

• To what extent was the design of the intervention and its results relevant to the needs and priorities of the beneficiaries? Was the choice of interventions relevant to the situation of the target group?
• To what extent is the intervention consistent with the national development strategies in the area of gender equality and women’s empowerment, and reflect national priorities and commitments on WPS and EVAW?
• To what extent has gender and human rights principles and strategies been integrated into the project design and implementation?
• To what extent does the UN Women possess the comparative advantage in the programme’s project’s area of work in comparison with other UN entities and key partners in Ukraine?
• To what extent the interventions are contributing to the implementation UN Women Strategic Note’s relevant outcomes?
• Do the interventions reflect and align to Ukraine’s national plans on gender equality as well as the country’s internationally undertaken obligations and/or best practices?
• To what extent the interventions are aligned with Ukraine’s UN Partnership Framework (UNPF) 2018- 2022 and nationalized SDGs?
• To what extent are the interventions aligned with international, regional and national agreements and conventions on gender equality and women’s empowerment, and more particularly on ending violence against women and girls?
• To what extent does the design and implementation process of the interventions include a collaborative process, shared vision for results-based delivery?  
• Is the implementation based on quality analysis, including gender and human rights-based analysis, risk assessments, socio-cultural and political analysis?
• To what extent the interventions have been flexible to adapt to the changes in the reform processes at the national, regional and local levels?

Coherence

Internal coherence:

• To what extent does the interventions fit within UN Women’s Strategic Plan and interrelated threefold mandate?
• Are there any synergies and inter-linkages between the interventions and also with other interventions of UN Women?

External coherence:

• To what extent are the interventions consistent with the national development strategies in the area of gender equality, gender mainstreaming and women’s empowerment, and reflect national priorities and commitments on GEEW?
• To what extent the project is in complementarity, harmonized and coordinated with the interventions of other actors’ interventions in the same context?
• To what extend the implementation of the project ensures synergies and coordination with Government’s and key partners relevant efforts while avoiding duplications? 
• To what extent are the interventions achieving synergies with the work of the UN Country Team?
• What is UN Women’s comparative advantage in Ukraine in the thematic areas of work?
• To what extent is project aligned with the UN Development Partnership Frameworks and nationalized SDGs?

Effectiveness

• To what extent have the expected results of the interventions been achieved on both outcome and output levels?
• What are the reasons for the achievement or non-achievement of the intended results? Has project achieved any unforeseen results, either positive or negative? For whom? What are the good practices and the obstacles or shortcomings encountered? How were they overcome?
• How effective have the selected strategies and approaches been in achieving results?
• How well did the interventions succeed in involving and building the capacities of rights-holders, duty-bearers, as well as the project partners?
• To what extent are the interventions’ approaches and strategies innovative for implementation of its interventions?  What -if any- types of innovative good practices have been introduced in the project for the achievement of the results?
• How adaptably and rapidly did UN Women react to changing country context and address the challenges? Have the interventions led to complementary and synergistic effects on broader UN efforts to achieve GEEW in Ukraine?

Efficiency:

• Have resources (financial, human, technical support, etc.) been allocated between the implementing partners strategically to achieve the intended outcomes?
• Has there been effective leadership and management of the programme including the structuring of management and administration roles to maximize results? Where does accountability lie?
• Have the outputs of the interventions been delivered in a timely manner?
• Does the project have effective monitoring mechanisms in place to measure progress towards results?

Sustainability:

• To what extent the interventions succeeded in building individual and institutional capacities of rights-holders and duty-bearers to ensure sustainability of benefits and more inclusive practices to local development and governance?
• How effectively have the interventions generated national ownership of the results achieved, the establishment of partnerships with relevant stakeholders and the development of in-country capacities to ensure sustainability of efforts and benefits?
• To what extent have the interventions been successful in making the linkages and agreements that would ensure the continuation of work on WPS, EVAW and Governance and Participation? What factors are/will be critical to sustainability?
• To what extent has UN Women in Ukraine been able to promote replication and/or up-scaling of successful practices?
• Do key national, regional and local partners including women’s movements and women’s organizations etc. have voice and influence in the context of the implementation of the initiatives?
• To what extent key national, regional and local partners were involved in the conceptualization and design process of the initiatives?
• To what extent the modality led to improved communication, coordination and information exchange within the relevant stakeholders?

These are preliminary questions. During the evaluation’s inception phase, it is expected that the Evaluation Team will develop an evaluation matrix, which will relate to the questions above (and refine them as needed), the areas they refer to, the criteria for evaluating them, the indicators and the means for verification as a tool for the evaluation. Final evaluation matrix with revised and refined evaluation questions will be validated and approved in the evaluation inception report.

The evaluation will be conducted in Ukraine, covering the sites where the projects have been implemented including Kyiv, Donetsk, Luhansk and Zaporizhzhia oblasts. Depending on the restrictive measures related to COVID-19, the evaluation is subject to virtual mode including possible travels to selected communities of the Donetsk, Luhansk and Zaporizhzhia oblasts. Maximum six domestic travels to three oblasts are foreseen only for the National Evaluation Consultant and National Gender Specialist/Coordinator.

Together with the Evaluation Team, the National Gender Specialist/Coordinator will examine all the relevant documents to be provided by UN Women, including logical framework of the projects, Monitoring and Evaluation Plans, annual work plans, reports to the donors, knowledge products produced in the frameworks of the projects, etc.

Evaluation Process and Methodology

The evaluation methodology will deploy mixed methods, including quantitative and qualitative data collection methods and analytical approaches to account for complexity of gender relations and to ensure participatory and inclusiveness processes that are culturally appropriate. Interviews and focus group discussions with all key stakeholders involved in the project implementation, including but not limited to UN Women project teams, local NGO partners beneficiaries, legislative body, central and local government partners, etc.  shall also take place. Led by the Team Lead, the evaluation team will present a detailed methodological approach in the evaluation inception report to be validated by UN Women.

In addition, UN Women is a UN-SWAP reporting UN entity and it has to be taken into consideration that the evaluations managed by UN Women are annually assessed against the UN-SWAP Evaluation Performance Indicator and its related scorecard. The evaluation will be conducted in accordance with UN Women evaluation guidelines and UNEG Norms and Standards for evaluation and the UNEG Code of Conduct for Evaluations in the UN System.

For the purposes of this assignment the evaluation process consists of 1) Inception, which will involve reconstruction of theory of change, evaluability assessment, online inception meetings with the UN Women project  teams, inception report and finalization of evaluation methodology; 2) Data collection and analysis, including desk research and preparation of field missions, visits to the national stakeholders in Kyiv, and visits to projects’ target communities in 3 regions (by the National Consultants only, subject to Covid-19 measures); and 3) Data analysis and synthesis stage, focusing on data analyzed, interpretation of findings and drafting of an Evaluation Report.

The Evaluation Team, comprising of the International Consultant and two National Consultants are expected to deliver the following products:

1)            Inception Report where the Evaluation Team will present a refined scope, a detailed outline of the evaluation design and methodology of the Cluster Evaluation of the two projects, evaluation questions, and criteria for the approach for in-depth desk review and field work (of the two National Consultants, subject to Covid-19 restrictions) to be conducted in the data collection phase. The report will include an evaluation matrix and detailed work plan. A first draft report will be shared with the Evaluation Management Group[1] and, based upon the comments received, the National Gender Specialist/Coordinator will support the Team Lead to revise the draft.  The revised draft will be shared with the Evaluation Reference Group[2] for feedbacks. The Evaluation Team will maintain an audit trail of the comments received and provide a response on how the comments were addressed in the final inception report – by 1 February 2021.   

2)            Conduct field visits to/online meetings with the target communities of Donetsk, Luhansk and Zaporizhzhia oblasts, conduct key interviews and focus group discussions and analyze data. The Evaluation Team will be in charge of arranging the logistics for the field visits and/or the online meetings. Field visits/online meetings must be completed by 15 March 2021. 

3)            Presentation of preliminary findings obtained through the data collection process should be validated through a stakeholder workshop with evaluation management and reference groups towards the end of the primary data collection stage by 30 March 2021.

4)            Draft Evaluation report which will be shared with the Evaluation Management Group for initial feedback. The second draft report will incorporate Evaluation Management Group feedback and will be shared with the Evaluation Reference Group for identification of factual errors, errors of omission and/or misinterpretation of information. The third draft report will incorporate this feedback and then be shared with the ERG for final validation. The Evaluation Team will maintain an audit trail of the comments received and provide a response on how the comments were addressed in the revised drafts – by 20 April 2021.

5)  Final Evaluation report taking into consideration comments and feedback collected from UN Women. The report shall include the following chapters: Executive Summary, Introduction and Background, Evaluation approach and methodology (including limitations), Findings, Conclusions, Lessons learnt, Recommendations and relevant Annexes - by 15 May 2021.

6) Evaluation communication products:  Final PowerPoint/Prezi presentation of the final key evaluation findings and recommendations, and a 2-pager on the final key findings, lessons learned and recommendations (Evaluation Brief, 3-4 pages) in a format preferably adjustable for individual project sites both in English and Ukrainian will be submitted together with the final evaluation report by 15 May  2020.

The Cluster Evaluation will be conducted within period of 15 December 2020 to 15 May 2021. All deliverables will be presented in English except the Evaluation Brief and the Evaluation Communication Product which will also be produced in Ukrainian.

Assignment Deliverables for the National Gender Specialist/Coordinator:

The Gender Specialist/Coordinator will provide inputs to the development of the detailed evaluation framework and all expected evaluation products in order to ensure the Cluster Evaluation is gender responsive, as well as will provide the overall coordination support to the evaluation, including establishing meetings, organising focus group discussions, interviews, etc., on both Kyiv and regional/community levels and participate respectively. The Team Lead is responsible for coordination during all phases of the evaluation process, ensuring the quality of outputs and application of methodology as well as timely delivery of all evaluation products in close collaboration with the Evaluation Task Manager and EMG. The National Evaluation Consultant will provide support to the Team Lead in all the aspects of conducting the evaluation.

In further detail, the duties and responsibilities of the Gender Specialist/Coordinator are as follows:

• Provide inputs to the development of the detailed evaluation framework and all expected evaluation products in order to ensure the Cluster Evaluation is gender responsive;
• Provide support to the National Evaluation Consultant in conducting the analysis of the collected national policy and programme documents, reports, studies etc. within the desk review;
• Provide inputs, along with the Evaluation Team, to identification and formulation of gender-responsive questions and considerations to be used during interviews, focus-group discussions, filed visits for data colllection;
• Provide support to the Team Lead and National Evaluation Consultant with overall gender expertise, ensure required inputs to the inceptions, draft evaluation, final evaluation reports and communication products.
• Support the Team Lead and National Evaluation Consultant with overall coordination of the evaluation process, including establishing meetings, organising focus group discussions, interviews etc, on both Kyiv and regional/community level and participate respectively;
• Support the National Evaluation Consultant in preparation and conduction and take part in field visits or online meetings with the projects’ targeted communities in Zaporizhzhia, Donetsk and Luhansk oblasts (government-controlled areas), Ukraine, in order to ensure necessary data collection through interviews, focus groups etc. with local partners, stakeholders and beneficiaries on both community and regional level.

Inputs:

• The Consultant is expected use her/his own computer.
• UN Women will provide the Consultant with the resource materials, provide assistance in identifying the stakeholders and facilitate contacts with the stakeholders.

Performance evaluation:

Contractor’s performance will be evaluated against such criteria as: timeliness, responsibility, initiative, communication, accuracy, and quality of the products delivered. The evaluation will be carried out and cleared by the UN Women Programme Specialist which will also be the basis for payment on a delivery by delivery basis to the consultant.

Financial arrangements:

Payment will be disbursed by four instalments upon receipt of the deliverables (as detailed above) and their certification by the UN Women Programme Specialist that the work submitted is of the required standards and as per the requirements set out in the TOR.

[1] UN Women will establish an Evaluation Management Group (EMG) comprising of Evaluation Task Manager and delegated staff member/s by UN Women Representative to oversee the evaluation process, make key decisions and quality assure the different deliverables. The EMG will quality assure and approve all deliverables.

[2] The Evaluation Reference Group (ERG) will facilitate the participation of key stakeholders in the evaluation process and will help to ensure that the evaluation approach is robust and relevant to staff and stakeholders. Furthermore, it will make certain that factual errors or errors of omission or interpretation are identified in evaluation products. ERG will provide input and relevant information at key stages of the evaluation including inception report, draft and final reports and dissemination of the results. The Evaluation Reference Group will be engaged throughout the process and will be composed of relevant National Government and local stakeholders, representatives from Civil Society Organizations.

Core Values

• Respect for Diversity
• Integrity
• Professionalism

Core Competencies

• Awareness and Sensitivity Regarding Gender Issues
• Accountability
• Creative Problem Solving
• Effective Communication
• Inclusive Collaboration
• Stakeholder Engagement
• Leading by Example

Please visit this link for more information on UN Women’s Core Values and Competencies: https://www.unwomen.org/-/media/headquarters/attachments/sections/about%20us/employment/un-women-values-and-competencies-framework-en.pdf?la=en&vs=637

Qualifications Education:

• A Master’s Degree or equivalent in social sciences, gender studies, political science, development studies or another related field; Or, Bachelor’s degree in these disciplines with minimum five years’ experience.

Experience:

• Minimum five years of progressive professional experience in the field gender equality and/or women’s empowerment in Ukraine;
• Proven experience in coordination, facilitation of work with various stakeholders;
• Knowledge and previous experience with monitoring and evaluation issues would be an asset;
• Previous experience with development agencies and the United Nations will be an asset.

Languages and other skills:

• Fluent in Ukrainian, English (working level) and Russian (working level).

Evaluation of Applicants:

Applications will be assessed on the three main categories: relevant education, language skills and relevant expertise and competencies, as reflected in the CVs. The categories will be assigned different weighting, which will total to 70 points (100 %). Technical passing score is 49 (70%) points.

Applications will be evaluated based on the cumulative analysis taking into consideration the combination of their qualifications and financial proposal. A two-stage procedure is utilized in evaluating the applications, with evaluation of the CVs being completed prior to any financial proposal being compared. The award of the Special Service Agreements is made to the individuals whose offer has been evaluated and determined as:

• Responsive/compliant/acceptable
• Having received the highest score out of a pre-determined set of weighted technical and financial criteria specific to the solicitation.

Technical Evaluation Criteria

Criterion A – Relevant education – max. 10

Criterion B – Language skills – max. 10

Criterion C – Relevant Experience with total for all stated criteria – max. 50

• Minimum five years of progressive professional experience in the field gender equality and women’s empowerment in Ukraine (25 points);
• Proven experience in coordination, facilitation of work with various stakeholders (15 points);
• Knowledge and previous experience with monitoring and evaluation issues would be an asset (5 points);
• Previous experience with development agencies and the United Nations will be an asset (5 points).

Financial/Price Proposal evaluation:

• Only the financial proposal of candidates who have attained a minimum of 49 points in the technical evaluation will be further considered and evaluated.
• The total number of points allocated for the financial/price component is 30.
• The maximum number of points will be allotted to the lowest price proposal that is opened/ evaluated and compared among those technical qualified candidates who have attained a minimum of 49 points in the technical evaluation. All other price proposals will receive points in inverse proportion to the lowest price.

Evaluation of submitted financial offers will be done based on the following formula: S = Fmin / F * 30

(S - score received on financial evaluation; Fmin - the lowest financial offer out of all the submitted offers qualified over the technical evaluation round; F - financial offer under the consideration).

Application and submission package:

The candidate’s application should include:

• A brief Letter of Interest containing the statement on candidate’s experience in the field of recruitment. Completed and signed UN Women Personal History (P-11) form, that can be downloaded from: https://www.unwomen.org/-/media/headquarters/attachments/sections/about%20us/employment/un-women-p11-personal-history-form.doc?la=en&vs=558.
• Proposal. The financial proposal shall specify a total lump sum amount with a breakdown of a daily professional rate per number of anticipated working days to include daily subsistence allowance, international travel related costs, and other costs (telephone, etc.).
• Interested candidates are requested to apply no later than  Ukraine midnight time on 23 November 2020 by submitting 2 attachments: 1) technical (P-11 and a Letter of Interest) as well as 2) financial proposals to the following e-mail address: [email protected] with a subject National Gender Specialist/Coordinator to Conduct the Gender-Responsive Cluster Evaluation of the Projects “Building Democratic, Peaceful and Gender Equal Society in Ukraine” and “Decentralization and Law Enforcement Reforms: Transformative approaches to Gender Equality and Women’s Empowerment in Ukraine”.
  • Applications without financial proposal will be treated as incomplete and will not be considered for further assessment.

UN Women applies fair and transparent selection process that would take into account the competencies/skills of the applicants as well as their financial proposals.

TOR Annexes

1. UNEG Code of Conduct for Evaluations [1]
2. UNEG Ethical Guidelines[2]
3. UNEG Norms for Evaluation in the UN System[3]
4. UNEG Standards for Evaluation in the UN System[4]
5. UNEG Guidance Integrating Human Rights and Gender in the UN System[5]
6. UN Women Evaluation Handbook[6]
7. UNDP Evaluation Handbook[7]
8. UNFPA Evaluation Handbook[8]
9. UN SWAP Evaluation Performance Indicator and related Scorecard[9]
10. Evaluation Consultants Agreement Form
11. M&E template for Evaluation Report[10]
12. Evaluation Matrix - 

    Table A. Sample Evaluation Matrix

    Relevant evaluation criteria	Qey Questions	Specific Sub-Questions	Data Sources	Data collection methods/Tools	Indicators/Success Standard	Methods for Data Analysis

[1]  http://www.unevaluation.org/document/detail/100

[2] http://www.unevaluation.org/document/detail/102

[3] http://www.uneval.org/document/detail/21

[4] http://www.uneval.org/document/detail/22

[5] http://www.uneval.org/document/detail/1616

[6] http://genderevaluation.unwomen.org/en/evaluation-handbook

[7] http://web.undp.org/evaluation/handbook/documents/english/pme-handbook.pdf

[8] https://www.unfpa.org/sites/default/files/admin-resource/eval_policy_e5_dp_fpa_2013%20%281%29.pdf

[9] http://www.uneval.org/document/detail/1452

       [10] http://procurement-notices.undp.org/view_file.cfm?doc_id=142960

Sample of Financial Proposal

The format shown on the following tables is suggested for use as a guide in preparing the Financial Proposal

Cost Breakdown per Deliverables*(Lump Sum, All Inclusive)

Deliverables - # of days and Percentage of Total Price (Weight for payment) -   Price, UAH

1   

2   

3   

 Total max. # of working days (100%)     UAH ……

*Basis for payment tranches