January 2010 MONITORING AND EVALUATION PROCEDURES VERSION 3 Collaboration for Agriculture & Rural Development Program Vietnam January 2010 CARD – M&E Procedures January 2010 TABLE OF CONTENTS Executive Summary List of Abbreviations 1 CARD M&E FRAMEWORK 2 1.1 Introduction 2 1.2 Overview 2 1.3 Project Level M&E 3 1.4 Program Level M&E 3 1.5 Institutional Level M&E 4 1.6 Where M&E Fits in the CARD Project Cycle 4 1.7 Purpose of this Document 5 2 BASIC PRINCIPLES OF MONITORING AND EVALUATION 5 2.1 Overview 5 2.2 The Five Key Questions 5 2.3 Different Projects, Different Approach 7 3 M&E IN THE CARD PROGRAM 7 3.1 Overview 7 3.2 Logical Framework (Logframe) Methodology 7 3.3 Intermediate and Final Outcomes/Impacts 9 3.4 Designing for Impact 9 3.5 When to Monitor and Evaluate? 10 4 TECHNIQUES OF M&E AND IMPACT ASSESSMENT 11 4.1 Performance Indicators 11 4.2 Information Sources and Timing of Impacts 12 4.3 Using Negative Findings 13 4.4 Baseline Information 13 4.5 Options for Comparison 15 4.6 Contribution Analysis 15 4.7 Specific M&E Tools 16 4.8 Impacts to be Assessed 18 4.9 Environmental Monitoring 19 4.10 Judging a Project’s Success 20 5 BENEFIT COST ANALYSIS IN M&E 20 5.1 Overview 20 5.2 Identifying Benefits and Costs 21 5.3 The Representative Farm Concept 22 5.4 Enterprise Budgets 23 5.5 Comparing Benefits and Costs 23 6 WORKPLAN FOR EVALAUTION OF THE CARD PROGRAM 24 6.1 Overview 24 6.2 Mid-Term Reviews 25 6.3 Project and Program Completion Evaluations 25 6.4 Ex Post Evaluations 26 6.5 Further Training needs 26 6.6 Conclusions 27 Attachment 1: Terminology and Definitions Attachment 2: Procedures and Terms of Reference for Mid-Term Reviews CARD – M&E Procedures January 2010 Attachment 3: Procedures and Terms of Reference for Project Completion Evaluations Attachment 4: Enterprise Financial Analysis Templates LIST OF ABBREVIATIONS AMC Australian Managing Contractor APR Annual Progress Report AusAID Australian Agency for International Development BCA Benefit Cost Analysis BCR Benefit Cost Ratio CARD Cooperation for Agriculture and Rural Development EIA Environnemental Impact Assessment EMP Environnemental Management Plan EOI Expression of Interest IFAD International Fund for Agricultural Development M&E Monitoring and Evaluation MARD Ministry of Agriculture and Rural Development MTR Mid-Term Review PCC Project Coordinating Committee PCE Project Completion Evaluation PCR Project Completion Report PMU Program Management Unit STED Science and Technology and Environment Department (of MARD) TAP Technical Advisory Panel TOR Terms of Reference ACKNOWLEDGEMENTS The CARD M&E strategy and procedures described in this document are based on AusAID recommendations for project monitoring and evaluation as described in AusGuide which is downloadable from www.ausaid.gov.au/ausguide. Material was also derived from “A Guide for Project M&E” produced by International Fund for Agricultural Development (IFAD) and downloadable from www.ifad.org/evaluation/guide. CARD – M&E Procedures January 2010 CARD – M&E Procedures January 2010 Sequence of Activities EOI Expression of Interest Feedback and Lessons Learned TAP Technical Adivisory Panel PCC Project Coordinating Committee M&E Stages PCR Project Completion Report Ex ante evaluation begins here And continues through the project design process Contract defines outputs, outcomes and milestones Monitoring continues through implementation period Initial evaluation undertaken at project completion Ex post evaluation follows some time later Project evaluations aggregated to evaluate overal portfolio Monitoring Data Used In Evaluation Process MONITORING AND EVALUATION IN THE CARD PROJECT CYCLE Proponent Organisation(s) Identify Project Ideas Prepare EOI and Submit to CARD TAP Evaluates EOIs Proponents Prepare Project Proposals and Submit for Peer Review Proposal Modified as Necessary and Submitted to TAP TAP Evaluates Proposals TAP Recommendations to PCC Final Review and Approval by PCC CARD & Proponent Agree Contract and Payment Milestones Project Implementation Undertaken by Proponents and Self- Monitored Independent Project Completion Evaluation Independent Ex-Post Evaluation Evaluation Reports - Relevance - Effectiveness - Efficiency - Impact - Sustainability - Lessons Learned Monitoring Reports - Baseline Study - Six Monthly Reports - Outcome Milestones - Technical Reports - PCR Lessons Learned Inform Future Rounds of Project Design and Implementation Independent Evaluation of Entire CARD-MARD R&D Portfolio PCC Reviews EOIs and Prepares Shortlist TAP Recommendations to PCC PCC Feedback Revise & Resubmit EOIs PCC Feedback Peer Feedback CARD – M&E Procedures January 2010 EXECUTIVE SUMMARY One of the main objectives of CARD is to strengthen the capacity of MARD to manage agricultural technology and knowledge development programs. Sound management of such programs depends on being able to monitor and evaluate Programs in terms of their relevance, effectiveness, efficiency, impact and sustainability. This recognises that R&D is an investment which needs to be evaluated alongside other investment opportunities, in order to ensure that the best investments are chosen from widely differing alternatives. CARD is in the process of building the capacity of MARD to undertake M&E of R&D projects, beginning with the projects currently supported by the CARD Program. As part of this process CARD provided the services of a M&E Specialist to undertake training and facilitation services for a group of personnel from MARD and its affiliated institutions. The first round of training was completed in April 2007, a second round was undertaken in September-October 2008, and a M&E review workshop was conducted in January 2010. The CARD M&E framework caters for M&E requirements at project level as well as the institutionalisation of CARD processes within MARD. Its key elements are:  Project M&E which aims to assess the progress and impact of collaborative research projects on raising smallholder productivity and competitiveness;  CARD Program M&E which aims to assess the progress and impact of the Program as a whole, both in benefits to smallholders and raising the capacity of research institutions, to undertake effective research projects;  M&E at the MARD institutional level in assessing the improvement in capacity in MARD (STED) in organisation and management of the MARD research Program. The CARD Program has been focused on monitoring at the individual project level through milestone reports and site visits. During 2008 and 2009 CARD initiated evaluation of eight on-going and 14 completed projects using procedures for Mid-Term Reviews (MTRs) and Project Completion Evaluations (PCEs) which were provided in an earlier version of this document. This represents the first step towards evaluation of the overall CARD Program. This document sets out the proposed approach and procedures for individual project evaluations based on the monitoring data already accumulated, and for subsequently aggregating these evaluations up to Program and finally to institutional level. The document was developed in parallel with a training program for a group of staff from MARD and its affiliated institutions which involved three workshop sessions and 21 case studies. The training represents significant progress in developing a group of competent evaluators which is capable of evaluating all CARD projects in the first instance, and subsequently strengthening the evaluation of the entire MARD R&D portfolio. CARD is approaching the end of its seven-year duration and it is therefore appropriate to carry out a number of systematic PCEs. This document sets out a schedule for undertaking these evaluations during the remaining life of the Program. CARD will assist by providing coaching and guidance to contracted evaluation teams as well as further on-the-job training in evaluation methodology. The benefits will be improved project design, the identification of areas of high (and low) return R&D investment, improved relevance and impact, and improved accountability and transparency in the allocation of resources to agriculture and rural development. CARD – M&E Procedures January 2010 2 1 CARD M&E FRAMEWORK 1.1 Introduction One of the main objectives of CARD is to strengthen the capacity of MARD to manage agricultural technology and knowledge development programs. Sound management of such programs depends on being able to monitor and evaluate Programs in terms of their relevance, effectiveness, efficiency, impact and sustainability. This recognises that Research and Development (R&D) is an investment which needs to be evaluated alongside other investment opportunities, in order to ensure that the best investments are chosen from widely differing alternatives. CARD is in the process of building the capacity of MARD to undertake M&E of planned and ongoing R&D projects, beginning with the projects currently supported by the CARD Program. As part of this process CARD provided the services of an International Monitoring and Evaluation Specialist to work with CARD’s in-house national M&E Specialist to undertake training and facilitation services for a group of personnel from MARD and its affiliated institutions. 1.2 Overview The purpose of M&E is to learn so that future development interventions can be more effective. M&E is not an examination or test. It is not an audit. Negative outcomes have value provided we learn from them. The learning process is essentially internal within CARD and its partner institutions. However internal learning needs to be balanced with external accountability. Projects have responsibilities to stakeholders and Vietnam society at large to account for expenditures, activities, outputs and impacts. There is an extensive literature on project M&E for agricultural development which is mainly oriented towards long term impacts of major investment projects such as those funded by the World Bank, ADB and IFAD. CARD comprises a suite of relatively small projects which are intended to generate benefits for stakeholders in both the short and long term. In this regard, CARD needs an approach to M&E which somewhat different to the standard textbook models, and which is certainly cheaper and less complex. The CARD Program M&E framework (see chart at the front of this document) requires an approach which caters for the M&E requirements of the program and project level as well as the institutionalisation of the CARD processes within MARD. The key components of the framework are:  research Project M&E which aims to assess the progress and impact of collaborative research projects on raising smallholder productivity and competitiveness;  CARD Program M&E which aims to assess the progress and impact of the Program as a whole, both in benefits to smallholders and raising the capacity of research institutions, to undertake effective research projects – this is in effect a sum of all project implementation impact; and  M&E at the MARD institutional level in assessing the improvement in capacity in MARD (STED) in organisation and management of the MARD research Program. The purposes of the M&E framework for the Program are to: CARD – M&E Procedures January 2010 3  make available timely and relevant information to support effective management decisions by the PMU, Program Coordinating Committee (PCC), AMC and AusAID.  measure progress of the projects and identify issues for which management can take necessary actions; and  provide information for internal management and external reporting. 1.3 Project Level M&E The M&E strategy at the project level revolves around the development of a simplified logframe for each project, and from these the development of output and outcome milestones and deliverables for each project contract. Output milestones are six-monthly progress reports which record achievement against the logframe activities and identify highlights and issues in project implementation. Outcome milestones focus on impacts at the smallholder and institutional levels. A key milestone for each project is the establishment of baselines on current practices, production levels and profitability. The final milestone for each project is validation of the impact of the project in relation to baseline levels and production of the Project Completion Report. Project reporting involves collaborating institutions self-assessing achievements against their own specific performance measures, detailed in the logframe and project milestones. The PMU monitors projects through appraisal of project output and outcome milestones. Once projects are completed, independent case studies of selected projects are commissioned to evaluate potential economic, social and environmental impacts. At the project level output milestone reporting includes, 6-monthly and Annual Progress Reports (APR) and Project Completion Reports (PCR). Each year the APR and at the end of the project, the PCR, reports against achievement of their defined performance measures. Delivery and payment of these milestones is tracked on the PMU database and any implementation issues are addressed through discussions between the PMU and the collaborating institutions. Outcome milestones include impact assessment at the smallholder and institutional capacity levels. Using baseline data on knowledge, skills and practices; and at the smallholder level physical and financial performance, all projects are required to validate their project outcomes at the completion of the project. 1.4 Program Level M&E At the program level the M&E strategy is to assess the sum of project impacts and to assess changes in research institutional capacity to prepare and implement high quality R&D projects. A key outcome milestone for all projects involves assessment of improvements on competency levels of research and extension workers. PMU project site visits to evaluate implementation and impact using standards assessment formats are integrated with TAP site visits to assess the quality of project selection. At the program level the PMU prepares an Annual Report for the financial year (July – June) to be presented to the PCC in March for finalisation prior to June 30 each year. The Annual CARD – M&E Procedures January 2010 4 Report provides the context for development of the Forward Annual Plan and enables the PMU/Technical Coordinator to take into account any significant interventions that will improve the ownership of CARD in MARD and through that CARD’s sustainability. In this way M&E is part of the planning process and is expected to result in continued improvement in implementation of the CARD Program. The Annual Report includes:  Implementation highlights, issues and options.  Number of projects started and the status of implementation of each project in each year.  Significant outputs from completed projects.  Research project summary sheets including objectives and milestones and acceptance and payment of project milestones.  Significant impacts of research outputs and capacity building arising from the Program as measured by research institution self-assessment, case studies and PMU Progress reports.  Summary of resource inputs and activities achieved against logframe estimates as well as qualitative ratios established from activity analysis (achievements against the CARD Program logframe activities.  Summary of institutional capacity building arising from analysis from internal and external activities involving MARD.  Issues, problems and recommendations. 1.5 Institutional Level M&E At the institutional level a series of performance indicators has been established to monitor and evaluate the institutionalisation of governance and management structures and processes within MARD. A key aspect of this is to measure the change in attitude, beliefs, behaviour and practices within MARD (STED) in relation to R&D policy, organisation and management. A survey was conducted early in the life of the Program to establish the baseline status of institutional capacity. This survey will be repeated during the second half of 2010 to assess the degree of institutionalisation of CARD systems, procedures and management practices into MARD and its affiliated institutions. 1.6 Where M&E Fits in the CARD Project Cycle CARD projects are designed and implemented according to a sequence of steps which can be described as the “CARD Project Cycle”. The second of the charts at the front of this document details the steps in the cycle. M&E begins early in the design process where the project logframe identifies the indicators of achievement and the means of verification. The project is also subject to independent ex ante evaluation firstly at expression of interest (EOI) stage, and again at proposal stage, to assess its likely outputs, outcomes and impacts. During implementation, projects are expected to gather baseline information and self-monitor their activities to provide the data needed for subsequent evaluation. Mid-Term Reviews (MTRs) CARD – M&E Procedures January 2010 5 may be undertaken when about half the milestones have been reached. Independent project completion evaluation (PCE) takes place at the end of the project implementation period, and ex post evaluation is conducted some time later when the full impacts of the project are likely to be apparent. The various M&E reports produced in this process are then available to undertake overall Program evaluation. 1.7 Purpose of this Document Until now the CARD Program has been focused on monitoring at the individual project level. Monitoring is on-going with submission and appraisal of milestone reports and site visits. MTRs and PCEs have been carried out in 2008 and 2009. No further MTRs will be undertaken, but a number of PCEs will be undertaken in 2010 which is the final year of the Program. This will allow aggregation of individual project impacts to the program level. This document sets out the proposed approach and recommended procedures for undertaking individual project evaluations based on the monitoring data that have been accumulated during implementation, and for subsequently aggregating these evaluations up to Program and finally to institutional level. 2 BASIC PRINCIPLES OF MONITORING AND EVALUATION 2.1 Overview M&E is an essential tool in the management of programs and projects in agricultural research and development and is an important part of the CARD implementation framework. This section describes the basic principles of M&E as applied in the CARD Program and is followed by a section which describes a number of techniques which can be used. Monitoring is defined 1 as the regular collection and analysis of information to assist in timely decision-making and provide the basis for evaluation and learning. It is a continuous function that generates data to provide project management and stakeholders with early indicators of progress and achievement of objectives. Monitoring provides data to generate insights about impact as part of the evaluation process. Formal monitoring involves gathering data about selected indicators and performance measures. However informal monitoring involving valuing and sharing impressions is also an important ingredient of the process. There can be no evaluation without some form of monitoring Evaluation is defined as a systematic (and objective as possible) examination of a planned, ongoing or completed project. It aims to answer specific management questions and judge the overall value of a project and generate lessons learned to improve future planning and decision-making. 2.2 The Five Key Questions Evaluations commonly seek to determine the relevance, effectiveness, efficiency, impact and sustainability of the project (see Box 1 below). Evaluation should provide information 1 See Attachment 1 for a complete list of M&E terminology and definitions. [...]... Benefits C=(B-A) Year 0 Year 1 Year 2 Year 3 Year 4 Year 5 Total 100 100 100 100 100 100 600 120 122 125 130 133 133 7 63 20 22 25 30 33 33 1 63 With Project Costs Benefits Net Benefits D E F=(E-D) 100 120 20 200 130 -7 0 90 140 50 85 150 65 80 170 90 640 870 230 Incremental Costs Incremental Benefits Incremental Net Benefits G=(D-A) H=(E-B) I=(H-G) 0 0 0 100 8 -9 2 -1 0 15 25 -1 5 -1 5 -2 0 20 27 37 35 42 57... preparation 8 Review of draft evaluation report by project partners and stakeholders 9 Finalisation of evaluation report taking partner and stakeholder comments into consideration a/ Depends on the number and geographic location of beneficiaries 0.5 0.5 0.5 2 1 1-2 1-2 1-5 a/ 3- 5 1-2 1 PMU Evaluation Team and Partner Institutions Partner Institutions Evaluation Team Ideally evaluation should be undertaken... sources and methods to cross check and validate data, where a single method does not provide an acceptable degree of accuracy 3 ATTACHMENT 2: PROCEDURES AND TERMS OF REFERENCE FOR MID-TERM REVIEWS 1 OBJECTIVES AND SCOPE OF THE MID-TERM REVIEW As part of its routine monitoring and quality assurance procedures CARD carries out a review of projects during the implementation period, generally around the mid-point... more to eat and sell resulting in higher incomes and living standards January 2010 8 CARD – M&E Procedures Objectives, outputs, activities and inputs are specified in the logframe and are relatively easy to estimate or measure However outcomes and impacts are more difficult to define, measure and evaluate This usually requires a degree of judgment about future adoption rates and the level and nature... evidence of outcomes and impacts  Sample Survey Methods: These involve first selecting the sample and then designing the questionnaires or checklists The sample may be a random sample, a stratified random sample, or a non-random/targeted sample The survey questions need to be January 2010 16 CARD – M&E Procedures carefully phrased and tested to ensure that people understand them correctly and that the questions... traders and input suppliers Enterprise budgets for annual crops or short-cycle livestock and aquaculture enterprises are the simplest to develop Longer-term activities such as perennial crops and forestry are more demanding to analyse and must cover the complete life cycle of the enterprise Standard formats for annual and perennial enterprises are given in Attachment 4 5.5 Comparing Benefits and Costs5... description of tools and techniques applicable to agricultural and rural development projects is the IFAD “Guide for Project Monitoring and Evaluation3 ” 4.1 Performance Indicators The CARD standard logframe must specify performance indications and the means by which they are to be measured Performance indicators are qualitative or quantitative factors/parameters that provide a simple and reliable basis... compare the total benefits and cost of a project Further detail on BCA is given in Section 5  Semi-Structured Interviews: These are face-to-face interviews with individual stakeholders or small groups using a series of open-ended questions and topics to guide the conversation Such interviews are critical in gaining an in-depth understanding of why things happened (or did not happen), and what people feel... completion  “With and without” – this involves comparison of project and non-project areas which are otherwise similar in agro-ecological and socio-economic characteristics The comparison sheds light on the question of “what would have happened in the project areas in the absence of the project” and adds weight to the attribution of benefits to the project interventions  “Participants and non-participants”... inputs and outputs If the design logic of a project is weak or uncertain, it usually proves very difficult to monitor and evaluate 2 .3 Different Projects, Different Approach Different projects have to be evaluated in different ways because the nature of the project activities and the benefits and costs they engender vary Therefore it is not possible to provide a standardised set of procedures for evaluation . Project Completion Evaluation Independent Ex-Post Evaluation Evaluation Reports - Relevance - Effectiveness - Efficiency - Impact - Sustainability - Lessons Learned Monitoring Reports - Baseline Study - Six Monthly Reports - Outcome. 27 Attachment 1: Terminology and Definitions Attachment 2: Procedures and Terms of Reference for Mid-Term Reviews CARD – M&E Procedures January 2010 Attachment 3: Procedures and Terms of Reference. 23 5.5 Comparing Benefits and Costs 23 6 WORKPLAN FOR EVALAUTION OF THE CARD PROGRAM 24 6.1 Overview 24 6.2 Mid-Term Reviews 25 6 .3 Project and Program Completion Evaluations 25 6.4 Ex Post Evaluations