EVALUATION: A PARTICIPATORY PARTNERSHIP Using PRA Tools and Techniques for Evaluation

PARTICIPATORY EVALUATION
An attempt at using PRA Techniques in Evaluation

This paper is a contribution to the MYRADA PRA-PALM Series by Ms.Debra Johnson of World Neighbours, Oklahoma, U.S.A.

Editor’s Note :

In November 1992, a group of persons involved with development from eight different countries came to MYRADA to participate in a workshop on the use of PRA tools and techniques; they spent four days at one of MYRADA’s projects in Mysore District of Karnataka State. One sub-group of participants chose to explore the use of PRA tools in evaluating a community-based programme (drinking water). Debra Johnson who was a member of this sub-group wrote up the experience and sent it to MYRADA. We thought we should share it with a wider group of people since it states in clear language the lessons we learnt that day. This is the first paper in our PRA-PALM series to document an experience in evaluation. We hope you enjoy reading it and find the lessons of use in your own work.

PRA theory is not much different from World Neighbours philosophy of empowering” people to help themselves. PRA stresses that outside organisations, as catalysts for this empowering process, have to stop taking information from the community, analysing the data, and returning only to tell the community members what their problems are and how they should solve them. PRA offers a basic theory of community development and “bag” of possible tools to encourage and ensure participation by the marginalised, “poorest-of-the-poor”, community members that the development program staff wants to reach. Social mapping and wealth ranking (to name a couple of tools) can help the community and the program leaders identify those families who have the most need. Once they and their particular needs are identified, the community, the program, and those individuals can work on finding solutions. They can also raise the awareness of all community members about the conditions of their neighbours.

Visualising Their Situation :

There are many areas in the program cycle where PRA tools and methods can be used to enhance the participation of the entire community : analysis of its current solutions within the community and outside; planning and implementing changes to improve the situation; and evaluating the progress/process of implementation. Many of the PRA tools are visual in nature : maps, timelines, matrices, rankings, etc. They can be used to portray the community’s situation in a visual form. In order for the community to present its information visually, it must begin to analyse what it knows. This visual presentation of their situation constitutes the first step in the participatory development process. For example, often they must talk about village boundary lines and the placement of houses and other significant landmarks. What is considered a significant landmark for some is not significant for others, so the community members generate informally criteria for what is considered a landmark. Once this information is presented in a visual form, others can give their input at anytime. The diagrams, matrices, and timelines can be saved on paper and used for evaluating and planning later.

Participatory Evaluation :

We saw how dynamic exercises like social mapping can be when well-facilitated. Very little has been written to date about the use of PRA tools and techniques to encourage participatory evaluation (and subsequent action) of program interventions. As a group, we had the chance to try these techniques out in an “impact evaluation” of a drinking water system in a small Indian village called Magudilu. By the time we had spent two days in the village, we discovered that the drinking water system was beneficial to some of the villagers, but that it had failed to provide close, safe drinking water to those who needed it the most – the poor or marginalised people in the village.

We used six main tools from the PRA bag : water system map, focus group discussions, time allocation drawing, seasonality of diseases, individual interviews, and an observation walk. Since we were focussing on evaluating the water system, we tried to limit the amount of information in order to ensure enough time for community participation.

Our main objective for the time in the village was to help the community and MYRADA evaluate the water supply system installed by the community, MYRADA and the local Government. The first afternoon we arrived at the village, we met members of various sanghas (community organisations). We asked some of the villagers who had gathered to illustrate the water system as it exists now. While they were working on the map, we talked with the other group members about the system. We discovered that before the water system was installed, the majority of the people went to the river for all their water needs. Since the installation of the system, they said there had not been any problems with their water supply. This did not sound possible, so we tried asking more probing questions to verify that there had not been any problems in the two years that the system had been serving the village. We asked questions like, “If the system broke down where would you go for water?”, “Who would fix the system if it did break down?”, etc. Still we received the same response. A couple of Harijan women (socially at the bottom of the caste system) told us a heartwarming story about the positive impact of the water supply system on their lives. Due to the water tap system, they were now able to make it to the fields in time to get casual work (temporary, agricultural labour available on a day-to- day basis). Before the taps, they had to walk an hour to get water in the morning. It was difficult to get back in time to complete their chores and arrive at the fields early enough to get work. If they were late, they were turned away and that meant a day’s worth of wages lost. After these discussions, we could have concluded that the system was working wonderfully — benefitting everyone.

Around dusk, everyone gathered again to share and analyse the information gathered during the afternoon. The main group shared the map they had developed illustrating the village’s water system. When they were finished explaining what they had done, we asked if anyone had any corrections. A lively discussion broke out. The spontaneous debate that followed the presentation included lower and upper castes, men and women, old and young. It was wonderful to see so much intra-community interaction. Since so many of the exercises were visual and used symbols chosen by the group, everyone participated, and no one paid much attention to the outsiders sitting and watching the debate.

The MYRADA translators could not keep up (which was fine because it was the community’s map and we did not need to meddle), so we just observed. After the dust settled, it turned out that the “perfect system” had three broken taps and two which had been moved to other areas of the village. By the end of the discussion, it was getting rather late. Many of the villagers had given us the entire afternoon, so we left saying thanks and giving our promise to come back the next day. We said we had learned much from them. They in turn thanked us for coming to their village and said they also had learned a lot. They said they learned that there were problems with the system that they had not realised or that they had forgotten (since the problems had not affected the village as a whole but only the poorer locality, as we later found out).

MAP OF WATER SUPPLY – MAGUDILU VILLAGE Dt.18.11.92

Later in the evening, other members of the village who were not able to participate in creating the map were given the chance to correct the map based on their knowledge. The result was the map found above.

The next day, three members of the facilitator group walked around the village with a small group of villagers to take a look at the system and to get a better idea of the layout of the village and the water system through our own observation. (The other facilitators were carrying out additional PRA exercises.) The walk proved to be very valuable because we met the “water boy”, who was in charge of cleaning the water tank and doing minor repairs on the system. He had not attended the meetings, but he added a whole new dimension to the water system evaluation. It turned out that the pump had broken down a few months ago. [The pump is needed to force the water from the main well upto the water tank. From the tank, the water flows back down to the village through four main pipes.] It took a month and a lot of pressure from the community, to get the pump fixed by the Government. During the repairs, everyone was forced to go to the river or the few boreholes in the village for their drinking water. The local Government has now taken over the management of the system.

When we returned to the main group, one of the smaller groups that had worked on displaying the main users of the water sources presented their work to the main group. After a lot of discussions and contribution from the community members and the PRA team, we found that 75% of the lowest caste group (there were nine castes in the village) was still using the river for their water. In the original water scheme there were only two taps allotted for the entire Harijan community (80 families — 40 families per tap) and one of them was broken. In other areas of the community there are 10 to 15 families per tap. The Harijans, due to their social status, are not allowed to use taps or boreholes meant for any other caste, so they collect water from the river. Furthermore, the Harijans felt the river water tasted better, and rice made with this water boiled quicker. They did not seem aware of some of the problems associated with using polluted water for their household needs.

If we had spent only the first afternoon in the village, we would have come away thinking that the water system was successful and that everyone benefitted more or less equally. As it turned out, the system benefitted the more affluent/ influential people, while the poor and marginalised did not receive the same benefits. By using PRA tools to help the community and program leaders more fully evaluate the success of the project, we were able to discover some serious flaws in the system, but ones that could be rectified. Although the group of facilitators were not able to follow up on the evaluation, it seemed that the community had already identified some actions that could remedy these flaws.

Despite the dramatic success of the participatory approach, there are some problems with the use of PRA tools, which stem from a poor understanding of group dynamics and good facilitation techniques :

1) Trying to get too much information too quickly. A PRA raises many more questions than can be pursued at first. In our case, we decided before going to the village that we would look at the management of the water system and the effect of the system on the incidence of water-borne diseases. Although the two subjects are linked, it was difficult to pursue both during a two-day PRA.

2) Insufficient self-criticism by the outside catalysts in the way the information is gathered, analysed and shared with the community. We were a group of five, each one anxious to pursue the subject that interested him/her the most. There was quite a bit of chaos, but we tried to keep focused without controlling the direction of the villagers’ discussions. We should have been better prepared and less determined to do what we wanted to do individually. It is the fine art of facilitating “controlled chaos”.

3) Lack of verification (triangulation) of the information through other sources and different means. This requires that the facilitators not only work with those who are able to attend village meetings but also that they go out into the village and talk to people who were not able to attend.

In the final analysis, the evaluative tools and techniques of Participatory Rural Appraisal empower the community to take an active role in determining its needs and ingenerating plans of action to address those needs. The use of visual, analytical exercises encourages participation from all socio-economic classes by removing literacy and numeracy as criteria for participation in a community’s self-analysis and evaluation of program activities. Evaluators become facilitators as the community critically examines the progress it has made and looks at ways to improve its performance. The performance evaluation of the water system in Magudilu has shown how revealing and motivational these tools can be in evaluating program activities.

However, the dynamic nature of PRA is at the same time its strength and a potential weakness — precipitate decisions must somehow be tempered by an outside, objective facilitator, who verifies that the community’s plans are needed by the majority, feasible, and known to all.

In the best of scenarios, Participatory Rural Appraisal breaks through the bureaucratic inertia of some outside development organisations and the internal social barriers of the village in community decision-making; marginalised groups (low castes and women), previously excluded or ignored, have an opportunity to take part in the development of their community. PRA represents a serious attempt to potentialise and realise measurable progress in developmental projects and programs.