13 evaluation tools
Instruments and techniques available for qualitative and quantitative monitoring and evaluation are varied. The selection of tools summarised below are well-suited to most stages of the process, impact and outcome evaluation of health-promoting radio interventions.
1. Central Location Intercept
++ Interviewer goes to a place frequented by members of the target Listener-learners and asks screening questions to ensure they "are" from the target group. Asks specific pre-determined questions.
Pro: Cost effective. Many interviews. Can choose different locations. Use of structured questionnaires allows quick analysis.
Con: May not reflect total context. Difficult with sensitive topics. Only a short time may be possible with each respondent.
2. Community Interviews
++ Facilitator questions, raises issues and seeks responses from groups of more than 15 participants. Main interaction is between the interviewer(s) and participants rather than among participants. Often takes the form of open, public meetings.
Pro: Direct interaction between investigator and a large segment of the participants. Interviewer can record verbal and non-verbal responses. Can generate quantitative data including community level statistics or quantifiable data about behaviour, opinions or attitudes of participants, gathered by tallying the ‘yes’ and ‘no’ responses to questions. Participants often correct each other, improving the validity of the data. Quick and cost-effective.
Con: Easily manipulated. Discussion easily monopolised by a few articulate people. Issues that can be discussed in individual encounters cannot be examined because of various social and political inhibitions.
3. Focus Groups
++ Small groups discuss issues raised by an interviewer or listen to program pilots. Evaluation based on unstructured or structured discussion or self-completed questionnaires
Pro: Quick results and analysis. Flexibility to follow up emerging unexpected issues / discussion. Usually stimulates quality, in-depth conversation and opinion.
Con: May not reflect total context or audience
4. Gatekeeper Review
++ Investigate the reactions of potential gatekeepers (intermediaries who influence or control components that affect Listener-learners)
Pro: Inexpensive. Provides direction for the intervention from a critical population.
Con: Often low response rate. Results based on small sample should be interpreted carefully.
5. Individual In-Depth Interviews
++ A trained interviewer probes the attitudes, beliefs and emotions of target Listener-learners
Pro: Opportunity to probe individual subjects in depth. Can obtain information on sensitive or emotional subjects. Good for interviewing hard-to-reach groups.
Con: Time-consuming to arrange, conduct and analyse results. Quantitative information should not be used to make broad generalisations.
6. Informal Listener Response
++ Asks questions of listeners who telephone or visit studio or office. Can be structured or unstructured discussion or questionnaire. Should include questions related to non-radio components
Pro: Can probe highly-motivated individuals in depth. Can discuss sensitive or emotionally laden issues without fear of embarrassment.
Con: Time-consuming. Cannot use information to make broad generalisations. Danger of courtesy bias.
7. Informal Surveys
++ Small-scale surveys of a Listener-learner sample of between 30-50 people. Usually analysed with descriptive or non-parametric statistics
Pro: Provides relatively reliable data quickly, conveniently and at low cost. Do not need too many interviewers.
Con: Does not provide in-depth information because they do not permit extended discussions. Danger of sample bias. Can’t apply results to broader contexts. Descriptive statistics on small sample sizes may be misleading.
8. Key Informant Interviews
++ A select group who have a very good knowledge of the Listener-learner context (doctors, health workers etc) discuss defined topics
Pro: Gain depth and inside information. Flexibility to explore new ideas. Can ask questions for clarification / understanding.
Con: Can be biased / subjective / inaccurate. May not have actual experience or knowledge of the situation and may have been chosen because of status.
9. Listener Questionnaire
++ A questionnaire sent to known Listener-learners (on mail files)
Pro: Identifies impact, trends and listening patterns.
Con: Courtesy bias. No information about unknown audience.
10. Observation
++ Visit and observe activities at locations where services are delivered (eg., clinics, schools, radio station and other appropriate venues) and evaluate with a checklist
Pro: Useful for informal monitoring and evaluation of "on-the-ground" delivery and outcomes.
Con: May not reflect total context.
11. Peer Reviews
++ People involved in radio program production
Pro: Sense of connection and group responsibility. A broad range of experience is helpful. Individuals can get assistance / advice. Group discussion stimulates ideas and creativity.
Con: Some producers may feel threatened to have their work evaluated by peers. Individuals can monopolise the meetings. Individuals can criticise peers in a non- productive way, eroding relationships.
12. Production Team Meetings
++ Give staff regular opportunities to discuss and evaluate programs and project components
Pro: Entire staff gains a sense of "ownership" and "involvement". A sense of intra-staff connection and group responsibility emerges. Avoids "top down" decision making. Can identify problem areas. Can identify areas of integration and cooperation.
Con: Reluctance to express views openly in front of leader or peers. Dominant leadership may curtail open expression.
13. Project or Partnership Meetings
++ Regular and frequent meetings of representatives of all agencies involved
Pro: Coordination and integration. A broader range of experience and views can be brought to bear on an issue. Can identify impact of radio programming on non-radio components.
Con: Structure may become unwieldy for planning and implementation. May slow down response to emerging trends / situations. Some agencies or individuals can monopolise the direction of the project, ability to make decisions, or discussions.