An overview of the process from the perspective of someone doing it for the first time

Karibuni! Welcome to our blog. I’m Katherine, the Scripture Engagement Coordinator for SIL Tanzania. 

Some background

Our Language Programmes Coordinator came to me with an idea: how about we do some research in communities that launched New Testaments in recent years to see what we can learn about the impact of our work and having NTs a couple of years on from the launch. This research can also provide an opportunity for staying engaged with these communities, helping with distribution and offering ministry. Outcome Harvesting may be a good way to go about it.

I was tasked with coordinating this project. Within the general goal of learning about the impact of our work, we also specifically want to learn about which aspects of our work have produced the most fruit, in terms of observable changes in behaviour, and who/what were the main contributors to those changes taking place. By discussing the outcomes of the research together with relevant groups in the community, we hope that the research can help guide future work in the area. We also hope it will be helpful for other projects within and beyond Tanzania and in talking to funders.

I attended a webinar on Outcome Harvesting and connected with the SIL Learning Towards Transformation team. We met several times online to make plans, and then two of the team (Phil and Robert) came out to facilitate an initial workshop, providing me and my colleagues with a chance to learn more about the Outcome Harvesting process as well as make a real start on research in the Malila language area.

What follows is a summary of key aspects of the process that I jotted down for future reference as we continue this work. (These brief notes can be supplemented with Phil’s more thorough description in earlier blog posts.)

Who

• The core team: Me, Frank (a colleague in the Mbeya Literacy/SE Department) and Majaliwa (a former colleague from the Malila language area, who will serve as a language area research assistant).
• Also present for most of the week: Two colleagues from the Mbeya Literacy/SE Department), except for field trip.
• 
  Present for some initial conversations before the field trip: Tumaini (the chair of the recently formed Malila NGO, formerly the Language Committee) and Juma (the director of the NGO).

Process

1. Get to know one another: Introductions, roles etc.
2. The journey: Start by outlining where we are heading, from ‘Connecting’ (getting to know one another) through to ‘What’s next?’ [Flipchart – overall plan diagram]
3. Purpose of research: Using the illustration of a farm with fruit trees, talk about the different kinds of results one may see, from a big tree with good fruit through to a small tree with poor fruit, and how examining these results may help maintain or improve fruit production. [Flipchart – tree images]
4. A theory of change for SE (SE change chain): From the ultimate goal of ‘Transforming community’, work backwards through each stage (Changed lives ← people seeing/hearing Scripture materials ← materials getting into the community (‘reach’) ← development of materials), to see how one leads to another. Talk about how we will be looking for change at all levels, all of which lead towards change at the community level. There’s also the ‘legs’ – supporting structures that enable the whole change process, such as key relationships or significant organisations. [Flipchart – theory of change diagram]
5. Timeline: Consider several areas of change/things happening: A) Community (stuff that could impact SE) – e.g. flooding. B) SE work (e.g. distribution, workshops). C) Individuals (e.g. someone starting to distribute or write songs or listen regularly to audio). D) Community Based Organisation (CBO) / NGO. (Note, we didn’t really look at C, focus was more on the others, as those were more tangible and easy to map on a timeline.) We started the timeline from when the NT was launched (Sep 2020). Once done, ask: Which thing/change is most significant in your mind? (Write down the reasons they give…start to see Outcomes emerging.) [Flipcharts stuck together to create timeline + different coloured post-it notes for each area of change]
6. Actor mapping: Pick up on actors already mentioned. Ask them to add more. It could include actors that haven’t been involved until now, but would be good if they were involved. Group the actors (e.g. church-related actors, government, diaspora etc.) [Post-it notes]
7. Actor contributions: What do you hope they will contribute to the transformation we want to see? Note: This would be especially important if we combine ‘Outcome Harvesting’ with ‘Outcome Mapping’. More information on this at the end.
8. Writing Outcome statements: Explain how to write the different components. Give an example. Everyone try to write one. Discuss our initial attempts and how to improve them. Then talk about ‘Contribution’ and ‘Significance’. Practise writing. [Flipchart – Outcome statements diagram]
9. Prepare for field trip: Discuss who will do what: introductions, Bible study, talking about fruit and the purpose of the meeting and asking for permission to share what we hear, one of the team to tell a story of change in their own lives or that they know about making sure they include all the relevant details of when, who, where, what etc), facilitating participants to share their own stories of change (or what they have seen) since NT launched. Make sure everyone on the team has paper and pens for recording what they hear.
10. Field trip: Meet with a small group (about 6 people), who have connected with work in some way or other in the past (e.g. pastor, literacy teacher, attended a workshop etc.). See above for what we did as we met with them. Took a good 3 hours, including chai time. As time permits, meet with individuals or another group. Next time: Also dig a bit deeper into what contributed to those changes and asking about any other signs that the work is expanding/the fruit tree is growing! And ask what has struck them from what they have heard. Also ask about when/where have things not worked out?!
11. Listing the Outcomes: Try and list out the Outcomes we think we heard (by comparing notes you can get a pretty detailed record of what was heard). Could do this onto flipchart or straight into a spreadsheet. Work on them to make them into proper Outcome Statements. Number them.
12. Sense-making activity A – Significance scale: Print off the Outcome Statements. Place them on a scale from low to high significance. Discuss why we placed each one were we did (record that on each statement). All of them are significant, but in relation to the size of impact or relevance to SE, some may seem more significant than others. To help identify what makes something ‘significant’ have a list of things to consider that would indicate greater significance’: It shows our goals/vision has been reached, it’s the first time this has happened – it’s something new, it has growth potential, it is surprising/unexpected but good, it was difficult to achieve, result of someone’s own initiative, a trigger – something small but it caused ripple effects. It’s also possible to have a negative one – a negative outcome may be significant because of what we can learn from it.
13. Thanksgiving and prayer: Each person to choose one of the Outcomes that they really want to praise God for and why. Praise God for what we have seen!
14. Patterns / findings / insights: Discuss what patterns we have started to notice. Just initial thoughts. We might be able to start identifying ‘codes’ for grouping the statements. (E.g. Several statements related to how people could understand the Scriptures more easily in Malila, so we could code all those statements with ‘Easy to understand’.) Can also start to comment on what we learn from all of this. Note: Keep adding to this as you continue through other sense-making activities.
15. Sense-making activity B – Actors involved: Group statements according to actor groups identified during actor mapping. Discuss which groups have been most impacted / involved. Count how many statements there were per group and add these numbers to the actor map. Which groups seem to have been most reached and which ones least? Note: This may just be because of the people we met with, and we need to meet with different kinds of people another time, or it may be a sign that we have failed to adequately engage with particular groups or that interaction with certain groups hasn’t led to change for some reason.
16. Sense-making activity C – SE change chain: Now group statements according to which aspect of the change chain they relate to. If something sits in more than one place, put them in the most relevant place, but also put their number in the other place. Where is most change happening?
17. Patterns / findings / insights: As noted above, with each sense-making activity, note ideas that come up in discussion about what issues we see and how to address them, or questions that we have that we want to explore further or things we want to look into more on the next round of research to check whether what we think we are noticing is true etc.
18. Contribution: Look at each statement and identify what contributed to the change – identify how SIL contributed and any other contributions. Note if there is anything to follow up (e.g. a contribution that’s not clear or we’re not sure exactly what the contribution was.) Then group statements again according to contributions and see what we notice about where the biggest contributions are or which kinds of contributions have led to which kinds of outcomes. Example to remember (not from Tanzania): Pastors using local language Scriptures in church – team thought the main contribution was workshops to train pastors, but when pastors were asked it turned out that of greater importance was the way project staff had attended their funerals and come alongside people so the pastors knew that they could be trusted and were willing to trust their encouragement to use the Scriptures in church. Note: In future, make sure the question of ‘contribution’ is discussed there and then with the people telling their stories.
19. Moving forward: Plan the next steps – who are the people/groups we’d like to hear from more? Which places? (We plan to visit 3 more villages, 2 of which we have done a lot of work in, 1 of which we’ve done very little in, in order to allow some comparison. This will all take place in one trip over several days.) What about ministry and distribution? How can we effectively fish for Outcomes? (We saw pros and cons to visiting groups versus individuals. Groups work for a mix of people who we have had plenty of engagement with, they help ensure stories are true and spark different stories. But others may not be confident to share in a group and may be better approached informally, one-to-one.) Set up a WhatsApp group for capturing Outcomes that come outside of official research trips. After the next research trip, go through the same process above (from #11 onwards) – any follow-up can be done by the language area research assistant (Majaliwa), by phone or during another visit for ministry with some follow-up. Note: If we had asked more about ‘contribution’ it would have been easier to identify what forms of ministry may be most useful.

Blending Outcome Harvesting with Outcome Mapping

Outcome Mapping starts by looking at the actor map. What changes do we hope to see in those different groups of people? List out those changes. Then go and see whether those changes are apparent, gather the evidence. This isn’t in the form of specific Outcome Statements, but rather gathering as many examples as possible of evidence for those changes. It feels like this method takes more preparation but is quicker to implement at the evidence-gathering stage. Also you can really make sure you assess all the different things that the project has done and whether your goals have been reached. However, you may miss outcomes that you hadn’t anticipated and so weren’t looking for, and you don’t get the nice, clear little stories of change.

We would like to blend the two approaches. Before our next research trip, we will think about what changes we would hope to see in the different actor groups (these are called ‘progress markers’) so that as a team we know what we are looking out for. We will then proceed with Outcome Harvesting, as before. However, if we don’t hear about certain changes we were hoping for, we can ask questions / invite stories that target those changes to see if there has been any impact in those areas.

Other things to note

• Data management: Need to make clear how data will be recorded, who can input into the spreadsheet, what needs to be included in the spreadsheet (e.g. a code for each outcome statement so they can easily be identified).
• Sense-making with the community: Plan to do some of the sense-making activities with community members at the end (e.g. the NGO), so that we can learn together from what we have seen.
• Sharing the data: Need to think about who to share the data with and in what form. Possible groups: Community members (e.g. NGO), SIL Tanzania, SE practitioners, funders, Community Based Organisations in other language areas.

Author: Katherine O'Donnell