Difference between revisions of "Most Significant Change (MSC)"

The most significant change (MSC) technique is a form of participatory monitoring and evaluation. It is participatory in the sense that many project stakeholders are involved in deciding the sorts of change to be recorded and in analyzing the data collected. It is a form of monitoring because it occurs throughout the program cycle, providing information to help people manage it. It contributes to evaluation because it provides data on impact and outcomes that can be used to help assess a programme’s performance as whole.

The MSC process involves the collection of significant change (SC) stories emanating from a particular project and the systematic selection of the most significant of these stories by panels of designated stakeholders or staff. The process is developed around positive questioning such as Looking back over the (concerned period), what do you think was the most significant change in [particular domain of change]? or From among all these significant changes, what do you think was the most significant change of all?

The MSC can potentially influence organizational learning through its horizontal and vertical processes. The horizontal dimension is between a group of participants engaged in discussing and selecting the most significant of a set of stories while the vertical dialogue involves exchanges of views between groups of participants at different levels. The vertical dimension is particularly important if the MSC process is to assist organizational learning throughout the organization. It will depend on good documentation and communication of the results of one group’s discussion to the other.

The creation and/or facilitation of the following contexts are important for a successful MSC implementation:

• An organizational culture where it is acceptable to discuss both successes as well as things that go wrong;
• Champions capacitated to promote the use MSC, including having good facilitation skills
• Willingness to try something different
• Time to run several cycles of the approach
• Infrastructure to enable regular feedback of the results to stakeholders
• Senior management commitment

Implementation steps of MSC technique:

1. Starting and raising interest
2. Defining the domains of change
3. Defining the reporting period
4. Collecting SC stories
5. Selecting the most significant of the stories
6. Feeding back the results of the selection process
7. Verification of stories
8. Quantification
9. Secondary analysis and meta-monitoring
10. Revising the system

Step By Step^[3]

Step 1: Starting and raising interest

A. It may help to use one of the following metaphors to explain the MSC:

• Newspaper: Newspapers are structured into different subject areas in the same way that MSC uses domains.
• School of fish: MSC helps the individual fish to communicate with each other and to swim in the same direction, towards what is good and away from what is not.
• Holiday memories: MSC helps teams to focus on memorable events and to use these events to help realign efforts towards achieving more of the wonderful things and less of the terrible ones.
• Restaurant menu: MSC presents a series of glimpses of what a programme is achieving. The stakeholders can then select from these glimpses in much the same way as they would select food from a restaurant menu.

B. Start small. It is a risky exercise to implement a huge and complicated MSC system without first piloting it on a small scale.

C. Identify key people (champions) who are excited by MSC. These champions can:

• Excite and motivate people
• Answer questions about the technique
• Facilitate selection of SC stories
• Encourage people to collect SC stories
• Ensure that feedback occurs
• Ensure that the stories are collected and organized and sent to review meetings
• Develop protocols to ensure confidentiality where necessary

Using domains of change helps organizations to group a large number of SC stories into more manageable lots, which can each be analyzed in turn. The “any other type of change” domain is a useful open category that allows participants to report significant changes that don’t fit into the named domains. Between three and five domains is a manageable number. The limiting factor is how much time participants are willing to spend in discussing each domain.

A domain can be identified before SC stories are selected or afterwards by sorting SC stories into meaningful groups. This depends on the extent to which the organization wants to be open to new experiences rather than continuing to be guided by past experiences.

The frequency of collection of SC stories has varied from fortnightly to yearly. Each organization has to make its own decision about the most appropriate reporting period, balancing the costs and benefits involved, and taking into account the reporting gaps that any existing monitoring and evaluation systems may be ignoring.

Experiences suggest that organizations tend to start MSC with more regular reporting and decrease the frequency as the process continues.

Step 4: Collecting SC stories

A. How to capture SC stories:

• Fieldworkers write down unsolicited stories that they have heard
• By interviews and note-taking
• Group discussion sessions
• Beneficiaries write a story directly

B. The information to be documented should include:

• Information about who collected the story and when the evens occurred
• Description of the story – what happened
• Significance (to the storyteller) of the events described in the story which is the key part of MSC

C. How long should the stories be?

Most MSC stories are a page or less in length, with some being up to two pages. Shorter MSC stories are quicker and easier to read, but they should not be so short that vital information is left out.

D. Ethics of collecting stories

When a storyteller tells a story, the person collecting the story needs to explain how the story is to be used and to check whether the storyteller is in agreement with its use. The storyteller should also be asked whether they wish their name to accompany the story. Even when consent has been given, it is good practice to check with storytellers before placing any stories in media such as newspapers.

Step 5: Selecting the most significant of the stories

The MSC approach uses a hierarchy of selection process. People discuss SCs within their area and submit the most significant of these to the level above, which then selects the most significant of all the SCs submitted by the lower levels and passes this on to the next level. The diagram below illustrates this process.

A. Planning the selection process

• How many levels of selection will there be above the staff who initially documents the SC stories? This usually depends on the number of layers of management that exist within the organization.
• At each of these levels, how many separate selection processes will there be? This will depend on the number of separate offices at each level (based on location or specialization).
• In each of these levels, how many SC stories can be managed by the staff involved? It is unrealistic to expect staff to meet and work on the selection of SC stories for more than two hours at the most. If there are four domains of change to review, this means 10 minutes for each. Within each domain, aim to read through and discuss no more than 10 SC stories.
• Who should participate in each selection process? This aspect is covered in more derail below.
• How often should the selection occur? Normally this choice would be dependent on the frequency with which SC are collected.

For each domain the group will select a story that they believe represents the most significant change of all. The selection process invariably begins with reading some or all of the stories out loud or individually. The key ingredients to story selection are:

• Everybody reads the stories
• The group holds an in-depth conversation about which stories should be chosen
• The group decides which stories are felt to be most significant
• The reasons for the group’s choice(s) are documented

The group must decide whether the criteria for selecting stories will be identified before or after reading stories. If MSC is being used to aid organizational learning, the selection criteria should emerge through discussion of the reported changes and not be decided in advance.

There are several ways of reaching a decision about which stories to select:

• Majority rules

  Read the stories, make sure everyone understands them, and then vote by show of hands. The main risk is that a choice will be made without any substantial discussion.

• Iteractive voting

  After the first vote, people discuss why they voted as they did. This is followed by a second and then a third vote, ideally with some movement towards consensus.

• Scoring

  Instead of voting, participants can rate the value of a SC story. The ratings for each of the stories are then aggregated and the story with the highest rating is selected as the most significant. The downside is the limited opportunity for dialogue, although explanations for rating can be given at the same time as the ratings.

• Pre-scoring and group vote

  The method is suitable for groups who are short of meeting time. Prior to the meeting, participants are asked to read SC stories and rate their significance. These ratings are summarized in a table and presented to the participants when they meet face-to-face. Participants discuss the score and cast their vote. The disadvantage is that all stories must be sent to participants some time before the meeting.

• Secret ballot

  Cast votes anonymously. Each person writes their choice of SC story on a slip of paper, and then the total votes are presented. This should be followed by an open discussion of the reasons for the choices. This process can be surprisingly useful, especially if there are power inequalities in the group, or if people are initially reluctant to cast their votes publicly.

It is important to remember that in the MSC transparency is an important way of making subjectivity accountable. Therefore, it is very important to add the second step of capturing and discussing the reasons for choices.

The documentation attached to the most significant story should record:

• The reasons for selecting an SC story as the most significant
• The process used to select the story

Stories that are filtered out should not be thrown away. They should be kept on file so that they are accessible to others within the organization using the MSC, for as long as they continue to use it, and arguably even for a while after that. This is to enable some systematic content analysis of the full set of documented SC stories.

Step 6: Feeding back the results of the selection process

The results of a selection process must be fed back to those who provided the SC stories. At the very least, this feedback should include the following points:

• Explain which SC was selected as the most significant and why.
• Provide information on how the selection process was organized.
• In some cases participants provide more comprehensive feedback in the form of tables showing who gave which rating to what SC story.

Why feedback is useful?

• Feedback about why a selection was made can expand or challenge participants’ views of what is significant.
• Feedback about the selection process can help participants to assess the quality of the collective judgments that were made.
• Information about which SC stories were selected can help participants’ search for SCs in the next reporting period.
• Providing feedback can potentially complete a communication loop between different levels of participants in an organization.

Providing feedback to the community brings benefits as well as risks.

Benefits:

• People can be motivated by reading success stories.
• Participants can gain ideas about how they may reach their goals.
• It can lift the morale of staff and participants.
• It can make the process more transparent.

Risks:

• Giving feedback to the community about which changes the program team does or does not value might be interpreted as the program trying to tell individuals and communities how they should develop.

Step 7: Verification of stories

Especially in larger organizations the reported changes may not reflect what has actually happened, but instead:

• be deliberated fictional accounts, designed to the same time or gain recognition
• describe real events that have been misunderstood
• exaggerate the significance of events

What aspects of the MSC stories should be verified?

• Description aspect: It is useful to consider whether any information is missing and to ask how accurate the facts are. Is there enough information to enable an independent third party to find out what happened, when and where, and who was involved?
• Interpretation aspect: It is useful to ask whether the interpretations given to the events are reasonable, and if what the reporter did after documenting the story is consistent with the contents of the story.

Step 8: Quantification

Within the MSC, there are three methods to collect and analyze quantitative information:

• The first is within individual stories. It is possible, as with any news story, to indicate how many people were involved, how many activities took place and to quantify effects of different kinds.
• The second method can be used after the selection of the most significant of all stories, possibly in association with the feedback stage. For example, if the most significant of all stories referred to a woman buying land in her own name, all participants could then be asked for information about all other instances of this kind of change that they are aware of. This one-off inquiry does not need to be repeated during subsequent reporting periods.
• The third means of quantification is possible during Step 9- Secondary analysis and meta-monitoring. It involves examining the full set of collected SC stories, including those not selected at higher levels within the organization, and counting the number of times a specific type of change is noted.

Step 9: Secondary analysis and meta-monitoring

Secondary analysis involves the examination, classification and analysis of the content (or themes) across a set of SC stories, whereas meta-monitoring will focus more on the attributes of the stories. Meta-monitoring can be done continually or periodically. Secondary analysis is a more in-depth look at the contents of all the stories; it tends to be done less frequently, such as once a year.

In preparation for both meta-monitoring and secondary analysis, it is useful to develop a supporting spreadsheet containing data about each of the SC stories, one per row.

Meta-monitoring: It does not require expert knowledge. There are four main types of measures that can be monitored:

• The total number of SC stories written in each reporting period and how this change over time.
• Who is writing stories and who is not, and how the membership of these groups changes over time.
• Whose stories are being selected and whose are not.
• What has happened to those SC stories?

Secondary analysis: It is a deeper analysis generally done in a non-participatory way by a research or a monitoring and evaluation specialist.

Step 10: Revising the system

Almost all organizations that use the MSC change the implementation in some way. Many of changes made by organizations using the MSC arise from day-to-day reflection about practice. The most common changes are:

• Changes in the names of the domains of change being used: for example, adding domains that capture negative changes, or “lesson learned”
• Changes in the frequency of reporting
• Changes in the types of participants
• Changes in the structure of meetings called to select the most significant stories

Meta-evaluations of the use of the MSC involve extra costs. These are most justifiable where the MSC has been implemented on a pilot basis with the aim of extending its use on a much wider scale if it proves to be successful.

Building capability for effective MSC^[4]

Regarding to the resources an organization may need to implement the MSC technique, three strategies are considered important:

The most important attributes for champions are enthusiasm and interest in the MSC technique. Good facilitation skills are also useful. Champions can develop a sound understanding of the MSC in various ways:

• Reading some of the existing documents on MSC and experimenting with MSC on a small scale
• Having a consultant visit the program office and work with the champions to introduce the MSC to the organization, as well as helping the champions to build their knowledge-base
• Going on temporary assignments to other organizations that are more experienced in using the MSC

If one person assumes the leadership for the MSC in an organization, it is strongly recommended to build the MSC capacity of a second or third person as well. So that when a champion moves to another job, the implementation of the MSC in that place will not fall down.

There are two main options available for building the capacity of program teams in the MSC:

• Through training

  Here are some tips of training people in MSC:

  • Use plenty of hands-on exercises, such as role-playing exercise
  • Ask participants to document their own stories in the training session. An effective training technique is to put participants in pairs and encourage them to interview each other to elicit their MSC stories. Choose a topic that everyone will relate to.
  • Compare MSC with other techniques such as case studies and conventional monitoring systems to help participants understand the differences.
  • Explain how MSC fits into the project or organization monitoring and evaluation framework.
  • Offer plenty of opportunity for questions and discussion. People often need time to absorb the MSC technique.
  • Run the training in conjunction with a facilitator who can focus on how the participants are feeling.
  • Once the initial training has been conducted, it helps to have a refresher session after the first stories have been collected and selected.
• Through mentoring and practice

  It helps to have someone with a very good understanding of the MSC who can answer questions, address any confusion and design systems to minimize frustration.

The MSC is time-consuming. Once MSC is going smoothly, it should become quicker and more streamlined. Organizations often choose to lengthen the reporting period after a year or so, which also reduces the amount of time the process consumes.