Shirah Hecht is Program Evaluator for Academic Technology Services at Harvard University. Dr. Hecht has conducted research related to Jewish education and community for over two decades, including as an evaluation and research consultant with CJP of Boston, Brandeis University’s Mandel Institute for Studies in Jewish Education, and as a staff member with JESNA.

I. Outcomes and evidence: How will we know it if we see it?

Before we shoot for a valued target, and in order to assess our efforts toward that goal, we first need to define the target. In other words, how will we know it when we see it? This is the basic connection between “abstract concept and concrete indicator” for those using research terminology. In lay language, to know if we got there, we need first to know what “there” is and then what it looks like.

This article responds to one of the central questions posed for this journal issue, “How can day schools assess whether or not they have been successful in their missions,” by addressing the question: How can day schools articulate their goals and the process by which they plan to achieve them? With this task accomplished, day schools can, like any program, communicate to others about their goals and processes, and ultimately measure their progress towards their valued ends. Non-profits, from professional associations to educational institutions to social service agencies, have long sought to track their work in order to know and serve their constituencies better and to argue for program funding. In recent decades, that process has become more systematized within the discipline of program evaluation, developed as an approach to help non-profit programs assess whether they are reaching their goals. Along the way, the field embraced the logic model as a tool to help programs and organizations define, understand and tell their story. As outcomes have become better understood as the guiding light and the logic model creation process has been developed, it has become a tool with high value for those seeking to understand, support and assess program efforts.

The following image summarizing the basic program evaluation framework suggests two themes essential to the logic model also: (a) a causal, logical view of programs and (b) the connection between process and outcomes.

II. Warm-up exercise: Program constituency mapping

Given the complexity of the logic model, it can be helpful to begin with a more brainstorming, overview process. Program constituency mapping is essentially a stakeholder-identifying activity, with an eye toward program understanding and data collection.

This activity, along with a basic understanding of the program’s main features and its history, is a good beginning point, to dip a toe into the water of defining and ultimately measuring progress toward defined outcomes.

Using a generalized education-related example as a model, the image below provides a shell for a program map.

To discover and understand the relevant groups for the map, consider these questions:

Constituencies:

• Who is impacted by your work?
• Who might have an investment or interest?
• Who might be an audience for reporting?

Questions to consider about the groups:

• What is their investment/interest in your work?
• Are they a source of data about your work?
• What would you like them to experience or do?

After the groups are identified, it may also be helpful to explore their perspectives. To “play out” a constituency perspective for the purposes of program understanding and evaluation, relevant questions might include:

• What is important to them?
• What do they “do” in relation to this program?
• What information might they provide about the program, based on their experience?
• At what point do they interact with the program?
• What might they want to know about the program, if anything?

III. Logic models

After this somewhat free-form approach, developing the logic model will be more structured and yet also draws directly on this exploration. It also requires “think-work” and some detailed attention to program process and mission.

Below is one version of the basic logic model that comes at the end of a much longer process. As the figure shows, the logic model mirrors the input-throughput-output approach of program evaluation, while filling in the details of process and results. While there is some variation in different sources on the logic model, the essential elements are comparable and the wording above provides brief guidelines. A completed logic model then provides a one-page snapshot in a non-time-oriented way, which captures much about a program’s basic dynamics and value. As such it is a tool that is holistic – visualizing the whole program at once – while also demonstrating an articulated process towards defined program goals. Below is a basic set of terms that define the columns and the added elements that many logic models incorporate.

Inputs: resources, contributions, investments that go into the program. This can also include a description of the target population for a program: they are a key “input.”

Outputs: activities, services, events and products that reach people who participate or who are targeted. It is good to consider various activities and services, and to state them clearly so that they are either verbs or refer clearly to actions the program takes.

Outcomes: results or changes for individuals, groups, communities, organizations, or systems. These outcomes can be short-term, medium-term or long-term.

Assumptions and Theory of Change: the beliefs we have about the program, the people involved, and the context and the way we think the program will work. The idea of the “theory of change” is a summary indicating the beliefs of the program provider, perhaps based on research evidence or experience or logic, about what leads to what; this logic explains why the program is built the way it is, to achieve its desired goals.

External Factors: the environment in which the program exists includes a variety of external factors that interact with and influence the program action. These may be the circumstances in which the program operates, which again explain why it is built as it is built, addresses the need it has identified, or explains the limitations of what the program can achieve.

A key decision in developing a logic model concerns granularity, since it can be developed at various levels: from a high level similar to that of an entire institution, with significant programs within it defined simply as “activities” in the model; to a detailed programmatic level that includes activities that may be carried out each day. To illustrate in the education field, a logic model could be developed for an entire school, with the outcomes for all students or even the entire school community defined at the end; or, at the other extreme, you might build a logic model for a one-time afternoon activity, aimed at a specific audience.

There are some pitfalls, innovations and value in the logic model as a tool. The “mental download” of the program staff to create a model often results in laundry lists of activities, and lacks a through-line. In a somewhat better scenario, a logic model can be initially developed by a program evaluator, drawing on the program’s written materials; a first draft is then reviewed by program staff for the content. Even better approaches highlight outcomes-first (“begin with the end in mind”) and good group process. In the “Tearless Logic Model” approach, the model is ideally developed from a shared group process including a range of stakeholders, having a discussion in a “safe” environment for sharing ideas, and relies on jargon-free direct questions.

With good process, developing a logic model offers value to a program and its staff in three ways:

ARTICULATE Helps program staff articulate their goals, thereby giving them a tool to clarify the mission and measure progress.

COMMUNICATE Conveys the goals of a program and how they will be achieved to external audiences; for support and eyes on the process, it demonstrates how the program works in a one-page overview document with details.

EVALUATE Helps create an “evaluation” or research plan, to collect data relevant to decision-making. Ultimate goal is an evaluation product as research.

With these goals in mind, the logic model can support organizations in these situations:

• Communicate to others, including donors, that you have a plan and that it works, along with what you do to achieve your desired and professed goals.
• Clarify your own thinking as you develop or field a program, about what you need to do in order to achieve your goals, and what goals are reasonable to expect, given your activities and resources.
• Identify how you will know if you have achieved your goals, by defining the expected outputs, outcomes, and what those outcomes will look like when observed or measured.
• Identify your outcomes and research opportunities, to design a program evaluation study.

IV. Logic model perspective, tips and tricks

Logic models are created from the mindset of an “outsider” in many ways – how do we begin from the beginning, to explain what we do, to someone as if they had no experience with our program? Consultants often help to create logic models, as they stand outside of the organization and ask the basic questions. Two analogies for a logic model may help explain its essence: it’s a “recipe” for a program, resulting in a somewhat variable but recognizable result, or a blueprint for the program as a car, getting from Point A to Point B.

At the same time, reflecting the earlier difficulties with the method, there are some challenges associated with building logic models. It is a challenge to figure out how detailed a model to create in these terms – School level? Program level? Sub-activity of a program? The difference between outputs and outcomes is also key to understanding the model. In simplified language, outputs are what “we” do as program providers, and outcomes are what “they” do as program participants and beneficiaries. In addition, it is also helpful to connect activities closely to outputs: all activities have results that can be counted as outputs. In contrast, when you consider outcomes, your perspective is more on the entire program, not sub-activities.

V. The day school case and logic models

How can we apply this approach to work in Jewish day schools? Schools are often locations for program evaluation, as is evidenced by the field’s development in the education sector. This section will explore some of the challenges of using the approaches described here in a school setting, as well as how they might apply. While program evaluation can be conducted at various levels, we will first consider the “school as program.”

Likely every Jewish day school has a mission or vision statement which it shares as a way of describing the school and its values. Each of the mission statements we find on the internet for day schools is an important guidepost to the school’s work and holds the values to which the school is committed.

Now, how do we make that concrete and measureable, so that the school can tell its story and keep itself on track toward its mission? The program evaluation framework and logic model will support the school in doing just that. Following this path, a school makes that somewhat abstract statement of values concrete by considering the results for individuals and for a larger situation, which flow directly from the school’s efforts. This approach allows a school to tell its story to itself and to others in a way that is true to both the details of education and the vision the school relies on to go forward. It also offers a way for schools to integrate the mission into shared thinking and action, making it possible to use that vision to guide detailed decision-making.

The following describes key paths you’ll follow when you work on a logic model for a school. While not a “recipe” since there are many details to consider, it provides the broad outlines.

Begin by taking your mission statement and considering what it means in terms of impact. These impacts will be on the populations involved in your “school as program” which include the students, their families, and perhaps others. At the same time, also consider the big picture of what you hope to change and achieve through these impacts, at the highest level. As an example, if the mission statement indicates the school creates a “community of learners,” the logic model process will guide you to defining: what would that look like? Who would be doing what, if they were members of an actual “community of learners?” Take that somewhat abstract goal down to the concrete level with questions like these:

• Who is in that community of learners? (students? parents? others?)
• What are they doing if they are in a community?
• What are they doing if they are learners?
• What connects them to the school, as evidenced by their behavior or attitudes?

Next consider what changes you would see in the short, medium and long-term towards that aspect of the school mission. What changes in behaviors, attitudes, skills and knowledge do you hope to see on the part of your primary beneficiaries, as you define them (likely, students and families)? Then play out the outcomes and impacts for all the populations you consider relevant to your school’s mission. As an example, we might consider the impact on the student.

What are the immediate changes we hope to see in that student? Would it lie, for example, in their attitude toward school attendance or learning behaviors?

What is the medium-term picture we have in mind for the student? Perhaps a year or two out after graduation, the student might show evidence that s/he was impacted by their experience at the school-as-program by remaining connected to events appropriate to their alumni status, or by their Jewish behaviors in college.

And finally, while we might not measure all of these impacts and outcomes and most notably this is the case at the highest level, it’s still important to ask: what is the long-term impact we hope will be evidenced for students? It might involve their continued involvement with the Jewish community, their use of their gained Jewish knowledge, or their connection with the school in some way. Again, we want to put these outcomes in concrete terms as much as possible: who will be doing what?

Note that there are many factors and variables affecting outcomes, particularly at the medium and long-term stage. It is therefore important to be humble about your program’s impact, even as you imagine the world you want to help create. Eventually you will pin those changes down that you have identified for your relevant populations to say: how will I know if those changes occurred? This is the stage at which you are designing an evaluation based on your logic model; those changes and outcomes you identified are what the school needs to find indicators and metrics for, by purposefully and systematically collecting relevant information and data. At this point, you will select those impacts and outcomes that are both measureable and also central to your mission, and then pursue them with data collection.

To conduct this data collection, you will need to enter another phase of this work: detailing the ways that you might collect information, through a variety of means, to discover and track your outcomes. However, at this stage, it is likely you will engage the services of a professional with research training and program evaluation experience.

But before you get to that type of evaluation, complete the logic model by answering the question: what are the activities needed, to achieve those outcomes, those goals? That is, what do you have to do to get there? The answer to this question, of course, relies on the outcomes that you identified for your program. Depending on the level at which you are creating a logic model, these might include:

• Conduct outreach and marketing activities
• Develop the curriculum
• Plan for alumni activities
• Provide teacher professional development

As you can see in this brief sample list, all of the hard work that staff do is valued as it contributes to reaching the school’s goals. And don’t forget, when doing the logic model use active verbs to describe these activities: real people are doing real things to get the school to its goals.

And then make sure to count what you do. This is the “outputs” column of the logic model. It’s the way you’ll know if you have followed your own “theory of change” – we believe that these activities will lead to these valued results. It also allows you to track that great work that is being done, as evidence and guidance for you, in developing your program and seeing the path to your final goals.

Drawing these different elements together into one “story” as told by the logic model will then put you in a position to understand your own processes. You will also be able to communicate to others that you are doing the job as you’ve designed it and towards the clearly identified goals.

VI. Logic model opportunities for schools and a challenge

There are many opportunities for program evaluation and logic modeling available in day schools, even beyond the case explored above of school as program. Think at different levels, such as: a consortium of day schools as a group; a single day school; a subprogram within the school designed to achieve a specific objective; a one-time program designed either for student-learning or for supporting the school in a way that ultimately leads to student learning. Approaching activities related to the school at all these different levels results in very different research questions that might be addressed. Bringing these tools into a day school also opens up the conversation about who the actual beneficiaries are of the school’s work.

There is one challenge related to applying logic models and evaluation in schools that is worth noting: thinking about the difference between program evaluation and what is more commonly called assessment. There is an aggregate “wholeprogram” aspect to the work described here that is important to note and that distinguishes it from other assessment work. While the goal is to identify excellent teaching and learning outcomes, program evaluation is not essentially about assessing the work or results of individual teachers or students.

In an important way, the tools described here are designed to track and measure the outcomes of a full set of activities, regardless of who carries them out and without regard even for the specific students impacted. In this approach, the topic being evaluated or judged is whether the basic “logic” of the effort – the planned activities and the theory of change which lies behind them – whether that “machine as a whole” correlates with our best understanding of what process works in general, towards valued ends. As such, program evaluation and the logic model are about planning for outcomes, and measuring milestones of activity given that plan, more than assessing individual efforts.

VII. Conclusion

The logic model ultimately serves both internal and external program purposes. The processes described here bring out a key and fruitful question of reporting and communication for schools. Who are the audiences for understanding the school’s methods, plans, theories of change, and results? And what does the school most need or want to communicate to those audiences? At the same time, there is value in creating a common language and a process for building internal shared understanding of any “program” associated with the school. The program may be the Hebrew language learning track, or it may be a fundraising evening or plan, or it may be outreach to parents in marketing the school. It also may be the “heart of the order” – student achievement and long-term outcomes. Overall, the logic model provides a program with an opportunity to step back and participate in a process that facilitates helpful discussion internally while creating a clear product for communicating about the program to others. It also importantly becomes a key tool in moving from defining “success” to measuring it through the next step of identifying indicators for those clearly developed and valued outcomes.