3:15 PM to 4:15 PM
Golden is the Sand: Memory and Hope in Policy, Theory, and Practice Plenary Session
Lois-ellin Datta
This is a time of images, unforgettable from the past week. Boats on the river- evaluation policies, down on the river. Pushme-Pullyou double llama head on each end can look inward or outward. Evaluation policies shaped by inner gaze but not always harmonious. Need of outer gaze in addition.
After darkness of 1930s, Americans wanted to have better life in the 60s. Also the beginning of evaluation, finding out on a broad scale... could hope lead to enduring change? Last year AEA developed evaluation policies and practices in regulation and expenditures. May become recognized player in initiating sound evaluation policies, can support sound evaluation policies others have done, if necessary avert/correct evaluation policies that seem unsound.
Awareness of great social, economic, environmental and political world as part of external gaze focus of evaluation.
Add two more: Who is involved? How is diversity represented?
Resource distributions: how are they directed for training, capacity building efforts?
OMV reviews all government agency programs, central conduit how an administration recommends programming. Efforts to influence policies are daily bread. Speak up! Don’t whisper and hope that you’ll be heard. We at AEA have been a little out of touch. Over 100 social science orgs work through COSA, AEA doesn’t. There is an upcoming reauthorization of NCLB Act, never prepared a recommendation on evaluation policy. What should eval policies be? Who decides what is promoted?
Hard to develop policies that are inclusive of all members and specific enough to help.
Gazing inward (Kitty with bug on nose picture)
What should we as evaluators promote as evaluation policy for programs?
Evaluation Areas of Likely Agreement
More discussion needed
Gazing to the…
Rise of the Rest
These need to be taken into account when setting evaluation policies. Systems-based approaches needed. Where we are today isn’t just the leadership of today’s leader but the steps taken by many people over a long amount of time & previous leaders.
Meta-evaluation criteria
4:30 PM to 6:00 PM
New Evaluator "How-to-Session": Methods in Evaluation Multipaper Session
Needs Assessments and Evaluations: Examples of Combining the Two
James W Altschuld, Professor Emeritus, Ohio State University
Publishing 5 volume series in how to do needs assessment next year
There are guiding principles, but he’s been at this a long time and has a humble addition to the list.
Altschuld’s added guiding principle: It doesn’t address thinking of evaluation as a profession or a calling, it’s more than a service. Do research and publish to add to the body of knowledge in the field.
Need is the measurable discrepancy between ‘what is’ or the present state of affairs in regard to the group and situation of interest and the ‘what should be’ or desired state of affairs (Witkin &Altschuld, 1995) Needs assessment is a systematic set of procedures undertaken for the purpose of setting needs-based priorities and making decision about organizational improvement and allocation of resources.
Periodically Reassess/Rexamine Needs (last step of the cycle) Table 7 Needs Assessment table Many times you don’t need to do this. There are a lot of things missing in the literature because no one is publishing.
If you don’t think beyond the quantitative data, you won’t understand the causes and problems.
Liffin method variation (?)
Altschuld loop (red circle between two groups of data) is his contribution to the field. Science instruction is consistently ranked mediocre to poor by minority students and not well by others.
Importance of Logic Modeling
Ralph Renger, University of Arizona
Needs assessment (N/A) has identified a problem. He hates N/A. Every time we do one, that’s where it ends because we’ve identified the problems then we do nothing for them. Important to ask what we do next.
When should program coordinators consult with an evaluator? In the planning stage.
What exactly are you supposed to do if you are consulted in the planning phase? All the work!
Help make sure program has every chance of being successful. Justify investments in evaluation. How is success/effectiveness defined? Goals and Objectives are met.
Why do programs fail? If we can understand the issues then we can help avoid them.
Activity Trap: Department of Homeland Security. Just got back from Lisbon and the world does hate us, a lot. We think we’re it, cultural unawareness, not even caring about understanding, we think our way is the only way to do things. Strategies of baggage check and 3oz bottles not addressing the problem.
What tool can you use to help the client avoid pitfalls? (step two, then a miracle occurs cartoon)
1st step: root cause analysis, also called program theory. Understand what the underlying conditions are.
(visual diagram of neighborhood ->lack of physical activity -> high rate of obesity) NEVER show this to a client!
Happy to share resources on how to engage in this process. 10-15 experts in the field, not other evaluators. Find the people who are content experts.
Next step, prioritization. Ask key questions: is it within your mission? Do you have funding?
Back it up with research, then you have an evidence-based framework.
Ask clients to write down their programs and strategies. Get the documentation in place as the basis for your process evaluation. You need to know what they want to do and how to do it. Convince them (wouldn’t it be helpful in an audit? Don’t you want to be a model program someday?)
Make sure the strategies are linked. Develop realistic objectives.
Final step is measurement: he thinks we take too much of a research approach to measurement when we’re doing service programs.
Program theory = we want to figure out what’s going on with your program and what it’s about when speaking with your client.