8:00 AM to 8:45 AM
Evaluation Policy and Evaluation Practice Plenary
What is evaluation policy?
Substantive vs Evaluation Policy
Goals above translated to programs that are then evaluated, evaluation policies guide how evaluation happens.
Evaluation policy questions are everywhere
What is evaluation policy?
Rule, regulation, law principle or norm that a group or organization uses to guide decisions and actions regarding evaluationuation.
How do we distinguish policies from guidelines, standards or theories?
What do evaluation policies look like?
Relatively short statements that can be collected together.
Who is involved in policy making? Who controls it?
Signal -> organizing device
Public stance -> transparency
Mechanism for learning -> evaluation policy analysis
Connection between theory and practice
Changing evaluation policy is more efficient than changing practices everywhere
Many evaluation controversies are policy struggles
How can evaluation policy be made?
Those with authority may fear sharing it.
A draft taxonomy of evaluation policies
Goals, Participation, Capacity Building, Management, Roles, Process & Methods, Use, Meta-Evaluationuation
A child policy should never be broader than its parent (encapsulation)
In hierarchical or multilevel orgs, collaboration works across levels. Inheritance and delegation move from higher to lower levels.
How do we avoid overformalizing policy?
How do we avoid senseless policy?
Evaluation challenges
How do we incorporate our core values?
Get more serious about evaluation policy
Emphasize the dynamic relationship between evaluation policy and practice
(tossed in a bunch at the last minute, room lights did bizarre strobe light effect at this point)
Why is it important?
How can effective evaluation policy be made?
Who’s involved? What approaches can we use? What types of evaluation policies are there? How is evaluation policy structured? What are the major characteristics of evaluation policy?
9:15 – 10:45 AM
Examining Published Literature to Better Understand Evaluation Theory and
Practice
(I have notes scrawled on a Powerpoint handout for this that I'll transcribe later)
10:55 AM to 12:25 PM
Roundtable Rotation I: Evaluating Online Learning: Focus on Usability, Instructional Design and Adult Learning Theory
Carl Hanssen, Hanssen Consulting based in Grand Rapids MI Most in science and math education, NSF work.
No survey work done, examined usability around the tool and instruction design of the actual course layered with adult learning theory. As one, you need certain things to keep you engaged with the class. What they found on basic level, tool that was used was not an online learning tool (Telesys, threaded discussion forum) but more like Facebook (communication tool, but was used to post course materials and PDFs in the discussion forum posts). Found across different courses, layout and how to deliver content was extremely erratic (syllabi, reading assignments, etc) There wasn’t anything special going on from adult learning perspective. NSF expects what they’re funding to be state of the art.
Usability heuristics similar to Andersen Consulting stuff. Instructional design – what the tool looked like. Adult learning principles. Online learning was nothing more than self-study from downloaded PDFs and have need for threaded discussions, modern day correspondence course. How are they going to rebuild the tool? Are they going to? How can we make evaluation more useful for them?
WebCT/Blackboard. Question if students would use it. Public Health agency in Canada CE program online modules 3x year, very interactive platform (submitted assignments, videos, discussion) Looking at way to enhance the content. Pre & post survey in modules (on content in modules, not national policies), case studies, looking for diffusion of learning of how students apply learning and how orgs are running as a result of what is learned. What to do about attrition (can be high as 25%). Value of modules with several assignments and open-ended guidelines vs. weekly lessons/metrics tracking. This model failed with undergraduates for another teacher, who then went to week by week basis.. built technology in Excel to check assignments and grades itself. Student surveys aren’t an effective measure of satisfaction with the modality.
Content analysis of blogs (heavy duty evaluation, labor intensive, is it worth it?) Worth it for Journalism field. What we’ve been given is helpful for course development, next step is how to assess student learning? College profs have content expertise. Online faculty member monitoring things? Content experts and technology experts, not having one person running everything. UK Health competencies, linking it to learning plans, accountability, under huge learning system. Rachel Roberts. ThinkLink, Bismark State College running online programs with interactive modules incorporated adult learning theory that are successful & they use eCollege.
Roundtable Rotation II: Regional Education Master’s Online Training in Evaluation (REMOTE): Lessons, and Questions
Second speaker – Chuck Giuli Masters degree in evaluation from Hawaii. Discrepancy between the promise and the problems, lessons and learning. Regional Education Masters Online Training in Evaluation.NSF funding. F2F really helps for orientation, students didn’t want to leave the format.
Lessons: from 19 original enrollees 12 remain. Competition for learning time is significant and sustained. All participants value the opportunity and experience. Contact with peers & teachers f2f is missed. Mixing book-based learning and online learning problematic. Computers and bandwidth often subpar. Synchronous learning valued but hard to schedule.
Implications: desire for synch defeats 2 assets of online learning: ability to work alone when convenient, ability to time shift. Academic & participation requirements need to be thoroughly explained and repeated. New structures required to overcome negative effects of isolation and abstract learning. More time ($) needed for keeping close/continual contact with students. Courses need to be redesigned, rigor maintained. Funding for long term evaluation needed.
Determine minimum standards for required hardware, software and infrastructure.
Discuss course expectations and likely challenges 3-4 times.
Frequently monitor student progress, include synchronous instruction and opportunity to interact with peers.
It is like face to face school, expectations to focus on material and not other factors (work, etc.) The Red Violin (movie) One way or another you need to demonstrate knowledge, through test? Demonstration? Show you have set of KSAs you didn’t have before. Cost of bandwidth a barrier to education access. How do you demonstrate that what you’ve learned is being applied to the job?
Collaboration excitement was not shared between sponsoring stakeholders. Critical to lay out a road map. If you hang in there through the survival part... online learning runs the other way, fun at first but then gets harder toward the end. How disruptive lives can be after 2-3 weeks of intense energy then going back home. One student was 1 assignment short, going through major life issue. Critical to build an excitement factor for making it through. If you don’t quite know after 500 miles you’ve gotten the hang of it will get easier, is it 1 or 50,000 miles remaining? Focus early on a clear problem and build on learning around. Pay attention to the journey & energy required to make it through online learning.
Financial requirement for education an incentive to finish it? You need to have support of your family, colleagues, etc. Distance learning is not for everybody. Extremely high self-motivation and direction needed.
What are the questions we as evaluators should be asking about distance education?
Why are people leaving?
Employers making allowances for education coursework during work hours?
Who is the ideal student? (family & work environment supported them)
Build in mentors or peer to peer support structures?
BYU - Video assignments posted by students as part of discussion, instructor reviews them and posts own videos & positive feedback of better connectivity in the class.
1:40-3:10 PM
New Tools for Old Methods: Using Technology for Evaluation Methodology and Technical Assistance
Previously did needs assessments via surveys, in 2006 at the end of training was a sheet with open ended questions which were limited or broad, not sure what specific training wanted or needed. Restrictions as government agency, review of paperwork. Developed web board of cardiovascular health, threaded discussions based on topics. What ways could they use this tool for meeting training needs? How to narrow scope of questions? NACCD: how do we get appropriate questions? Asked evaluation component of heart disease & stroke to develop eval plan and logic model, evaluate hypertension project.
Listed broad level questions and provided specific examples, what level of training they need in specific settings of work. One of the biggest issues, how to adapt the tool? Weren’t able to apply survey checkmarks for threaded discussion board input. Opened a question and left it there a week, opened a new one. Participants would come in and answer question.
Needs assessment question example: Describe 2-3 major eval activities you or your program will do in next 12 months, include questions and tools you use.
Sent out invitation to web board, notified of when it opened, what the password/userid were, received notification when it opened. Staff positioned questions in it, emails once a week to remind them to come in and answer questions. One thing they noticed is immediate response worked best, box that would notify original poster if someone posted in response to them helpful.
Didn’t receive much feedback about the tool, disappointment. 5 questions, 9 total responses, 6 people participated (half of those contacted). Many more views of questions than what was posted due to tracking.
Challenges: participation, losing connections with people (not logging in for over a week). This was the first time they had used the web board. They probably won’t do this again unless they could figure out how to get higher participation. Postings were not anonymous, would participation increase if so? Thought they signed up, but actually weren’t, keeping track of yet another login/password, deterrent to participating in new websites. Online survey option instead, do you need people’s discussion or people’s answers?
Future considerations: web board still there focusing on general networking for health department staff, not training needs assessment, not heavily used currently either.
Traffic was higher at the beginning, tapered off at the end.
Visceral Reactions to Virtual Panels
DHDSP funds states to support oolicy and system change but lack of practice-based evidence on how state health departments can intervene. Lack of evidence for successful policy or systems change strategies. Expert knowledge needed were in high demand with little time. ‘Electronic meeting system’ 90 minute session for panel with phone/computer. 4 specific questions for 9 people, They listened to presenter and typed responses visible to all, no verbal communication from them. Spontaneously started talking with each other on the tool but they didn’t interact much on the brainstorming questions. After 15 minutes moved on to the next idea. Questions were provided to them in advance. As they progressed people were more comfortable and laughed, more conversations among themselves but not responding to comments others had, asked them to submit ideas one at a time instead of long narrative.
Received 144 recommendations in about an hour, rated them on impact to public health. They all completed it and took it seriously. Then could access report function of the tool to see higher ranking questions.
Ranked 63 recommendations combining suggestions, etc. Followup session open for a week.
9 Strategies ranked highest on feasibility, impact and evidence.
Benefits: all enjoyed it, interactive and kept people busy for the entire time they were online, no time to get bored or distracted, all had an equal voice, all have equal opportunity, responses anonymous or attributed, concurrent input from all participants. Real-time viewing, voting, prioritizing and sharing results.
The organizers didn’t have to do notetaking, statistics, etc. 100% attendance, saved time and money.
Keys to success
Know your audience, know your tool, personal invitations and follow up, have it be a new experience.
Professional Networking for Advanced Evaluators
(I’m not even a rank amateur, so at this point I spaced out entirely)
3:35 – 4:20 PM
Improving the Collection, Analysis and Reporting of Survey Data
Gary Miron, Western Michigan
“Are you just pissing and moaning, or can you verify what you’re saying with data?” cartoon
Survey construction
4 areas that can be addressed with surveys
Know the purpose for every item, standardize the scale, judicious use of open ended items, pilot testing, pilot test the instrument to ensure cultural validity, confidentiality, sharing results back.
Examine survey for sensitivity, specificity, ease of response, consistent direction, flow of items, use of vocabulary and readers’ level of education, avoidance of embarrassing/intimidating items, ethical issues.
Questionnaires should not be too long, be careful not to collect unnecessary data, include brief cover letter (who is collecting, why, how findings used, thanks) Parent-teacher surveys 10 minutes, children 5 minutes to complete.
Place sensitive items at the end of the survey, in recall questions ask about events in recent past so that memory can be more reliable, use questions that have been previously validated; retrospective pre/post test items are good for capturing change in knowledge but not for attitude or beliefs (some may rank selves highly at first then realize they don’t know at all and results trend downward)
Collection
Sampling frame, creating respondent ID codes, incentives (instantaneous most helpful), follow ups, establish system for multi-year surveys.
Paper surveys (ID codes-create meaningful ones (district/school/teacher#), contact information, follow-ups, entering data)
Electronic surveys (ID codes, contact information, follow-ups) – much higher response rate, they haven’t researched it yet.
Common problems: paper (removal of IDs, length of time from sendout to receiving, data entry errors
Electronic – mistakes on large scale, computer glitches
Analysis
Investment in creating system for analysis
Excel – quick and easy, best for dealing with aggregate
SPSS – easy but requires a bit more effort, best for disaggregating data
Open ended analysis.
Reporting
Easy to read, informative, user-friendly; standard templates, graphics.
Frequency bars – bar chart colors to the right, horizontal quick to interpret. Primer to explain what statistics mean. Written quick evaluation summary with a few quotes, not extensive content analysis. Include all results but edit to remove personal identifiers (i.e. complaint about no art supplies is clearly from an art teacher, modify to ‘supplies’)
4:30 – 6:00 PM
(I didn't realize this session was a think tank setting involving group collaboration. Even if you had expert knowledge, could you honestly handle a think tank at 4:30 after 5 conference sessions? I bailed quickly, dashed into the session next door and it didn't grab my interest either, scanned the program and couldn't find anything else that interested me, so I took that as a sign that I was done for the day and bailed to have an excellent Italian dinner with my aunt. She is awesome!)