8:00-8:45 AM
(I have written notes for this, nothing earth-shattering but I learned my lesson from the first plenary to not use the outlet behind my chair because every single person will trip over the cord throughout the entire session eventually yanking it out. I needed to save my laptop's battery for the next session which was not guaranteed a power outlet, but thankfully had one. This Hyatt is great for meetings but awful for access to power outlets.)
9:15-10:45 AM
Client interested in how people identify themselves as Jewish in 20-30s demographic, suggested starting a blog. People don’t want to answer long questions in surveys but don’t mind in blogs, identify themes in postings. Another person was concerned with education in physics, science middle school teachers mostly 40s – depends on the person, some really enjoy writing and others don’t on blogs. “Does anyone really read this? Is it of value or a waste of time?” Didn’t see use to it of posting opinions for public view. Could age demographics be a factor? Having a clear objective and audience a clear point to a blog.
Aboriginal justice programs in rural communities, posting lessons learned in a blog over time instead of filling out a survey. Incorporating video/pictures etc. Challenges: making sure everyone has access, training/guidelines on what to blog about.
Can guide discussion/moderate discussions in blogs, how to get data that isn’t relevant/useful, delve into things in interview but not so much in response to blog posts. Teachers suggested using a blog instead of meeting f2f, was a total failure “younger people blathered on, older people never got on, they knew it was going to be used by the evaluators.” The data, depending on writing ability in print which is different medium than conversation.
Have to change old way of practice & methodologies, how can I twist the blog? No, that's the wrong approach; what can the blog give me that I can’t get in interviews? This technology wasn’t meant for interviewing. Asynch format is both strength and weakness. What are you reaching for and what are you trying to get at?
Who’s reading it is very important, social desirability in responses, things mentioned that were never told to faculty. What information is available on which people are most likely to blog? It’s changing. The Grandma Blog. We can’t zero in and just say it’s young people, interests are wide ranging. How well does the site facilitate making comments? Video blog for individual student feedback in classes, typing out didn’t work well. Not as easy to analyze the data. Keep the blog a blog, don’t have people merely dump data there that may be better for email. Anonymity or real name posting?
BYU student blog for professors to respond on. Google Analytics data helped to see percentages of people that are hitting certain topics and areas. Having a free for all at the beginning helpful to then refine subjects of focus later.
Usability and web metrics – you don’t know that they’re reading the rest of the post.
Evaluation of comments, spike based on current post? Additional response to things tagged with the same topic. Blogs with participation vs. those that aren’t, different set of data than general numbers.
Are students doing 5-10 minutes a day in online class or going through half the class in one sitting? Adapt curriculum in response.
How much time do you spend looking at blogs looking for qualitative data? Blogs are the fastest way to find out about new technology. There’s no harm in signing up for a blog so there are large readerships. Don’t make your blog exist just to get data, have a healthy blog that’s fun. It’s tricky to build an audience, readership & people who want to comment a lot.
How to deal with vulnerable populations and IRB factors with blog? Had to choose anonymity to get users to use the site. Therapeutic benefit to posting on blog vs. seeing counselor a possible factor.
10:55-11:40 AM
Research & eval needs – various geographic locations with large distances between them, quantitative/qualitative tools and strategies, flexible design parameters (math/science partnership, state dept of education asked for eval, RFP landed on their lap but there was no actual program yet).
Database tool: File maker - Reformed Teaching Observation Protocol (RTOP, from Arizona State) lesson observations that include narrative, 25 inquiry items, cite evidence for rating. Moved from Word to dbase, organized data for storage, verification, review and analysis. Clone copy of RTOP form, mirror protocol, customize layout & include export capability via email. Can re-ex/import to modify data without wiping original source data. Integrate radio buttons (1-4 ranking) Working on exporting data to InVivo. There are so many car accidents in Las Vegas that data was routinely lost when papers would fly out of the evaluator’s car! Encouraged evaluators to immediately email import data instead, internet access onsite required as a result. Staff issues in using it? They used to email their Word documents anyway so it wasn’t that much of a process adjustment. Is spell check in it? Adding it as a feature now.
Participant Log: Moodle – statewide math-science partnership MSP evaluation, teachers log monthly PD, program goal is 100 hours per year. ‘Conned’ the teachers, they are filling out 3 surveys simultaneously but they think they’re answering 15 questions on who they are like Facebook profile setup.
Pre Post-test: Forms Manager, matching, geographic location, IRB protocol, student friendly. Students like putting their friends’ birthday in the data? 12 born on the same day in one class, not reality.
Technology = Building Capacity. Simultaneous data collection, increased reliability, decreased staff time, travel, paper expenses; integration with program activities, addressed stakeholder needs, gained rapport; increased access to data management, consisten IRB protections. “If I’m in an extremely boring meeting, I can log on and work on my data collection instead.”
Challenges: pre-post test matching, time for introduction to instruments, exporting data requires forethought, sustainability at program close, participant access to technology, risks in trial and error, more participant communications, excess data.
1:35-3:05 PM
New evaluator, new eval department of 1 for multi-site agency with 10 regular program and 4 federally funded programs, how to align the program outcomes. What to do? Logic models provide clear guide. U Wisconsin Extension program (online learning modules, downloadable handbook) http://www.uwex.edu/ces/pdande/evaluation
Logic Model
Horizontal-linear model design
Model elements included inputs, outputs (activities, learners), program objectives (added), outcomes/impacts (short term, intermediate, long term and longer term; assumptions, external factors.
Big Hairy Audacious Goal (B-HAG) (I love this. Susan Barnes loves this. We suspect Maryanne Blake will love this. From now on I will ask what the B-HAG is of a project or program)
Ongoing collaborative process
Developed draft & revised with executive leadership
Large scale org model rolled out to all staff with emphasis on program staff
Developed individual program models collaboratively with program directors/staff using org-wide model as the framework.
(pretty color coded left/right flowchart logic model! Inputs Outputs Program Objectives Outcomes/Impacts headers. Bottom has Assumptions <->Inputs/Outputs and External Factors <->Outcomes/Impacts)
Org Challenges
Defining individual program outcomes, measuring short and long term program impact, building internal eval infrastructure, engaging all stakeholders, securing staff buy-in (you’re doing great work, how can we show funders?), working with limited resources and capacity.
Benefits and Learning
Internal benefits: Unified staff vision (I see how my program fits & we’re working for the same goal), alignment of outcomes, identify gaps in programming and services, informed org’s eval plan and practice.
External benefits: easy tool to communicate to funders and partners, streamlining eval and measurements for reporting, legitimacy of programming and measurement.
Logic Models applied to federal programs (same org)
(specific to Talent Search & GEAR UP MA program evaluators)
Logic model same; information/awareness, self, environment, action boxes under short & intermediate outcomes/impacts. Information/awareness and Self boxes blank)
(Karyl Askew & Michelle Jay, GEAR UP NC Logic Models from the perspective of external evaluators)
Evaluation fair – allows stakeholders to come together to share their evaluation strategies and program accomplishments.
Include a Logic Model Terminology definition page.
Benefits of collaborative evaluation
Stakeholders’ knowledge and insights are acknowledged and valued
enhanced evaluation knowledge and how-to
improved data collection, data quality and report writing
Enhanced capacity to utilized and conduct evaluations
Additional conclusions
Collaborative evaluation approach is a developmental process for both client and evaluator
Use of logic models allows evaluators and program staff to gain a more concrete understanding of a program and its evolution
Can be used as a communication tool in addition to evaluation tool
3:25-4:10 PM
(I tried to understand these presentations. I really did. The first one had no Powerpoint, and I am simply not that fast of an audio-to-text writer so I failed miserably trying to get the first one down in any coherent fashion. The second one was about evidence-based practice (EBP) and I thought that I'd understand that, and did for the first part, but the evaluation concepts were completely unfamiliar with terminology that baffled me. I have a copy of the second presentation and am not sure if I'll try to make sense of it later or not. It includes phrases such as 'Purpose: To critique the epistemological foundation of the current "gold standard" for generation of evidence-based practice' right from the beginning.)
Multiple evaluation demands
Multiple levels, multiple needs (planning, conducting, using, consulting, training)
Why focus on eval at org level?
Walk the talk – expectation of funding partners, building capacity and skill to evaluation complex programs, build culture of evaluation and evaluative thinking.
“Don’t accept your dog’s admiration as conclusive evidence that you are wonderful” Ann Landers
Evaluation Capacity Building (ECB)
Intentional work to continuaously create and sustain overall org processes tjat make quality evaluation and its uses routine
Evaluation is an ongoing effort, routine.
Challenges of ECB
Complex org, provide vision & focus for evaluation, desire for information and solutions now, provide useful info on important questions, pressure to support scientifically rigorous approaches, sustain evaluation resources and activities.
Key Strategies
Implemented across org, involve various levels, responsive to emerging issues in timely manner
Portfolio of Eval projects
Short term tends to be less rigorous but yields important data for decision making, can build credibility.
Long term projects address critical questions to org goals, best/promising practices at state/natl level
Key Infrastructure
Dedicated funding, personnel/staffing. Staff time to participate/contribute & provide feedback to eval, learning environment and evolving approaches, leadership buy-in and approval. We are working in a supportive culture, people willing to hear what they have to say about evaluation.
Lessons learned
Evaluation conducted in phases, stresses eval is a journey and not an endpoint, flexibility – adjust plan to needs of org, manage resources to meed need – prioritize projects, maintain focus, be flexible, say No; need & challenge of securing support of internal stakeholders. Important to gain support across the org (substantial time & effort, difficult to coordinate/communicate, tension from being an evaluator)
Internal evaluator role is to identify and highlight issues, not necessarily solve them. (That is my own emphasis there. This was a major aha moment for me. I don't know squat about evaluation & I thought I had to solve everything as part of the job.) You are not a program manager or org consultant, build capacity of leadership to use the evaluation data. Evaluation question should drive the method, strong desire to start with performance measures but didn’t meet need and answer questions.
Need short and quick victories, but can’t sacrifice longer term projects, need to balance ability to implement project with its potential benefit (avoid hard to implement and low benefit projects, build capacity over time to implement more complex and high benefit projects, expand expertise to meeting demands of program or advances in evaluation.
We Think That We Already Know
‘The greatest obstacle to discovery is not ignorance – it is the illusion of knowledge’ Daniel J Boorstin
Development of Strategic Planning Process to Complement an Existing Evaluation System
Jill Elnicki
Evaluation Technical Assistance
CDC Division of Adolescent and School Health (DASH) Eval Research Team
Strategic planning is a process that results in decisions and actions to guide what your program is, what it does, and why it does it (Bryson, 2004) Do you really want evaluators doing this?
Why Strategic Planning?
DASH wanted to suggest a process to their funded partners to use to improve
understanding and connection to 5 year goals, data-driven goal setting at the
beginning of the 5 year funding period, improved communication between DASH
project managers and funded partners.
Challenges of Strategic Planning for DASH Funded Partners
Planning already occurred for them, some were newly funded but most were
previously funded and ongoing, at different points to program success.
Connecting Strategic Planning to Existing Tools
Logic models, annual workplan, process eval, outcome eval, all of these = lots of planning
Their job was to create an umbrella over the top of these systems, connect these pieces. Had to be mindful of terminology usage, study how to best connect them and have it make sense to the partners.
Developed Strategic Planning Kit for Funded Partners
Accommodates where they are in their journey, builds on traditional strategic planning, provides tools to help complete key steps tailored to program components and data sources instead of creating a new component, used existing planning and evaluation system to drive the strategic planning process.
There was a sense that there was already too much planning going on in the first place, continuous planning and not evidence the plans were being used for anything.
Use of existing Planning/Eval Systems
Data source matrix - categorizes data sources as internal or external to the program, complete SWOT analysis using internal (Strength/Weakness) and external (Opportunity/Threats) data.
DASH Program Goal Review
Lots of technical assistance provided and resources/guides on how to write things, but partners’ workplan goals are often rambling and confusing. Strategic planning provides the basis for improving program goals, partners’ identify program effect within 5 years, target population and strategies. They then use SWOT analysis to improve goals.
Strategic Plan also Guides Program Evaluation
Strategic planning often reveals weaknesses in program evaluation (data not collected, eval findings not incorporated in future workplans, poor communication to stakeholders), strategic planning highlights importance of good planning and eval data, new strategic plans include strategies for improving evaluation.
Lessons Learned
Division-wide collaboration key, creating example strategic plan from real data helped reveal problems, providing training to DASH staff prepared them to be frontline tech assistance providers, ongoing training (webinars) & tech assistance needed.
Strategic plan is at www.cdc.gov/HealthyYouth/evaluation/sp_toolkit.htm