8:00 - 9:30 AM
Distance Education: Course and Program Level Evaluation
Evaluation Methodologies for Innovative Distance Education Programs
Debora Goetz Goldberg VCU Medical College
Need for eval of distance programs
Growth in # & diversity distance ed, most published evals of distance-based grad programs at course level, lack of confidence in distance-based programs.
PhD program in health sciences
Interdisciplinary, hybrid (2 weeks beginning of program, dissertation meetings on campus), 10 years. Methods: on campus lecture/seminars, interactive multimedia texts, 1-1 communication, asynch & synch group discussion, computer learning modules, course/podcasting, telephone/video conferencing, online assessment. Most are working family adults, program gave them iPods to access materials.
Step 1: Define quality in distance ed
Institutional, student (access to materials & tech to do program), faculty support (tech & tools to teach in online environment); course development & structure, instruction and curriculum
Step 2: program aspect for review
Program goals achieved, skill develop for student, advision function of program, instructional technology adequate, overall eval of program
Step 3: data collection approaches
Course eval, alumni assessment (1-3 years and 5-10 years after graduation), student feedback of specific technologies, instructor assessment of programs (semi-structured interview), employer feedback (informal interview). Haven’t done the last two yet but plan to.
Step 4 Data collection & analysis
Internal eval of program (no additional funding to do this). Timeframe: 1st phase –course eval, assess tech & alumni, 2nd phase rest of previous ones
Step 5 eval of PhD program
100% response rate on alumni questionnaire (did it meet stated goals/objectives? Students’ objectives?) Based on lit review. High rankings for format and tech (100% on 3/5 questions, upper 90s others) Skill development high except public speaking, cultural competency teaching and quantitative analysis.
Findings
Improvements needed to statistics (hard to teach via distance learning) and curriculum design classes, majority said class sequence was good.
Step 6 improve programs
Changes in curriculum, enhanced use of distance ed (started taping all onsite courses, additional videos of subject experts), additional instructor training on distance technologies, supplementary use of TAs in statistics to provide more personal assistance to students.
Questions: no external perspective, ask about work/life disruptions to education? Would have liked to do external but no funding. Facing challenges of legitimacy and leadership value of distance education. TAs with quantitative backgrounds assisting students, who have wide background & often don’t have these backgrounds.
Building Evaluation Program into Online Teaching
Juna Snow, InnovatEd Consulting
From instructor perspective, design & evaluation of distance course. Was adjunct faculty at two large universities, designed 2 courses in fall 2007 and taught them spring 2008, 100% online courses as 100% online Master Ed programs. Used Blackboard in 2 ways, asynch plus asynch/synch. Used web boards for reading/reflection time.
Evaluation approach
Internal, participatory evaluator role: action research. Participatory, qualitative, reflective. Innovation evaluation (study in situ, context-laden), interpretivist research paradigm.
Guiding questions
To what extent do the students find the course meaningful? (curriculum, readings, assignments, discussions) In what ways should the course be improved? (work load, pace)
Design
Opportunistic,purposeful, practioner-oriented, reflexive
Method
Artifact meta-analysis
Web based discussion board postings, student work, curriculum review, achievement scores/assessment methods
Performance Management Portfolio
Personnel evaluation from human resource literature as alternative to annual review, created by her in 2005 to foster empowerment within individual staff performance. Adapted in 2008 for student performance assessment, goal-oriented, learner-centered formative curriculum evaluation. Gives place and space for students to record what they are doing, learning and account for time.
Portfolio entry structure
Weekly student performance, patterns in performance, records tasks, tracks time, reflects on task assignment & performance, synthesizes multiple aspects of performance experience.
“It is good to see how much time I’m actually spending on this course, and it’s not just my imagination”
Analysis
Ongoing content analysis, emerging patterns, critical reflection, iterative responsiveness
Actions & Outcomes
Schedule synch, reduce amount of synch events, focus content, reduce reading assignments, clarify assignment instructions, share assessment rubrics in advance, provide examples.
In assessing student work, can see variations in performance, had a few students who just weren’t getting it while others were getting it & doing good work. Read 55 pages of text, formulate a response for discussion board, read others, respond to 2 others = takes a lot of time, questioned how some could plow through that much in 1.5 hours where the others were spending 4.5 hours.
Use of Participatory Multimethod Approach in Evaluation Distance Ed in 2 Developing Countries
Sabrina Liccardo, University of Johannesburg & work in Bangladesh
OLSET radio learning programs nonprofit NGO, develop English language competencies to broadcast to schools that involved teachers as partners in teaching & learning for 1st through 3rd grades, all materials provided to schools by OLSET.
(more in handout, didn't take extensive notes)
Evaluation of Interactive, Computer Based Instruction (CBI) in 6 Universities: Lessons Learned
Rama B. Radhakrishna, Penn State
3 year project funded by USDA, development of 11 forage crop online modules. IT has changed the way we do business in education, use of computers & 24/7 Internet increased tremendously.
Hybrid courses increasing in popularity (Young, 2002) because of quality student-faculty interaction (Navarro & Shoemaker, 2002) & enhancing learning outcomes (Tuckman, 2002).
Allows student to determin their pace & instruction at a time (Shoener & Turgeon, 01) Computer based useful for classes requiring integration of text & images.
Needs
Rising costs of higher education, demand from funding agencies for collaborative funding opportunities, students learn at own pace & time, capture research & teaching expertise of numerous faculty.
Purpose & Objectives
Describe process of developing modules, determin effectiveness of them for knowledge acquisition, evaluate them for accuracy, readability, navigability, etc.
Module Development
Spent 1.5 years developing it based on extensive lit review.
Method Procedures
Created 20 knowledge questions for each question, module evaluation survey (Likert, yes/no, open ended scale of measurement)
Pre and Post Tests: all students had significant increase of knowledge gain in all modules.
Navigability, Content Accuracy, Design & Layout: all students ranked it high except they could not print particular parts of modules because they couldn’t find Print icon on module.
Conclusions
Working collaboratively works, people happy about it because they saved quite a bit of time and resources. Students gained knowledge as shown in pre/post tests. Students agreed that navigability, content accuracy and design/layout were good, also indicated it is up to date, logically organized and visually appealing. Integrating visuals in text was challenging. Print capability was rated low, modified icon to make it more visible.
Lessons Learned
Will help subsequent projects be more efficient, offered challenges and opportunities (try out other courses in agricultural sciences), meeting of project teams was challenging (IRB among 6 universities & crop faculty had no clue what they were!), some institutions were not ready or understood this process. Implementation fidelity major concern (things not done as planned), faculty didn’t understand eval process and protocol, instructions to collect data not followed by some, statistical analysis limited because of low numbers. There is a relationship between knowledge gain and course satisfaction.
Implications
Project has stimulated faculty discussion to do more CBI projects, allows for reaching large audiences efficiently and effectively, helps to bring expertise of many faculty/educators to share resources and reduce duplication of effort.
9:50-10:35 AM
Social Network Analysis 101: A Brief Demonstration
Stephanie Reich, UC Irvine
What is Social Network Analysis? (SNA)
Study of social relations, focus on interactions/patterns (reason for formation of patterns, consequences of patterns, stability or dynamics of them)
Really looking at groups and the interdependence between things vs individuals, based completely on dependence on everyone else in the network
Freeman, 04
Social network approach is grounded in the intruition notion that the patterning of social ties in which actors are embeeded has important consequences…
Key Elements of SNA
Focus on relationships between actors rather than attributes of actors, sense of interdependence: molecular rather than atomist view Stucture of connections affects outcomes
Relations (Graph Theory)
Social relationships can be graphed, mathematical structures can be used to model these social relationships (matrix and list structures. (Facebook app)
Basic Terminology
Network (group linked by a specific type of relation (class, conference attendees, etc)
Actor (persons objects or events/nouns. In graph form, an actor is a node.
Tie (edge/acr) something that connects actors (evaluation, kinship, physical location, shared associations, common activities, formal relationships. In graph form edge (undirected) or arc (directed) can be reciprocal (two actors pick each other)
Strength of ties
Dichotomous (present or not, you like them or not)
Valued (degree of tie, really like them or not, talk to them daily etc)
Sources of data
Questionnaires (roster of organization, nominate top 4 friends)
Interviews (high risk things, feelings towards classmates)
Observations (watch who is talking, what’s going on)
Archival data (records, rentals/ratings)
Interesting connections in networks
Dyads (relationship between 2 people), triads (3 actors, everyone connected = strong connection, 2 ties but not 3rd likely to become friends because of mutual friend, idea transitivity (friends of friends are likely to become friends), diamonds (really cohesive, likely to stay together, some triangles but not others are likely to dissolve), stars, sub-groups, networks
Matrix (mapping who to whom)
Population (closed) network
Study universe (not inferential, like classes, businesses)
Sample Network (ego-centered)
Study sample of a target people. Ask one actor who they’re connected to, ask them who they are connected to (who do you share needles with?) CDC does this a lot to look for high risk behaviors
Modes
One-mode (network of actors of the same type)
Two-mode (two types of networks that connect to actors to each)
Things to Calculate
Individual indices (importance in network)
Network properties
Modeling of networks
Individual Indices
Network importance
Centrality (affect flow of information, power, resources (degree of popularity, closeness/proximity, betweeness/go-between & hub)
Degree Centrality
In-degree = Popularity (coming in to me or out from me, how many pick me, amount of ties to actor)
Out-degree = Expansiveness (amount of ties from actor)
As compared to ideal: Star (everybody connected to you directly)
Standardized Degree Centralities
(scary formula to the bottom right)
Closeness Centrality
Close proximity to other actors in network, efficiency for transfer of information/power, distance to all actors in comparison to if on every shortest path (geodesic)
Standardized Closeness Centrality (more scary formula)
Geodesic (what is the shortest path to get to everyone?)
Betweenness Centrality
Strategic location in network (number of paths compared to being on all shortest paths)
Hub for transfer (information/power/innovation, can facilitate or block transfer)
Standardized Betweenness Centrality (yep, more scary formula)
Some Positions are Powerful
Cut points/Bridges (connect isolated parts of the network, essential to be one network because if you removed them it would disconnect components)
Prominence: Prestige
Who you know matters. Own prestige is a function of the prestige of your friends (linear composite is an Eigenvector (rank order or prestige).
Importances of Connections
It’s not just how many connections you have but how and with whom you’re connected
Networks Level Measures
Density, centralization, components, sub groups, role equivelencies
Network-Level Structure
Does the structure affect function of the network? Early studies of the effect of structure on: efficiency, leadership, satisfaction (Bavelas, Levitt, Freeman)
Network Level
Centralization (network revolves around core people, degree to which network cohesion is organized around central points)
Dense Networks
Many connections between actors, few central points
Components
Sub-groups based on reachability, can define different types (cliques, clans, clusters, all measures of distance to what makes it a group)
Structural Equivalence
Actors are in same or equivalent positions (doesn’t matter who they are, identifying meaningful patterns)
Different Power Distributions/Hierarchies
Simple (top down)
Branched
Mixed
Model Testing
Prediction (across network, over time)
Comparison (between networks and/or times)
Explanations (patterns and relationships)
Networked individual kids with obesity, over time the obese kids stay friends with each other and the thin friends stay friends with each other
P* Models
Model testing based on Markov random graphs, determine which patterns and structures occur beyond what would be expected by chance.
Connections, Journal of Social Structure & Social Networks – Journals in SNA
Analytictech.com
Insna.org (small group really friendly)
SIENA software website
(This is where I take my scheduled sanity break which involved seeing a Veteran's Day parade, taking a mile-long walk to Tattered Cover (LoDo location), spending hours browsing, buying books and laminated pressed Colorado wildflower bookmarks, walking back & grabbing a burger with pumpkin pie cheesecake ice cream at some place I can't remember the name of, and basically enjoying the gorgeous weather before diving back into the hotel for 3 more sessions)
1:20-2:50 PM
Developing Multicultural Evaluators
Evaluating Culture in Youth Eval Project
Various stakeholders, what does the process of getting better look like in terms of culture>
Metholodogy
Constructivist grounded theory study
Single conversational interview
Data Analysis
Codes from literature review related to resiliency, recovery and systems of care (approaches in mental health services)
New codes emerged
What they did to evaluate culture
Check how many time the codes identified in the literature were found in the transcripts (who made the references in the transcripts, what concepts were mentioned in culture)
Culturally sensitive outlook
Developing competency
Self awareness
System policies
Strong emphasis related to administrators and therapists to culture, more important for them to know the whole spectrum. Culture might be more important/aware where it’s closer to home
Teachers, parents/guardians and youth don’t mention a culturally sensitive outlook
(Why, was the question formulated off? Not critical? Is it an important result?)
Missing Codes
Stigma: no one mentioned it, no stakeholders acknowledging tradition
Developing competencies (no stakeholder mentioned using cultural experts
Overlapping areas indicated major themes of Cultural Differences, Identifying Needs, Cultural Sensitivity, Communication
Not all the codes in other concepts seem to be related to culture as we coded it. The areas where there is no relationship show where more work is needed.
No mention of culture’s influence on youth understanding resources or systems, or making services available but these are all key aspects of systems of care. No mention of culture’s relationship with community supports. Omissions: Cultural influences on education or coordination around medications, resources, information/making servings available
Conclusions
Reinforced relationship between Culture and Recovery, Resiliency and Systems of care. Insights involving each stakeholder emphasizes cultural influences on getting better, few references to culture influences in the classroom.
What we understand from omissions
Multiple levels of cultural competency with interviewees (some are deeply involved, some don’t think it’s possible) Issues with way questions were asked?
Need to find better ways to ask about cultural influences with some stakeholders. Relationsip between recovery and culture, importance of spirituality in recovery among adults, how to translate this to youth and family services to investigate that relationship?
Presentation available at
www.outcomesmhcd.com/pubs/publications.htm
Preaching to the Choir: Black Evaluators on Cultural Responsiveness
Tamara Bertrand Jones, Florida State University
Purpose
Maria Stewart (1833) When I cast my eyes on the long list of illustrious names that are enrolled on the bright annals of fame among whites, I turn my eyes within and ask my thoughts, ‘Where are the names of our illustrious ones?’
Research Questions
What have been the educational and professional experiences of Black evaluation professionals?
How do Black evaluation professionals define cultural competence in evaluation?
What skills do Black evaluation professionals believe are necessary to develop to become culturally competent?
Hood (01) yielded a list of 25 Black doctoral recipients in the evaluation field, first man & woman in the 1930s. 15 published articles in scholarly journals, 10 published evaluation studies and reports, or discussions of eval theory and practice
Literature Review
AEA Building Diversity Initiative (development of guiding principles for evaluators to work across cultures) Cultural competency in evaluation definition? Depends on who is defining it, one’s worldview as a scholar.
Study Participants
4 Black evaluators chosen to interview based on publication in eval, about cultural implications in evaluation, balanced genders. Each had at least 5 publications.
Characteristics of Cultural Competency
Context (as the professional’s understanding of cultural values important to evaluator and client, add to understand of the program’s working, cultural systems affecting program’s environment, establish rapport and trust with stakeholders , willing to be immersed in culture)
Methodology (take the entire context into account when evaluating, key aspects of evaluation, responsive to context of program, match eval techniques and models that will produce the richest data & incorporate them into evaluation, rigorous and approaches are relevant)
Evaluator (defining who the evaluator is, have understanding of their social location and interpersonal skills, assess themselves, see how their abilities match up to social context)
Culturally Competent vs Culturally Responsive
Can you ever be culturally competent in some other culture other than your own? What would be the minimum standard for cultural competency level? – Dr Roberts
Changed track to responsiveness as desired term, was interchangeable for the presenter after the study was done.
Tools for Culturally Responsive Evaluator
Awareness of cultural signals, building rapport through shared lived experience, ensure validity through various measures (doublecheck that your recording is accurate of them), practice! You have to have necessary evaluation skills and also the ability to frame culture in setting.
Developing More Evaluators of Color
Role of race in evaluation, role of diversity training programs, socialization, mentoring. Being black is not a temporary thing, it’s who a person is.
(Next presenter scheduled never showed up?!)
Developing Multicultural Evaluators: Building Capacity in Culturally Relevant Evaluation – Portfolio of NSF Award
Darnella Davis, Cosmos Corporation
Paradigm Shift
Proposes establishment of a culture learning protocol for evaluation researchers, useful in the US and internationally, that will guide evaluation research.
(asked her to email me presentation, very relevant information about indigenous culture and non-profit orgs & I was so absorbed that I wasn't getting notes down)
3:05-3:50 PM
IHEs Confront Technology's Impact: Online Surveys and Digital Resources
Evidence for Changing from Paper to Online Student Ratings: Evaluation Practice Shapes Policy
Chris Migotsky
University of Illinois UC
(I was the single freak in the room who had done online evaluation forms as both a student and instructor)
Paper: rising paper costs, long processing turnaround, tedious form request process, class time for evaluation.
Web based: completely web based, fast access to results, flexibility in item creation, no class time ‘wasted’, available during the last 3 weeks of the class & open during final exam week.
Did Pilot work?
Eval data: actual student ratings, surveys from faculty & students, focus groups with faculty & students, workshops across campus with Q&A.
Online response rate 56% vs 66% for depts. Standard deviations higher online (only hear from those who love/hate the class?)
Survey Results
Students said: incentives to participate, no pressure, fast/easy to use, more reflective answers, 89% prefer online
Faculty said: response rates low, easy to use, good to save paper, quick access to results, 67% prefer online
Focus Group
Students: tell us why it’s important, reminders with prizes, ensure confidentiality
Faculty: stop evals before finals, delete open-ended responses, attendance & evals
Policy/Practice Changes
Video tutorials (start to finish of whole process), no reversing of scales (some 1 to 5, some 5 to 1), change text of student emails (emphasize/explain), new norm comparison groups (SD changes), end evals before exam week, access to raw data for instructors, ability to move items on the form, ‘guarantee’ open-ended only for faculty.
All evaluation information comes via email to students from evaluation office, not instructors.
Googled: What Evaluation Reveals about How we Use the Web
Flora
Online Educational Resources (OERs): Setting the Context
Digital libraries, online educational resources, how to evaluate impact of OER’s on teaching and learning, how might they evaluate impact on future?
What happens when moving resources from place to educational digital libraries.
NSF has spent 100 million on digital libraries since 1997, colleges & universities invested heavily in distance and hybrid courses. How are faculty using OERs and to what end?
Question of Impact
Phase 1: National study, 11 focus groups oversampled community colleges, expert users of digital libraries. National survey (cold called) faculty 120 institutions > 4600 respondents, all disciplines
Phase 2: refining instrument for individual digital library use
Phase 1 Findings
Few differences among traditional demographic variables, pretty homogenous. People who responded valued them highly, but there are differences out there.
Asked about use of online scholarly resources (top box score reporting very frequent use 49%), digital images (mostly for powerpoint slide usage, 42%), teaching/learning exercises (28%), data sets (22%), animations/simulations (11%).
Why Faculty use OERs (most/least frequent)
Improve student learning, stay abreast of prof development, helps keep me fresh, incorporating digital resources in class is fun, saves me time, helps me develop my teaching skills, students expect me to, help me better accommodate students with disabilities, helps students learn difficult concepts, usefulness of digital resources are overrated
Searching & Finding OERs (scale of -2 very unlikely to +2 very likely)
When familiar with topic in field Collection 1.08 Google 1.14
New development in discipline Collection 1.09 Google .56
Barriers to using OERs
Use if had more time, more useful digital resources available, better training in use of them available, institution rewarded me for using them. Far less important: tech dependability, access, priority at institution
Phase 2 Refining
Is there a difference by discipline? What about communities of users?
How have changes on web since YouTube, blogs, Wikipedia, etc affected users?
Time, what does it really mean?
Faculty report being concerned about intellectual property in focus groups but not survey, always concerned about the students’ appropriate usage of resources.
Tool available & presentations to look at
http://serc.carleton.edu/facultypart
4:00-5:30 PM
Communication and Cognition in Evaluation Utilization
How do people process data reports?
Steve Lee U Kansas, Jill Lohmeier U Mass Lowell
What are traditional methods of reporting findings?
Personal/oral briefings, scholarly presentations, videotape, Q&A, written reports & briefings
Typical written reports/briefings
Narrative, tables, graphics, summaries, highlighted content
Why is the method important?
Too complex (not understandable), too much (not efficient/too long), not engaging, not memorable, doesn’t match stakeholder expectation, Type 3 error (appears to solve wrong problem)
What is the best way to report evaluation data?
Relevant studies Brown & Newman (1982) more agreement w/recommendations for percentage/graph data rather than statistical, advocacy affected reading and understanding of data. Sanfey & Hastie (1998) Textual displays were more salient than graphic displays in ordering (weighting) or cues for the purpose of making predictions.
Evaluators’ goals in presenting findings
Understanding, Main points, remember findings, remember finding accurately.
Method
Interviews with undergraduates, how they use data in various ways. Followed think aloud (TA) while reading a one page report, then asked where they looked first and most often, why they looked at those parts, what they learned from the report. Took the sheet away, asked what they remembered from the report they had just read.
Findings: where did they look?
What did they mention first in TA (title 48%, source 1 (18%) Chart 15%
What did they say they looked at first? Chart 31%
TA had chart mentioned most (95%),
Was what was looked at first remembered best?
Significant relationship between section participants reported looking at first and their memory for that section, not what expected
(more results I spaced out on waiting for conclusions)
67.5% accurately remembered the chart
Conclusions
During TA procedures students mentioned sections that clearly displayed data, that chart was mentioned & recalled by almost all of them. Charts & easy to read highlight boxes with bulleted facts best.
Making Evaluation Meaningful: Creating Recommendations that Clients Can & Will Use
Caitlin Scott, Northwest Regional Educational Laboratory
(No PowerPoint, nice written presentation handout)
35% of evaluations contained no recommendations in a national K-3 reading initiative .
Criteria for determining good recommendations
Recommendation based on a “major” finding of the eval, supported by more than 1 source of data?
Recommendation important enough that it made the conclusions and/or executive summary of the report? Only 4-5 recommendations total?
Reccomendation about a phenomenon that the evaluation data explain adequately? Reasonable idea about what would improve it or what the client should do to further investigate?
Recommendation something the client can do something about?
Language of recommendation clear, actionable, and non-threatening? AVOID ACRONYMS AND JARGONY TERMS! (yes!) Avoid judgement (‘fortunately’, ‘unfortunately’), demonstrate knowledge of the program and context it works in?
Develop Recommendations collaboratively with the client
Send the full draft report, without recommendations, include questions (What surprised you in this? What 3-4 findings concern you the most? What plans do you already have to address some of the issues raised here?)
Schedule 2 hour meeting/conference call to discuss report & questions. Prepare draft list of recommendations before the meeting
Facilitate a discussion during the meeting about report & questions.
Soon after the conversation, write the recommendations adding more details as necessary
Send final report, with recommendations, to the client.
Exploring the Role of Communication in Evaluation Utilization
Janet Lee, UCLA, Graduate School of Education & Information Studies
Increasing era of accountability in today’s educational environment, programs designed to raise student achievement, rare that the information from an evaluation gets to the staff.
Purpose of study: examine evaluation use in school setting, specifically communication as a factor that facilitates evaluation use. Other factors include evaluative inquiry, participation. Communication is a dynamic process, information flows from one person to another. One person read report and found it valuable, another didn’t but had good conversation with the first person about it.
Research questions
What are teachers’ views and perceptions of evaluation?
How do teachers participate in the activities?
Do they use the information gained from the evaluation process?
What do communication networks of teachers look like?
Relationships between different networks?
Data Sources
(attention spacing out here….)
Survey 41% response rate, 37 teachers responded to both surveys, limited analysis as a result
Findings
Strongly agree with statements about evaluation in general, 50%+ disagreed with statements about evaluation at their school. For the most part, teachers didn’t participate in evaluation (40%), 18% didn’t know. Some evidence of process use, changed the way they thought about small learning communities & schools. Can’t analyze network features or individual network data (social network analysis!) They’re rethinking this since they have a 50% response rate for some areas. Long pipeline of information, small clusters.
How Bad Presentations Happen to Good (?) free book, how to graphically present