1 of 16

CyberAmbassadors:�Results from Pilot Testing a New Professional Skills Curriculum

Astri Briliyanti, Julie Wilson Rojewski, Katy Luchini-Colbry and Dirk Colbry�Michigan State University

PEARC 2020

2 of 16

CyberAmbassador Project

NSF training grant (#1730137)

Provide professional skills training to �CyberInfrastructure (CI) professionals
Focus on communication, teamwork, leadership
Overarching goal to support interdisciplinary research

http://colbrydi.github.io/cyberambassadors

3 of 16

The idea for this program came from two of my passions. First, I am a research facilitator and have been part of the campus champions since way back in the teragrid days. I have a passon for helping researchers to use computers to advance their science. I also happen to be an engineering future’s facilitator which is a program inside of the Tau Beta Pi engineering honor society. I have been teaching Engineering futures for over 15 years and the program has been around since the eighties.

�There is a vast amount of research in how to provide professional skills training. There is general consensus that face to face practice with real world examples is most effective. We adopted best practices and adjusted example exercises to focus in our area of Cyber infrastructure.

We didn’t really want to make yet another organization or group. Our goal is to develop the curiculum and work with other content developers, funding agencies and groups that lead our target audents and provide the materials to them to adapt and in ways that are best for their organization.

Content Developers
Funding Agencies
Target Audiences

4 of 16

CyberAmbassadors Project Timeline

Year 1:

Develop and pilot test curriculum (Communications, Leadership, Teamwork)
Develop partnerships for professional skills training

Year 2:

Continue to test, refine and evaluate impact of the curriculum
Develop and test professional skills facilitator training

Year 3: (now)

Continue to refine the curriculum and expand for new audiences
Develop infrastructure and funding models for program sustainability beyond the NSF grant

5 of 16

The CyberAmbassador Curriculum

INTRODUCTION

The CyberAmbassadors Program (55 Min)

LEADERSHIP

LEADING THE TEAM�Understanding Style and Personality (70 Min)
LEADING THE CHANGE�Equity and Inclusion (60 Min)
LEADING WITH PRINCIPLES�Ethics (50 Min)

COMMUNICATIONS

FIRST CONTACT�Communicating with a Purpose (70 Min)
LET’S TALK�Communicating about Problems (70 Min)
IT’S COMPLICATED�Communicating About Complexity (65 Min)

TEAMWORK

TEAMING UP�Effective Groups and Meetings (65 Min)
LEVELING UP�Problem Solving and Decision Making (70 Min)

6 of 16

The CyberAmbassador Curriculum

INTRODUCTION

The CyberAmbassadors Program (55 Min)

LEADERSHIP

LEADING THE TEAM�Understanding Style and Personality (70 Min)
LEADING THE CHANGE�Equity and Inclusion (60 Min)
LEADING WITH PRINCIPLES�Ethics (50 Min)

COMMUNICATIONS

FIRST CONTACT�Communicating with a Purpose (70 Min)
LET’S TALK�Communicating about Problems (70 Min)
IT’S COMPLICATED�Communicating About Complexity (65 Min)

TEAMWORK

TEAMING UP�Effective Groups and Meetings (65 Min)
LEVELING UP�Problem Solving and Decision Making (70 Min)

7 of 16

Example Case Study

The Context�Dana is a PI who stopped by HPC office hours. Researcher-facing CI-Professional Kim is there, and listens as Dana says:

“I am SO frustrated! I have a lot of data I need to process using the Sanborn-Stratta method, which is better than the more common FCFD approach. However, my transient error is much larger than the latent error. I think I can reduce the transient error by increasing the Tallholm threshold, but then the software will not run on my computer. So, I need you to make it run on the supercomputer!”

How should Kim respond?

8 of 16

Pilot Testing Modules

Teaching Method	Instructor �Mode	Participant �Mode	Number of Sessions	Total �Participants
In Person	In Person	In Person	4	58
Online Type 1	Virtual	Virtual	2	53
Online Type 2	Virtual	In Person	1	29
Hybrid	In Person+Virtual	In Person/Virtual	1	254
		TOTAL	8	393

9 of 16

Group	Modality	Respondents	Total Participants	Instructor	Module(s)
1	In Person	8	12	A	5
2	Online Type 1	5	25	B	5
3	Online Type 1	11	28	B	3
4	In Person	9	12	B	2, 3, 4
5.1	Hybrid	7	37	B	2, 3, 6
5.2	Hybrid	14	217	B	2, 4, 6
6	Online Type 2	3	29	B	2, 3
7	In Person	8	22	A&B	2, 3, 4, 7
8	In Person	4	12	A	5
	TOTAL	69	393

10 of 16

Pilot Study Respondents

69 respondents, out of 393 participants (18%)

42% of respondents in person
28% of respondents online
30% of respondents hybrid formats

Respondent Demographics

Ages ranged from 18 to 60
54% identified as male
64% had prior professional skills training

11 of 16

Pilot Study Methods

Reliability Analysis
Independent t-tests
One-way ANOVA

5-point Likert Scale and open response.

Satisfaction with:

Curriculum content
Training Implementation
Videoconference tools (online participants ONLY)

Data Collection: Quantitative & Qualitative

Survey

2. Observation

Host-learner interactions
Effectiveness of teaching
Participants’ reactions during training

Kirkpatrick Framework

Knowledge Improvement (before & after training)

(based on learning objective of each module)

Data Analysis: SPSS

Quantitative & qualitative data

The survey questions were developed using Kirkpatrick’s framework.

All instrument data were analyzed using the Statistical Package for the Social Science (SPSS). We created constructs based on the learning objectives of each module. In doing so, we conducted the reliability analysis to test internal consistency of each item using Cronbach's alpha. We found that each scale is greater than 0.80; thus, the scale is reliable and appropriate for statistical analysis. After constructs were created, the differences between posttest and pretest scores were calculated to capture learning gains. The differences in learning gains between in-person, online, and hybrid groups were analyzed using independent t-test and a one-way ANOVA. Demographic background was analyzed using descriptive statistics.

12 of 16

Pilot Study Results - Reaction (Satisfaction)

All respondents were satisfied with content, structure and pace of training

Hybrid group had slightly higher satisfaction scores overall
Online group had lowest average scores for each component

Training met respondents’ expectations
Respondents were willing to recommend training to colleagues

“I felt it was a very well-done training session, and really appreciated the warmth and networking with peers. The content was good, concise, and surprisingly novel!”

Respondents were asked to rate their satisfaction with the training content, structure, and pace, as well as whether the overall training program met their expectations and whether they were willing to recommend this training to colleagues. Table 3 provides mean, variance, and the results of the F-test for these measures of satisfaction. We found that the hybrid group consistently reported a slightly higher satisfaction score, with the lowest variation among the three groups. Meanwhile, the online group had the lowest average score for each component, except for their satisfaction with the pace of the training. Our analysis found no significant differences at the p<.05 level for any of these satisfaction measures, suggesting that all participants, regardless of the training modality, were satisfied with the content, structure, and pace of the training. Overall, the CyberAmbassadors training also met their expectations and respondents were willing to recommend it to their colleagues. These quantitative results are supported by the open-ended, qualitative responses, which generally expressed appreciation and positive feedback. For example, one respondent shared that “I felt it was a very well-done training session, and really appreciated the warmth and networking with peers. The content was good, concise, and surprisingly novel!”

13 of 16

Pilot Study Results - Learning Impacts

Respondents asked to compare skills and knowledge before/after training

Positive gains in learning across all three training modes
Learning gains were higher in hybrid and in-person than online training

No significant differences in learning gains across modules

We sought satisfaction feedback because we were piloting this curriculum and wanted to modify it for maximum effectiveness. We also sought evidence of impact, and assessed how much participants learned (or believed they learned) from the training. Respondents were asked to assess their skills and knowledge before and after the training, and Table 4 presents mean, variance, and the results of the F-test for scales measuring changes in learning for each of the training modalities. Overall, the average improvement between post- and pre-test is 0.89, which indicates a positive gain on respondents’ understanding and skills. The average improvement for the in-person and hybrid group is 0.94 and 0.96, respectively, while the average improvement for the online group was lower (0.73).

Independent t-tests and a one-way ANOVA were used to compare the learning improvements between the three groups. The results indicated no significant difference between the learning improvements in each module. Generally, there was also no significant difference at the p<.05 level for overall learning gains [F(2,61)=0.537, p = 0.567], although respondents reported an increase in knowledge and skills regardless of the training modality. Given the fact that the majority respondents were satisfied with the training experience, it is not surprising that they would also report gains in learning [49]. These results suggest that hybrid and online professional skills training can be as effective as in-person experience, when participants are satisfied with the training content and how the training was implemented. The qualitative responses suggest that many respondents found online training a good use of time and money and/or preferable for busy people who are unable to travel. Videoconferencing was generally seen as an appropriate and effective medium for professional skills training – although a significant caveat is that the CyberAmbassadors curriculum is highly interactive, with small- and large-group activities regardless of the training modality. Participants who are using videoconferencing software were divided into groups for activities conducted in smaller videoconference “rooms” before being brought back into the main training (virtual) space.

14 of 16

Next Steps

Original Project Ends October 31

Summative analysis across all trainings
Finalizing and distributing the curriculum, including the “Train the Trainers” module
Adapting the curriculum for Tau Beta Pi
Developing partnerships for sustainability �(TBP, CIMER, CaRRC, etc.)

No-Cost Extension Request

Pandemic led to cancelled in-person trainings, unspent funds
Hope to add another year of facilitator training and support activities

15 of 16

Additional Acknowledgements

The CyberAmbassadors Project Team:�Dirk Colbry, Katy Luchini Colbry, Julie Wilson Rojewski, Astri Briliyanti, Lillian Gosser, T.J. Nguyen, Nolan Feeny

This material is based upon work supported by the National Science Foundation under Grant No. 1730137. �

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

16 of 16

CyberAmbassador Fellows