1 of 42

Ian Robertson�Ava Firth

EBVM Project

Vets Now

2 of 42

Presentation Structure

Background to:

EBVM PD Program
Program Evaluation Project

Formative Evaluation Findings
Next steps
Conclusion

RCVS Knowledge Presentation

2

3 of 42

Purpose-built electronic patient record software

Launched mid-2012

~ 100,000 cases per year

4 of 42

2014 Target Grant

5 of 42

Quantitative Analysis of the Caseload Presenting as an Emergency in First Opinion Practice �

Ava Firth, Aoife O’Sullivan, Gavin Allison, Michelle Dawson, Dave Leicester, Helen Bowes, Laura Playforth, Amanda Boag

6 of 42

80,000 dogs, Presenting Complaint, by Body System/ Category

7 of 42

38,000 Cats, Presenting Complaint, by Body System/ Category

8 of 42

Building EBVM Skills and Culture

Your pet emergency service

8

9 of 42

Building EBVM Skills and Culture …. In Private Practice x 50+

Your pet emergency service

9

10 of 42

Teaching & Learning EBVM

Literature search →

veterinary students
physicians/ nurses

Veterinary practitioners - ?

Veterinary nurses - ???

RCVS Knowledge Presentation

10

Ava – how to build the skills, develop awareness and EVENTUALLY how to influence clinical practice through the use of evidence-based knowledge??

No one has done this before in private practice.

----------

Mills (2015) posits that establishing evidence-based-veterinary-medicine (EBVM) as a normative culture in first-opinion practice requires a thorough understanding of the learning processes that occur and the teaching processes required to support the learners.

Our literature search shows a paucity of research that considers how EBVM skills and principles might be developed in non-academic settings and first-opinion small animal practice. There are examples of studies that investigate veterinary surgeons’ participation in continuing professional development (see for example, Dale, Pierce, & May, 2013) but these do not specifically address the practice of EBVM. Existing publications focus on the teaching and acquisition of EBVM skills to and by veterinary students (see for example, Fajt, Brown, & Scott, 2009; Weller & May, 2013)

Whilst there are publications that provide detail on how to undertake EBVM (Cockcroft & Holmes, 2003) there are few, if any, published studies of the issues and obstacles encountered by postgraduate veterinary surgeons or veterinary nurses as they attempt to acquire EBVM skills and apply EBVM to their practice. The EBVM professional development program evaluation reported here goes some way to addressing this gap in the literature.

11 of 42

Vets Now EBVM Projects

12 of 42

EBVM Cohort

15 vets, 4 nurses enrolled at end of 2015

From 18 different clinics or sites

Job roles ranging from clinic vet to senior management

Only 5 of them not in a clinic or company leadership position

All working full-time, 12 of them on an after-hours rota

Voluntary, outside of working hours

Your pet emergency service

12

13 of 42

Project Development

Projects/ PICO questions generated by participants

Discussed with mentor

Initial project plan developed

Revised after initial literature search/ data retrieval/ pilot survey

Your pet emergency service

13

14 of 42

Projects (1)

Retrospective Studies

Grape/raisin ingestion in dogs (800)
Cats presenting with dyspnoea (1440)
Dogs with dystocia (1400)
Azotaemia in cats with cat bite abscess
Seasonal canine illness

Prospective Studies

Effective dose of apomorphine in dogs
Use of biomarkers in dogs with small bowel obstruction

Your pet emergency service

15 of 42

Projects (2)

Knowledge Summaries

Effectiveness of hypertonic saline in traumatic brain injury in dogs
3 others in preliminary stages

Survey

Synthetic cannabinoid intoxication in dogs

Your pet emergency service

16 of 42

Project Support

Regional meetings every 3 months – North, Midlands, South

One-to-one meetings – in person, phone, Skype

Webinars

Intranet resource pages

Email

File sharing

Reference management

Your pet emergency service

16

17 of 42

2015 Network Grant

Vets Now EBVM Projects

Investigation of the Teaching/ Learning Processes of Initiating an EBVM Culture in a Large, Private, First Opinion Veterinary Company�

18 of 42

Ian Robertson

Ava Firth

RCVS Knowledge Presentation

18

19 of 42

Purpose of the Evaluation Project

To document … the learning process, issues and solutions encountered by a cohort of 20 veterinary surgeons and veterinary nurses as they work through the discovery and application of EBVM in a multi-site, first-opinion small animal practice.

RCVS Knowledge Presentation

19

20 of 42

Evaluation Project Design

RCVS Knowledge funded
External evaluator
Case study approach

Surveys
One-to-one interviews
Observation of webinars and communication

Formative (process) and summative (outcomes)

RCVS Knowledge Presentation

20

21 of 42

Weaknesses

Small cohort
Single site
Evaluation process may influence participant experiences
Incomplete program (at this time)

Strengths

Focusses on participant experiences
More in-depth than simple surveying
Allows for iterative development of ideas that emerge from the data

21

RCVS Knowledge Presentation

There are a number of strengths and weaknesses to this evaluation approach. Some would criticise the approach on the basis that findings are based on a small cohort of participants at a single organisation. The benefits of this evaluation approach are that it allows collection of qualitative data from participants using a range of strategies over the duration of the professional development program. This prolonged engagement with program participants allows for iterative development of ideas that emerge from the data. It is an opportunity to illuminate the experiences of the participants in the EBVM professional development program. In doing so, this approach also accepts that the relationship developed between the evaluator and participants may influence the participants’ experiences.

22 of 42

Formative Evaluation Findings

RCVS Knowledge Presentation

22

23 of 42

Formative Evaluation Findings

Anticipated Outcomes
Emerging Themes

Program Design
Program Support
Individual Issues

Time

RCVS Knowledge Presentation

23

Likelihood of completion	N =
Confident	8
Possible	7
Non starter	1
Late starter	2
Left employment	1
Withdrawn	1
TOTAL	20

In terms of participant numbers, the EBVMPD program has engaged a total of 20 participants. Of those, we are optimistic that 8 will complete their desired project within the 12 month period. At present we believe that 7participants are at risk of not completing their desired project in the 12 month period. It is anticipated that a number of these will continue with their projects beyond that time.

1 employee is best described as non-starters. Non-starting classification is the result of maternity leave or change of employment. Two (2) participants are late-starters. That is, they entered the PD program some months after the program had commenced. At this time, only 1 participant has formally withdrawn once they have commenced in the PD program – this was due to enrolment in a Masters program.

Currently we know that at least six (6) participants will present their project findings at the 13^th Emergency and Critical Care UK Annual Congress which is to be conducted by VetsNow in Harrogate between 09th and 11th November 2016.

24 of 42

Formative Evaluation Findings

Anticipated Outcomes
Emerging Themes

Program Design
Program Support
Individual Issues

Time

RCVS Knowledge Presentation

24

Program Design

Program Support

Individual Issues

Time

25 of 42

Time

I feel that I've been a bit slow - life getting in the way - but I have the right motivation to complete! (#11)

Any improvements would have been purely time management related, and are personal to me and not associated with the project. (#6)

I still wish to proceed but a promotion has limited the hours that I can spend on it due to less free time. (#8)

RCVS Knowledge Presentation

25

For example, a universal issue which emerges is that of time. Participants consistently reflect on the time available to them to complete their projects, and, their ability to successfully manage that time to best effect. Many participants also commence their projects with unrealistic expectations about the quantum of time required to complete the project and the duration of time that it will take. In practice, this optimism can lead to a sense of slow progress and a belief that the participant’s progress is slow in comparison with other program participants. A number of participants reflected on the importance of deadlines in achieving outcomes. They indicated that they were very task focussed.

(comment from Ava - working on a project 2 hrs/ week for every week of the year adds up to ~ 100 hours. This is the equivalent of 2 ½ weeks full-time – how big of a project can you fit into that time?)

The following citations are examples of the way in which time has been interpreted.

I feel that I've been a bit slow - life getting in the way - but I have the right motivation to complete! (#11)

Any improvements would have been purely time management related, and are personal to me and not associated with the project. (#6)

I still wish to proceed but a promotion has limited the hours that I can spend on it due to less free time. (#8)

26 of 42

Individual Issues

Personal Circumstances
Work Circumstances
Participant

Motivation
Existing Skills
Expectations
Demeanor

RCVS Knowledge Presentation

26

Program Design

Program Support

Individual Issues

Time

27 of 42

Individual Issues

Personal Circumstances
Work Circumstances
Participant

Motivation
Existing Skills
Expectations
Demeanor

Existing or sudden onset
Expected or unexpected
Ongoing or temporary

RCVS Knowledge Presentation

27

28 of 42

Individual Issues

Personal Circumstances
Work Circumstances
Participant

Motivation
Existing Skills
Expectations
Demeanor

Workload has been extremely busy in recent times [but] I still expect to be able to … complete the project (#5)

My workload has increased massively and I haven't had any time to commit to the project … [recently] my workload has decreased again, so I am hopeful I will find time again to move things forwards (#6)

I still wish to proceed but a promotion has limited the hours that I can spend on it (#8)

RCVS Knowledge Presentation

28

29 of 42

Individual Issues

Personal Circumstances
Work Circumstances
Participant

Motivation
Existing Skills
Expectations
Demeanor

I am enjoying my project and definitely want to complete it! (#2)
Very exciting results from my project, now writing up, very motivated. (#14)
I am gaining a great deal of professional satisfaction from my project. (#3)

RCVS Knowledge Presentation

29

Interview data shows that participants were internally motivated by a desire to improve their professional practice, improve animal welfare, and, to contribute to the veterinary profession more generally. Participants were positive about undertaking their EBVM professional development project and when asked if they would withdraw, if given the opportunity, they responded in the negative. The personal satisfaction that participants are gaining from completing their projects is reflected in the following citations.

I am enjoying my project and definitely want to complete it! (#2)

Very exciting results from my project, now writing up, very motivated. (#14)

I am gaining a great deal of professional satisfaction from my project - this has filled a void in my career, for which I am very grateful. (#3)

30 of 42

Individual Issues

Personal Circumstances
Work Circumstances
Participant

Motivation
Existing Skills
Expectations
Demeanor

Research
Writing
Data retrieval and analysis
Survey design

RCVS Knowledge Presentation

30

31 of 42

Individual Issues

Personal Circumstances
Work Circumstances
Participant

Motivation
Existing Skills
Expectations
Demeanor

Progress is very slow having to focus on only one tiny aspect at a time (#7)

I would have preferred to have been able to progress the project more rapidly (#3)

RCVS Knowledge Presentation

31

32 of 42

Individual Issues

Personal Circumstances
Work Circumstances
Participant

Motivation
Existing Skills
Expectations
Demeanor

[Need to have more] confidence in self (#1)

I need to be more prepared to ask for help (#11)

RCVS Knowledge Presentation

32

33 of 42

Program Support

Facilitation Style
Ethics and Data Retrieval

Is ethical clearance required?
Is there a mechanism for seeking ethical clearance?

RCVS Knowledge Presentation

33

Program Design

Program Support

Individual Issues

Time

34 of 42

Program Support

Facilitation Style
Ethics and Data Retrieval

Is ethical clearance required?
Is there a mechanism for seeking ethical clearance?

I found the setup of the project perfectly helpful (#9)
The project has been progressed with the … guidance of the mentor (#3)
Support has been great (#10)
Skype sessions help motivate (#11)

RCVS Knowledge Presentation

34

The EBVM professional development program can be characterised as workplace learning that involves participants with multiple job roles, and variable working hours in multiple locations. Such programs are particularly difficult to facilitate (see for example Smith & Sadler-Smith, 2006). Where the facilitator is employed on a part-time basis this adds further complication. Despite these difficulties, data from surveys and interviews shows that there is a high level of appreciation for the assistance that the facilitator has provided. Notwithstanding the suggestions for improvement in processes that are found within this document, participants are largely satisfied with the level and style of facilitator support provided in the professional development program. For example

I found the setup of the project perfectly helpful. (#9)

The project has been progressed with the support and guidance of the mentor. (#3)

Support has been great. (#10)

Skype sessions help motivate me. (#11)

35 of 42

Program Support

Facilitation Style
Ethics and Data Retrieval

Is ethical clearance required?
Is there a mechanism for seeking ethical clearance?

Gathering valuable information has been a challenge (#10)

My project hit a little stumbling block in that when I started analysing my data I realised I needed more information … than I had requested in my initial search of the patient data (#2)

RCVS Knowledge Presentation

35

In the conduct of this evaluation it has become apparent that there are some projects that raise ethical concerns particularly in respect to issues of privacy and access to personal or commercially sensitive information. There are two issues here. Firstly, where projects require ethical clearance there is a need for some mechanism to achieve this. Secondly, those projects that require ethical clearance can experience significant delays in completion. In time limited projects this can have a severe impact on the ability to complete.

A number of participants have registered some frustration in terms of data retrieval. This stems from a lack of skill in: determining useful search criteria; data retrieval processes; or, analysing the retrieved data. For example:

Gathering valuable information has been a challenge (#10)

My project hit a little stumbling block in that when I started analysing my data I realised I needed more information … than I had requested in my initial search of the patient data (#2)

Given that these are all skills that are legitimately being developed in an EBVM professional development program these frustrations are not surprising. In the current program, data retrieval from the in-house system has been facilitated by technical staff and through the support of other participants who have offered to assist.

A further frustration experienced by participants searching the broader literature for publications on specific topics has been the frustration of finding that there is a lack of high quality research into their topic. An experienced researcher would interpret this finding as important in and of itself. The inexperienced researcher can be baffled by the experience.

36 of 42

Program Design

Individual/Group
Scope and Achievability
Content and Process
Clarity of Project Outcomes

RCVS Knowledge Presentation

36

Program Design

Program Support

Individual Issues

Time

37 of 42

Program Design

Individual/Group
Scope and Achievability
Content and Process
Clarity of Project Outcomes

Doesn’t feel like I am part of a group (#15)
I am now collaborating, rather than it being a solo project which will encourage and support me (#6)

RCVS Knowledge Presentation

37

Some participants have discussed the impact of individual and group based projects in influencing their experiences. Individual projects allow participants more autonomy is terms of the selection of their project and the manner in which it is completed. But individual projects can also lead to a sense of isolation and concern about how well an individual project is progressing in comparison with others. One participant commented as follows:

It doesn’t feel like I am part of a group. (#15)

Alternatively, group based projects provide an opportunity for collaboration and support. Members of the group can support each other emotionally and in terms of the actual work of completing the project. For example, team members are likely to have different skill sets. However, group based projects add the complication of requiring negotiation over the specific topic and work practices. One participant reflected on the benefits

I am now collaborating, rather than it being a solo project which will encourage and support me. (#6)

38 of 42

Program Design

Individual/Group
Scope and Achievability
Content and Process
Clarity of Project Outcomes

I have a very contained project and the only reason I haven't finished already is that I am too … lazy. (#9)

When I formulate a question it is never possible to answer, then the question I end up with is one which is possible to answer, but not the one I wanted to ask. (#7)

RCVS Knowledge Presentation

38

Scope and achievability of projects has emerged as a critical issue in influencing participant’s experiences in the professional development program.

In terms of initiating the project topic, almost all projects in this professional development program are being conducted on the initiative of the individual participant. In a few cases, participants have been unsure about what they would like to investigate. In these cases the project topic has arisen as a result of discussion between the facilitator and the participant. And, in all cases decisions about the project topic have always been with the positive agreement of the participant.

In many cases participants are completing projects that are achievable. For example

I have a very contained project and the only reason I haven't finished already is that I am too … lazy. (#9)

Data from surveys and interviews also shows examples where the scope of a project is beyond the available resources such as time and existing skill level. This has resulted in anxiety and delays. Where these cases have been identified, the facilitator has endeavoured to work with the participant to rework the scope of the project in a manner that renders it more achievable. In some cases, participants are reluctant to rework a project for which they feel a strong commitment. For example:

When I formulate a question it is never possible to answer, then the question I end up with is one which is possible to answer, but not the one I wanted to ask. (#7)

However, where advice is taken, this can result in a positive outcome.

I have now stepped back from my project as I realise I cannot do it … I am much less stressed as a result. (#12)

39 of 42

Program Design

RCVS Knowledge Presentation

39

Through conversations with individual participants it has become clear that they have a focus on the content of their project rather than the process of completing a PD project that intends to develop skills in EBVM. This is not unsurprising given that participants have selected a topic in which they are intrinsically interested and which will sustain them over the duration of the project. It is also a characteristic that is common in many post-graduate students who are completing post-graduate research into a topic of their own choosing. Whilst acknowledging the importance of content, it is important to establish a balance between content and the processes for achievement of the outcome. By focussing on the processes that are important in undertaking the project the participant is more likely to develop the skills that will endure and allow them to complete other projects in the future.

40 of 42

Program Design

Individual/Group
Scope and Achievability
Content and Process
Clarity of Project Outcomes

Essential in project setup and monitoring
Should consider

existing skills and resources
Personal and work circumstances
Scope and achievability of project
Balance of content and process
Formative deadlines
Communication schedule

Monitoring provides opportunity for revision

RCVS Knowledge Presentation

40

The specific issues identified to this point reinforce the importance of project proposals as a design feature that can be critical in optimising the likelihood of positive project experiences and outcomes. In the EBVMPD program discussed here, the facilitator encouraged participants to develop project proposals, some were formally documented. However, In programs such as the EBVMPD program where participation is voluntary and participants complete projects in their own time, one can encourage participants to develop such proposals, however, compulsion is somewhat more problematic.

Development of a project proposal provides an opportunity for a structured conversation regarding the participant’s existing skills and resources, and, personal and work circumstances that might be important in undertaking the PD program. Developing a project proposal also provides an opportunity to discuss the importance of balancing content and process in a PD program. These conversations can be important in shaping the scope and achievability of the proposed project, the strategies to be adopted and the resources required. The project proposal also provides an opportunity to establish formative deadlines and a schedule of communication. If interpreted as a document that can be revisited on a regular basis a project proposal can be used to assist monitoring of how well the project is progressing over time as well as providing an opportunity for revision as circumstances change.

41 of 42

Next Steps in Evaluation

One-to-one discussions with participants
Observation of participant presentations
Audit of project outcomes (deliverables)
Focus groups to validate and rating the importance of issues
Survey of EBVM skills at completion
Final report to RCVS Knowledge

RCVS Knowledge Presentation

41

42 of 42

Conclusions

PD Program Outcomes

Successfully engaging ~ 20 participants
Diverse topics and sources
Measurable outcomes being achieved

Evaluation Outcomes

Successfully revealing a range of factors that influence participant experiences
Three primary themes identified

42

RCVS Knowledge Presentation

In conclusion, analysis of formative outcomes shows that the EBVM professional development project is being successful in engaging 20 veterinary surgeons and veterinary nurses in a diverse range of topics. Some projects are based on in-house or practice-based data, others are collecting information from published sources. At least six participants are planning to present EBVM project outcomes at a professional congress in veterinary and emergency critical care, other participants are on track to complete knowledge summaries.

Workplace based professional development programs that are conducted with geographically distributed participants who experience irregular work hours and are completing projects in their own time are notoriously difficult to facilitate. This evaluation of an EBVM professional development program reinforces that reputation. The findings presented also demonstrate that the evaluation project is being successful in revealing a range of factors that participants perceive as important in influencing their experiences within the EBVM program. Broadly these factors can be classified according to three primary themes: program design; program support; and, individual circumstances.