Catalyst Sector Challenge Programme:

Evaluation Report

May 2021

Introduction: The aim of this report is to outline key outcomes and learning from the Sector Challenge Programme. A mixed method approach guided the data collection process, where both the charities and digital agencies contributed with their perceptions and reflections about their experience in the programme.

Data collection methods: The following data collection methods have been deployed to evaluate the Sector Challenge programme:

​

​

​

​

​

​

​

​

​

Data Limitations: there were 21 online survey responses from charities, from the 31 taking part, although this still represented a large sample. In order to reduce the number of data collection points with charities, only one survey was used, which did lead to a longer survey than intended, and may have impacted on data quality as respondents reached the end of the survey.

​

​

​

1

Introduction

1

Digital partner motivations for taking part in the programme & importance to the sector

• Empathising with the various topics on the programme was the most common theme drawn out of the mid-point interviews with the topic mentioned four times during the seven interviews with digital partners. Five respondents in the interviews identified that their motivation to take part was related to a personal connection to the cause or an opportunity for real impact during the programme.

​

• From the mid-point interviews conducted, ‘reuse’ was the theme which was highlighted the most by digital partners as both a motivation to take part in the programme, but also as a reason why the programme is important to the wider charity sector. By the end line interviews this theme was extended with four out of seven digital partners stating how their motivation comes from a desire to develop a solution that is ‘useful’ to the sector and can add value to multiple organisations.

​

• Connections and partnerships were mentioned in three out of seven of the mid-point interviews and in seven out of seven of the end line interviews with digital partners. Initially connections and partnerships were mentioned as a motivation to participate and by the end of Sector Challenge the collaboration with other digital agencies, as well as the charities and the Catalyst/CAST network, were described by all the digital agencies as a key benefit to participation.

​

• In addition to the connections, in the mid-point interviews one participant outlined that they were interested in learning from others through this process. During the end line interviews, four out of seven digital partners also revealed how they built on existing skills or learnt new ones during Sector Challenge. Three digital partners also revealed how they learnt from the process via exposure to Catalyst’s tools (such as Calendly), through Catalyst’s approach in allowing digital agencies and charities to connect ‘organically’, as well as through the ‘matching system’.

​

• Two digital partners in the mid-point interviews viewed taking part in the programme as a good opportunity to use their current knowledge and expertise. By the end of the programme four out of seven digital partners stated how they had a better understanding of where and how they could add value to the sector. “It’s shown a value to our company as well for this process and the processes that we do, being solution-diagnostic and these kinds of ways are really useful and they can work with the non-profit sector very well”.

�

​

​

2

Why charities joined the programme?

• Chart 1 reveals how charities joined the programme at different stages of their digital journey, where most (ten out of twenty one) identified as ‘starting out’.

​

• Charities were asked why they joined Sector Challenge and out of twenty one respondents the most common response cited by nine charities was that they wanted to use digital to better help and reach their service users and/or they wanted to improve service delivery. Charities talked about helping service users and improving services in the form of more effective support, tools and communication. One survey respondent gave the following example: “we wanted to explore ways to best help those that we support who are experiencing digital exclusion and are struggling to support their children with home learning”.

​

• Five out of twenty one charities also stated that they joined the programme in order to meet other charities with similar challenges and to share learning with others in the same sector, whilst making new contacts and forming peer networks.

​

​

​

• Four charities responded to this question by stating how they joined Sector Challenge in order to improve internal digital systems or digital strategy, however two charities also mentioned here that this expectation was different to the reality of the programme.

​

• Two charities revealed how they joined Sector Challenge to see if they could better support their staff during the Covid-19 pandemic.

​

• One charity mentioned how they wanted to help share sector wide change.

​

• One charity stated how they joined the programme in order to work with a digital partner, which they said was something their organisation had not done before.

​

3

Charities’ perception of digital progress

• Charities were asked about how their digital status had changed as a result of the programme. Chart 2 reveals how charities were somewhat divided in how they felt Sector Challenge had influenced their digital journey, with eight charities stating their digital status had changed and six stating that it had not.

​

• Charities were asked to describe how the programme had influenced their digital journey, and the most common responses (answered by four charities) were that charities had more of an awareness about how to use digital tools and platforms and the potential that it has to help the sector, users and staff. Four charities also stated how the programme had also built up their digital capacity and confidence.

​

• Two charities stated how they had made improvements to services through the use of digital tools and platforms.

​

• Chart 3 demonstrates how fifteen charities out of twenty one felt that Sector Challenge had an important impact in helping them to identify other areas within their organisations that could be supported or improved as a result of digital design approaches.

​

• Nineteen out of twenty one charities gave details about how various digital tools, skills or knowledge could be applied within respective charities after the programme. The tools mentioned included Slack, Airtable, referral/feedback forms and apps, and five charities stated how digital design approaches could be used and applied broadly across organisations.

4

Developing digital processes and prototypes

Sector Challenge charities were broken down into seven cohorts (there were originally nine cohorts, but it wasn’t possible to recruit enough charities for two of the cohorts). The end products of each of the cohorts are briefly described below, taken from interviews and final playbacks with the digital agencies.

4

Developing digital processes and prototypes

• Overall, from the 21 charity surveyed, the majority were satisfied to some degree (81%) with the current status of the process or digital prototype they worked on in their cluster.

• In written responses in the survey charities were generally happy with the final output of the process or digital prototype, however, there was some frustration in several cases that the project did not finish with a working prototype, or it was unclear what would happen next with the project.

4

The process of developing a digital prototype (digital partner perspective)

• User research and/or testing were described by all of the digital partners as a key part of the process that shaped the development of the digital prototypes.

​

• Six out of seven charities explained how they utilised user testing and how the process helped to narrow down solutions that may work, to evolve existing designs, to see what was “worth keeping”, and to move forward with a prototype that worked, got good feedback and addressed the most ‘pain points’. In one case the digital partner describes how they went beyond user testing to empower a service user to be involved in the project “we thought we would pay people to do testing for consultation but when we realised the incredible skill set our lived experienced designers had we thought to use the money on that. People have said it was life changing for them to have the space to do that, and for the group who had been quite reluctant to bring them in”.

​

• In the seven interviews with digital partners user research was referred to four times as a baseline from which charities worked to understand problems staff and volunteers had in more depth. One partner stated how user research helped charities to focus on what needed to be addressed, as opposed to what charities wanted to address: “in some cases despite what charities said; they were pulling us in one direction and the actual user research was pulling us in a slightly different direction, so we trusted the user research”.

​

• Four out of seven partners stated that exiting software was used, in particular Google Forms and Docs for data collection and Miro which was described as an “obvious tool for collaborative online work” that was mentioned by three out of seven digital partners.

​

• Four out of seven partners talked about ‘trust’ within the interviews. Digital partners referred to how a high level of ‘trust’ is necessary within a design process, with a key challenge being that the end-point is ambiguous, however when the group do ‘trust’ in the process it creates a strong group bond and what one partner described as those "momentum-building moments”.

​

• Three out of seven digital partners referred to the ‘double-diamond’ approach as a useful anchor and a simple process for charities to follow when trying to define/narrow down to one problem.

​

5

Satisfaction with the process and/or prototype developed

​

Charities’ perspective:

• As revealed on Chart 4 seventeen out of the twenty one charities that responded were either ‘very satisfied’, ‘satisfied’ or ‘somewhat satisfied’ with the process or prototype that was developed during the programme.

​

• Six charities stated how their expectations were not being properly managed from the start which led to confusion about the goal of Sector Challenge and what they could and couldn’t achieve on the programme. All six of these charities revealed how their expectations differed to the reality of the programme and to the end result.

​

• Five charities made reference to how the prototype and/or process was unfinished and that they felt dissatisfied that there was no end product that could be utilised.

​

​

• Four charities stated how they had hoped for a more tangible outcome given the time and money that had gone into the project, three charities mentioned how there was a need for more funding or for an unknown party to take it forward which they felt was disappointing.

​

• Three charities expressed positive feelings about the prototype developed and stated that they were either using it or planned to use it in future.

​

• Two charities shared the view that the prototype developed did not encompass what each charity needed to support users or beneficiaries.

​

​

6

Satisfaction with the process and/or prototype developed

​

Digital partner perspective:

​

• Six out of eight of the digital partners explained that charities would have liked for the prototypes to be further developed at the end of the programme, there was a shared feeling that there was more room for progression and automation.

​

• During the digital partner workshop three out of eight digital partners stated that the charities they worked with were ‘happy’ with the prototype developed, one digital partner stated that their cluster was ‘really happy’ with the prototype and that it was something they could now easily implement.

​

• Three out of eight of the partners also pointed to how charities were more satisfied with the process of learning new digital skills from the process than the prototype itself. Two partners stated how they felt for the cost of the programme, it could have been better focused at learning only, if that was the purpose: “it was nice and helpful but not transformative. But if it was definitely about learning we could have taught them more for 50,000”.

​

​

​

• Three digital partners agreed at the workshop that the charities’ expectations could have been managed better from the offset. One stated how their cluster expected more from the process in terms of the end product, and another explained how the cluster “didn’t know what they were going to get”. For one digital partner they also spoke about how their own expectations weren’t properly managed as they themselves expected that the prototype would have been at a later stage by the time the programme came to a close, they stated “I thought they might have been a little more shiny and further along”.

​

• Two out of the eight digital partners at the workshop stated how one or more charities within their clusters were dissatisfied with the final output, in one instance it was because the prototype could not be integrated within one charities current system and for another cluster it was due to a lack of expectation management from the start.

​

​

7

Engagement beyond Sector Challenge

Charities’ perspective:

​

• When charities were asked about next steps for their cluster nine charities out of twenty one either ‘disagreed’ or ‘totally disagreed’ with the statement that there were clear next steps to continue collaborating on the shared challenge after the programme.

​

• Two charities stated how at the end of the programme the clusters did not discuss next steps or continued work together, one charity stated how they felt their was a ‘loss of collectiveness’.

​

• Two charities revealed how there was a need for someone to take the product forward and apply for additional funding, both charities stated how they were not aware that this would be expected of them.

8

Engagement beyond Sector Challenge

Digital partner perspective:

• To support the progress of the digital processes or prototypes developed six out of seven digital partners mentioned how funding had some influence over whether or not the next steps could take place.
• Three digital partners mentioned how they or a charity within their cluster were considering applying for additional funds for their digital prototypes. Two partners described funding as a challenge; one stated the portal app developed would require a great deal of funding and the other stated how they would be parting ways with the project as funds would only cover the charities’ involvement and not the digital partners. Another interviewee stated how the platform they developed would require a small amount of funds to actively improve it, otherwise it would be free as it is.
• Digital partners expressed how in two cases individual charities within clusters appeared to be taking ownership of the digital prototypes, however the interviewees opinions on this differed significantly, with one stating how it was unfortunate than not all six charities within the cluster would end up using it. The other partner stated how they only approach one of the charity members on the matter of continued funding stating “we’ll go to one of the charity members and talk with them, the rest of them I’ll tell them what is going on but I won’t get them involved, particularly, as they are not a net benefit to the project, they are questioning, dragging things down, slowing the process, not because we know the answer but because we know the process”.
• Two digital partners expressed how they felt that for the digital prototype to hold value to the sector then it should be owned by a singular cluster of charities who can collaborate on its development going forward.
• Two digital partners mentioned how charities were taking the next steps in isolation from the digital partners, one interviewee stated how they advised the charity to trial it and scope out whether they are able to manage it in terms of effort and workload in isolation from the digital partner.
• One digital partner stated how further testing would be required to test the applicability of the concept within a range of communities, they mentioned testing with: “black-African communities, with different sexualities, and with the disabled community”, depending on its applicability the digital partner explained how adjustments would then need to be made, including to language.

​

​

​

​

9

Sustainability of the solution (as identified by charities)

• Charities were asked about how sustainable they felt the prototype or process would be once launched/completed. The most common response by eight of the charities was that securing more funding would be needed to make it sustainable in terms of maintenance and keeping it running.

​

• Four charities also stated how ongoing updates to the processes or prototypes and how it was necessary for one charity to take ownership, and how this would work, were also seen as challenges in the product/prototype's sustainability.

​

• One charity felt that the biggest barrier to long term sustainability of the product was that it was not original and there were competing products already on the market.

​

• Another charity stated how the adoption of an open source set up was an important consideration that their cluster had made in ensuring that the data created was sustainable.

​

10

Interactions between clusters of charities

Charities’ perspective:

​

• All twenty-one charities felt that a key benefit of working within the cluster was that they were able to share learning within the group, alongside digital partners (Chart 6).

​

• The most common response from charities (mentioned eight times) regarding their interactions and relationships was that they enjoyed the collaborative, shared approach within clusters where they could better understand the challenges that other charities were facing and work on finding solutions together, as well as sharing best practice.

​

• Four out of the twenty-one respondents to the survey explained how some the charities within the clusters were of very different sizes and had different focuses, so there was some misalignment within the group and this restricted what charities could work on together and in order to find something they could two charities stated how they had to compromise. One charity also outlined how it helped to have distinct focuses/sizes of charity within the groups, as it meant there was less of a competitive feel between them.

​

11

Interactions between clusters of charities

Digital partner & convener perspective:

• During the mid-point interviews participants stated that working with other similar charities was also said to increase the confidence and stance of those within the cluster: “I think it helps to make them feel more confident, more assured and more relevant to be involved with a bunch of other organisations facing the same challenges.” During the interviews with digital partners, the main theme which arose was the usefulness and enjoyment of the charities being able to work and interact with similar initiatives and the sense of comradery that came with this. Two conveners also explained during the workshop that they felt that there was less relationship breakdown and mentioned the following reasons. The first was that digital partners were not having to work with existing dynamics, but rather with new ones and the second was the strength of the application process, where applicants selected based on how they would contribute to a successful team, rather than the strength of their application alone.
• For those interviewed at the end of Sector Challenge they described in details how participants’ perspectives shifted to understanding the value of collaboration: “I think there was this feeling that we all have the same problems and working together we can conquer them, working independently we can’t”. Two participants at the conveners workshop stated how the collaboration between charities worked well and two conveners also mentioned how the relationships started off challenging, where charities were hesitant to interact with one another but were described as ‘strong’ by the end.
• Five out of seven digital partners reported that throughout the programme the charities built relationships, and in some cases communities. One participant said that the benefits appeared to be more profound in terms of sharing and working in the open, and at the end point their cluster had realised the possibility and had the know-how to build a national or even international community.
• One respondent stated in the mid-point that the charities within the cluster were open with sharing data. By the end of the programme two digital partners mentioned how charities “were open and willing to share with one another”.
• One digital partner mentioned at the mid point and end point how they found that relationships were built up through a weekly workshop session. At the end point two digital partners spoke in details about how the final workshop offered an opportunity for charities to open up, bond and reflect on the process. However, two digital partners also reflected on how in general the process did not offer enough time for charities to build relationships outside of the workshop structure.
• One interviewee stated at the mid-point how the interactions between charities in the cluster allowed the digital partners and charities to develop their knowledge and understanding of the sector as a whole. Another digital partner supported this perspective at the end of the process and stated how Catalyst allowed for charities to connect and therefore learn from one another easily.

​

​

Identified by digital partners & conveners:

• Four digital partners at the mid-point and three at the end-point reported that the charities benefited from sharing methods and being open with each other as well as with the sector. Within the workshops one digital partner and one convener highlighted how charities realised they gained more from sharing, than holding back, in terms of building connections and learning how their challenges differ or relate. One convener gave the example of how their cluster is now sharing an output with another team who are working on a shared service layer.

Identified by charities:

• A key area that charities felt benefited themselves and their cluster is revealed by Chart 9 & 10. Charities felt that they learnt more about the process of establishing shared goals and gained a better understanding of the value of working collaboratively with others to address these shared goals or outcomes.
• Eight charities in the survey stated how the collaborative, shared approach within clusters where they could better understand the challenges that other charities were facing and work on finding solutions together, as well as sharing best practice were a key benefit of taking part. One charity wrote in the survey: “It was really valuable to talk things through with people working in the sector but delivering different services for similar beneficiaries. I feel we learnt a lot from each others experience. This way of working also allowed a more 'open' dialogue than I have experienced before”.
• Two charities in the surveys also stated how open learning was new to them and how the programme highlighted the importance of this to track progress and collaborate effectively.

​

12

Interactions between clusters of charities

• By the end point five digital partners outlined how they had adapted to the challenges that emerged and one interviewee explained how they saw their role as striking a difficult balance in order to support the cluster effectively: “there’s an element where as the digital partner you have to educate and reassure at the same time, you have to bring challenge and say that it’s all going to be ok whilst saying they are not quite right”.

​

• For three digital partners during the end line they felt a key challenge they faced during the interactions was aligning all of the different perspectives to form a collective view, this also echoed by two conveners during the workshop. They felt that this was predominantly due to their different needs and sizes as individual charities. “The difficulty in getting there was the differences between the three groups; they weren’t the same shape, the same scale, the same internal challenges, so it felt very tenuous. I think the added uncertainty was not having a clear alignment on the same goal”. This was elaborated on during the workshop with digital partners where two people explained how they felt there was a ‘mismatch’ in terms of charities or as individuals, and also by three conveners at the workshop, where one participant described the charities as ‘divergent’, in terms of what they wanted from the process.

​

• Three digital partners and one conveners at the workshop sessions agreed that charities came into the process with a clear idea of what they were going to do and already had a clear solution in mind for their own charity before they had interacted with others in the cluster. One partner stated how a charity had already budgeted the solution and another explained how the charities were telling others in the cluster what they were going to do.

​

• From the start to the end of Sector Challenge four digital partners stated that they faced a challenge with getting the charities in their cluster to work as equal partners in the design process. One digital partner during the end line interviews and two convener in the workshop stated how there was confusion about the roles of digital partners and the cohort of supporting people in how much the charities decision making process they were taking control of and also the alignment between them all.

​

• Two digital partners during the workshops agreed that there was some tension with charities not fully understanding and appreciating the digital partner’s roles, in one scenario a partner explained a challenging scenario where they day rates of charities and digital partners were compared openly in the group: “day rates of teams came up between digital teams and charities which was really awkward and those who don’t work in it don’t realise how much has gone into it. Maybe a need to explain the value a little bit more to help them understand”.

​

• Two digital partners also added (one at the mid-point and one at the end point) that they faced a challenge in getting the charities to take ownership over the programme and the solution due to it not being initially valuable to all of them.

​

Challenges that emerged working with the charities/between the charities

13

Sharing (internally or externally)

Charities’ perspective:

​

• As demonstrated by Chart 7 when charities were asked about sharing internally or externally, of the twenty-one charities that responded to this question all stated they either shared inside or outside of their organisation.

​

• The most common response amongst charities was that they shared the most internally (with colleagues and service users) throughout the process and at the end, which was mentioned by nine out of twenty-one charities. This was done to ascertain a wider pool of views, to provide support through new digital/design processes or to gauge next steps.

​

• Six charities stated how either the prototype, tools, new ideas or learning had been shared externally with the wider sector in order to help others in the sector, as well as to test useability.

• Four charities mentioned that through the programme they had acquired new ways of working and had shared this with the wider sector, these included the use of Slack to improve internal communication and the application of design and development methodologies in new services and products.

​

• Four charities stated that they shared updates on the process, progress made and learning, as well as general updates from the project.

​

14

Sharing (internally or externally)

Digital partner perspective:

​

• During the first set of interviews one digital partner highlighted that they were going to continue to share parts of their programme publicly, as they had a positive response to this approach “So at half way through, we are at the point where we have started to share publicly and, beautifully, it seems that coming out of our user research is the observation that there is a desire to share more, which is lovely.” At the end point interviews, six out of seven digital partners stated how parts of the programme continued to be shared publicly on You Tube, social media, and through webinars. One interviewee stated how there was a possibility for the creation of a recipe to be shared on the CAST website and another stated how they needed some time to fix some bugs before sharing it widely.

​

• Five out of seven digital partners stated how they documented learning in the form of how to guides, booklets and online files. Four out of seven explained how the purpose of the documentation was for sharing with the wider sector. One digital partner explained how they even documented learning and transformed this into an individualised book that was sent to the cluster for the final day as a memoir of learning that they could share with others.

​

• Four out of seven of the respondents across all the interviews said that they had shared some aspects of the programme through a blog post or through blogs on their websites. One respondent stated how these blogs were then re-shared through social media.

​

• At the mid-point three interviewees reported that they were sharing parts of the programme internally with others working on similar projects. Another individual mentioned, “we’ve shared our ‘Diary of a collaborator’ with the Dot Project and it made its way to the Owls.”

​

• Two respondents also mentioned at the mid-point that they had written week notes, which had been shared publicly. This was reflected during the end point interviews with three digital partners referring to the writing and sharing of the written week notes, however one partner did highlight how there was a lot of fear and uncertainty attached to this process as well.

​

• In two instances digital partners felt that the clusters had limited capacity and time to share learning. For one interviewee, they stated how they would have liked to attend the shared learning events but time was a barrier and for another they revealed how they had to ‘push’ charities to share due to time limitations.

15

Partnerships & connections

Charities’ perspective:

​

• Chart 8 reveals how charities’ understanding of how to build connections with digital partners had improved, with fifteen out of twenty-one charities stating that they either ‘totally agree’, ‘agree’ or ‘somewhat agree’ that this was a positive outcome of the programme.

​

• As an extension to the question on Chart 8 charities were asked to explain how their understanding had increased. The most common response answered by three charities was that they hadn’t worked with a digital partner in the past, so they had nothing to compare it to, however that it would be something they would recommend to others.

​

• When charities were asked about the digital partners they worked with, eighteen out of twenty- one respondents described their partnership in a positive way, with key words to describe them including ‘professional, approachable, knowledgeable, respectful and open’.

​

16

Partnerships & connections

Digital partner perspective:

​

• Connections and partnerships were mentioned in three out of seven of the mid-point interviews and in seven out of seven of the end line interviews with digital partners. By the end of Sector Challenge the collaboration with other digital agencies, as well as with the charities and the Catalyst/CAST network were described by all seven digital agencies as a key benefit for participation.

​

• During the workshop with digital agencies the partners described their partnership with charities using the following descriptors: collaborative, positive, constructive, engaging, dynamic and warm. Digital partners also rated their partnership with the clusters at a an average of 7.8 out of 10 (on a rating scale of 1 bad, 10 excellent).  

​

• Four out of seven digital partners alluded to how they had found synergy with others working in a similar field, one person described this as taking place within the ‘tech for good’ space, another stated how they had been working in the field for a number of years and were surprised to have met new faces through the Catalyst network.

​

• Three of the seven digital partners at the mid point said that the charities benefited from building relationships with each other and at the end line digital partners reflected how charities had understood the value of co-production.

​

​

17

Challenges or barriers that have emerged in running the programme

(experienced by the digital partners and conveners)

• The main challenge identified by digital partners and conveners in running the programme was that expectations were not clear, and not clearly communicated. This was highlighted fourteen times across interviews with digital partners and was also raised by two conveners in the workshops. They all agreed that the expectations differed between the charities and that some charities joined the process with the expectation that they would be addressing their own challenges and did not realise that it was a collaborative building process. In both the mid point and end line interviews ten partners stated how the end goal could have been communicated more effectively to charities and (in some cases) to digital partners, one interviewee explained: “it would have been good if I’d known at the beginning of the process to say “it’s quite likely that we won’t end up with a usable digital tool” but I didn’t say that because I didn’t know to say that”.

​

• Another challenge digital partners experienced was resistance from the charities. This was mentioned seven times in the midpoint interviews, six times in end line interviews and was discussed twice during the digital partner workshop. Resistance was outlined in two strands; resistance to the approach of the programme, including to new tools/solutions that were introduced by digital partners and in the processes which took place – in particular to the level of ambiguity behind the data collection and design process. One partner suggested a way to address this may be to outline the roadmap more clearly and where the pivots may be.

​

• Limited digital confidence was mentioned seven times during all the interviews with digital partners and five times in the workshop with digital partners, as a key barrier in running the programme. This limitation in digital confidence was in the use of new online tools and processes. In relation to using a new digital tool as part of the project, one interviewee highlighted that “They just hated it, and it made them feel inadequate and they just wanted to stop using it.” One interviewee at the end point stated how design thinking and jargon made people feel stupid and led to group members quickly switching off.

​

18

Challenges or barriers that have emerged in running the programme

(experienced by the digital partners and conveners)

• Across both sets of interviews user research was described by a total of five partners as a key challenge with limitations on users to speak to, and issues accessing them through digital means during the pandemic.

​

• The time scale of the programme was also described as a challenge by two digital partners at the end line interviews, by three conveners in the workshop and was reflected on twice by digital partners in their workshop. One digital partner in the end line interviews stated this in terms of striking a balance between progress and the time/space required for reflection. Another partner stated how the only downside for them was that they couldn’t spend more time with the group to develop more of a group bond. Three conveners agreed and described the timing of the programme as ’rushed’, ‘crazy’ and that the short time period required to complete the application process and to form the clusters created a lot of challenge.

​

• Two significant challenges that emerged from the mid point interviews with digital partners, echoed by one convener at the workshop, was the need to properly introduce charities to digital partners.

​

• Two conveners at the workshop spoke about how competing priorities were a challenge in that some support staff were trying to focus more on the problem and others more on relationship building with the clusters, another convener explains how: “competencies, partners, infrastructures, support needs” were competing with one another which was seen as a challenge. In a similar sense two digital partners also agreed that they felt they were being pulled in three different directions (by Catalyst, the sector and the cluster) and highlighted how it wasn’t clear who the main client should be.

​

• Two conveners and two digital partners mentioned the eligibility of the application process as a challenge, in terms of reaching the right member of staff, the timing of the process and the way in which the tendering made charities arrive to the process with ‘solidified’ expectations as to what they would get out of the programme.

​

• One digital partner said that the charities came in “fearful” of the challenge and of what design would entail, and this created an initial barrier within the first few weeks of the programme, towards the end, one digital partner mentioned how a participant seemed to feel worst about their technical ability than when starting the programme.

​

​

​

​

​

19

Challenges or barriers faced whilst taking part in the programme

(experienced by the charities)

• The most common challenge shared by eleven out of the twenty one charities that responded was related to the ‘short and fast’ timeframe of the project and the pressure that this put on charities, which was said to have a negative impact on the final output. One survey respondent explained: “It felt very rushed throughout the programme, with insufficient time for each section. More time to try out ideas, step back and reflect and the re-group would have been beneficial”.

​

• Six charities revealed through the survey how another key challenge was the lack of clarity in the programme brief and on the purpose of Sector Challenge. This led to misunderstandings of what the charities expected to do during the programme and what they thought they would get out of it. Misunderstandings were around what ‘co-production’ meant, who would get funds and what for, and more generally related to the layout/structure of the programme. One survey respondent states: “It was hard to buy in to the process initially but not sure how that can be overcome without going through the experience. In hindsight it all made sense, as it was happening it felt quite painful”.

​

• Three respondents wrote about how they struggled to buy into the process at the start of the programme which was down to the slow start and a lack of understanding as to what clusters were working towards. Two charities stated how in the end they felt this was a valuable learning experience despite it being described as ‘painful’ at points.

​

• Two charities made reference to how the Covid-19 pandemic was a barrier for them trying to work remotely during lockdown and the fatigue that came with virtual meetings at the time of the programme.

​

​

20

What could be improved? (digital partner & convener perspectives)

​

• The most common improvement mentioned by conveners related to a lack alignment and communication breakdown. The group stated that these issues could have been resolved by organising an internal kick off to clarify different roles and perspectives, as well as regular check ins throughout the process to encourage more ‘cross-pollination’.

​

• Four digital partners concurred in the workshop session how they felt the issues of ambiguity and fear within the clusters of charities could have been addressed better in the following ways: 1) Charities going through another more introductory programme first, such as a Design Hop to help with grounding and understanding of the design process 2) Creating more time in earlier sessions to state how charities will feel at each stage of the process.

​

• Three conveners stated how three financial wellbeing challenges and the two early years challenges did not go ahead due to the number of applicants for certain challenges being too low. Conveners felt that it was difficult to pinpoint why this was but that ‘infrastructure issues’, how translatable the issue was and how to find the right fit in a short period of time were all identified as disconnects and areas to be improved.

​

• Three digital agencies in the workshops agreed that more clarity and transparency on the purpose of Sector Challenge and on who the programme was being delivered for would be instrumental in improving it. Two partners stated how it was not clear whether the purpose of the programme was for charities to learn or whether it was to develop an effective product for the wider sector. The partners questioned whether it was OK to end up with a bad product if the charities have learnt something. And if so, was the client really the sector?

​

• Two conveners in the workshops stated how the level of involvement required from them in terms of interactions with digital partners and the clusters could have been clearer. Two conveners discussed how they adopted a ‘hands off approach’ and that some of the solutions they proposed at the mid-point were not utilised by digital partners but could have been useful to course correct.

​

• Two conveners felt that the timeframe of the programme (four weeks for applications and all the team bonding) intensified the pressure on the team and to overcome this one convener felt that the team building could have been conducted up up front to save on time.

​

​

​

21

What could be improved? (charities’ perspective)

• Three digital partners in the workshop spoke about how Catalyst could play a more instrumental role in explaining to charities about the foundations of what a digital partner does and in encouraging charities to shift their mindsets about digital so that they appreciate/trust the work digital partners do that can then have a longer term sustainable impact of a solution.

​

• Two digital partners spoke about the tendering/application process as an area that could be improved, which differs from the way in which the conveners perceived this process, where they identify it in their workshop as an area that worked well (previous slide). Two partners felt that the expectations that charities arrived with at the programme to address individual challenges could have been addressed by changing the questions asked and also by including digital teams in the deciding the suitability of charities.

​

• During the surveys the most common area of improvement touched upon by the charities (mentioned fifteen times) was that there was a need for more clarity around the project brief and purpose of the programme. These charities widely shared the view that their expectations could have been managed better, in stating that products may not be finalised by the end of the programme and in the potential need for one group to own the product and/or apply for additional funds for it.

​

• Another area of improvement revealed by five charities during the surveys was that there was a need for a more ‘directive start’ to the programme. In order to overcome teething issues, charities felt that there was a need for more conversation at the beginning to gain a better understanding of other charities within the clusters and to define a shared understanding of the workshop format and co-production.

​

• Three charities also stated during the surveys that they felt the digital partners could have focused more on the charities’ users with a focus on ethics and the inclusion of people with lived experience in the process, as opposed to focusing so much on the tech aspect of the programme. One survey respondent explains: “I don't think we did enough data analysis & user journey mapping. Didn't respect people with lived experience - ignored a colleague with lived experience who kept offering to take part in user research”.

22

Outcomes for charities: increase in use of digital skills, tools & capacity

Identified by digital partners & conveners:

• The charities benefited from learning to use new digital tools and to apply these skills. During the end line interviews five out of seven digital partners interviewed made reference to a new tool or skill developed and three conveners from the workshop also made reference to new skills that the clusters had picked up, varying from managing a project, journey mapping, utilising Miro, ideation and prototyping, or utilising double-diamond methodology.
• Two digital partners discussed in the workshop how the use of the double-diamond approach was something that charities found very useful for shaping future projects.

�

​

​

​

Identified by charities:

​

• When charities were asked in the survey to outline whether they had benefited from the Sector Challenge programme, the most common response outlined by ten charities was that they had built up new digital skills and/or capacity. Five of these charities further outlined how these new skills, tools and capacity had enabled them to apply new ways of working. One charity gives an example of how charities are doing this within their organisations: “we are applying some of the user research and design approaches to our new website development, involving our participation group and partners in testing the staging version of the website”.

​

• Chart 11 reveals in more detail the types of skills that charities feel they have acquired as a result of their participation in Sector Challenge. The most common skills acquired during the programme was ‘collaboration’ (mentioned eighteen times) with others in the cluster, including other charities and digital partners. ‘User testing’ (mentioned fourteen times), ‘developing a prototype’ (mentioned thirteen times) and ‘creating a user journey’ (mentioned thirteen times) were also skills that charities felt they learnt as result of taking part.

​

23

Outcomes for charities: increase in digital confidence

Identified by digital partners:

• A central theme was how charities had built digital confidence over the course of Sector Challenge, for example in using design thinking processes and agile development which was highlighted ten times by digital partners during end line interviews and three times by conveners during the workshops. One interviewee at the end point explained how the charities transformed their own ability to think critically and find appropriate solutions: “in terms of confidence in developing these new skills and learning about these new tools they became more confident idea generators”.

Identified by charities:

• Seventeen out of twenty one charities felt between ‘somewhat confident’ to ‘very confident’ about undertaking digital, data and design work in future (Chart 12).

​

​

• When charities were asked how the programme had impacted on their levels of confidence, five charities stated how they had more understanding of the processes involved in developing a digital prototype and more confidence when thinking about how to approach digital and data design.

​

• In answer to the same question three charities explained how the programme had created less fear in utilising new tools and more of a willingness to experiment and explore what digital can offer.

​

• Two charities stated how their previous involvement in other Catalyst programmes, including by being part of a Learning Group via Beyond and the completion of the Discovery programme, had given them confidence to apply for the Sector Challenge funding and familiarised them with digital and design approaches and processes.

​

24

Identified by digital partner:

• Six out of seven partners alluded to how charities were confident using digital and design terminology by the end of the programme, including terms such as ‘prototype’ and ‘agile’.

Identified by charities:

• Chart 13 demonstrates that fifteen out of twenty one charities who took part in the survey felt more confident utilising digital and design terminology after the Sector Challenge programme.

​

​

Outcomes for charities: increase in comfort with digital design terminology

• Eight charities stated in the surveys how they had increased their level of awareness and knowledge with regards to digital and design terminology. Examples of the terminology charities were more familiar with included ‘discovery’, ‘development’ and the ‘double-diamond’.

​

• Four charities felt that they had increased their confidence in understanding digital terms and their application in a digital context, and one charity explained how their new knowledge as to how to apply digital terminology had helped to facilitate the next steps in their digital journey by helping them to commission a digital project.

25

Identified by digital partners & conveners:

• Three of the seven digital partners at the mid point said that the charities benefited from building relationships with each other and at the end line four digital partners reflected on how charities had understood the value of co-production.
• Three conveners at the workshops at the end of Sector Challenge revealed how clusters had formed fruitful bonds and that they mentioned that collaboration would continue between charities in terms of the next steps for their prototype, applying for new funding or setting up participation groups.

​

Outcomes for charities: building relationships & co-production

Identified by charities:

​

• Chart 14 shows how seventeen out of twenty one charities were motivated to continue working with others in the sector after the programme.

​

• Eight charities stated how a key benefit of their participation in Sector Challenge was being given the opportunity to make new connections, build up their networks and in turn to learn from others, particularly in how other charities approach and solve problems. Three of these charities stated how they felt these connections provided ongoing opportunities to collaborate on projects in the future.

​

• Two charities wrote in the surveys that they felt this was also a good opportunity to network with professionals in other sectors, both gave the example of the digital agencies they had partnered with.

​

26

Identified by digital partners & conveners:

​

• Three digital partners in total said that the programme encouraged the charities and digital partners to develop their knowledge of specific issues, in particular the way in which user research helped charities to gain a different perspective on the problems their service-users face.

​

• Three conveners revealed at the workshop how the prototypes developed (from challenges 1-3) were all functional and could be utilised by the sector or other parts of an organisation within the cluster’s service delivery.

​

• Two conveners felt that more ‘powerful’ progress had been made with the engagements and relationships with key stakeholders (such as the DWP).

​

Identified by charities:

​

• Six charities also stated how they felt that the prototype/product were useful and transferable tools for the sector, at the point of delivery. However, three of these charities stated how they would only be useful to the sector if they were to be completed and launched.

​

• The second most common response, by five charities, as to why they thought the programme was useful was that the learning, structure or process that they went through at Sector Challenge was adaptable and could help charities to work more effectively within the sector in future. One charity stated in the survey how: “the process was enlightening and could be really useful to the sector to help them best identify the best solutions and implement these”.

​

• Four charities described the work they had done at Sector Challenge as either ‘very useful’ or ‘very helpful’ to the wider sector within the surveys.

​

​

​

​

​

​

​

​

Outcomes for charities:

contributing to knowledge & progress within the sector

27

Outcomes for digital partners

• Connections and partnerships were mentioned in three out of seven of the mid-point interviews and in seven out of seven of the end line interviews with digital partners. By the end of Sector Challenge the collaboration with other digital agencies, as well as with the charities and the Catalyst/CAST network were described by all seven digital agencies as a key benefit for participation. Four out of seven digital partners alluded to how they had found synergy with others working in a similar field, one person described this as taking place within the ‘tech for good’ space, another stated how they had been working in the field for a number of years and were surprised to have met new faces through the Catalyst network.
• Digital partners spoke about how Sector Challenge gave them an opportunity to ‘add value’ (to the sector/their own organisations) or to ‘learn their own value’. The theme of ‘value’ was mentioned in this context five times during the end point interviews. Interviewees on all five occasions spoke about how digital solutions are transferable to the sector and gave examples of how playbacks, solution-diagnostics and user research had added value to the projects. One participant explains how they realised their own value through the programme: “so a huge benefit was understanding where we can be valuable. For a long time we would work anywhere because there’s an inherent value that can be transferable but there was more than that there was a specific value”.
• During the end line surveys four digital partners discussed how they had learnt a range of new skills from the programme, these included: co-working with an experienced designer, becoming experts in running workshops digitally (mentioned twice), and reviewing how other agencies diverged design thinking processes. One of these respondents stated that as a result of what their organisation had learnt from participation in seven Catalyst projects, they were redefining their organisation’s approach. One respondent explained at the mid-point interviews that they had become a better project manager, while another said that they learnt how to facilitate conversations effectively within the cluster.
• Three respondents during the mid-point interviews reported that they benefited from learning about different sectors through the programme and the problems and realities of different fields, including the charity sector. Similarly, two out of the seven digital partners at the end of Sector Challenge stated how they had gained deeper insights into the charity sector; that one digital partner stated was beneficial as their agency plans to embark on more work with charities in future.
• One digital partner also stated how the programme was an opportunity to also apply existing skills: “For us it’s been a chance to practise what we preach with bringing in lived experience, really meaningful participation, online co-production, but this was an example of how to do it and we felt that we did it really well and it did work”. One convener during the workshop session also reflected on how it contributed to a better sense of wellbeing for the partners when the digital partners felt that the solution resonated with them.
• Working effectively with CAST/Catalyst and teh conveners was also mentioned as a benefit by one digital partner who was initially apprehensive about how there involvement would work with the fear of the group feeling ‘observed’ or ‘judged’, but found that this was not the case with their relationship with Conveners: “they were so lovely and respectful and nobody felt observed, nobody felt looked at, nobody felt judged, it was really, really well done”.

​

28

Would digital partners take part again?

During the digital partner workshop the group were asked: “would you take part in the sector challenge programme again if it was repeated?”

A question that was raised prior to the group answering this question by one of the digital partners was whether this would be inclusive of the suggestions, learnings and improvements provided through this evaluation. The Catalyst member of staff present said that it would. Based on this, the group responded as follows:

• 7 said “yes” they would take part again
• 2 said “I would probably take part”

​

29

General comments

• Seven charities within the general comments section expressed their gratitude to everyone involved in the programme and felt positive about the potential that the programme has to impact the sector in a positive way. Two examples of the comments received:

“Thank you for the opportunity to be part of this project. When it's done well, social prescribing is an effective way to help people and there are so many opportunities to better digitise social prescribing and ultimately help many more people”.

“Just want to express massive thanks to allow us to be part of the programme.  What is has achieved is far beyond our expectations.  It has moved us out of our 'comfort zone' to a place where we can be braver and see how digital solutions can help us address our issues in the future”.

• Three charities mentioned that they thought it was important to get a clearer description of the what the project entails from the application stage, including clarity on funding and time commitment. One charity stated:

“We have really enjoyed and appreciated taking part in the Sector Challenge. Some hiccups around communication and expectations, particularly relating to the grant and budgets, has perhaps put us off a little from applying again at least for now, but we appreciate that these are probably our fault if anyone’”.

• One charity wanted more clarity on the next steps for their prototype: “I felt that we haven't been explained how the prototype can benefit our organisations, for an example if it'll became paid for service, would we get a discount?”
• One stated how they would have liked to have the opportunity to change course slightly on the programme, based on the outcome of user research: “I think we should have the ability to refine the challenge if user research tells us that we are not solving the right problem”.

30