| A | B | C | D | E | F | |
|---|---|---|---|---|---|---|
1 | SCORE | |||||
2 | CRITERIA | 1 = definitely advance (best) | 2 = likely advance | 3 = consider if other strong reasons to advance | 4 = reject (worst) | |
3 | BREADTH & DEPTH OF IMPACT | Breadth of Impact | If affecting individual humans: 10,000,000+ people If affecting the environment: A large biome (e.g., the Amazon, the Pacific Ocean) or the entire world | If affecting individual humans: 1,000,000–10,000,000 people If affecting the environment: A considerable subset of a large biome; endangered species w/ critical ecosystem role | If affecting individual humans: 10,000–1,000,000 people If affecting the environment: Small, localized eco-system; endangered species w/ non-critical ecosystem role | If affecting individual humans: <10,000 If affecting the environment: Hyperlocalized eco-system or small subset of a non-endangered species |
4 | Depth of Impact (see footnote 1) | Deaths or disabilities averted. | Permanent, substantial increases to household consumption (whether through education, improved agricultural productivity, etc.). | Permanent but small increases to household income; temporary but substantial increases to household income. | Small, one-time increases to household income. | |
5 | IMPLEMENTATION POTENTIAL OF SOLUTION | Algorithm Feasibility | Useful AI solution of sufficient accuracy is already in production. Note: "Sufficient" accuracy varies by context; an AI solution that is only fairly accurate but that enables unprecedented progress may earn a score of 1. | Proof-of-concept model exists with sufficient accuracy. It should not be difficult to replicate or scale-up. | Proof-of-concept model exists, but will require substantial investment to obtain sufficient accuracy. | It is unlikely, given problem type, that useful models of sufficient accuracy are possible. |
6 | Differential Potential for AI | The introduction of AI solutions to the area will unlock unprecedented impact potential. For example, the AI solution may: • enable service delivery to previously unreached populations by automating a key decision process; • operationalize previously untapped data for which existing analytics methods are of little or no use (e.g., parsing large textual corpus). | The introduction of AI solutions to the area hold promise, though they will not necessarily change the sector. Non-AI technical solutions (such as software development) or better logistics/human coordination would remove most of the obstacles to progress. | Some use cases might unlock potential. But on average, in the near- and mid-term, the best investments into this area should prioritize non-AI technical solutions (such as software development) or better logistics/human coordination. | AI solutions will not engender meaningful progress in this area. In the near- and mid-term, the best investments into this area will prioritize non-AI technical solutions (such as software development) or better logistics/human coordination. | |
7 | Data Availability (see footnore 2) | High quality, representative data exist publicly, "in nature," with little to no need for human labeling; or human labeling on foundational corpus already completed (enabling immediate use). | Any of the following barriers inhibit data availability: • Data must be collected/labeled, with limited time/cost investment required. • Data already exist but are proprietary/siloed across institutions; can be accessed with small time/cost investment. | Any of the following barriers inhibit data availability: • Data must be collected or manually labeled, with substantial time/cost investment required. • Data already exist but require substantial time/cost investment to access (e.g., deeply siloed within bureaucracy). | Data are not available and unlikely to be available in the near future. | |
8 | Barriers to Adoption | If end-users are beneficiaries: AI solution can be made easily and publicly available to intended users. Scaling to new users is a straightforward task. | If end-users are beneficiaries: AI solution requires training or special tools/technology to use for which there may be some time/cost investment. At least a sizable portion of intended users should be able to access the tools/technology (i.e., costs are not prohibitive). | If end-users are beneficiaries: AI solution requires training or special tools/technology to use; only small portion of affected base can access training/tools/technology in near- to mid-term, given the time/cost investment required. | If end-users are beneficiaries: An overwhelming majority of intended users lack the resources or capacity required to use the solution. | |
9 | If end-user is institution: • Leadership shows substantial demand for solution and, if applicable, can find funding for it. No incentive misalignment between solution's intended purpose and leadership's goals. • No new training or special tools required for use. | If end-user is institution: • Overall, leadership shows demand for solution. • AI solution may require training or special tools/technology for which there is small time/cost investment. • If solution is a tool for low-level agents (such as healthcare workers), it can be scaled with minimal time/cost investment to all relevant agents. | If end-user is an institution (at least one of the following apply): • Leadership does not show native demand for solution, but can be feasibly convinced with some time/cost investment. • Leadership shows some interest, but adoption requires substantial time/cost investment. • If solution is a tool for low-level agents, it requires notable time/cost investment to scale. | If end-user is institution: • Intended user(s) do not show demand for solution or are not able to afford it in the near- to mid-term. • May be deep misalignment between solution's purpose and user's goals. | ||
10 | Distance Between Adoption and Impact | AI solution provides output/insight that clearly motivates the intended impact outcome. There are not many stakeholders left to engage and users can immediately access the resources required to act on the output/insight. From the point of solution delivery, there is high probability of intended impact. | AI solution provides output/insight that clearly motivates the intended impact outcome, but notable intermediate steps are required to achieve intended impact. | From the point of AI solution delivery, there are substantial & cumbersome steps required to achieve intended impact. | Users are unlikely to have access to resources required to act on the output/insight. Substantial time/cost investment would be required, or bureaucratic coordination is too severe to enable action in the near- to mid-term. | |
11 | Algorithm Generalizability | Both of the following are true: 1. AI solution will require no or trivial updates across geographies or implementing partners. 2. AI solution will require no or trivial maintenance to mantain performance over time. No concerns about exploitable vulnerabilities in model behavior (e.g., no opportunities to "game the system"). | At least one of the following concerns is a hindrance: 1. AI solution will require some updates across geographies or implementing partners. 2. AI solution will require some maintenance over time, such as retraining on new data, but it will not be too costly. If there are concerns about exploitable vulnerabilities, there is a clear protocol for overcoming them. | At least one of the following concerns is a hindrance: 1. AI solution will require substantial reworking across geographies or implementing partners. 2. AI solution very likely to require updates to maintain performance or invulnerability to exploitation over time. | At least one of the following concerns is a hindrance: 1. AI solution cannot be adapted to new geographies or implementing partners. 2. AI solution will require frequent, costly updates within its first few years of operation. | |
12 | RISKS FOR SOLUTIONS | Bias / Fairness / Transparency Concerns | Issues of bias or fairness are not a substantial concern. | There may be bias or fairness concerns, but they can be easily mitigated with existing protocols. | There is substantial potential for bias or fairness concerns, and any implementation will have to consider robust mitigative protocols and external certification. | The potential for bias or fairness concerns is so high that mitigation efforts are unlikely to correct for problems and thus the effort should be abandoned. |
13 | Need for Human Involvement | No need for human referees to adjudicate whether results are problematic. | Solution will require human referees to adjudicate whether results are problematic, but minimal time/cost investment required to make those humans available. | Solution will require human referees to adjudicate whether results are problematic, but substantial time/cost investment required to make those humans available. | Solution will require human referees to adjudicate whether results are problematic. At scale, those humans will not be available. | |
14 | Other Risks (see footnote 3) | AI solution has no other foreseeable risks. | AI solution may have some other foreseeable risks, but they are mitigable with existing protocols. | AI solution has other foreseeable risks, and require substantial investment to mitigate. | AI solution has other risks that cannot be mitigated with existing protocols or without prohibitive investment. | |
15 | OPPORTUNITY AREA SYNERGIES | Collaboration Opportunity | Multiple organizations are working on similar AI solutions in this area, but there is not serious collaboration at present. There are existing industry working groups or other venues that could facilitate collaboration. Standardization across actors would likely lead to major improvements. | There are multiple orgs who could work in this space, but only some organizations are considering AI. Some connections exist in the ecosystems, but they are not highly concentrated. There are likely some benefits to standardization across actors. | There are multiple orgs who could work in this space, but only 1 or 2 of them are currently exploring AI. Few existing connections between key actors in the ecosystem. There may be some limited benefits to standardization across actors. | There are only one or two organizations, if any, that would use AI for this use case and they are unlikely to cooperate. Standardization / collaboration are unlikely to bring any benefit. |
16 | Catalytic Fixed Investment Opportunity | A large investment ($1MM+) in one or more of the following would eliminate a key binding constraint to progress, such as: • a common software standard / platform • a benchmark data set or data sets • a pre-trained algorithm that serves as a base model | A large investment ($1MM+) in a common good would be useful to a large number of actors, but not the primary barrier to deployment. | A large investment ($1MM+) in a common good would be useful to 2-3 actors, but not the primary barrier to deployment. | No large investment would have ecosystem-level effects. | |
17 | Anchor Organization | There is at least one clear, respected organization that works at scale who could serve as an initial partner to Google.org or other funders. | There is no predominant organization, but several medium or large organizations who could take leading role in collaboration. | The ecosystem is nascent or highly fractured with few to no organizations reaching substantial scale or prominence within the sector. | The ecosystem is nascent or highly fractured and there is no foreseeable trajectory for any actors to achieve prominence. | |
18 | Funder Expertise | Funder-in-question has world class expertise in the technology or subject area underlying the topic. For Google.org, for example, this may be: • Natural language processing questions • Satellite imagery / geospatial questions • Large scale search or social media questions | Funder-in-question has relevant expertise, but may not have world class skills or background knowledge. | Funder-in-question has some relevant expertise, but lacks understanding of the sector and may have hard time identifying and supporting high performing potential grantees. | Funder-in-question has no expertise on the technical details of the technology or subject matter expertise. | |
19 | ||||||
20 | Footnotes: | |||||
21 | (1) This metric was adapted from the Global Innovation Fund's depth of impact framework: https://www.globalinnovation.fund/practical-impact-assessment/. | |||||
22 | (2) Unlike many categories, availability of data is not a fixed characteristic, it is an obstacle that funders can help grantees overcome by providing resources to do so. | |||||
23 | (3) There is no exhaustive list of potential risks associated with technology. Challenges that may fall into this category include risks around: privacy, surveillance, shifting power from marginalized to non-marginalized groups. | |||||
24 | ||||||
25 | Notes: This framework was developed in a collaboration between IDinsight and Google.org | |||||
26 | Last Updated: April 23rd, 2021 | |||||
27 | Corresponding Author: Ben Brockman, IDinsight (Ben.Brockman@IDinsight.org) | |||||
28 | ||||||
29 | ||||||
30 | ||||||
31 | ||||||
32 | ||||||
33 | ||||||
34 | ||||||
35 | ||||||
36 | ||||||
37 | ||||||
38 | ||||||
39 | ||||||
40 | ||||||
41 | ||||||
42 | ||||||
43 | ||||||
44 | ||||||
45 | ||||||
46 | ||||||
47 | ||||||
48 | ||||||
49 | ||||||
50 | ||||||
51 | ||||||
52 | ||||||
53 | ||||||
54 | ||||||
55 | ||||||
56 | ||||||
57 | ||||||
58 | ||||||
59 | ||||||
60 | ||||||
61 | ||||||
62 | ||||||
63 | ||||||
64 | ||||||
65 | ||||||
66 | ||||||
67 | ||||||
68 | ||||||
69 | ||||||
70 | ||||||
71 | ||||||
72 | ||||||
73 | ||||||
74 | ||||||
75 | ||||||
76 | ||||||
77 | ||||||
78 | ||||||
79 | ||||||
80 | ||||||
81 | ||||||
82 | ||||||
83 | ||||||
84 | ||||||
85 | ||||||
86 | ||||||
87 | ||||||
88 | ||||||
89 | ||||||
90 | ||||||
91 | ||||||
92 | ||||||
93 | ||||||
94 | ||||||
95 | ||||||
96 | ||||||
97 | ||||||
98 | ||||||
99 | ||||||
100 | ||||||