Evaluating Open Source Ventilator Projects

Copyright Robert L. Read 2020. Released under CC0 license.

Created by Public Invention

I propose we use a 5-point scale for each attribute, but we give a precise definition of what it means to be at a certain level for each scale.  Then we can add footnotes, but the overall table will be short and understandable.

Attributes are:

1. Openness
2. Buildability
3. Community Support
4. Functionally Tested
5. Reliability Tested
6. COVID-19 Suitability
7. Clinician Friendly

Openness

1. Not Open
2. Declared to be open, but no plans published
3. Have a repo with at least some plans
4. Has a clear license strategy, regular updates to plans
5. Fully open, everything document, responsive community, clear license

Buildability

1. Unbuildable
2. Documents available but they require guesswork
3. All software and hardware transparent and documented. Some manufacturing instructions, such as a build video
4. Complete documentations suggested reproducibility
5. Has been successfully reproduced by another team purely from documentation

Community Support

1. Inactive; not point of contact
2. Point of contact, but unresponsive
3. Responsive leader or manager, more than one volunteer
4. Active community, weekly activity and reports, git repo or other shared documents
5. Large, active, open community

Functional Testing

      0.  In Design Phase, Not listed/tested

1. Makes a bag move
2. Tested with a test lung
3. Tested for pressure and volume limits, with breath rate control
4. Tested for alarms, multiple modes, O2 mixing
5. All test green (if asserted as a feature)

Reliability Testing

      0.   Not Listed

1. Operates for one hour
2. Operates for 12 hours
3. Operates for 12 hours passing all functional test acceptably (low exception rate)
4. Independent team operates for 48 hours passing all functional tests, data logs reviewed
5. Mean time between failure data starting to become meaningful

COVID-19 Suitability  

      0.  In design phase/Not listed

1. Operates with supplemental oxygen
2. Pressure or volume control or both
3. PEEP
4. Sophisticated alarm capability and sanitizability of all patient contact points, supports spontaneous breathing modes
5. Meets British RVMSv1 standards

Clinician Friendly

      0.   Unknown controls

1. No controls
2. Breath rate and volume control, standard ports
3. Breath rate, volume and pressure control easy to set, standard ports, clear external labeling graphically and in the language of choice
4. Alarms easy to set and understand; wholesale replication of an existing UI or conformance to a TBD UI standard
5. Data logging, informative, easy control, battery backup for moving. No training needed in normal operation due to similarity with familiar designs.

EMC (Electromagnetic compatibility) Testing

1. In Design Phase, Not listed/tested/addressed
2. EMC requirements have been considered for functional design
3. Risk assessment for EMC is conducted
4. EMC Testing is being conducted
5. EMC Testing passed and certification obtained

National Approval Agency/ EUA approvals

0.   Not listed/tested/addressed

1. Not seeking approval
2. Haven’t yet applied for approval
3. Applied for approval
4. Denied first round of approvals
5. Approval passed and certification obtained

Usage in field

      0.        

1. Not deployed
2. Sold/ sent for use in hospital
3. Feedback provided by the doctor/physician with improvement suggestions
4. Feedback provided by the doctor/physician with only positive comments
5. Used on a patient or patients successfully

Financing

0. Not listed/addressed (or research-only device)

1. Blocked by need for investment
2. Financing secured for 100 units
3. Financing secured for 1000 units
4. Financing secured for 10,000 units
5. Sufficient investment secured

Manufacturability (1000s)

Note: This is usually zero until Buildability of (1) unit (separate column) reaches 4.

0. Insufficient plans to duplicate a single unit.

1. Bill of Materials (BOM) clear.

2. 2d parts, 3d parts, code, all clearly documented.

3. Electrical schematics and air circuit schematics clear. PCBs if any present and documented. Wiring if required fully documented.

4. Basic instructions and special instructions present. Video instructions helpful. Documented in language of choice, preferable more than one. Basic description of “smoke tests” and simple quality assurance present.

5. Either evidence of BOM availability in units of 1000, or supply-chain flexibility of parts suggesting same. Documentation for handling supply-chain disruption present if minimal. Plans include manufacturing issues for non-expert workers. Detailed quality assurance plans present.

Manufactured Products/ produced products/ produced units

0.   Not listed/addressed

1. Manufacturing planning in progress
2. Manufacturing facility is setup
3. Manufacturing is in progress
4. More than 25 pieces manufactured
5. More than 500 pieces manufactured

Guide for Evaluation

1. In generally, evaluating a ventilator should take between 20 minutes and one hour. A higher score will require a longer time to evaluate.
2. Evaluate based on information which can be reached on the web. In some cases, you can contact the point of contact and ask questions about plans, but in general you should be able to evaluate based on what is publicly accessible.
3. You will probably have to follow links to repositories in one way or another.
4. When examining repos, look for LICENSE statement and clear licensing policy. Many teams forget to put licenses on many of their engineering documents.
5. To evaluate community support, consider all sources. If they have a slack, enter the slack and evaluate the size and activity of the community. For GitHub repos, evaluate the number of contributors and how recently they have been updated. It can be especially challenging to evaluate projects in Languages other than English---you may have to find a native speaker to assist you. We do not want to have favoritism toward English-language sites.
6. Teams do not always announce when they have sought national authorization or approval. Nonetheless, look for a statement of such submission. If you learn from personal contact of their submission, note that.
7. Try to evaluate the “drive” mechanism, which requires a bit of engineering understanding. This can often be determined from videos, but not always. Engineering diagrams will usually make it plain, but you have to dive into them to find it.
8. COVID-19 Suitability is a particularly difficult category, because the view of medical professionals has been evolving. If the team has not demonstrated measurement of pressure-volume curves with a test lung, they are unlikely to suitable. However, some devices may be planned for only non-invasive or only invasive ventilation. If a device is suitable for only one phase of the disease, they should not be penalized. For example, a machine that provides CPAP support only has limited utility, but if it does it well it may save many lives, even though many patients will progress to needing more sophisticated ventilatory support.  Support for therapeutic oxygen and an understanding of that is critical here.
9. It can be difficult to evaluate “clinician friendliness”. Few teams have gone so far as to produce training materials, so you generally have to guess from photos and incomplete videos.
10. Be on the lookout for “modules” -- pieces which are smaller than full ventilators but which can be reused or composed. For example, many teams have made “flow sensors” due to the worldwide shortage of those projects. If you an identify a module which is documented sufficiently well that it can be used independent of the “parent” project, try to add it to the spreadsheet as a separate project.