[Public] Tidepool's "101 Questions" Self-eval with CQOE Principles
 Share
The version of the browser you are using is no longer supported. Please upgrade to a supported browser.Dismiss

 
Comment only
 
 
ABCDEFGHIJKLMNOPQRSTUVWXYZAAABACAD
1
101 QuestionsDuh?Major CategoryMinor CategoryFDA Excellence Principle 1FDA Excellence Principle 2FDA Excellence Principle 3Perspective: UsersPerspective: ProcessPerspective: LearningPerspective: OrgWeightTidepool ScoreTotalWhat Tidepool does well - LINK TO EVIDENCEWhat Tidepool could do betterPatient Safety Flag
2
Do you have a repeatable build system? Can you internally replicate a build from any point in time (e.g. based on a user complaint with a specific version)?Duh.Writing Code and Building SoftwareReleases and DeploymentProduct QualityPatient SafetyXX155All code stored forever.Confirm we store all dependencies.1
3
Do builds happen quickly and automatically with every check in, e.g. using a continuous integration system like Jenkins, Travis CI or CircleCI?Writing Code and Building SoftwareReleases and DeploymentProduct QualityPatient SafetyProactive CultureXX1551
4
Do you permanently store (and back up) your build artifacts, including dependencies?Writing Code and Building SoftwareReleases and DeploymentProduct QualityPatient SafetyX155NEED TO CONFIRM.1
5
Can you release to a test environment that is separate from your production environment?Duh.Writing Code and Building SoftwareReleases and DeploymentProduct QualityPatient SafetyProactive CultureXXXX155Prod. Int. Staging. Dev.Multiple staging environments.1
6
Can you do A/B testing of new functionality? On multiple environments?Writing Code and Building SoftwareReleases and DeploymentProduct QualityPatient SafetyProactive CultureXX111We don't do A/B testing.1
7
Are your automated tests robust/complete enough that you can do Continuous Deployment?Writing Code and Building SoftwareReleases and DeploymentProduct QualityPatient SafetyProactive CultureXXX133Lots of automated tests.Not nearly enough to do continuous deployment with confidence.1
8
Do you clearly document the requirements or use cases for each piece of functionality that you write code for? Can you clearly trace code changes back to those requirements?Writing Code and Building SoftwareRequirements and Functionality TraceabilityProduct QualityX144Use cases on every Trello card. Requirements stay with card.MVP requirements are not always sufficient for independent testing.0
9
How do you know that the code only does what it is supposed to do without side effects? Is the code clean and readable, written in a consistent style? Does it have unit / system / integration tests?Writing Code and Building SoftwareRequirements and Functionality TraceabilityProduct QualityPatient SafetyXX133Lots of unit tests. Some integration and system tests.Could have more integration and system tests.1
10
Do your engineers perform peer reviews of each other's code, or do pair programming?Writing Code and Building SoftwareCode Quality and Code ReviewProduct QualityPatient SafetyXXX155All Pull Requests require peer review1
11
Do you use coding standards? Are they documented somewhere where everyone can find them? Does everyone follow them?Writing Code and Building SoftwareCode Quality and Code ReviewProduct QualityPatient Safety144POINTER TO CODING STANDARDS1
12
Do you use a software version control system (e.g. Git / GitHub, Subversion, Perforce Helix, Mercurial)? (Seriously? Subtract 10 points if you don't…)Duh.Writing Code and Building SoftwareVersion ControlProduct QualityXX155GitHub0
13
Do you use a clearly defined and documented branching strategy? Is all new functionality developed in a separate branch? Is the merge tested before before it is integrated with mainline?Duh.Writing Code and Building SoftwareVersion ControlProduct QualityXX144XXX Link0
14
Dependencies, and dependency validation: You probably depend on a lot of other software. Would your tests catch if that underlying software changed in a way that could break your assumptions? For example, what if a math or date library, changed?Writing Code and Building SoftwareDependency ManagementProduct QualityXX133Validate that libraries (e.g. moment) include unit tests.More unit tests to validate assumptions?0
15
Do you have a way to reconstruct a build with external dependencies? Would you be able to reconstruct a build from a year ago if a dependency were not available?Writing Code and Building SoftwareDependency ManagementProduct QualityXX155We keep all builds forever. CONFIRM.0
16
Do you manage dependencies in a repeatable, reproducible way so that you don't inadvertently get an update that you weren't expecting? (e.g., use of yarn.lock)Writing Code and Building SoftwareDependency ManagementProduct QualityXX155Link to example yarn.lock file.0
17
Do you build working, functional prototypes?Listening to your users: Functionality, User Experience and UsabilityProduct QualityProactive CultureXXX144Prototypes for key things. Framer, InVision.Could do more prototypes.
http://share.framerjs.com/0dhmy2gnustq/
0
18
Do you test the prototypes with typical users and incorporate their feedback prior to delivery, e.g. during alpha and beta programs, if not ongoing?Listening to your users: Functionality, User Experience and UsabilityProduct QualityProactive CultureXXX133Meh, not bad. Could always test with more users.0
19
Do you do interviews with real users on a regular basis? Do you document the results of those interviews and collate the results back into your product requirements? (If you do more than 1 per week, on average, prove it and give yourself up to 5 points here.)Listening to your users: Functionality, User Experience and UsabilityProduct QualityProactive CultureXXX155Definitely crush this. See Google Drive notes. Have done thousands of user interviews.0
20
Do you continue to test your software with real users after shipping to production? Do you incorporate feedback on a regular basis?Listening to your users: Functionality, User Experience and UsabilityProduct QualityProactive CultureXXX155Crush this. We get feedback on a regular basis and incorporate on a regular basis.0
21
Do you do "hallway" usability testing with real users?Listening to your users: Functionality, User Experience and UsabilityProduct QualityProactive CultureXXX133Could do more. As a remote team this is hard to do, we have no hallways, hard to access real users.0
22
Do you do formal usability testing with real users?Listening to your users: Functionality, User Experience and UsabilityProduct QualityProactive CultureXXX111We have not done formal usability tests with video cameras, two way mirrors, proctors, etc.0
23
Do you have an automated test harness that runs with every build?Quality and TestingAutomated TestingProduct QualityPatient SafetyProactive CultureXXX155We crush this. See TravisCI and CircleCI examples, e.g. in Platform and Uploader repos. 1
24
Do your automated tests run quickly enough that they are useful to developers during development iteration?Quality and TestingAutomated TestingProduct QualityPatient SafetyProactive CultureXXX155Yes. CONFIRM with all devs.1
25
Are you able to simulate scenarios (e.g. fake device input data) without involving real users?Quality and TestingAutomated TestingProduct QualityPatient SafetyProactive CultureXXX144See BLOB input for Medtronic devices.Could do more with more devices.1
26
Can you automatically simulate use of your software/device without involving real users?Quality and TestingAutomated TestingProduct QualityPatient SafetyProactive CultureXXX144See GhostInspector. Could do more automated tests.1
27
When a bug occurs, do you ask yourself why an automated test didn't catch it, and if possible add a new test?Quality and TestingAutomated TestingProduct QualityPatient SafetyProactive CultureXXX133Sometimes, but not always.Could ask this question as a part of every bug analysis.1
28
Do you have a policy around what and how code gets tested? Unit tests? Integration tests? Functional tests?Quality and TestingAutomated TestingProduct QualityPatient SafetyProactive CultureXXX144Policy is: Developer discretion but should always have unit tests, PR reviews, Exec approval.Could have more integration and system tests.1
29
Are there documented manual tests for functionality that cannot be tested automatically?Quality and TestingManual TestingProduct QualityPatient SafetyXX144All tasks have test plans. Begin with test strategy.Could write the tests before work begins, do truly TDD.1
30
Do you conduct Alpha and Beta programs and document the results?Quality and TestingManual TestingProduct QualityPatient SafetyXX155We crush this. In depth Alpha and Beta programs1
31
When a new bug is found that could not have been caught by automated tests but could have been caught by manual tests, do you add a new manual test or review your testing process? DUPLICATE OF 4 QUESTIONS ABOVE.Quality and TestingManual TestingProduct QualityPatient SafetyXX00Duplicate.1
32
Do you have a bug tracking system and a single place where all bugs are tracked?Quality and TestingManual TestingProduct QualityPatient SafetyXX144Trello for bug tracking.Could have more clearly documented bug policy (priorities, risk).1
33
Is your risk analysis process formally documented and does it get used for all bugs?Quality and TestingManual TestingProduct QualityPatient SafetyXX155See Risk Analysis SOP.1
34
Do you have a mechanism for prioritizing bug fixes along with new work?Quality and TestingManual TestingProduct QualityPatient SafetyXX133No clear rubric. Discretion of Product/CEO.1
35
Do bugs from your Alpha and Beta program (pre-market) get documented, quantified and incorporated into your process?Quality and TestingManual TestingProduct QualityPatient SafetyXXX155See alpha/beta homework examples.1
36
Do all bugs reported by your users (post-market) across all possible inbound systems (support desk, phone, social media) get documented, quantified and incorporated into your process?Quality and TestingManual TestingProduct QualityPatient SafetyXXX155See User Feedback Reports.1
37
Do you have a documented process for quantifying the risk for every feature, bug or complaint? DUPLICATE OF 33.Processes and Continuous Process ImprovementRisk AnalysisPatient SafetyProactive CultureXXXX050Duplicate.1
38
Do you have regular reviews of your processes, e.g. sprint retrospectives or post-release retrospectives?Processes and Continuous Process ImprovementCorrective and Preventive ActionProactive CultureProduct QualityPatient SafetyXX144Sprint Reviews and Kanban retrospectives.They don't always occur monthly.1
39
Do you document the things in your process that could be improved?Processes and Continuous Process ImprovementCorrective and Preventive ActionProactive CultureProduct QualityPatient SafetyXX155See #process in slack, and Retrospective reviews, and Retrospective board.1
40
Do you implement those things, and then later measure their effectiveness?Processes and Continuous Process ImprovementCorrective and Preventive ActionProactive CultureProduct QualityPatient SafetyXX133Qualitative review.Could establish quantitative metrics for improvements, but we don't.1
41
Do you have a mechanism for prioritizing process fixes amongst all of the other work that needs to happen? DUPLICATE OF 34.Processes and Continuous Process ImprovementCorrective and Preventive ActionProactive CultureProduct QualityPatient SafetyXX00Duplicate.1
42
Is your software development process documented in such a way that a new person can come up to speed and follow it with minimal reliance on "tribal knowledge"? (Add 1 bonus point if you are an open source project and an outside developer can come up to speed and build your project with minimal help.)Processes and Continuous Process ImprovementProcess DocumentationProduct QualityXX133Much improvement (with Docker, documents, etc.But we are still a complicated project and hard to come up to speed on.0
43
Is your documentation publicly available so anyone, even people outside your organization can inspect and comment on it?Processes and Continuous Process ImprovementProcess DocumentationProactive CultureXX155Crush this.0
44
If an employee or partner wanted to escalate an important issue internally, would they be welcomed? Is vocally raising issues internally encouraged? (If an employee would be shunned in any way, subtract 1.)Processes and Continuous Process ImprovementOrganizational EmpowermentProactive CulturePatient SafetyXX155Totally encouraged. We are as open as they come.1
45
Is it clear whom outside your organization an employee could escalate to, e.g., a board member or to the FDA?Processes and Continuous Process ImprovementOrganizational EmpowermentProactive CulturePatient SafetyXXX155See Employee Handbook.1
46
Are there clear mechanisms for employees and partners to raise issues? Do those issues get documented and prioritized against all other work?Processes and Continuous Process ImprovementOrganizational EmpowermentProactive CulturePatient SafetyXX155Everything goes to support@tidepool.org1
47
Do you encrypt all data at rest and in transit?Operational ExcellenceCybersecurityCybersecurity ResponsibilityX1550
48
Are all secret keys stored in a protected place and is it easy to rotate them? Is the process documented?Operational ExcellenceCybersecurityCybersecurity ResponsibilityX144Protected place. Could rotate keys more frequently.0
49
Is your software digitally signed? Do you have a mechanism for knowing if your software got tampered with, esp. for software running on devices?Operational ExcellenceCybersecurityCybersecurity ResponsibilityPatient SafetyX144Software is signed by GitHub.I don't believe we check signatures after it has been installed on servers.1
50
Do you offer 2-factor authentication for your users?Operational ExcellenceCybersecurityCybersecurity ResponsibilityX111Will have this when Auth0 integration is complete.It's coming!!!0
51
Do all of your employees use 2-factor authentication for all activities?Operational ExcellenceCybersecurityCybersecurity ResponsibilityX155See audit checklist.0
52
Do you use an external agency to do penetration testing?Operational ExcellenceCybersecurityCybersecurity ResponsibilityXX111Do not use an external agency.0
53
Do you have an active Responsible Disclosure Program?Operational ExcellenceCybersecurityCybersecurity ResponsibilityXX155Active responsible disclosure program.0
54
Is your code open source and available for public review?Operational ExcellenceCybersecurityCybersecurity ResponsibilityPatient SafetyX155See GitHub.1
55
Do you maintain configuration info, including deployment keys, independent of source code? Is access to those keys limited to specific people who do software deployments?Operational ExcellenceCybersecurityCybersecurity ResponsibilityXX155Four people with server access.0
56
Are your servers locked down to configurations that keep ports and network access limited as much as possible?Operational ExcellenceCybersecurityCybersecurity ResponsibilityX155See security whitepaper.0
57
Do you review available security patches and update configurations on a regular basis?Operational ExcellenceCybersecurityCybersecurity ResponsibilityXXX144Nominally quarterly.Need to be more rigorous about this.0
58
Do you know about https://www.owasp.org? Do you review this in relation to your software?Operational ExcellenceCybersecurityCybersecurity ResponsibilityXXX133Aware of it, review it and follow most guidelines.Not a formal part of our development process.0
59
Do you document and prioritize security issues based on a documented risk analysis process?Operational ExcellenceCybersecurityCybersecurity ResponsibilityXX155Security trello board.0
60
Are all of your security issues documented in one place?Operational ExcellenceCybersecurityCybersecurity ResponsibilityX155Security trello board.0
61
Do you evaluate and prioritize security issues regularly along with all of the other work that needs to be done?Operational ExcellenceCybersecurityCybersecurity ResponsibilityX155Yes, based on risk.0
62
Are your servers in multiple data centers or availability zones?Operational ExcellenceContinuity of OperationXX1550
63
Do you create regular backups and have you documented and tested the restoration process?Operational ExcellenceContinuity of OperationX1550
64
Do you use a high-availability, fault-tolerant system like AWS, Google App Engine or Rackspace (as opposed to trying to build/host your own systems)?Operational ExcellenceContinuity of OperationXX1550
65
Do you use automated logging and alerting systems? Do you have a 24x7 ops team or an on-call rotation?Operational ExcellenceContinuity of OperationXXX155Data Dog, Pager Duty, SumoLogic0
66
Do you have multiple, fault-tolerant instances of your production environment? Will your app/service keep working fine if hardware goes down?Operational ExcellenceContinuity of OperationXXX155Multiple availability zones in US-WEST-2.Multiple regions.0
67
Do you have a system that allows your users to intuitively and easily submit issues, complaints or support tickets?Ongoing Feedback and Post-market AnalysisPost-market Quality AnalysisProduct QualityPatient SafetyXXX155support.tidepool.org1
68
Do you have an automated mechanism for collecting issues that your users are having (e.g. via logging or crash reporting)?Ongoing Feedback and Post-market AnalysisPost-market Quality AnalysisProduct QualityPatient SafetyXXX144Rollbar.Could use Rollbar in more places.1
69
Can users easily identify what version of your software they are running?Ongoing Feedback and Post-market AnalysisPost-market Quality AnalysisProduct QualityPatient SafetyXX155In the footer of every screen.1
70
Do you analyze those issues on a regular basis, including doing risk/hazard analysis?Ongoing Feedback and Post-market AnalysisPost-market Quality AnalysisProduct QualityPatient SafetyXX144Risk analysis done on all bugs.No automated process for pulling Rollbar issues into Trello (yet).1
71
Do you create and prioritize new tasks based on the analysis, including the risk?Ongoing Feedback and Post-market AnalysisPost-market Quality AnalysisProduct QualityPatient SafetyXX1551
72
Does your software automatically report back when it encounters errors?Ongoing Feedback and Post-market AnalysisPost-market Quality AnalysisProduct QualityPatient SafetyXXX133Uploader using Rollbar.Need to use Rollbar in Web, Mobile.1
73
Do you log issues with your software and analyze them regularly?Ongoing Feedback and Post-market AnalysisPost-market Quality AnalysisProduct QualityPatient SafetyXXX144Bug board.Some low-risk issues sit stale for too long.1
74
Have you ever had a report of a critical, severe or catastrophic hazard due to your software? (Subtract 10 points or more for each one.)Ongoing Feedback and Post-market AnalysisPost-market Quality AnalysisProduct QualityPatient SafetyXX-1000Never ever.1
75
Do you claim specific outcomes based on your software? If so answer the following questions: (If not, give yourself 5 free points. This means that your software is, for example, a "Medical Data Display System" (MDDS) or Electronic Health Record System. You are just moving information around but not making any medical recommendations or claims).Ongoing Feedback and Post-market AnalysisPost-market Outcomes-based AnalysisClinical ResponsibilityPatient SafetyProduct QualityXX155No health claims. We liberate data.1
76
Can you reference published, peer-reviewed studies that show your claimed results?Ongoing Feedback and Post-market AnalysisPost-market Outcomes-based AnalysisClinical ResponsibilityPatient SafetyProduct QualityX100n/a1
77
Could someone else run a study and replicate your results? (Be honest! If you prevent researchers from doing comparative studies of your product, subtract 2 points.)Ongoing Feedback and Post-market AnalysisPost-market Outcomes-based AnalysisClinical ResponsibilityPatient SafetyProduct QualityX1001
78
Has someone else replicated your results?Ongoing Feedback and Post-market AnalysisPost-market Outcomes-based AnalysisClinical ResponsibilityPatient SafetyProduct QualityX1001
79
Does your logging/metrics system gather data that allows you to validate your claimed results?Ongoing Feedback and Post-market AnalysisPost-market Outcomes-based AnalysisClinical ResponsibilityPatient SafetyProduct QualityXX1001
80
Do you have a process for following up with the public if your software might not support your claims, e.g. via social media or email campaign?Ongoing Feedback and Post-market AnalysisPost-market Outcomes-based AnalysisClinical ResponsibilityPatient SafetyProduct QualityXX155Vibrant social media presence.1
81
Do you make it easy for people to let you know how it's going with your product? Via email, social media?Ongoing Feedback and Post-market AnalysisPost-market Outcomes-based AnalysisClinical ResponsibilityPatient SafetyProduct QualityXX155Vibrant social media presence.1
82
3303113060
83
94.2%92.7%0
84
0
85
0
86
0
87
0
88
0
89
0
90
Score:0
91
5We crush this. We do it better than anyone else or as good as it can be done.0
92
4Doing well here. A few improvements needed.0
93
3OK, meh, not bad.0
94
2We are phoning it in.0
95
1Yeah, we suck at this.0
96
0Not applicable.0
97
98
99
100
Loading...
Main menu