State of Simulation Testing - 2015 (Responses Cleaned).xlsx
 Share
The version of the browser you are using is no longer supported. Please upgrade to a supported browser.Dismiss

 
View only
 
 
ABCDEFGHIJKLMNOPQRSTUVWXYZAAABACADAEAFAGAHAIAJAKALAMANAO
1
Timestamp
In what capacity do you work?
What industry do you/your clients work in?
How big is your company?
How would you categorize your company?
What is at risk if your system fails to operate correctly?
Describe the impact a system failure would have on your company or its customers
Have you used or considered using simulation testing at your company?
What factors led to this decision?
What other testing strategies does your company employ?
What environment is the target of your simulation test?
How would you rate your team's skills with regards to simulation testing tools & techniques
Who introduced the idea of simulation testing to your organization?
Where did they learn about it?
What is the platform/tool-chain of the system under test?
Did you experience difficulty getting business buy-in for simulation testing?
Did you experience difficulty getting buy-in from your team for simulation testing?
Did you experience any difficulty implementing your simulation test?
What were your goals in implementing simulation testing
Did you write simulation tests against an existing, legacy system, or a greenfield project?
What library did you use for implementing simulation tests?
How often do you run them?
What made you decide against simulation testing?
What other testing strategies does your company employ?
What is the platform/tool-chain of the system under test?
Do you think simulation testing would be a good fit for your company?
Have you tried proposing simulation testing in your organization? Was there pushback?
What is the platform/tool-chain of the system under test?
What other testing strategies does your company employ?
Is there anything you think could make simulation testing easier to implement or use?
Do you have any general comments on simulation testing?
2
2/4/2015 7:54:57At a company
Online entertainment
100-500 employees
Small-to-medium Sized Business
We stand to lose customers
We have some business SLAs, but main impact would be dissatisfied customers.
Considered, but decided against
Seemed like a lot of investment to try the approach. Concerns over whether we would ever get like-live results after simulating so much.
Unit Tests, Integration Tests, Headless Browser Tests, Manual Validation
Clojure
3
2/4/2015 0:00:07At a companyEducation6-25 employeesStartupWe stand to lose customers
We are a self-coaching tool, and if we're down or buggy when folks need to use this capability, those folks lose confidence in our tool and associate that with our approach and concept. To ensure that we succeed in the long term, we need to succeed at every interaction along the way.
Considered, but have yet to implement a simulation test
We have lots of existing user data (as event-sourced event logs) which we can use to feed a simulation test. We also have a large system that would frankly take a very long time to test with hand written integration tests. We feel that end-to-end system testing is the best way to exercise the whole lot. And, being able to reuse the tests for load testing, for exploratory what-if scenario testing by altering the generative model is just a huge win.
Unit Tests, Headless Browser Tests, Manual Validation
A CI server running the system
Familiar with the basic concepts
I did!
Stu's first talk on Simulant
Clojure
No. Being the CTO, deciding that we'll use this tool isn't the issue. Finding time to do it is!
No. Everyone immediately understood the benefits; we were just not clear on how to model things. Now that Michael Nygard has great modelling sample code available, we know how to proceed.
Michael Nygard's webinar and example code go an incredibly long way. I don't think he's actually got it in a completely working state. I think that if his example code is finished, thoroughly documented, and adopted as an official Simulant example and communicated as such, that'd go a long way to helping to get people started.
Wish I had the time to use it to test everything!
4
2/4/2015 2:27:57
At a consulting agency
Financial Services6-25 employees
Small-to-medium Sized Business
We stand to lose revenueirritation, less sales, frauds
Considered, but have yet to implement a simulation test
because we can generate more possible test cases

needs a more stringent modelling of the business to be able to do so
Unit Tests, Integration Tests, Headless Browser Tests, Manual Validation
A version of the system running on a developer's computer
Familiar with the basic concepts
A DeveloperFrom a blog post
java .net clojure in some combo
yes, the expected amount of work needed is prohibiting
yes, it is hard to make it not abstract
without an eventdriven business logic in place Its hard!
from java entity beans to sanity

more successful examples of eventmodelling
5
2/4/2015 7:08:24At a companyAutomtovie500+ employeesLarge EnterpriseWe stand to lose revenue
Considered, but have yet to implement a simulation test
Demos by Stu Halloway and Michael Nygard
Unit Tests, Integration Tests, Manual Validation
A staging environment
Familiar with the basic concepts
MeVideosClojureNoNo
Not implemented yet. Still working on a timeline.
A "how to plan" document would be great. We have several existing projects. We use Clojure, ClojureScript, and Datomic. We want to start simulation testing, but we aren't sure how long it's going to take to set up a simulation, plan our simulations, or run our simulations. If simulations take a long time to plan, set up, and run, then we're less likely to absorb that cost.
We love the idea. We just need to get the ball rolling.
6
2/4/2015 7:29:00At a companyHealthcare1-5 employeesStartupWe stand to lose customersToo much of that and we lose our customers, revenue, developer time and... business. We can't afford to have our systems failing badly.

At the same time, it'd be very difficult now to take time implementing this dreamed simulation testing robot for most of the reasons given above: too much revenue and developer time invested in this endeavor would imply that we... lose our customers who want more, and thus revenue, and lose our business!
Considered, but have yet to implement a simulation test
Those systems are critical and we need continuous simulation testing of them; not just testing new builds, but also testing live systems. We're talking about some kind of simulation-based monitoring.
Unit Tests, Manual Validation, Headful Browser Tests
Probably all of those!
Familiar with the basic concepts
A DeveloperFrom a blog post
Python (& JavaScript)
Officious buy-in was easy to get. Budget is yet to come.
Some kind of automated testing robot was already in their mind.
Not yet started.
Definitely more online teaching resources for Simulant-based testing.
Wow, I so much want that in my tool chain!
7
2/4/2015 7:46:18At a companyE-Commerce500+ employeesStartupWe stand to lose revenuelost sales
Considered, but have yet to implement a simulation test
complex shystem hard to test otherwise
Unit Tests, Integration Tests, Headless Browser Tests, Manual Validation, QA (Internal)
A staging environment
Familiar with the basic concepts
Upper-management
From a conference
Clojure
8
2/4/2015 10:14:02
At a consulting agency
Professional Services
25-100 employees
Small-to-medium Sized Business
We stand to lose revenue
Considered, but have yet to implement a simulation test
We haven't done simulation testin. I only pondered it privately.
Unit Tests, Integration Tests, Headless Browser Tests, Manual Validation, QA (Internal)
N/A
Never heard of simulation testing
N/AN/ARuby & Ember
9
2/4/2015 10:24:56Self-employedCharity1-5 employees
Small-to-medium Sized Business
We stand to lose customers
Considered, but have yet to implement a simulation test
Reading different people's opinions
Unit Tests, Integration Tests, Headless Browser Tests, QA (Internal)
A staging environment
Familiar with the basic concepts
A DeveloperFrom a blog post
Clojure, Ruby, JavaScript/NodeJS
10
2/4/2015 11:42:52At a companyE-Commerce500+ employeesLarge EnterpriseWe could face legal action
Our customers' inventory flow would be incorrect, either too much or not enough inventory moving between trading partners.
Considered, but have yet to implement a simulation test
Seeing Michael Nygard's talk at Strange Loop 2014 has convinced me utterly that simulation testing is a vital part of any well-rounded software QA effort.
Unit Tests, Integration Tests, Manual Validation, QA (Internal), QA (Third-party)
Ideally I want to build simulation tests against the same VM/Docker environment that runs on developer machines, in CI, and a staging environment
Familiar with the basic concepts
A Developer
From a conference
Clojure, Java, JavaScript/NodeJS
I will let you know in next year's survey.
The biggest challenge I foresee trying to introduce simulation testing in my team will be crafting a model of our business flow that we are confident covers everything, but that is something of a unique challenge due to the system we are building and I would argue is a worthwhile exercise in itself.
11
2/4/2015 12:06:35At a companyTechnology500+ employeesLarge EnterpriseWe stand to lose revenue
A system failure could mean losing a sizeable amount of revenue for failing to deliver services in a timely manner.
Considered, but have yet to implement a simulation test
We're interested in exploring new avenues of testing to increase the confidence of our application stack.
Unit Tests, Manual Validation, QA (Internal), QA (Third-party)
A staging environment
Familiar with the basic concepts
Cognicast
From the author of Simulant!
Clojure, Java
We haven't left the initial evaluation stage for simulation testing, although our progress so far is promising.
No, we're developers who care about quality software!
Not yet.
12
2/4/2015 15:26:10At a companyMobile25-100 employeesStartupWe stand to lose customers
Considered, but have yet to implement a simulation test
Lack of time.
Unit Tests, Integration Tests, Headless Browser Tests, Manual Validation, QA (Internal), QA (Third-party)
A staging environment
Familiar with advanced concepts
Upper-management
From a conference
Clojure, JavaYes.More knowledge-base articles, walkthroughs, talks, etc.
13
2/5/2015 5:46:38At a companyHealthcare1-5 employeesStartup
We stand to lose developer/business time
Considered, but have yet to implement a simulation test
Have automated, more realistic test coverage
Unit Tests, Integration Tests, Headless Browser Tests, Manual Validation, QA (Internal)
A version of the system running on a developer's computer
Familiar with the basic concepts
Upper-management
From a conference
Clojure, RubyNoNo
Not implemented yet.
14
2/5/2015 13:22:43At a company
software development
1-5 employeesStartupWe stand to lose customers
Customers would be unable to use the product, and would probably not trust it to work again in the future.
Considered, but have yet to implement a simulation test
Heard Michael Nygaard talk about it on the Cognicast podcast.
Unit Tests, Integration Tests, Manual Validation
A staging environment
Never heard of simulation testing
A Developer
Nygaard on Cognicast podcast
ClojureWe haven't done it yet.
We haven't discussed it yet.
We haven't even planned it yet.
Greenfield project
15
2/6/2015 16:37:47At a companyAll500+ employeesLarge EnterpriseWe stand to lose revenue
Considered, but have yet to implement a simulation test
Knowledge and understanding of the benefits, plus a couple things gleaned from the presentation Michael Nygard did.
Unit Tests, Integration Tests, Manual Validation, QA (Internal)
A version of the system running on a developer's computer
Never heard of simulation testing
A DeveloperFrom a blog postJavaDidn't ask.Haven't tried yet.
Am working on it. WIsh there were more examples, like the one Michael Nygard was working on
Some more examples of how things are wired together.
16
2/8/2015 11:01:24Self-employedFinancial Services1-5 employees
Small-to-medium Sized Business
There would be no substantial impact
Considered, but have yet to implement a simulation test
Curiosity, task complexity.
Unit Tests, Integration Tests
A version of the system running on a developer's computer
Familiar with the basic concepts
A DeveloperFrom a blog postClojure
17
2/8/2015 21:17:04At a companyapartment rentals500+ employeesLarge EnterpriseWe stand to lose revenue
We would fail to generate rental leads for our customers, reducing revenue for us and our customers.
Considered, but have yet to implement a simulation test
I haven't yet finished the implementation. I'm working on it in my 10% time.

I've decided to work on a simulation test, because I wanted to see how the system under test behaves across a variety of partial-failure conditions and under differing user behavior.
Unit Tests, Integration Tests, Headless Browser Tests, QA (Internal)
a combination of local dev machine and CI database
Familiar with the basic concepts
A Team Lead
From a conference
Clojure
Not yet. The test implementation is not yet ready to demonstrate.
No.
I'm still working on it. It's early days, so I can't quite say yes or no to this question yet.
18
2/10/2015 8:51:01At a companyIT / data analytics1-5 employeesStartup
We stand to lose developer/business time
We are currently pre-launch on our data visualization webapp, VisClay.com, so we would not have any real impact at the moment. Later, we could lose customers and revenue.
Considered, but have yet to implement a simulation test
The time it takes to understand simulation testing is a cost that I haven't paid yet.
Unit Tests, Integration Tests
no simulation testing at present
Familiar with the basic concepts
Upper-management
Cognitect's recent webinar announcement
Clojure
Haven't tried yet. Still not sure that simulation testing is worth doing for us.
Again, n/a as yet.
N/a, haven't implemented any simulation tests.
19
2/10/2015 19:17:41Self-employedFinancial Services1-5 employeesStartupWe stand to lose customers
Considered, but have yet to implement a simulation test
Seems like a good approach to catch unknown failures but have not yet had time to try it.
Unit Tests, Headless Browser Tests, Manual Validation
A CI server running the system
Familiar with the basic concepts
A DeveloperFrom a blog post
Clojure, JavaScript/NodeJS
20
2/16/2015 11:33:59Self-employedE-Commerce1-5 employees
Small-to-medium Sized Business
We stand to lose developer/business time
Considered, but have yet to implement a simulation test
Because Simulant, Datomic, and Cognitect are awesome!
And I agree with their motivation for Simulation testing.
Unit Tests, Integration Tests, Headless Browser Tests, Manual Validation, QA (Internal)
A staging environment
Familiar with the basic concepts
A Team LeadFrom CognicastClojure, Ruby
I didn't try, but I will, when the application will be big enough to worth effort
I didn't try, but I will, when the application will be big enough to worth effort
I didn't try, but I will, when the application will be big enough to worth effort
Simulation testing is obviously valuable and I was very glad to hear in Cognicast, that the whole idea of Datomic was born from the work on Simulant!
21
2/16/2015 14:44:12At a companyE-Commerce6-25 employeesStartupWe stand to lose customers
Customer would loose important documents or unauthorized person could get access to system.
Considered, but have yet to implement a simulation test
Critical application (clients sharing encrypted data). If something goes wrong, client will lose data.
Unit Tests, Integration Tests, Headless Browser Tests, test.check cross tests between CLJ (Java), CLJS (JavaScript) and iOS implementations
A CI server running the system
Familiar with the basic concepts
A Developer
Clojure community, video with great talk by Stu Halloway
Clojure, Java, JavaScript/NodeJS, ClojureScript, iOS application, other mobile platforms in future
No. Everybody understands importance of simulation testing for our application. But with looming deadlines it is difficult to find time for proper testing.
No. Everybody understands importance of simulation testing for our application.
Not yet implemented. I assume that most difficult will be data model, in which different actors create, share and receive data.
Greenfield projectGreat and clever idea. I fell in love with simulation testing first time I saw talk by Stu Halloway.

I was sold at the moment when he explained, that he can run simulations and write tests (Datomic queries) later.
22
2/16/2015 19:38:47At a companyMusic6-25 employeesStartupWe stand to lose customers
Considered, but have yet to implement a simulation test
Not a believer in TDD. Believer in writing Unit Tests post code. Integration and Functional tests always seemed broken. Simulation testing feels like a better solution.
Unit Tests, Integration Tests
A production environment
Familiar with the basic concepts
A ManagerFrom a blog postClojureNo. I make the decisions.
No. They trust me.
I have not implemented it yet.
23
2/19/2015 2:51:13At a companyFinancial Services6-25 employeesStartupWe stand to lose customersloss of face.
Considered, but have yet to implement a simulation test
over the years i have come to value having as much reality as possible in testing without actually having real customers as unwitting testers
Unit Tests, Integration Tests, Manual Validation
A staging environment
Never heard of simulation testing
me. I am CTOrelevance podcastClojure, JavaExisting systemnone yet
just need time to read more and have a go. may yet decide against.
24
2/4/2015 3:26:09
At a consulting agency
Financial Services6-25 employees
Small-to-medium Sized Business
We stand to lose customers
Dependant on nature of failure, this could mean losing customers, or legal action.
Have not considered
Having first heard of simulation testing on the Cognicast, I thought it maybe useful for one of our large projects due to the complex, highly concurrent nature of the system.
Not yet, I need to learn more about it and it's successful usecases, methods etc.
Java
Unit Tests, Integration Tests, Headless Browser Tests, Manual Validation, QA (Internal)
These things might already exist but.. A good book, or tutorial series / video series etc describing how to start out, and common pitfalls, bad paths to avoid when learning.
It sounds like an interesting, and highly valuable technique for certain problems.
25
2/9/2015 6:06:47At a companyE-Commerce1-5 employeesStartupWe stand to lose customers
Large retailers rely on our system to make analytical decisions in a timely manner, mostly about how to price their products. If our system failed once or even occasionally, we believe the customers would stick it out. But if it failed regularly, they would probably seek out one of our competitors.
Have not considered
I'm not sure. We haven't considered it.
No.Clojure
Unit Tests, Integration Tests, Headless Browser Tests, Manual Validation
A great introduction and explanation of the benefits. A guide for how to get started (not just setup). What decisions do I need to make?
Benefits are unclear.
26
2/4/2015 3:02:24
At a consulting agency
E-Commerce25-100 employees
Small-to-medium Sized Business
All of the above
Implemented a simulation test
Extremely large and opaque system responsible for much revenue.
Unit Tests, Integration Tests, Headless Browser Tests, Manual Validation, QA (Third-party)
A CI server running the system
Implemented other simulation tests
A DeveloperUnknownJava
Like many things, simtesting doesn't make development especially cheaper -- instead it magnifies its impact and protects the system under test from certain kinds of failure. The key seems to be presenting the expense as it relates to those benefits.
No.Modelling can be difficult.

Understanding a large distributed legacy system's behavior well enough to drive a simtest against it is time-consuming and can require a high degree of participation from the business' subject-matter experts.
To provide the customer a means of detecting undesired behavior across sub-systems before deploying revenue-impacting bugs into a production ecommerce system.
Existing system
An in-house solution modelled after Simulant.
On a scheduled basis (Every X days)
More examples. More blogposts. A flexible open-source platform.
27
2/4/2015 12:03:12At a companyE-Commerce25-100 employees
Small-to-medium Sized Business
We stand to lose revenue
Implemented a simulation test
Unit Tests, Headless Browser Tests, Manual Validation
A staging environment
Familiar with the basic concepts
Upper-management
_Clojure, JavaExisting systemfork of Simulant
On a scheduled basis (Every X days)
28
2/6/2015 22:34:43At a company
Healthcare, Professional Services, BPO
100-500 employees
Small-to-medium Sized Business
We stand to lose revenue
We systems add value and generate money for us and our customers but they are not mission critical. We optimize existing processes with machine learning and fall back on old behavior when our systems are down.
Implemented a simulation test
We needed a DES for sensitivity analysis anyways and so we decided to conduct the simulations on the real system itself. This was in 2012 before simulation testing really took off and so it was kind of by mistake that we ended up with this awesome integration end-to-end test. :)
Unit Tests, Manual Validation, QA (Internal)
A CI server running the system
Varied skill levels.A Team Lead
it fell out of other needs as noted earlier
ClojureNo, it was required for other reasons.
No, it was fun to build!
It required us to change our design early on but for the better.
Traditional operations research reasons.. testing the system was a nice added bonus.
Greenfield project
We wrote our own DES libary (two actualy, one event based and the other process based using core.async))
It used to run per comit but with some recent changes it has fallen into disrepair. :(
Our approach is quite a bit different than simulants in some regards. With our system the generation of the events and the playing of them are not decoupled as they are in simulant. I think the simulant approach is better and preferred but for our needs it wasn't possible. Again, we are using the simulation for other reasons outside of just testing code paths. We are using the simulation to tell if our system makes certain decisions better what overall effect will it have on the business metrics. Because the decisions made by the system effect the users we have to have the event generation occur at runtime.

Anyways, with that justification out of the way.. the way we end up writing the simulation is nice in some regards. We use a process based approach using a modified version of core.async. This means each process/agent is a go process and you get all the nice benefits of core.async that you normally get. Namely, the ability to reorder messages using alts to build up a FSM. The fact that we use core.async in our normal code means that writing simulation code is natural. I won't lie though, this can get complex. However, after writing some event-based DESs I greatly prefer process based ones (no more callback hell).

Another thing that we did that made the simulation awesome was to create a visualization tool for it. It makes watching the sims fun and we basically got a free ops dashboard out of it. I think some of this could be generalized so a minimal viz of simulations could be provided out of the box for new sims. I haven't built another system in this manner yet so it is had to say for sure how well things would generalize.
I think I've said enough. :) The next large system I build I will use simulation testing again. Right now I'm primarily doing data science tasks not involving building systems.
29
2/15/2015 1:25:45Self-employedFinancial Services1-5 employeesStartupWe stand to lose customers
Implemented a simulation test
Interest in the technique and recognition of its power. The hope it would be less work than regular unit testing.
Unit Tests, Integration Tests, Manual Validation, QA (Internal)
A version of the system running on a developer's computer
Familiar with the basic concepts
Founder
From a conference
Clojure
Just a regular discussion
Yes, I found a bug that was not considered critical, but was not trivial to fix, and which prevented my simulation testing from continuing.
Proof of concept for the technique.
Existing systemtest.check + customizationWhen issues ariseThe ecosystem needs to mature quite a bit.<redacted: identifying information>
30
2/27/2015 14:46:41
At a consulting agency
Media applications6-25 employees
Small-to-medium Sized Business
We stand to lose customers
If there was any more than a 5 minute outage, we could lose a large contract and subsequently a large portion of our business.
Implemented a simulation test
We have an outage in our system when we built the first implementation some 10 years ago. After that, we began doing simulation testing.
Unit Tests, Integration Tests, Headless Browser Tests, QA (Internal), Simulations
A production environment
Familiar with advanced concepts
A Team LeadRich Hickey
Clojure, Java, .Net, JavaScript/NodeJS
No. The business people had experience with bugs (the 2000 general election bad call) and system outages (2002 - their system did not work at all) so there was plenty of prior experience to inform their decision to allow near 50% of resources to be spent on more sophisticated testing.
No.No. We simulate all activity in the system over time and can compress time ranges when needed.
Currently we generate test data statically at the beginning of the test with very specific specs. We are interested in improving this with generative style testing in the future.
Existing systemcustom - simulant did not exist
On a scheduled basis (Every X days)
This is something we are currently researching.
We are believers. I think the hurdle is getting business people to understand how to value this vs more traditional styles of testing they are used to.
Loading...
Main menu