I’m not sure how up on current news you are, but there’s been some splash (as of early February) about an amber alert in the state of Texas about a missing child. The suspected kidnapper? Chucky the doll. You know, Chucky, of Child’s Play horror infamy.
On the one hand, awesome news! They were testing their amber alert system. That’s terrific! On the other, a critical alerting system emailed an unknown number of people in the state of Texas (with a population of upwards of 29 million people) with an obviously fake and possibly insulting message. That’s… pretty bad.
So how do you avoid something like this? NOT TESTING is NOT THE ANSWER. The answer is having a testing plan and strategy to ensure something terrible like this can and will never happen.
What is a Testing Strategy
If you want to really want to do this right, a testing strategy should outline all the things you’re going to do to test your application. This would include:
who is going to be doing what testing (and where)
what is going to be tested (and how)
which standards will be tested (e.g. regarding accessibility)
which browsers (or other technologies) will be used to facilitate this testing
what content / data will be used to facilitate this testing
where will test cases be stored
when will regression testing be completed
and so on.
Personally, I don’t think any of the above questions are particularly difficult or time consuming to answer. This is all information you should have to actually build your application. For instance, “what content will be used to test your application?” Well, do you have any of the “real” content that will go into the application? If not, can you generate some fake content that is representative? If so, is any of it sensitive (e.g. user data) that would be preventative for testing? If so, can you use part of it, or sanitize it?
This should be a pretty logical process that again, you should be thinking through anyway. An example of how Acquia’s Build and Launch Tools (BLT) handles this, is that it automatically sanitizes all user data outside of the production environment. So, if we are going to synchronize “real” content out of the prod environment (including user data) the very first thing that is done is sanitizing that user data (including passwords and emails) to ensure that as development and testing occur, there is no danger of accidentally exposing this user data or spamming real user accounts.
So, in the example with Chucky, this could have been mitigated if a testing strategy clearly stated:
do not execute testing in the production environment
verify that all user accounts have been sanitized prior to performing any testing
For something as sensitive as an emergency alert system, I might also recommend taking the additional step of blocking all outgoing email from the test server, or redirecting all outgoing email to a very specific email (vs. whatever is on the user account(s)). This is an additional safety precaution just to be 110% sure you cannot spam real users if the sanitization fails.
Who Owns The Strategy?
As with anything on a software project, this is a tricky question to answer because there’s a lot of involved parties. The developers and the quality assurance team are responsible in most cases for actually implementing the strategy. Your architect / technical project manager is likely responsible for making sure it gets created and written (with input from many stakeholders, including the QA team).
At the end of the day, it doesn’t really matter in my opinion who “owns” the strategy as long as the team comes together and agrees on it and everyone is executing towards it.
When and How to Implement a Testing Strategy?
The testing strategy is something that, much like requirements for a project, should be collected and documented prior to the developers starting work on a project. The goal being that as soon as a developer starts work they have clear guidance on how they personally will be contributing to testing (hopefully by writing some automated testing) and what the development workflow will be to actually get things tested.
My recommendation is usually something like…
developer does their work
developer writes some level of automated testing coverage (the type of work will dictate the level of coverage and framework used)
developer opens a pull request which runs continuous integration which runs the automated testing suite against the new code, including any new tests
someone does manual code review of the work (not QA focused testing, but still “testing”) — typically done in a local environment
someone does manual QA of the work to proactively try to break it and approach it like a “real user” — typically done in an integration (or dedicated QA) environment
someone does manual user acceptance testing (UAT) of the work to ensure that what was built lives up to expectations — typically done in the production environment (pre-launch) or a dedicated QA/UAT environment.
Again, this process is highly variable depending on your team, the type of project you’re working on, etc. But the bottom line is that you have to talk through it and define how you’re going to do it!
What Happens If You Don’t Have a Strategy?
Well first and foremost, a lot of people don’t, so you’re not an on island. The best thing you can do is to start working with your team to establish one. Once you have a plan drafted / established, you need to communicate this plan with your team and start enforcing it as soon as possible.
There’s not anything revolutionary or particularly challenging about testing plans. In my opinion, they are mostly common sense! The thing is, they are frequently pretty big behavioral changes for your dev team, and possibly workflow changes as well. So if you don’t communicate and enforce these changes, then your testing strategy is just a piece of paper (and it’s not going to actually help you).
In Conclusion
Testing is a critical part of any project, but as our Texas example illustrates you can do a lot of damage if you test the wrong application in the wrong way. Make sure you take some time to think about the critical details so that you can do the testing you need without accidentally spamming all of your users with horribly and obviously fake content.
Related Content
Sometimes when testing an application your cross contaminate test content with real data (or user contact information). This is bad! Let’s talk about how to avoid this type of situation.