How to Reveal the Blind Spots in your Medical Device where Critical Defects are Hiding…
This is how to Identify all the High Risk areas in the Application & Logically Expose, Isolate and Mitigate Critical Defects Lurking in your Medical Device.
I’m going to layout the process I learned from Experts at KPMG, when I was a QA Lead on a Mission Critical System being updated – which could’ve ruined the Client company, if done improperly.
This process is based on Failure Mode & Effects Analysis, (FMEA), which has been around since the 1950’s and is as effective today as it was back then. It’s utilized to mitigate high risks in mission critical devices & systems, (caused by code changes), that are usually not discovered during typical software testing.
This wonderful process only needs to be performed once per Project, and is a huge time saver – since it greatly reduces costly rework & prevents critical defects from escaping to production.
Typically, these critical defects are generated by unexpected user interaction and/or abnormal system behavior.
To show how simple this process is to perform, I’m referencing this image of a whiteboard I previously used to explain this approach.
As you see in our company services, we are a process improvement firm – for biotech & pharma companies – across both engineering and quality teams.
For a typical medical device project, we’d conduct a gap analysis of R&D, Mfg & Supplier Quality teams across an entire organization, in order to identify all the gaps in their Quality & SDLC Processes.
What I’m going to walk you through is geared towards the Engineering Team because, it’s a specialized process for R&D.
This is our Risk ID Process, which can be utilized for new products or existing applications – here is how it’s performed.
First, assemble a cross-functional team of people with diverse knowledge about the process, product and customer needs. Functions often included are: design, development, quality, testing, reliability, maintenance, sales, marketing, clinical and customer service.
Next, get a large room with wall sized whiteboards, and management support to ensure all folks attending – contribute.
Let’s consider an existing medical device that’s been out in the field for a couple of years, and your customers are asking for additional features. You want to reduce the risk (of making changes to the code) as much as humanly possible – to ensure the updated product provides the features your customers requested without negatively impacting existing functionality.
People get funny, when you break features they’re accustomed to and depend on – especially on life saving devices.
Here’s the million dollar question: “how do you make changes to existing code (add/remove/change features) without breaking existing functionality…?”
What’s the secret? (it’s the magic which we’ll describe below)
Let’s start with the whiteboard shown above; you see the following sections:
- Feature
- Code
- Potentially Impacted Feature
- What if Scenarios
- Impact vs Prob
- High risk focus areas
You’ll need a Facilitator who everyone will listen to, and understands how to get people to contribute.
First, create a list of all the features/requirements changes for the new project. If you look at the Feature section, you’ll see New/Changed/Removed Features with alphanumeric indicators next to each of them. These are important and we’ll get back to them soon.
Create the list vertically, just like the whiteboard image.
Then to the right of those features, ask the developer responsible for making those code changes, to list the areas of the code (he would need to touch) to implement the new/changed/removed features.
The items on the whiteboard image labeled Code A1, Code A2, etc, are examples of that list.
Identify the High Risk areas.
After the developer lists the areas of the code (that he would touch) ask him to list – to the right of his vertical list – all the existing features that could potentially be impacted by the code he would touch. What is the surrounding functionality that could potentially be impacted.
See the Potentially Impacted Feature list on the whiteboard image above.
Create Scenarios which become targeted negative Test Cases.
Now it gets interesting because, this is where you start creating the steps to discover the hidden critical defects lurking in the code.
For every Potentially Impacted Feature, the entire team will participate in creating ‘what if’ scenarios.
For example, an existing feature on an infusion pump could be the Bolus button which the Nurse could use to override/increase the infusion rate in order to deliver a single dose (drug/medication) all at once.
This ‘what if’ scenario describes what could happen to the infusion rate functionality because of the code change.
Scenario 1.
The nurse enters all the new infusion settings on the device, including syringe size, rate & volume, and the infusion is started.
The nurse selects the Bolus button and holds it steady for 10 seconds.
The nurse deselects the Bolus button, however, the infusion rate has been permanently changed to the same rate as Bolus rate. (defect)
Team Contribution creates the Magic.
It’s expected that Engineers & Non Engineers (meaning – everyone) will be required to contribute scenarios throughout this process.
You’ll notice, once you get started, the more scenarios that are created, the more people in the room will think of additional scenarios as well.
Before too long these All Important Scenarios will start filling the whiteboard – which is the plan.
The marketing, clinical, tech support, program management, and engineering teams, all have unique knowledge of your products and programs – and it’s this Combined Knowledge which is the greatest contributor to this process.
The Magic.
After you’ve exhausted this process, then it’s time to Prioritize the Scenarios.
This process is actually pretty fast.
Draw a 4 Quadrant Chart (like the whiteboard image above) and label each Quadrant similarly.
Have the Facilitator go down the list of scenarios, and ask everyone to rate each scenario.
Scenario 1
If a nurse used the Bolus Feature on the infusion pump after the code change was made, what is the Probability this problem could occur?
High Probability (much used feature)
Also, if this problem does occur – how High is the Impact?
High Impact (unexpectedly permanently changing the infusion rate is dangerous to the patient)
It should look like this:
Impact vs Probability
| High Probability | Low Probability | |
| High Impact | 1 | |
| Low Impact |
Now you have Prioritized Scenario 1 as High Probability/High Impact – potentially very high risk to the device and the patient.
As another example, there could be a scenario testing the font color change in the help section, because it could be existing functionality impacted by the code change.
However – because it’s not impacting Core Functionality (nor critical user workflow) it’s considered low impact, regardless of probability.
Go through & prioritize the remaining scenarios, as shown in the whiteboard image above.
After you’ve completed prioritizing all the scenarios, only keep these scenarios:
High Impact / High Probability
High Impact / Low Probability
Save the other scenarios for future testing, especially if you end up having extra time on the project.
Justification
In a regulated environment, every approved requirement must be tested and added to the traceability matrix.
So for these scenarios, convert them to test cases and label them as additional test cases related to the requirements. Then you can trace these as well.
Benefit
By utilizing this process, you’ve identified all the high risk areas of the product (existing functionality potentially impacted by code changes) using a logical approach.
You created scenarios to unearth the critical defects typically caused by unexpected user interaction or abnormal system behavior (which could potentially appear in the clinical environment).
Then you prioritized them, so now you’re focused on testing areas of the application you’ve already determined were high risk, which means you’ll discover all the critical defects lurking in the system…that you wouldn’t have found otherwise. You save additional time, not wasted on areas deemed low risk.
The High Risk focus areas listed on the whiteboard graphic above are the areas of Greatest Risk – and you just made that determination as well.
That means you are properly focused on the highest risk areas of the application, and your test scenarios will ensure you are able to find & mitigate those risks.
Imagine the high degree of confidence you’ll have at product release!
