How to Combat Test Inflation to Optimize Performance

More than half of product organizations feel they are underperforming, according to our research. Here’s what’s causing the problem and how your test team can turn the tide.

Most people like to believe they’re good at their job. Test leaders are no different. But the pressures they face to fill the gaps of underperforming product organizations make it tough, according to a NI study of 200 test leaders. 

Conducted in 2023 with test leaders from the U.S., Germany, and China, NI’s study found a whopping 76 percent of organizations are underperforming or merely meeting expectations on the three most important metrics for senior leadership today—product quality, reliability, and speed of test. “It was the most surprising finding from the report,” says David Hall, director of global business units at NI. “We knew test leaders felt stuck—but not to this extent.”

Here’s what you need to know about the state of play for test and what it means for your business.

1. Demands on Test are on the Rise, but
Resources Aren’t (“Test Inflation”)

Let’s put this in the context of inflation—something we’re increasingly focused on today. “Many test teams have seen anywhere from a 100X to a 1,000X increase in number of test cases for increasingly complex products. On top of that, these teams are being asked to validate that increase in complexity in a fraction of the time,” Hall says. 

For example, the front end of a 5G mobile phone has significantly increased in complexity over the past decade. In fact, in 2013, there were just a handful of waveforms and frequencies that needed to be tested,” he says. “Today, for a 5G device, it’s more than 600 combinations of waveform and band, and each combination requires testing. And from those tests, the data must be collected and analyzed to make the right product decisions.” To make matters worse, the typical development time for many devices has shrunk from years to months. As time to market releases become dramatically faster, today’s test teams are being asked to do a whole lot more in less time—and often, with budgets that don’t match the expectations of the team. 

So, what happens when companies ask test teams to test exponentially more complex products in a significantly shorter amount of time with the same team size and budgets as before? “I presume they’re making tradeoffs to keep up,” answers Jay Guilmart, principal solution manager for enterprise software at NI. “If I have 100 test cases to validate a product, and 99 percent pass, I’m only debugging and root-causing one,” he says. “But if I have tens of thousands of test cases, like I might for a spacecraft or autonomous car, that one percent failure rate is much more difficult to address. I’m left with hundreds of issues to debug and might feel forced to make tradeoffs or let project schedules slip, and both options have serious business implications.”

Simply put, “test inflation” leaves test teams feeling concerned about how they’ll balance the quality of their work with the growing headwinds they face every day. 

2. Test Inflation Leads to Stress and Mistakes

On top of the data and delivery overload they face, test teams are being asked to measure something that hasn’t been measured before. For example, they must decide if the spec sheet is right and what the tolerances should be of increasingly complex devices. “Without increases in personnel or test investment, you’ll have to put more pressure on your teams to keep up, which leads to missed deadlines, mistakes, and employee stress,” corroborates Hall.  

And that’s exactly what’s happening at the majority of companies, he says. Test team headcounts and budgets aren’t being increased to match the growing complexity and challenges of the test function. Their work is viewed more as a checklist item than something that can add value to the product or the business. Unsurprisingly, the 24 percent of best-in-class test organizations that are beating the market on senior leadership’s most important metrics—product quality, reliability, and speed of test—are doing so with a greater investment in test. 

That includes some investment in early artificial intelligence and machine learning features. “We’re just beginning to leverage AI and ML in test,” Hall says. “While we’ve successfully used AI for some data analytics tasks—the future is using these technologies to write test software as well.” These technological advances in test will allow test teams to focus their efforts—to be more productive and efficient with their time.

But what’s the stopgap until more intelligent technology fully matures?

3. Automation and Data Sharing is Key

There’s still a lot of room in the lab environment between manual and automated testing. “You can often get 10X, 20X, and up to 100X improvement just by automating test instruments with software—something that many test teams still aren’t doing,” Hall says. “It would be unheard of not to use software automation in manufacturing test applications, but validation teams are playing catch-up—all while product complexity is driving up test cases. And while future AI and ML capabilities may provide significant improvements, they also come with a significant investment. Instead, you can start with a variety of manageable investments that deliver immediate, tangible business results but are also critical in supporting the architecture of more advanced tools. Automated testing, centralized system management, and a test data pipeline all address a near-term need and are required in the long run to build more advanced technologies. It’s a win for today and sets you up for advancements in the future.   

“Beyond automation software, today’s best-in-class test organizations are demonstrating the value of test better than the rest,” Guilmart says. “They do this by sharing test data and product insights with broader audiences inside the company. This increases participation and buy-in to the test function and produces better overall results.”

In short, they toot their own horn and ask the rest of the company to play along. But it involves more than sharing. It’s about scale. Connected test transfers the insights gathered from test to other parts of the engineering process to make better products more efficiently. It pulls test into the process earlier to inform future designs and mitigate lost time when a test isn’t passed down the line. But to truly succeed, connected test requires company-wide participation. 

4. Test Success = Bottom-Line Results (“Return on Test”)

So, what does connected test look like today? Consider Allegro Microsystems, a leading semiconductor company. After discovering that their chip validation was taking half of the total production time, the company created a new validation test team and standardized their software and instrumentation process. By doing so, they saved 40 percent on their test times and 20 percent on total development time. “The Allegro validation team is a great example of how having a deliberate test strategy can be a strategic differentiator for your business,” Hall says. 

In another example, Philips saved millions of dollars and hundreds of test hours after adopting a connected test system from NI. “Now their measurements are empowering strategic business decisions,” Guilmart says. “And it’s resulted in faster development times and increased test coverage and quality.”

But what about organizations that are still playing catch-up? “Many companies we work with are starting to see the value of standardizing test systems across their various labs,” Hall says. “We play the role of a ‘test connector’ and help them self-organize to get efficiency out of sharing test approaches and software.”

Today’s product leaders are using connected test throughout their product development processes to improve both product and business outcomes. They’ve turned test into a competitive advantage, rather than a figurative or literal checkbox. 

5. A Shift from Instruments to Outcome-Based Test Systems

As products have increasingly become commoditized, the temptation exists for some companies to treat test as a common, unspecialized task. In an increasingly complex world, however, this is a mistake.  

“Testing is harder than ever but also more important than ever,” Guilmart concludes. “Because of this, NI is providing exponentially more than the instruments needed for test. We’re offering our customers an entire system of systems that includes hardware, software, and data tools to deliver valuable outcomes to the business, along with the integration services to deploy them effectively. This takes test from a siloed, task-oriented function to an integrated, value-add to the entire organization.” 

Modern test is a balancing act that attempts to identify the correct, most insightful signals from more noise in a shorter amount of time. Consequently, today’s test leaders are rethinking product development lifecycles and how to use test data throughout to make faster, more reliable decisions. “Our customers are the product experts,” Hall concludes. “But they often struggle to capture and utilize good data amid so much test inflation. We exist to make that easier.”

To learn more, read NI’s new report on the state of test.