Key Takeaways:

• Independent testing ensures AI-based biometric systems perform reliably across real-world conditions—far beyond vendor claims
• Testing-as-a-service eliminates costly, time-consuming in-house efforts, accelerating deployment without compromising rigor
• OMB mandates performance-based acquisition; external testing supports compliance while reducing operational and procurement risks

Biometrics performance is mission critical

Border and law enforcement agencies rely on AI-enabled biometric technologies to verify the identities of millions of travelers at airports, land crossings, and seaports every day. These technologies strengthen program integrity and reduce national security risks—if their performance remains consistent with the intended use. For this reason, reliable, independent test and evaluation is mission critical.

Given the operational importance and national security implications of AI-enabled biometric systems, agencies must vet them before procurement and during deployment. The technologies must perform consistently in real-world conditions, adapting to various environmental factors, different use cases, and other external variables. Decision makers need clear performance metrics to understand the operational impact and risk exposure.

The challenges of in-house performance testing

Performance testing is as challenging as it is important. The ubiquity of digital identity verification for everything from opening bank accounts online to accessing telehealth services has flooded the market with options. Countless vendors claim to offer the best accuracy, speed, and responsiveness, and the sheer number of solutions is overwhelming agency buyers.

While buyers need answers, in-house testing is prohibitive. Not only is it expensive to build and maintain the necessary testing infrastructure and environment, gathering representative data sets is time-consuming. When testing is conducted as part of procurement—which it often is—complex processes and requirements extend the timeline even further. It can take years. This is a significant barrier to getting mission-ready technologies to the border at the speed of change. There are also testing challenges that reflect the nature of AI. These technologies are not commodity software. They are black boxes lacking model transparency and traceability. Their performance varies across devices and demographic groups, and there are always new ways to train the algorithms. The more that biometrics technologies leverage AI, the more essential critical standards-based testing with robust datasets becomes.

Testing-as-a-service is a faster way to mission efficacy

There is growing urgency for border and law enforcement agencies to address these challenges. Leaders are under pressure to comply with OMB guidance. At the same time, easy access to sophisticated generative AI is making it easier for imposters to create fake identity documents and deceive biometric identity systems.

In this environment, ensuring the robustness and fit of these systems is a priority. As decision makers explore performance testing options, independent testing as a service is an alternative to in-house or vendor testing. The “borrow not buy” engagement model offers precision, speed, and cost savings. It also offers three pillars of effective performance testing for AI-based biometric technologies:

1. Testing that is in the loop, not in the way. With testing-as-a-service, testing is not an encumbrance to operational duties. Consistent, repeatable testing processes run in realistic conditions in the background. Leaders set the testing and reporting cadence that meets their operational requirements. Insights are readily available, but there is no need to maintain the testing infrastructure, hire the technical experts, or continually innovate to maintain testing rigor.
    
2. Testing that is done early and often. Ongoing system assessments are critical to ensure that performance continues to meet requirements. Any changes to a system after testing during procurement can impact performance after deployment. If testing isn’t continuous, standards-based, and completed with scalable testing frameworks, agencies can’t ensure that performance remains consistent with intended use, which is essential to measure and protect mission efficacy.
    
3. Testing on sequestered data. When vendors test their AI-enabled biometric technologies, they often use the datasets that train them. This can deliver unreliable results due to inherent bias in the data. It makes it difficult to confidently verify performance and predict operational failures on data collected by border and law enforcement agencies which can vary dramatically across use-cases. This is why selecting best-fit technologies for operational use requires testing with sequestered data that are representative of operational use-cases. Sequestered data helps to ensure more reliable and trustworthy testing results, which in turn, makes for more robust biometric solutions.

More agile procurement processes: Better, faster, more informed

The availability of testing as a service provides opportunities for more agile procurement processes for acquiring AI-based biometric technologies. Imagine if commercial solutions were regularly tested independent of any specific procurement, and performance results were widely available. Not only would the technologies be less of a “black box” to agency buyers, but decision makers would have more insights earlier in the process about whether or not performance is consistent with intended use. These insights could shape and shorten today’s lengthy procurement cycles, giving agencies the flexibility to adapt to rapidly changing technology and threat landscapes.

AI-enabled biometric technologies play a critical role in securing the nation’s borders—this role is only going to increase. Testing-as-a-service is a faster, more reliable, and less expensive way to evaluate the performance of these technologies to facilitate the movement of legitimate travelers while preventing unauthorized entry.
 

Learn more about SAIC's Border Innovations