Arnold Punaro's Record Testimony

Prepared for the Subcommittee on Commerce, Justice, State and the Judiciary U.S. Senate Committee on Appropriations

On Thursday, February 3 at 2:00 p.m., ET, Arnold Punaro, SAIC Executive Vice President and General Manager - Washington Operations, was to testify before the Senate Appropriations Subcommittee on Commerce, Justice, State and Judiciary. Before Punaro was able to make his opening statement and provide his full testimony, the hearing was recessed. Punaro's testimony was to address SAIC's performance on the FBI's Trilogy Virtual Case File system. Subcommittee Chairman Judd Gregg, R-N.H. promised to hold future hearings on the subject, which will give SAIC a chance to make its case.

Chairman Gregg and Senator Leahy:

It is a privilege to appear before you today to testify concerning our portion of the work on the Trilogy Project for the Federal Bureau of Investigation. Mr. Chairman, I ask consent that my entire statement be entered into the record and with your permission I am prepared to summarize.

I. Introduction and context

At the outset, let us say clearly that SAIC understands and appreciates the overwhelming demands and difficulties that the FBI has faced since the attacks of September, 11. While we disagree with the Bureau over aspects of the Trilogy program history, we have only the greatest respect for the dedication with which the Bureau has pursued its mission of defending our nation under the enormous, and sometimes conflicting, pressures that surfaced in the aftermath of the terrorist attacks.

SAIC, with 45,000 employees, is the largest privately owned research and engineering firm and one of the largest government contactors in the nation. As employee owners, we have prided ourselves since our founding 36 years ago on our ability to assist the U.S. Government on programs of national importance. Our dedication to work that matters is further reflected in an aggressive and pervasive ethics program. How our company operates and how we are perceived are matters of vital, personal interest to each and every employee. We have grown to become a very successful and sought after company by providing quality products and creating satisfied customers.

In that respect, let me mention several major, illustrative software engineering projects successfully designed and deployed for the FBI to illustrate the work we've done.

• The Combined DNA Index System (CODIS) is a national DNA database system for use by U.S. and international law enforcement authorities by creating DNA profiles and by matching unknown profiles found in the course of criminal investigations to profiles stored in local, state, and national databases here and overseas.
• The FBI Interstate Identification Index (Triple-I) is the U.S. national criminal history system that maintains more than 40 million data entries (the largest and most accurate criminal history database in the world) and is used every day by state, local and federal law enforcement agency in the United States.
• The National Instant Criminal Background Check System (NICS) implements the Brady Act. SAIC was contracted to develop, deploy, maintain, and support the federal, state, and local governments in checking a citizen's eligibility to purchase a firearm (handing in excess of 30 million purchases to date). It handled more than four million calls per year from firearms dealers checking purchasers against the national database. To quote Mr. Michael D. Kirkpatrick (FBI Assistant Director in Charge, Criminal Justice Information Services Division at the time of the work) in his letter of appreciation to SAIC in January 2004, "Not only is the successful implementation of the NICS directly attributed to the hard work and dedication of the SAIC staff, numerous post-implementation challenges were met head-on and overcome with SAIC's support...you have been a trustworthy, customer-oriented partner."
• Law-Enforcement Online (LEO) is a 24 hours a day, 7 days a week, online, real-time, controlled-access web portal (more than 43,000 users) providing a focal point for electronic communication, education, and information sharing for the law-enforcement, criminal-justice, and public-safety communities nationwide.

In sum, SAIC comes to this issue with a record of outstanding achievement in challenging projects, including specifically for the Federal Bureau of Investigation. We point this out not to boast, but to provide the context for considering some of the issues that have marked the public discussion of Trilogy and the manner in which SAIC has performed on this contract.

The Results and the Reasons

• Trilogy began in a pre-9/11 world with very different circumstances and requirements than those that exist now.
• The events of 9/11 caused massive and continuing change in the project while the FBI dealt with enormous post-attack pressures and demands.
• The FBI's requirements for the project-the list of what the FBI wanted the project to have and do-grew and changed continually while turbulence in FBI program management worked against stability and definitive guidance.
• A key FBI decision to drop a controversial, high-risk plan for a one-step conversion to a new system opened the way for a sensible developmental approach of incremental improvements in capability.
• The FBI and SAIC renegotiated the contract in summer 2004, coming to firm agreement on requirements for the incremental improvement through what is called the Virtual Case File (VCF) Initial Operating Capability (IOC).
• SAIC acknowledges some areas where we made mistakes and particularly where we failed to adequately communicate our concerns to appropriate levels of management, to include the Director of the FBI.
• SAIC delivered, and the FBI approved and accepted, VCF IOC within the allocated budget and ahead of schedule to industry-standard quality, offering FBI agents significant new tools in their counter-crime and counter-terror roles.

Currently, the contract has a negotiated value of $130.3 million and a funded value of $123 million. To date, SAIC has been paid $115.2 million. We expect to be paid the funded value of $123 million at completion. In conjunction with this work effort, the company has invested $3.9 million of its own money to support the Trilogy program.

Aerospace Corporation

Before presenting SAIC's testimony about the course of its work on Trilogy in detail, I want to speak briefly to the report by the Aerospace Corporation. While we have not been given a copy of this report, we were allowed to read a copy last week at the FBI. We appreciate that opportunity. Aerospace Corporation did not inform us, nor attempt to discuss in any way its findings-a lapse we find both inexplicable and contrary to the practices of inspectors general, the General Accounting Office, and other scientific groups, who find that comments from those reviewed contribute to a more balanced and useful report.

The Aerospace Corporation produced a report on the wrong software while failing to concentrate on central issues that determine system performance.

Had they asked us for comment, we could have told them they examined the wrong software. Mr. Chairman, I mean that in a literal sense. Aerospace Corporation explicitly evaluated a snapshot in time of the software as if it were a finished product when in reality, as everyone should have known, it was still being developed. The Aerospace Corporation says it found "evidence of incompleteness" and "failure to optimize." This is hardly unexpected in a work in progress that was still months away from its delivery date. In academic terms, it was as if we had been assigned a paper due December but then graded it the previous summer.

The product we presented to the FBI in December 2004 is not the product evaluated by Aerospace Corporation. VCF IOC was rigorously tested and accepted by the FBI after meeting 100 percent of its requirements.

Because the software evaluated was different from the software delivered, SAIC believes that the Aerospace Corporation report is not an adequate basis for deciding on a future course of action concerning VCF.

This is not to say we accept Aerospace Corporation's judgments about the product that was evaluated. We emphatically do not. The Aerospace Corporation is a national asset in its realm of expertise: aerospace. The Trilogy project is something else, altogether. We respectfully-but strongly-urge this subcommittee to consider that Aerospace Corporation did not bring a sufficient understanding of the uniqueness, complexity, and scope of the FBI undertaking to evaluate our software product.

Central to the Aerospace report is criticism of requirements documentation. Time and again, in the Aerospace report we reviewed, we saw instances where criticisms about requirements were based not on the substance of the requirements or whether or not the product satisfied the requirements, but rather on ancillary data such as syntax in documentation. How well the product satisfied requirements was not a part of their evaluation. Based on examination of the documentation they concluded they were not assured the product would meet requirements and went no farther.

In particular, SAIC categorically rejects the assertion that its work lacked engineering discipline, an assertion that appears without support in the document we read. This kind of assertion, without rigorous-or even specific-support should be unacceptable in an endeavor of this importance. For instance, Aerospace Corporation did not look at the software development folders, which are key documents on how the code was designed and written. These comprise the "Bible" for software developers. In a football analogy, it was as if Aerospace Corporation was asked to scout another team which had made available its playbook. They didn't bother to read it. In fact, they scouted the wrong team.

Even so, Mr. Chairman, we would welcome the opportunity, late though it may be, to discuss the findings with Aerospace Corporation. It could only benefit the FBI, which is our aim here.

II. SAIC's participation in Trilogy

The FBI's Trilogy program is a massive, multi-part, multi-contractor program for broad-based modernization and improvement of its information technology. In June 2001, SAIC was competitively awarded a cost-plus-award fee developmental contract for the Trilogy User Application Component (UAC). This is an appropriate contract type because the project involved first working with the customer to develop and agree on what was needed (the requirements) and then execute the agreed tasks. The complexity and uniqueness of the missions of the Bureau also argued for this approach. Some of the public discussion of the Trilogy contract has been conducted as if the required tasks were well known at the start, and easily achievable. At no point in time has either condition existed.

At the time of award in June 2001, the contract scope for SAIC called for development of a web front-end to the existing legacy applications used to manage case information When this effort was complete, SAIC was to define an Enterprise Case Management System. This was a measured low-risk approach building on existing, or legacy, systems within the Bureau.

The attack of 9/11

The September 11, 2001, attacks had as profound an affect on this project as it did elsewhere in the nation. Following 9/11, the Bureau faced enormous and sometimes conflicting pressures. Prior to the attack, the Bureau was dealing with revelations that a spy, Robert Hansen, had plundered FBI secrets. Security and integrity of information is a fundamental issue for the FBI. After the attack, it faced three often conflicting demands:

• The need to share information in the post-9/11 world so authorized personnel could both see and connect the dots to analyze and exploit intelligence.
• The need, in the post-Hansen world, to prevent all but a few specifically authorized people from seeing truly sensitive information.
• The need to ensure admissibility of investigative information in court in keeping with the complex body of legal, policy, and Attorney General Guidelines under which the Bureau operates.

Thus, the FBI faces a task of great difficulty and complexity in building an information technology system that simultaneously meets all three imperatives.

Trilogy after 9/11

Following the attack, the Bureau fundamentally reexamined the project. The earlier, measured approach of June 2001 called for improving legacy systems. In the wake of the attack, the FBI correctly determined that the legacy applications should be replaced to make the Bureau more effective in responding to terrorists' threats as well as to improve the efficiency of the continuing criminal investigative mission.

In the months following 9/11, the Bureau conducted an independent review of available Commercial Off-The-Shelf (COTS) systems and Government developed systems, and determined they could not satisfy the requirements. Therefore, SAIC was tasked to in February 2002 to develop the replacement for the legacy systems using the original contract. The SAIC UAC contract was restructured to incorporate an aggressive development plan first conceived in February 2002. This became the electronic Virtual Case File (VCF) contract. Thus, the FBI shelved 6 months of work that no longer fit the post 9-11 world, and directed SAIC take on a much more ambitious, high risk project.

The Trilogy VCF was a large and complex enterprise-level undertaking. There are no other criminal investigative management systems of this scale in the world. In terms of size, the VCF DELIVERY 1 system was to manage millions of case files on Day One with an annual growth of hundreds of thousands of cases per year. At start-up, the VCF DELIVERY 1 system was to store and index more than hundreds of millions of documents in a wide variety of formats. The VCF DELIVERY 1 system would support 30,000 users geographically dispersed across the United States and other countries. FBI agents, analysts, and support personnel would rely on the VCF DELIVERY 1 to conduct nearly all the business functions that support the criminal investigative process. The VCF DELIVERY 1 was also to provide hundreds of interfaces to legacy systems. The VCF DELIVERY 1 system would manage this workload while providing a 3-second response to users as well as high system availability. This would not be an ordinary case file management system.

The VCF was intended, in sum, to provide the next generation system supporting the FBI's case file management concept. It would be, as the Justice Department Inspector General has reported, "the first real change in the FBI's workflow and processes since the 1950's". The VCF would move the FBI from its slow, paper-based processes into the twenty-first century with electronic work flow. VCF, it was envisioned, would support real-time coordination among agents, allow secure access to, and reporting of case information for all those authorized to receive it, regardless of organization or location. VCF would support a dispersed community of users in creating, accessing, and managing centrally stored electronic case file information. It would provide the foundation upon which the FBI could migrate its disconnected business processes into an integrated and seamless work environment.

Following the 9/11 attacks, time was of the essence. SAIC was asked to devise an approach to deliver VCF in record time-on an even more aggressive schedule. The new challenge was to define, develop, and deploy a bureau-wide enterprise-level case management system in just 22 months. Without defined requirements or an enterprise architecture for the FBI IT systems, this was a high risk approach that reflected the post 9/11 atmosphere. Here is where SAIC made honest mistakes. We should have made known that this approach was too ambitious.

VCF and "flash cutover"

One of the key issues in the new VCF development strategy was the so-called "flash cutover" approach. That meant, simply, that the new VCF, in spite of its then undefined requirements, would not be implemented via a low risk, evolutionary strategy, but rather would be built as a grand design in record time and be implemented all at once in a "flash cutover" from the legacy systems to the new VCF. SAIC informed the Bureau this was a high-risk strategy. It was here that SAIC should have made its concerns known to the Director. The FBI insisted on this aggressive approach because of its critical need to improve information sharing and case management. SAIC agreed to undertake the challenge. In hindsight, this approach was a fundamental error and, in May 2004, the National Research Council Computer and Telecommunications Science Board was highly critical of the flash cutover approach and instead argued in favor of an incremental deployment model with prototyping and adequate time for test. From 2002 through mid-2004, the Bureau was committed to the flash cutover approach; however, after the Academy report, the Bureau agreed to a low-risk, incremental strategy.

During 2003 and 2004, the Bureau's understanding of how it should respond, of what mechanisms and process it might need, and how it should adjust the IT infrastructure to meet the challenges of fighting terrorism continued to evolve. Not surprisingly, the impact on the VCF program was continuing and significant. In the testimony of the Department of Justice Inspector General before this Subcommittee in March, 2004, the IG identified "poorly defined requirements that evolved as the project developed" as one of the reasons for the delays and cost increases in the Trilogy project. In fact, as recently as 4 months ago, the FBI had a team working to define, confirm, and refine their case management requirements.

When the flash cutover approach was adopted, SAIC formulated an approach to meet the aggressive schedule. SAIC used eight development teams working in parallel and a program staff that reached 250 full-time equivalents. The risks associated with the multi-team, parallel approach became apparent in the fall of 2003. With multiple teams working on vertical slices of the system at breakneck speed, SAIC did not adequately enforce coding standards across the teams and this resulted in less than uniform code. In addition, this approach resulted in some level of duplication of effort in the code with different approaches used to solve similar problems. This, however, did not compromise the system.

Another matter affecting the VCF software development was significant management turbulence. Since November 2001, there have been 19 Government management personnel changes that had a direct and significant impact on the management of this project (11 FBI Changes and 8 FEDSIM Changes). This lack of continuity among key Government managers contributed to the problems of ensuring the effective and timely implementation of this system. Each change brought new directions, a different perspective on priorities, and new interpretations of the requirements.

In its report on Trilogy last year, the National Research Council spoke directly to the difficulty of developing software in the absence of specific, settled requirements. As the Council noted, "[I]t is essentially impossible for even the most operationally experienced IT applications developers to be able to anticipate in detail and in advance all of the requirements and specifications."

Probably the most damaging aspect of this development environment was the ever-shifting nature of the requirements. SAIC development teams would meet with the FBI agents assigned to the project to elicit system requirements, then SAIC would translate that into software designs. Often, however, the agents would look at the development product and reject it. They would then demand more changes to the design in a trial-and-error, "we-will-know-it-when-we-see-it" approach to development. The turbulence was not limited to the immediate changes demanded. They would ripple though the related parts of the software design. This cycle was repeated over and over again and prevented SAIC from defining system acceptance criteria and suitable test standards until requirements were finally agreed under VCF IOC this past summer. SAIC expressed concern over the affect of these changes on cost and schedule; however, we clearly failed to get the cumulative effect of these changes across to the FBI customer. We accept responsibility for this failure to elevate our concerns.

The most significant of these changes, occurring during the period when the flash cutover strategy was in place, was to the Records Management System. SAIC had actually selected a commercial off the shelf (COTS) solution and the FBI had agreed to it. Then, late in 2003, FBI representatives decided they wanted a different approach, which would require changes to another COTS software package. The new COTS vendor would not be able to modify the software until a new release of the software was available in spring 2004. At this point, the grand design approach of the flash cutover strategy had begun to fall apart.

In December 2003, we delivered an evaluation copy of the VCF system. The FBI reviewed the product and identified 17 deficiencies, some of which were actually more changes in requirements. These deficiencies and changes were addressed by SAIC, and an updated version of the system was provided in March 2004. The FBI then asked SAIC to assess the cost and schedule impact of incorporating accumulated changes and finishing Delivery 1. SAIC complied with this request in April 2004, but the FBI chose not to undertake this course of action. The goal established early in 2002-define, develop, and deploy a bureau-wide, enterprise-level case management system in 22 months-was now clearly in jeopardy and behind the aggressive schedule.

From VCF to VCF IOC

In May, 2004, a series of meetings between SAIC, the FBI, and FEDSIM took place to define a new strategy. What emerged from these meetings was a significantly different plan.

In these meetings, the Bureau agreed to modify its flash cutover approach in favor of an incremental approach, allowing deployment of new capabilities. Second, instead of replacing its legacy systems at this juncture, the Bureau agreed to focus on creating new capabilities based on legacy systems. Finally, the new approach was christened VCF Initial Operating Capability (IOC) and it was set for Delivery in December 2004. The fundamental understanding between the SAIC senior leadership and Director of the FBI that enabled SAIC to go forward on the VCF IOC was agreement, for the first time, on a fixed set of requirements and defined acceptance criteria.

III. What the FBI received in VCF IOC

In December of last year, SAIC delivered VCF IOC. The project was successful. It:

• Delivers significant new capabilities,
• Complied with the December, 2004 delivery date,
• Was within the budget allocated for IOC,
• Met 100 percent of requirements established by the FBI for IOC,
• Passed a rigorous testing phase,
• Was accepted by the FBI,
• Meets or exceeds industry standards for quality, and, most importantly,
• Is working well today for FBI agents in New Orleans and Washington Headquarters.

Functional capabilities

With VCF IOC the FBI has a system that will move agents from a slow, paper-based system to a twenty-first century system for their key investigative efforts. In the past investigative information was often held-up in Field Offices, captured in agent notebooks, stored away in filing cabinets, and generally held in different ways and different means all across the country. VCF IOC makes critical information available instantaneously, in a uniform, easy-to-access manner, to all who need to access it regardless of their physical location. Additionally, these new capabilities build a foundation for migrating now-disconnected business processes into an integrated work environment and provide the infrastructure required to add the additional case management capabilities. Specifically, the functional capabilities of IOC include:

• Investigative document import for the FD-302 and related documents (the current mainstay of FBI investigative effort) and National Security Letters.
• Electronic workflow, validation, and approval meeting legal, policy, and Attorney General Guideline standards to ensure admissibility in court.
• Upload of approved investigative documents into the appropriate case files as serials in the legacy Automated Case Support (ACS) system.

Infrastructure capabilities

If widely deployed, the infrastructure capabilities within IOC would take the Bureau from its current paper-based circumstances into a modern web-based environment. Specifically, IOC delivers:

• A modern 3-tier web based computing infrastructure (as a migration target from the legacy mainframe).
• An effective web-based user interface, already well received by agents who have seen and used it.
• Organizational Hierarchy maintenance infrastructure, which matches IT infrastructure to the Bureau's organization.
• Automated interface to the legacy ACS.
• A significant part of the underlying infrastructure for security, access control, auditing and logging.
• System management & integration with the FBI's Enterprise Operations Center (EOC), a 24-7 monitoring and support center.

The functional and infrastructure capabilities in IOC enable the rapid expansion of VCF capabilities, both to add new features and to integrate software developed for Delivery 1 but not included in IOC. As evidence of this, in November 2004, the FBI tasked SAIC to extend the capabilities of the IOC system to provide a significantly broader capability to the Agent users. These extensions were successfully implemented in less than three months and provided to the FBI pilot users, where they have been quite well received.

We believe the FBI would be well served by expanding these capabilities beyond the pilot sites, even as an interim solution to its urgent needs.

Beyond the capabilities and infrastructure active in IOC, SAIC has done substantial work toward meeting the full set of requirements articulated to date for the Bureau and enterprise-wide version of VCF. The product of that broader work can be categorized in three groups. In the first category are capabilities where implementation was complete (or nearly complete), where integration and test were underway, and where routine software problems were being identified and fixed. These specifics of work done in these categories include:

• Case Management
• Leads
• Intake and Report of Investigative Activity (RIA)-which is a different way of approaching the import documents in IOC
• Document Management
• Notifications & Ticklers
• Source Management
• Text Search
• Most of the Reporting Generation Capabilities
• Case Classification Hierarchy Maintenance Infrastructure
• The remainder of the underlying infrastructure for security, access control, auditing and logging including complex business rules address the potentially conflicting pressures to share information post-9/11 and to implement need to know restrictions post-Hansen.

Beyond completing the integration and test effort, additional work would be required to deploy these capabilities focused on (a) resolving outstanding requirements or implementation issues, and (b) adapting the capability away from the flash cutover approach to the incremental deployment strategy.

The second category represents capabilities where implementation was in progress but engineering or requirements issues required resolution before implementation could be completed, including:

• Evidence Management
• Analysis & Techniques and the remainder of the report generation capabilities.
• Name search
• Resource tracking & management
• Crisis Case management

The third category includes capabilities that were late requirements additions or implementation approach changes and preliminary engineering efforts were in place. This would include records management.

In addition to these capabilities, SAIC performed substantial analysis and engineering efforts to document the complex and largely undocumented legacy environment that has evolved over the years. That effort was critically important to the FBI's information technology initiative. In a December, 2002 report, the DOJ IG noted that the lack of documentation for the legacy systems would limit "how rapidly UAC can be developed and deployed" since "the FBI must know what it has before it can define the right solution to fix the problem". The SAIC team made significant progress in this area producing

• Over 300 Interface Control Documents (ICDs) covering the interfaces between internal FBI systems and also with external systems.
• Extensive analysis and mapping of largely undocumented legacy data to a relational model in preparation for migration into VCF.

IV. Conclusion

In conclusion, SAIC has spent the last 36 years working hard and ethically to support important work for the U.S. government and our nation. We have been successful because we have delivered good work for our customers. We followed a difficult path to get there. The Bureau faces difficult choices in difficult and challenging times. Unfortunately, the flawed report from Aerospace Corporation does not provide a sound basis for making decisions about VCF IOC.

The information technology assignment that the FBI envisioned and that SAIC accepted in June of 2001 changed dramatically after the terrible events of 9/11. As the FBI struggled to respond to new missions and conflicting demands, new technology requirements also evolved, and we attempted to keep up. Finally, it became clear to all that the grand design envisioned in the full version of Virtual Case File was collapsing. The FBI agreed, instead, to an incremental approach that would - and did - produce immediate and tangible results. With the delivery of VCF IOC, SAIC has given FBI agents new capability today-not at some uncertain point years from now, but today as they work to combat both crime and terror across this nation.

SAIC pledges to the Committee and to the FBI that we stand ready to work at cost with all parties to recognize the full potential of all of the extensive documentation, analysis and code that has already been provided to further enhance the capabilities of the FBI to perform its vital tasks.

If the FBI's goal is to provide its agents enhanced capabilities as soon as possible and at relatively low additional cost, then we strongly recommend that the FBI continue to deploy VCF capabilities to the agents using the highly successful incremental approach utilized for the VCF IOC delivery and to evolve it along with their emerging enterprise architecture. Using IOC should bring dramatic productivity improvements now while the bureau develops a new system.

If, however, the primary goal has shifted to meeting the new requirements of the new Federal Investigative Case Management System (FICMS), or to adopt the latest technology and COTS components that did not exist when VCF began, then the FBI's agents will have to wait until these new programs deliver as yet undefined capabilities in three or more years. The Trilogy IOC provides much needed capabilities today that are scalable across the entire FBI and provides the foundation to quickly add other required capabilities incrementally over the next year.

Business Contacts:
Use our Contact Form or call 1-800-430-7629.
Press/Public Relations Contacts:
Ron Zollars - San Diego, CA 858-826-7896 zollarsr@saic.com		Connie Custer - McLean, VA 703-676-6533 custerc@saic.com