Skimping on test and evaluation in defense acquisition is not a recipe for effective reform, it’s a recipe for disaster.
by Robert F. Mortlock, Ph.D., Col., USA (Ret.)
From 2009 to 2015, the Army conducted the most comprehensive testing of combat uniform camouflage in history, leading up to its selection of the Operational Camouflage Pattern on the Army combat uniform. The testing included a combination of controlled testing—relying on photo-simulation techniques (Soldiers viewing photos of camouflage uniforms on computer screens) that used modeling and simulation—and Soldier field testing. The testing measured the performance of camouflage patterns based on probability of detection and blending, with performance scores depending primarily on distance, environmental background, movement, lighting and the actual camouflage pattern.
The goal of the program was to select the best-performing camouflage pattern. The testing included a combination of developmental test and evaluation—highly controlled testing in an environment that typically is not combat-realistic, in this case the testing that relied on photo-simulation techniques—and operational test and evaluation, in this case the Soldier field testing, which takes place in a combat-realistic environment. The test and evaluation (T&E) strategy was considered a major shift in approach because it weighted developmental test and evaluation results more heavily than operational test and evaluation results, the reverse of a typical acquisition program. The developmental test and evaluation was statistically robust and tightly controlled the distance, background, lighting and movement variables, such that changes to detection and blending scores were attributed primarily to differences in the camouflage patterns.
Yet senior Army leaders consistently placed more credibility on the field testing than the photo-simulation results. Each time the program came forward for a decision, leadership demanded more expensive and time-consuming field testing, unnecessarily driving up program costs and delaying schedules. The scope of the T&E effort grew not because it was necessary but because senior leaders remained risk-averse and unwilling to make a decision. The bottom line is that they never fully appreciated the T&E paradigm shift and never really trusted the project manager.
That project manager was me, and the experience highlighted the importance of clearly understanding and synchronizing test objectives within the acquisition strategy, especially as T&E is a target for savings and efficiencies in current efforts to reform acquisition. It is a fool’s errand to try to chase the “right” amount of T&E cost for a particular acquisition effort. In each case, the T&E strategy must be optimal to provide the knowledge that decision-makers need for the most informed decision possible.
An extensive, comprehensive T&E program is a big part of building trust with warfighters, and acquisition professionals cannot afford to sacrifice that trust. The last thing the defense acquisition profession wants is for a Soldier, Airman, Marine or Sailor in combat to question whether their equipment works as required. Warfighters must be focused on their mission and must know implicitly that their equipment works.
Rigorous T&E in defense acquisition projects provides information and knowledge to decision-makers and warfighters about how the equipment will work in a combat environment, and provides confidence that the equipment is safe to operate and works as intended. Efforts to marginalize the importance of T&E within acquisition, along with the perception that acquisition programs have bloated, wasteful and redundant T&E activities, erode the integrity of defense acquisition as an institution—further compromising legitimate reform initiatives focused on process improvements.
The muzzle blast from a M2 .50 caliber machine gun tears through camouflage as gunners from the 173rd Airborne Brigade Support Battalion conduct base defense training in March in Slovenia. Though the search for a new camouflage pattern was not a major defense acquisition program, it attracted a lot of high-profile interest from Army leadership and Congress, as well as from the Soldiers who would be wearing the new uniform. (Photo by Lt. Col. John Hall, 173rd Airborne Brigade)
PUTTING COSTS INTO PERSPECTIVE
A comprehensive and rigorous T&E program comes at a significant cost for any category of acquisition. A meaningful measure of the appropriate scope, value and return on investment of a T&E effort might be the T&E costs as a percentage of total life cycle costs—the sum of a program’s research, development, testing and engineering (RDT&E), procurement, and operations and support costs. The following are two examples of using total life cycle costs to measure the value of T&E:
The Joint Common Missile (JCM) program—the predecessor to today’s Joint Air to Ground Missile program—was a major defense acquisition program under development in the mid-2000s. It sought to develop a missile that could be fired from Apaches, Cobras and Super Hornets, replacing HELLFIRE, Maverick and aviation tube-launched, optically tracked, wire-guided missiles. From the JCM joint cost position, the JCM total life cycle cost estimate was $7.275 billion ($970 million in RDT&E, $6.023 billion in procurement, $267 million in operations and support) with a T&E effort costing $293 million, or 4 percent of the total life cycle costs.
The Ground Combat Vehicle (GCV) program sprang from the ashes of the Future Combat Systems Manned Ground Vehicle program and was meant to replace the Bradley Fighting Vehicle. From the GCV program office estimate, the GCV’s total life cycle cost estimate was $83.982 billion ($8.195 billion in RDT&E, $38.952 billion in procurement, $36.835 billion in operations and support) with a T&E effort costing $417 million, or 0.5 percent of the total life cycle cost estimate.
Which program managed its test budget best? It’s not as simple as looking at which spent the least amount of money. Different kinds of equipment require different evaluations: testing a missile is always more expensive than testing a vehicle, since the missile is destroyed during the test process and the program goes through many of them. So simply imposing arbitrary caps on dollars spent or percent of total budget allocated to testing is not a smart way to lower acquisition costs, the author explains. (SOURCE: the author)
Did the T&E effort for JCM provide a better value than that for GCV? One might conclude that the GCV program’s T&E was a better investment, at less than 1 percent of program costs compared with 4 percent for the JCM. However, the numbers alone don’t provide a complete picture of whether either T&E effort was appropriate in scope or a good return on investment.
The T&E program for a missile inherently involves more expensive, destructive testing—the object being tested is generally destroyed during the test—and requires a great number of missile firings for statistical confidence. Additionally, the total life cycle cost profiles for a missile program and a vehicle acquisition are drastically different because the missile program requires far less in operations and support costs—the missiles are simply stored until fired and then destroyed.
Returning to the camouflage T&E effort, which represented 0.167 percent of total program life cycle costs, did the Army get its money’s worth for $10 million spent over six years of testing camouflage patterns? The cost numbers don’t account for the other benefits of improving camouflage, including reduced casualties, increased mission effectiveness, greater force protection and safety and improved Soldier confidence.
This program was not a major defense acquisition program, but it did have considerable interest from Soldiers, senior Army leaders and Congress. The proliferation of camouflage patterns on uniforms had garnered a lot of attention from Congress because of Government Accountability Office (GAO) concerns that the services were wasting money on duplicative research, development and testing of camouflage uniforms.
With respect to combat camouflage uniforms specifically, the September 2012 GAO report (GAO-12-707), “Warfighter Support: DOD Should Improve Development of Camouflage Uniforms and Enhance Collaboration Among the Services,” highlighted the fragmented approach taken by the Army, Navy, Air Force and Marine Corps. The report stressed the potential to save tens of millions of dollars in development, testing, logistics and inventory control for combat uniforms.
For the Army in particular, the decision to change camouflage patterns on uniforms and equipment affects an approximately $6 billion inventory of uniforms and equipment.
So, how much in T&E costs is enough to properly support an acquisition? The best answer for a seasoned acquisition professional is, “It depends.”
Photo simulation—where a camouflage pattern is viewed and manipulated on a computer—pattern selection has statistically significant data and controls distance, movement, background and brightness. In short, it’s a good way to select a camouflage pattern that will protect Soldiers. But senior leadership did not trust the lab tests, which highlights the importance of incorporating a robust, well-designed test plan from the very beginning of an acquisition. (SOURCE: The author)
T&E IN ACQUISITION
There is no silver bullet or secret untapped method to fix T&E within defense acquisition because radical change is not needed. Incremental improvements can and do work, and the T&E processes within defense acquisition already use the best practices in commercial industry. In my experience, T&E efforts increase in scope not because of the project manager or testing agencies but because senior leaders don’t understand either the program or the T&E strategy, and thus demand more testing before making a decision.
Recent Army acquisition reform and directives targeting the reduction of T&E do not improve acquisition outcomes. In fact, they may increase risk and potentially jeopardize Soldiers’ safety. The Army doesn’t need additional directives stating goals to “streamline T&E and minimize redundant testing.” That’s inherent in the project manager charter to develop a strategy that meets the program’s objectives by balancing cost, schedule, performance and risk.
The testing itself is all about gaining knowledge, and the independence of the T&E agencies is necessary so that command relationships don’t overshadow important technical perspectives. Comprehensive testing reduces risk and provides indications that there may be issues to address—perhaps requiring more testing.
Cost overruns and schedule slips are the result of immature technology, poor cost estimating, integration and interoperability challenges, and Murphy’s Law. The business of acquisition involves risk, and it takes time to work through the multitude of issues that arise in the development of new capabilities.
In the T&E arena, the best use of time and energy for DOD senior leaders and decision authorities is to become knowledgeable about the acquisition effort in question and about acquisition T&E best practices in general.
Soldiers assigned to 2nd Armored Brigade Combat Team, 1st Infantry Division, at Fort Riley, Kansas, move into nearby woods during a platoon combined arms live-fire exercise at Presidenski Range in Trzebian, Poland, March 26. The author, who was program manager of the search for a new camouflage pattern, explains that there are two ways to go wrong with test and evaluation: Cutting back on testing too aggressively, in pursuit of cost savings, puts the program at risk, while demanding more expensive field testing in the mistaken belief that it’s more accurate drives the budget up. (U.S. Army photo by Spc. Dustin D. Biven, 22nd Mobile Public Affairs Detachment)
CONCLUSION
The perception that the testing community, particularly operational test agencies, are causing program cost overruns and schedule slips is misguided and counterproductive—undermining the credibility of a dedicated acquisition T&E workforce.
The nature and extent of T&E is one of the main areas of policy emphasis in defense acquisition reform initiatives. On the one hand, senior leaders demand T&E efficiencies to fix the perceived high costs and long schedules of T&E efforts. On the other, decision-makers typically demand more, higher-quality T&E to satisfy their desires for the most complete information on a program’s progress and risks as they prepare to make the most informed decisions possible.
Project managers must balance those demands with cost and schedule constraints while ensuring compliance with applicable statutes. Early engagement and collaboration with all stakeholders in T&E planning, execution and analysis can help allay criticisms of the T&E program’s adequacy. Thus, the best answer to the question, “How much T&E is enough?” is one that recognizes and accommodates the interests of all T&E stakeholders.
The appropriate T&E acquisition reform initiative might be to empower project managers and then trust them to build appropriate T&E strategies—and, of course, hold project managers accountable if the program fails to meet cost, schedule and performance goals. Above all, the most appropriate metric for T&E efforts is to measure whether those efforts provided enough information about the critical technical parameters, critical operational issues, effectiveness and operational suitability for an informed decision by senior leaders. More oversight, new commands and additional reporting requirements are not effective acquisition T&E reform initiatives.
For more information, contact the author at rfmortlo@nps.edu.
ROBERT F. MORTLOCK, PH.D., COL., USA (Ret.), managed defense systems development and acquisition efforts for the last 15 of his 27 years in the U.S. Army, culminating in his assignment as the project manager for Soldier protection and individual equipment in the Program Executive Office for Soldier. He retired in September 2015 and is now a lecturer for defense acquisition and program management in the Graduate School of Business and Public Policy at the Naval Postgraduate School in Monterey, California. He holds a Ph.D. in chemical engineering from the University of California, Berkeley, an MBA from Webster University, an M.S. in national resource strategy from the Industrial College of the Armed Forces and a B.S. in chemical engineering from Lehigh University. He is also a recent graduate of the Post-Doctoral Bridge Program of the University of Florida’s Hough Graduate School of Business, with a management specialization.
Related Links:
This article will be published in the July – September 2018 issue of Army AL&T magazine.
Subscribe to Army AL&T News, the premier online news source for the Acquisition, Logistics, and Technology (AL&T) Workforce.
