Captain Kerr is a Mecha11ica//Nuclear/Acoustic Engineer by training, nuclear submariner by qualification, combat systems Program Manager (PM) by assignment and systems acquisition professional by choice. He’s done a little of everything: traditional jobs aboard submarines flag officer’s staff; repaired and built submarines and been a Fleet Repair Officer (repairing surface ships, submarines and one helicopter.) Since May 2003 he has been the Program Manager for Submarine Acoustic Systems (PMS 401) where he is responsible for all sonar arrays and processing systems (including the ANIBQQ-10). He has a B.S. degree in Mechanical Engineering, an M.S. Degree in Engineering Acoustics and an M.S. Degree in National Resource Strategy. He is DA WIA certified level III in Program Management and in Production and Quality Management.
Lieutenant Commander Miller is a retired submariner officer who served as both Engineer and Navigator prior to transferring to the E11gineering Duty Officer community. Since retiring from the Navy, he has worked for Anteon Corporation providing engineering and program management consulting services to PMS 401 for the AN/BQQ-10 so11ar program. He has a B.S. and M.S. degrees in Electrical Engineering and is a certified professional engineer.
PMS 401 is the Submarine Acoustic Systems program office 1111der the Program Executive Officer Submarines. PMS 401 is responsible for all acoustic array development and in-service support as well as developme11t and i11-service support for all submariner sonar systems. The ANIBQQ-1O(V) Acoustic Rapid COTS /11sertio11 (A-RC/) sonar system is PMS 40 l’s effort to field a high performance, easily upgraded, easy to maintain sonar system across all submarine classes.
The AN/BQQ-1 O(V) sonar system, commonly referred to as the Acoustic Rapid COTS Insertion (A-RCI) system, is the Submarine Force’s answer to the detection challenges posed by increasingly quiet foreign submarines. Currently installed on 44 submarines and scheduled to be installed on all active submarines, the BQQ-10 uses COTS technology to provide a technologically up-to-date and sophisticated sonar at an affordable price. Most importantly, the BQQ-10 is designed to be periodically updated with new COTS hardware via a process known as technology insertion. The technology insertion process eliminates the two major pitfalls that would be present if a static hardware baseline was maintained. The first pitfall of a static baseline is its inability to take advantage of new detection algorithms that require ever-increasing amounts of computer power. The second pitfall is the difficulty in providing spares and maintaining a system operational when many of the internal components are obsolete. The disciplined technology insertion process used for the BQQ-10 has proven itself a success in mitigating these two pitfalls and is now the example for all COTS-based systems used in submarine combat systems.
Why is Technology Insertion Required?
Sonar signal processing applications are similar to other software programs in that each new generation requires increasing amounts of computer processing power. In the case of sonar signal processing, this need is driven by new operational requirements and the development of increasingly sophisticated algorithms to better find the quiet target signal in an ocean full of noise. Under the Advanced Processing Build (APB) process pioneered by the BQQ-10, annual software updates are developed to answer emergent Fleet needs, improve previously introduced functionality, and to add new functionality to make the sonar system more effective. Each APB is targeted for installation on specific recent BQQ-10 hardware baselines that provide the necessary processing power. If the hardware baseline for the BQQ-10 remained static, the APB process would soon grind to a halt due to a lack of headroom to accommodate the increased processing requirements. However, because of the ongoing technology insertion process, BQQ-10 systems installed in the Fleet are capable of receiving new APB functionality when it is delivered.
The second rationale for technology insertion in the BQQ-10 is mitigation of COTS obsolescence issues. By its very nature, COTS technology is driven by the commercial marketplace, which demands ever increasing processing power, ever faster network speeds, ever larger storage, and ever cheaper prices. These demands force component manufacturers to constantly upgrade their products to remain competitive. However, this results in increasingly rapid obsolescence of their products. Once a component becomes obsolete, it becomes harder to obtain, and support for it by other component vendors disappears. The interrelationship between components means that a single obsolete component in a system will usually soon result in software incompatibility and an inability to update the system application software. In addition, a method must be in place to provide ongoing logistics support. For obsolete components, the only options are to buy large quantities of spares prior to shutdown of the manufacturing line or to repair failed components, a very expensive proposition for the inexpensive COTS products used. The BQQ-10 technology insertion process prevents these problems by eliminating obsolete components before they become unsupportable.
Establishing the Technology Insertion Process
One of the essential enablers for the BQQ-10 technology insertion process is the use of Multipurpose Transportable Middle ware (MTM) to isolate the system application software from the underlying hardware. Developed and still maintained by Digital Systems Resources (DSR), now a part of General Dynamics Advanced Information Systems, MTM is a freely licensed set of software utilities that provide a standardized interface between the application software and the various generations of system hardware. This design limits the impact of any hardware change to the MTM that was specifically designed to handle it instead of impacting the large amount of complex system application software. Without MTM, any hardware improvements provided by the technology insertion process would be un affordably due to the cost of updating the system software. With MTM, the BQQ-10 sonar system has been able to successfully upgrade the system hardware five times to reduce system cost and complexity and improve system performance.
The key parameter of the BQQ-10 technology insertion process is the two-year cycle length. What this means is that the hardware baseline for new system procurement and for updates of installed systems changes every two years. It does not mean that every installed system is updated every two years -an impossible task. There are several reasons for the choice of two years for the length of the technology insertion cycle. Procurement of new systems is driven by an annual budget cycle and it is advantageous to have a common hardware configuration for all systems procured in a given year. This results in a cycle based on intervals measured in years vice months. Finally, it takes time to evaluate new technology and to update the equipment configurations and system software to use the new technology. This task is too difficult and expensive to accomplish every year. Instead, it is better done on a less frequent basis where the effort and costs are spread out over more system procurement. However, as discussed earlier, the short COTS obsolescence cycle precludes an excessively long technology insertion cycle. The experience of the BQQ-10 program has been that COTS components are typically available for purchase for about eighteen months, fixing the maximum lifetime for a given hardware generation. The two-year cycle now used by the BQQ-10 program has been shown to provide the best balance between these competing factors.
The two-year technology insertion cycle used by the BQQ-10 program also supports meeting the operational requirements levied by the Chief of Naval Operations. The fundamental requirement levied is that every submarine deploys with an updated APB software build. This requirement coexists with a ground rule of the APB process that each hardware generation be capable of supporting three APB updates. The first APB is delivered as the initial software baseline for the hardware generation allowing the system to receive two additional APB updates. Under the typical eighteen-month submarine deployment cycle, this results in the ship deploying once with its original APB and then receiving an updated APB software-only update prior to its next deployment. Four to six years after its initial hardware installation, the ship would get a technology insertion to allow it to receive the latest APB prior to its next deployment. With the two-year technology insertion cycle, this results in a ship getting a hardware update every two or three generations, helping to keep the system procurement costs affordable and upgrade scheduling manageable.
For a two-year technology insertion cycle to work, the training, logistics, and contractor support infrastructures must be updated to support it. Instead of traditional crew training using shore-based sonar systems at the submarine training facilities, the BQQ-10 maintenance course is taught using Interactive Multimedia Instruction (IMI) where the system hardware is presented virtually and the student is taught common troubleshooting techniques and the skills to use the system technical documentation to find and fix faults. This troubleshooting technique must be supported by appropriate system level fault monitoring and localization functionality and an optimum lowest replaceable unit (LRU) selection. By sparing the COTS components at the relatively inexpensive box level, the technician does not need to troubleshoot to the lower board level, thereby reducing the time to correct faults. In addition, the system contractors’ hardware design, production, logistics support, and software development infrastructures must be set up to rapidly implement the design changes to maintain a two-year timeline. This effort initially required a significant paradigm shift for the system contractors and it is still a difficult task for the submarine planning yards that develop the installation packages.
Once the infrastructure has been updated, a process for selection of new components must be put in place. Over the last eight years, the BQQ-10 team has developed a process that de Ii very the maxi-mum capability while supporting system procurement timeline requirements. One of the unique attributes of the BQQ-I 0 team is that multiple organizations and hardware contractors work together to deliver the full system. The technology insertion process takes advantage of this broad based expertise by using an integrated product team (IPT) made up of members from each of the primary contractors and software developers to investigate new technology and make a consolidated recommendation to the program office. Involving each of the contractors in the hardware selection ensures that their specific concerns and requirements are addressed and that they buy in to the chosen technology. The process by which this IPT chooses the next generation of technology takes approximately twenty-four months and, therefore, under the two-year cycle, usually commences as soon as the first system is delivered from the previous generation.
The Steps in the Process
The first step in the technology insertion process is to establish performance requirements, cost goals, and system environmental limitations for the next generation of hardware. As part of this step, the previous hardware generations are looked at to identify where the technology has fallen short and where significant improvements could be made. In addition, the current hardware procurement costs are analyzed to determine where significant cost savings can be achieved. Knowing that the first procurement of the next generation technology won’t occur for about eighteen months, the IPT then conducts a market survey of current and upcoming technologies and selects candidate products for further investigation. Starting about twelve to eighteen months prior to delivery of the first system to the Navy, critical item performance testing of the candidate technologies is conducted as well as an evaluation of the impacts to system environmental characteristics (e.g., power, cooling, shock qualification) and system logistics. During this testing, the various system contractors port the system application software to the new hardware using MTM and verify that performance improvements are achievable. After conducting an iterative testing and evaluation process to tradeoff competing requirements, the IPT selects the most promising technologies for implementation. This is a very important part of the cycle as problems with potential candidates need to be discovered early and either resolved or the candidate technology eliminated from further evaluation. Final system testing of a fully built system is not the time to discover that a chosen technology has irresolvable issues as it is too close to Fleet delivery.
Approximately nine to twelve months prior to the date when the next generation system will be delivered to the Navy, the IPT makes its recommendations to the acquisition program office. Once the program office approves, the detailed design process starts where both cabinet physical design and system software porting is done. As the cabinet design progresses, testing continues to ensure all environmental requirements will be met. Design information is also provided to the submarine planning yard to start development of the Ship Alt package that will install the upgraded system. In addition, development of the logistical package is started to help ensure its readiness at the time of system delivery. Although most aspects of the next generation hardware are determined at the time of the IPT recommendation, certain attributes such as processor clock speed and disk drive capacity are deferred in order to better capture current technology when the major production orders are placed. These deferred attributes are limited to items that do not affect the cabinet design work in progress. The last minute attributes are specified six months prior to the first system delivery to the Navy when production orders are placed with the component vendors.
What We Have Accomplished and Lessons Learned
The BQQ-10 sonar has undergone five technology insertions since it was first introduced in 1997. As an example of the improvements that have been achieved, the display console processor was originally a HP744 VME card. In 1998, it was upgraded to a commercial HP J5OOO workstation to allow displaying the 3-D images used by the sonar system and, in 2000, further updated to an HP 15600 due to obsolescence issues. In 2002, the display processor technology was changed to a Dual 2.2 GHz Intel Xeon architecture using the Linux operating system. Finally, in 2004 the processor was again upgraded to a Dual 2.4 GHz Intel Xeon workstation. Although the 2004 processor variant has only a slightly higher clock speed than the 2002 processor, it is a more modem processor variant with other features that provide a higher throughput. However, it is not the fastest available processor because the shipboard power and cooling infrastructure is now at maximum capacity. The other processing components in the BQQ-10 system have also been updated as has the system network where obsolete Fiber Data Distribution Interface (FDDO networks and complex Asynchronous Transfer Mode (A TM) networks have been replaced by the commercially ubiquitous Gigabit Ethernet network.
The last eight years have not been without many lessons learned. An early lesson was that an ongoing technology insertion process is a cost of doing business with COTS and must be incorporated as an essential component of the program budget. Failing to provide periodic system upgrades will soon leave a submarine worse off than it was with the old legacy MIL-SPEC equipment due to the lack of support for the obsolete COTS components. The technology insertion process and the associated hardware procurement and shipboard installation processes must also be disciplined and streamlined to support the two-year technology insertion cycle. Failure to hold to the required timeline will result in last-minute impacts to the shipboard installation design and late equipment deliveries to the ship. Another lesson is that the system contractors must be incentivized to explore new technology and to use the best performing and most cost effective technology available while providing a method for the Navy to share with any cost savings. This is difficult to do in a fixed-price contract so the BQQ-10 program now uses cost plus award fee contracts that provide for sharing between the contractors and the Navy of the savings I costs associated with implementing new technology.
The technology insertion process has also identified that the existing submarine power and cooling infrastructure limits the further introduction of new hardware. The increase in heat given off by new processor chips has outstripped the ability of the ship’s fresh water cooling system to remove it. There is an initiative underway to upgrade the fresh water cooling system but it will be many years until all ships have been upgraded. In a parallel effort, advanced chip cooling methods are being investigated to allow for continued performance improvement.
Flexibility in component selection is also very important in the technology insertion process. Technology that seems like a good idea with an unlimited future one year may become a white elephant when abandoned by the commercial marketplace the next. The technology insertion team has to adapt and not feel locked into a previously chosen technology with no future. In the case of the BQQ-10, the original decision to go with modified 8-way Pentium III servers for the main signal processors was reversed in the next generation due to the high cost of the larger servers and the loss of procurement flexibility associated with being tied to a single vendor. Instead of these relatively expensive semi-custom servers, signal processing is now done on more mainstream Dual Xeon servers at about one-tenth of the cost.
Although the BQQ-10 technology insertion process has been successful to date, the future holds many challenges. The cost per server has leveled off and there will be little procurement savings available to fund the efforts to implement future technology generations. In addition, the total cost of the COTS components is now only a fraction of the overall system cost so any savings are small compared to the total system cost. Both of these developments mean that less complex technology insertions will be the norm and additional funding may be required to support cabinet design changes. As discussed above, shipboard power and cooling limitations must also be overcome.
Despite the lessons learned and the challenges ahead, the BQQ-10 technology insertion process has been a well-received success and is the model for nil submarine combat systems. The new AN/BYG-1 tactical control/weapons control system will use an identical technology insertion process and other submarine systems are in the process of leveraging the efforts of the BQQ-10 team. It is only by using this innovative process that the United States Navy will be able to maintain its superiority in anti-submarine warfare at an affordable cost. The men who sail in harm’s way deserve no less than the best we can provide them.
