Assessing Target Search Technology

Fall 2005

With the advance of satellite technology and near real-time sensor data from unmanned air vehicles, military commanders increasingly rely on digital imagery to help achieve their missions.

In the past, human operators searched large volumes of data produced by these technologies to detect significant enemy activity and to support targeting missions. More recently, automated target detection (ATD) is being used to aid the military with these tasks.

Although automated target detection technology offers the potential to process and exploit large volumes of sensor data, a major challenge is to develop sound methods to evaluate ATD and human observer performance for search problems.

In his SAIC Executive Science & Technology Council (ESTC) Award-winning article, SAIC's John Irvine presents a method for evaluating search performance that applies to both target detection algorithms and human observers.

The method

The method ― free response operating characteristic (FROC) model ― generalizes a standard method (known as the receiver operating characteristic or ROC) to search tasks. Used when multiple detections and multiple "false alarms" are a possibility, the FROC model quantifies the tradeoffs between target detection probability and false alarm rate. Receiver operating characteristic analysis has been applied to a variety of signal detection problems in medical imaging and other areas. However, the receiver operating characteristic model breaks down for search tasks where multiple detections and false alarms are possible. Search tasks of this nature arise in numerous signal detection settings besides automated target detection; this includes detecting the location of lung nodules in radiographs and detecting submarines from SONAR.

In addition, free response operating characteristic uses a two-stage approach for characterizing search performance for general searches (when an image contains multiple possibilities for targets and false alarms). The first stage models the opportunities for false alarms, which you can think of as separate bits of clutter that appear to be targets. Given this model of false alarm opportunities, the second stage describes the separability between targets and false alarms.

A general statistical framework

In conclusion, free response operating characteristic provides a general statistical framework that models search performance and provides a comparison across observers and automated target detection algorithms.

Irvine's paper, "Assessing target search performance: the free-response operator characteristic model," was published in Optical Engineering.

(Irvine directs imagery systems at SAIC and is also the Senior Scientist for the National Geospatial-Intelligence Agency's STAR Program, where he is responsible for development, evaluation, and prototyping of technology for automated and semi-automated fusion and exploitation of imagery data.)

*Note: PDF documents are viewed using Acrobat® Reader®.