The primary purpose of the process described and illustrated in this SERDP Exploratory Development (SEED) study is to confirm that the detection and classification decisions made prior to and during a remedial effort remain reasonable once ground truth information is garnered and taken into account. Specific objectives were to investigate (1) a practical method for assessing target detections and (2) procedures for determining the likelihood of false negatives given some amount of ground truth information.

The problem of unexploded ordnance (UXO) residual risk is naturally divided into the issues of target detection and, once detected, classification. With regards to the assessment of missed UXO detections, this study presents a practical and straightforward tool that involves inserting synthetic signatures, appropriately chosen for the UXO types actually recovered, into the reconnaissance survey data, processing using standard detection schemes and thresholds, and evaluating the results in the light of ground truth information. This approach naturally incorporates site-specific realistic noise levels and sampling schemes.

The second issue, that of classification failures, results when targets of interest (TOI) “look” more like what they are not than what they are, and are classified accordingly. In fact, because the decision metric is based on a library of expected TOI signatures, the target does not look “enough like” a TOI to be considered TOI. To investigate the possibility of classification failures, this work (1) performed a probabilistic risk assessment using polarizabilities and ground truth information from Camp San Luis Obispo, Camp Butner, and Camp Beale; (2) conducted a cluster analysis to look for groups of targets with similar signatures; and (3) calculated a decision metric based on clutter signatures and compared it with the decision metric based on the library of expected TOI signatures.

Based on a comparison of the actual depth distribution of the UXO recovered at San Luis Obispo and results of the synthetic seed study, this work concluded that all of the UXO, at least with regard to the four included in the study, were successfully detected. The deepest target known, based on the available dig results, was an 81mm at a depth of 0.52m. However, the synthetic seed exercise suggests that 100% of the 81mm mortar seeds would have been detected, using the same detection scheme, for burial depths of up to 0.77m. Thus, the detection process applied to ESTCP’s Classification Study at San Luis Obispo, California, was validated, given all available ground truth information resulting from excavations.

Using data and ground truth information from Pole Mountain, it was found that roughly half of the clutter would be dug at a threshold corresponding to a 1% residual risk of misclassified TOI, if extreme value theory is used to estimate the likelihood that one or more TOI may be misclassified. This application of statistics appears misguided. The mathematical formulations are correct but the logic is not. The outliers that dominate the risk calculations do not reflect random sample variations in the decision metric, but rather are due to inversion failures or other pathological events. To pursue the nature of the outliers, a cluster analyses and the calculation of a decision metric based on clutter signatures were used to show that, with the exception of a single TOI at Camp Beale that doesn’t match anything (a singleton), the TOI versus non-TOI classification decisions were reasonable.

Although assumptions can be made based on land use, expected munitions, expected clutter, and site conditions prior to conducting a geophysical survey, significantly more information is available after the geophysical survey and characterization is complete.

Information about the site is learned during all phases of the effort; from the reconnaissance survey, to the anomaly characterizations, through the classification, and finally, when ground truth information is gathered. This additional information can and should be exploited, using the processes and methods presented in this study, to re-evaluate the detection thresholds and classification decisions prior to site closure. By adopting such procedures, stakeholder confidence and acceptance can be increased.