PROGRAMProf. Tomek Strzalkowski, Principal Investigator
Department of Computer Science,
University at Albany, SUNY
Albany, NY 12222.
Email: tomek@cs.albany.edu
Phone: 518-442-4275; Fax: 518-442-5638
Summary
HITIQA (High-Quality Interactive Question Answering) is currently being developed to assist analysts in finding answers to complex intelligence problems, efficiently and thoroughly. The system uses event-based, data-driven semantic processing and natural language dialogue, coupled with an advanced information visualization interface, to deliver accurate answers to the analyst's questions along with related contextual information. The first version of the system has undergone a series of preliminary evaluations with analysts from the US Naval Reserve, producing valuable usage and performance data. These evaluations suggest that HITIQA creates a measurable cognitive augmentation effect for the analyst. The second more advanced version of the system is currently being demonstrated.
Summary of Project Accomplishments
This section summarizes the key accomplishments of the HITIQA research project funded by ARDA since the project inception in December 2001 until the end of May 2005. HITIQA has been developed at SUNY Albany in collaboration with Rutgers University as a subcontractor.
Intellectual Accomplishments
The AQUAINT program provides a unique environment to understand the information and technology needs of intelligence analysts and to capture this understanding in the system design. We believe HITIQA captures the dynamics of the analytical process to the extent greater than any existing factoid QA system. This is because HITIQA does not simply attempt to find a literal "answer" (whatever that may be); instead, it conducts multi-modal interactive dialogue with the users in order to help them rapidly assemble comprehensive reports addressing complex, analytical problems.
The HITIQA project has produced significant intellectual and scientific advances, including:
- Analytical question answering for complex, multi-faceted intelligence problems
- Multi-modal, problem solving human-computer dialogue
- Rapid extraction and structuring of event information from text
- Automated classification of content by aspects and information quality
- Novel methods for evaluation and introduction of advanced technology to the IC
System Capabilities
HITIQA is a new analytical question answering system that has been built entirely during the AQUAINT program. It is a fully automated interactive question answering system, which combines language-based dialogue and visual navigation of the information space.
HITIQA answers complex analytical questions, explanatory questions, and hypothetical questions - capabilities that are not available in factoid QA systems built before the AQUAINT program began. The answer is delivered in the form of a compact draft report, which includes excerpts of original text sources organized into topically labeled electronic folders.
HITIQA conducts multi-modal dialogue with the user, mediating between the analyst's information need and the available data in order to converge on the desired answer. The system anticipates and explores related, adjunct information, allowing the analysts to rapidly assemble comprehensive reports. Robust, data-driven semantics for natural language questions allows the system to operate efficiently in both open and specialized domains. We have developed novel methods for abstract visualization of open-domain information, in such as a way as to support the user in efficient exploration of the answer space. Furthermore, the visualization has been integrated with the language-based dialogue, allowing for multi-modal interaction (Figure 1).
Figure 1: HITIQA Multi-modal Interaction Panel
In addition to returning topical information, HITIQA assesses the quality of information sources and alerts the analyst if the data might be questionable in some respects. For complex topics, the system also classifies information into aspectual facets such as political, military, financial, etc., to further speed up answer comprehension.
HITIQA is a fully implemented prototype system, now available in version 2 (HITIQA-2).
IC Deployment Potential
The HITIQA team has been working very closely with intelligence analysts to assure that the technical development matches the needs and expectations of the IC users. HITIQA has received a strong positive response from a diverse group of IC analysts. The system's prototype has been installed at MITRE, SAIC, PNNL, and AFRL.
We have also established a series of on-line drills with USNR analysts from across the country. These analysts access HITIQA on-line via internet and conduct simulated analytic drills on a regular basis. This process increases analysts' exposure to the technology and provides valuable feedback to the developers. Thus far the response to HITIQA has been very positive, and these drills will continue with increased intensity involving more analysts from different IC agencies.
Analytical drills are monitored through novel evaluation instruments that collect usage data and analysts' assessments of the technology acceptability and transferability to everyday use. Our user-centered evaluation methodology has been successfully tested in multi-day sessions with active duty analysts in 2003 at Albany, NY, and in 2004 at the ARDA-sponsored 2004 challenge workshop at Richland, WA. The assessment of HITIQA technology ranks ahead of other currently available analytical tools (Figure 2).
Figure 2: "...the system as it stands now, in my mind, gave me enough information to try to put together an 80% solution..." (Analyst 2, Tape 7, 10/03)
Further tests indicate that HITIQA provides measurable cognitive augmentation for the analysts by helping them to produce better reports in less time and using fewer resources than currently practiced. Figure 3 shows that preparing a quality intelligence report with HITIQA requires 50% less time than it would take using Google (yellow bars). Furthermore, during this time fewer questions need to be asked of HITIQA (red bars) because more of them are turning up usable intelligence (blue bars). Specifically, 66% of HITIQA questions are productive vs. only 35% for Google. This makes HITIQA interactive approach nearly twice as efficient as Google search
Figure 3: HITIQA cognitive augmentation effect (06/04).
More information about HITIQA can be found in some 20 publications including a recent paper presented at the International Conference on Intelligence Analysis, cited below.
Strzalkowski, T., S. Small, H. Hardy, B. Yamrom, T. Liu, P. Kantor, K.B. Ng, N. Wacholder. HITIQA: A Question Answering Analytical Tool, Proceedings of International Conf. On Intelligence Analysis, McLean, VA. May 2-4, 2005.
Click here to Download Paper: https://analysis.mitre.org/proceedings