The ability to quickly find an expert or to establish an expert team is increasingly becoming a competitive advantage in our global economy (1). This article describes automated tools created for a global enterprise, The MITRE Corporation, that are used to discover and map corporate experts and areas of expertise within MITRE.

The Need to Find Experts

MITRE has thousands of staff and sponsors distributed across the globe working on solutions to pervasive, cross-organizational problems facing government in civil aviation, tax administration and national security. Expertise management is thus a core function in achieving MITRE's objectives of enabling innovation, integration and collaboration within and across public sector agencies.

Figure I illustrates our knowledge management process, in which teams initiate a discovery process for experts and knowledge, formulate expert teams if the knowledge they seek does not exist, then collaboratively create it and, finally, capture their knowledge to archive and disseminate it to the sponsor. Our efforts are aimed at providing full life-cycle knowledge management, to include web-based expert finding and knowledge discovery services, support of expert teams (e.g., via experts and expert team finding), tools for the creation of knowledge (e.g. team facilitation/collaboration), and delivery of knowledge to a consumer. We are also exploring new virtual organizational models that will permit the efficient distributed teaming of expert talent.

To enable effective distributed knowledge management, we employ a range of technologies, including global video teleconferencing, an award-winning MITRE Information Infrastructure (Mll) intranet, and public key infrastructure (PKI) enabled extranet services. Knowledge resources such as "ask the expert," on-line knowledge repositories of risk management experience (RAMP), and system engineering lessons learned (Systems Engineering Program Office) are also available. Tools for staff and project discovery, information sharing (e.g., transfer folders) and virtual place-based collaboration such as our Collaborative Virtual Workspace (cvw.mitre.org) are used for agile team formation and support.

As important as these services are to support expertise management, however, it has been just as important to cultivate a culture of collaboration, encapsulated in one of the three core corporate values: "people in partnership." If people are to collaborate, they need mechanisms for rapid team formulation, including the discovery of knowledgeable teammates and expert consultants. There are four basic functions an expert-finding service must support. These include characterizing expertise (i.e., the features), associating levels of expertise to particular experts (i.e., assigning feature values), locating experts (i.e., a search problem), and finally, selecting (using specific criteria consistent with a task need).

MITRE has explored the use of two prototype systems to support expertise discovery: Expert Finder and XperNet. Expert Finder is a dynamically-updated system that uses name extraction technology and other tools to exploit the intellectual products created within MITRE for automated expertise classification. XperNet addresses the problem of detecting extant or emerging areas of expertise without a priori knowledge of their existence. Both tools combine to detect and track experts and expert communities within a complex work environment.

Related Work

A variety of systems address the issue of finding expertise (2). Among them are many directory services that let people nominate themselves as experts in a chosen field (3-6 or services that compare new questions against previously-answered questions and find related experts ( 7, 8). Other systems automatically establish expertise through the analysis of sources such as email messages (9,10,19), USENET postings (11), and source code files (12). Most similar to the MITRE approach are systems that query citations, papers written, and other available information sources to determine expertise (13,14).

MITRE's approach is unique in combining a dynamic search/indexing system with a set of community-of-practice discovery and tracking tools. This approach also differs from previous systems by offering named entity extraction to find experts mentioned in unstructured text.

MITRE's Expert Finder

Expert Finder (15,16) attempts to overcome the limitations of manually-populated expert databases, which are difficult to maintain and can quickly become outdated. It dynamically mines expert publications and activities on the MII and provides end-users with results in an intuitive fashion. In addition to having low maintenance costs, this approach to finding expertise is also egalitarian in that it lets anyone be an expert based on their contributions to the system. This clear association between contributions and expertise rankings also makes it easy to find supporting documents to verify the expertise.

Figure 2 illustrates the system in action, in which a user is trying to find data mining experts at MITRE. Upon entering the search term "data mining," Expert Finder ranks employees based on the frequency with which a term or phrase co-occurs with an employee name either in corporate communications (e.g., newsletters) or based on what they have published in their resume or document folder (a shared, indexed information space of an employee's publications).

The documents used by Expert Finder fall into two main categories: documents about a topic that are published by an employee (these could be for internal or external audiences but are currently those that appear on MITRE's internal web) and documents that mention employees in conjunction with a particular topic. For the self published case, Expert Finder relies on the number of documents published by an employee about a given topic to provide an "expert score" for that employee. The only exception is the employee's resume, which is given additional weight as a self definition of the individual's expertise.

Managing the second class of documents, those that mention employees and topics, is more complicated. Whereas with the self-published documents there is an explicit linkage between the documents and employee (they are indexed by employee number), with documents that mention employees, this linkage must be derived from the underlying text. The first step after the documents have been returned from the search engine is to locate the proper names within the text using a name tagger (17). Once the names have been located, the next step is to associate them with the query topic. All the documents returned by the search engine contain the query string, but there are several distinct types of documents, and each type has a structure that must be exploited differently. After a document has been examined, the evidence gathered about each person found is combined into a single score for that person.

Overall, the results obtained by Expert Finder are encouraging. Our original goal was to place a user within one phone call of an expert. However, in the majority of the cases tested, reasonable "experts" are listed as the top three or four candidates, where the likelihood of randomly selecting a correct expert is the total number of experts divided by 4,500 total staff, often significantly less than a I -percent chance of getting any right. In experiments (18) comparing 10 technical human resource managers' performance with Expert Finder on five specialty areas (data mining, chemicals, human-computer interaction, network security, and collaboration), on average, human inter-subject agreement was not high (e.g., 63 percent of the time at least one of the manager's top five selections agreed, and this dropped to 11 percent if we measured agreement of three out of the top five).

We then measured Expert Finder precision, the degree to which a staff member found by Expert Finder is considered expert by humans, and recall, the degree to which a priori human-designated experts are found by the Expert Finder (i.e., the number of actual experts found divided by the total number of experts reported, and the number of actual experts found divided by the total number of experts, respectively). Over the five subjects, Expert Finder performed at approximately 40 percent precision and 30 percent recall. This performance was the result of a number of factors, primary among them being the lack of published data (e.g., resumes, documents) by some experts. Where data about experts were available, Expert Finder performance was typically 60 percent precision and 40 percent recall. Nevertheless, this exceeded the original goal of getting to an expert within one telephone call.

Forthcoming enhancements to Expert Finder include allowing users to offer feedback about the experts found by the system and allowing experts to indicate their availability to address questions in their field of expertise.

XperNet

In contrast to the query-based Expert Finder tool, MITRE's XperNet focuses on identification and tracking of expert communities using clustering and network analysis techniques (16). Networks of individuals with related skills and interests are created by processing information derived from staff projects, publications, personal web pages, and other work-related activities. Expertise indicators (e.g., explicit reference or citation, network centrality) as well as counter-indicators (e.g., being a member of the administrative staff) are used to assess a particular individual's level of expertise. The underlying algorithm uses adaptive clustering to create core clusters that form the basis for

automatically-expanded expertise networks generated using project and other organizational information. Because each individual or cluster member has an associated expertise indicator, as in Expert Finder individuals can be ranked according to their level of expertise or highlighted within a network visualization in which each node represents an individual and geographic locations are indicated by color coding.

As with the Expert Finder evaluation above, XperNet performance was benchmarked against human performance. User surveys were administered to approximately 10 percent of MITRE's Information Technology Center technical staff who identified experts from across the corporation. Nominated experts fell into four technology areas-collaboration, knowledge management, advanced instructional training, and language processing-roughly corresponding to four departments in the Center. Standard precision and recall measures were used to assess the overlap between the manual and automatically-generated networks. This was done at particular cutoff points since the surveys did not rank experts. Approximately 70 percent of the top 10 automatically-identified experts were in the manually-identified list. Precision dropped about 10 percent when computed over the top 20. Recall, at the top-10 cutoff, was also fairly high, but this is partly a function of the relatively small number of experts identified by humans. Looking at the top 30, approximately 75 percent of the experts were identified automatically with approximately 50 percent accuracy. These preliminary results are encouraging.

Research Focus

The MITRE expert and community finding tools represent a valuable first step toward achieving our objective of automated discovery and mapping of expertise. However, there are several system and user issues that require further investigation, including sources, processing and usage. Effective expert and expert team discovery raise a series of complex issues that our prototypes have only begun to explore. Currently there are more questions than answers. These include, What is an expert? How do we characterize various levels of expertise or competence? How do we authenticate or validate an expert? How do we characterize the evolution of what it means to be an expert or an individual's level of expertise over time? How can we systematically support this in a cost-effective manner? How can we evaluate expert recommender systems?

We are currently investigating a general, component-based expertise management architecture that supports a wide range of services including an expert registration service, a finder service, a qualification service, a selection service, and a question/answering service, that enable the management of expert knowledge and experts in an enterprise (16. We hope this will provide a basis from which we can discover answers to some of the above questions and facilitate the creation of more robust tools for automated expert discovery.

Acknowledgments

David House, David Mattox, Inderjeet Mani, Barbara Gates, and Chris Elsaesser developed MITRE's Expert Finder. Ray D'Amore and Manu Konchady developed MITRE's XperNet tools.