HPKB lessons learned

From: Adam Pease (apease@ks.teknowledge.com)
Date: 01/30/02


I sent this out Monday but it bounced, so I'm resending.  Sorry for the delay.


Folks,
   I volunteered to provide a few references of "lessons learned" from HPKB 
that might be relevant to the DAML experiment.

- Example metrics.  A key point that Paul Cohen made during HPKB is that 
many research efforts in AI are just "existence proofs" i.e. did the 
program work or not.  This is very different from quantitative metrics 
about how well or how fast a system worked.  Key HPKB metrics were 
questions answered correctly in particular domain (that was well-specified 
and scoped in an experiment document of nearly 100 pages for the final 
version) as well as rates of knowledge creation.

- Bob Shrag's final HPKB presentation gives a good set of example test 
questions, scores and 
metrics. 
<http://reliant.teknowledge.com/HPKB/meetings/FinalPImeeting/Schrag-Results/sld001.htm>

- Level of detail.  I think there was some consensus during the telecon 
that although the work done to date to specify the experiment was necessary 
and helpful, it was not at a sufficient level of detail to ensure that the 
experiment isn't either (1) very hard, "AI-complete" or open-ended or (2) a 
trivial thing that could be done in Java or with a relational 
database.  The HPKB Crisis Management spec is a good example of the level 
of detail and scoping needed in an experiment spec that researchers can act 
on.  A full set of those materials can be found at 
<http://www.iet.com/Projects/HPKB/Y2/>

Additional material in the same vein for the RKF project can be found at 
<http://www.iet.com/Projects/RKF/>

Adam


At 12:42 PM 1/28/2002 -0500, John Flynn wrote:

>All,
>
>
>
>This is a reminder that the teleconference to discuss the DAML Experiment 
>is scheduled for today at 3:00 pm East Coast time. A briefing to support 
>these discussions has been posted at:
>
>
>
><http://www.daml.org/JF>http://www.daml.org/JF
>
>
>
>Dial in info:
>
>Domestic: 1-800-403-2024
>
>International: 865-673-6716
>
>Participant code: 671932
>
>
>
>Thanks,
>
>
>
>John
>
>
>
>John Flynn
>
>(703) 284-4612
>
>DAML Integration and Transition PM
>
>BBN Technologies
>
>

Adam Pease
Teknowledge
(650) 424-0500 x571


This archive was generated by hypermail 2.1.4 : 03/26/03 EST