Heuristic Sample Selection to Minimize Reference Standard Training Set for a Part-Of-Speech Tagger
Open Access
- 1 September 2007
- journal article
- research article
- Published by Oxford University Press (OUP) in Journal of the American Medical Informatics Association
- Vol. 14 (5), 641-650
- https://doi.org/10.1197/jamia.m2392
Abstract
Part-of-speech tagging represents an important first step for most medical natural language processing (NLP) systems. The majority of current statistically-based POS taggers are trained using a general English corpus. Consequently, these systems perform poorly on medical text. Annotated medical corpora are difficult to develop because of the time and labor required. We investigated a heuristic-based sample selection method to minimize annotated corpus size for retraining a Maximum Entropy (ME) POS tagger. We developed a manually annotated domain specific corpus (DSC) of surgical pathology reports and a domain specific lexicon (DL). We sampled the DSC using two heuristics to produce smaller training sets and compared the retrained performance against (1) the original ME modeled tagger trained on general English, (2) the ME tagger retrained on the DL, and (3) the MedPost tagger trained on MEDLINE abstracts. Results showed that the ME tagger retrained with a DSC was superior to the tagger retrained with the DL, and also superior to MedPost. Heuristic methods for sample selection produced performance equivalent to use of the entire training set, but with many fewer sentences. Learning curve analysis showed that sample selection would enable an 84% decrease in the size of the training set without a decrement in performance. We conclude that heuristic sample selection can be used to markedly reduce human annotation requirements for training of medical NLP systems.Keywords
This publication has 12 references indexed in Scilit:
- dTagger: a POS tagger.2006
- Domain-specific language models and lexicons for taggingJournal of Biomedical Informatics, 2005
- A comparison of parsing technologies for the biomedical domainNatural Language Engineering, 2005
- MedPost: a part-of-speech tagger for bioMedical textBioinformatics, 2004
- Evaluation of a Deidentification (De-Id) Software Engine to Share Pathology Reports and Clinical Documents for ResearchAmerican Journal of Clinical Pathology, 2004
- Evaluation of a Deidentification (De-Id) Software Engine to Share Pathology Reports and Clinical Documents for ResearchAmerican Journal of Clinical Pathology, 2004
- Extracting structured information from free text pathology reports.2003
- The sublanguage of cross-coverage.2002
- Evaluation of negation phrases in narrative clinical reports.2001
- Automatic Structuring of Radiology Free-Text ReportsRadioGraphics, 2001