We use a few cookies

Essential cookies keep the library working. With your permission, we'd also like to use analytics to see what's useful and improve the experience. Read our Privacy Policy or Cookie Policy.

Skip to main content
Home/e-Resources/Artificial Intelligence & Machine Learning/Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again
Conference ProceedingsArtificial Intelligence & Machine Learning

Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again

2022

Checking your access to this record.

Abstract & Overview

Large pre-trained language models (PLMs) such as GPT-3 have shown strong in-context learning capabilities, which are highly appealing for domains such as biomedicine that feature high and diverse demands of language technologies but also high data annotation costs.In this paper, we present the first systematic and comprehensive study to compare the few-shot performance of GPT-3 in-context learning with fine-tuning smaller (i.e., BERT-sized) PLMs on two representative biomedical information extraction (IE) tasks: named entity recognition and relation extraction.We follow the true few-shot setting (Perez et al., 2021) to avoid overestimating models' few-shot performance by model selection over a large validation set.We also optimize GPT-3's performance with known techniques such as contextual calibration and dynamic in-context example retrieval.However, our results show that GPT-3 still significantly underperforms compared to simply fine-tuning a smaller PLM.In addition, GPT-3 in-context learning also yields smaller gains in accuracy when more training data becomes available.More in-depth analyses further reveal issues of in-context learning that may be detrimental to IE tasks in general.Given the high cost of experimenting with GPT-3, we hope our study provides helpful guidance for biomedical researchers and practitioners towards more practical solutions such as fine-tuning small PLMs before better in-context learning is available for biomedical IE. 1

Cite this record

Bernal Jiménez Gutiérrez, Nikolas McNeal, Clay Washington, You Chen, Lang Li, Huan Sun, et al. (2022). Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again. https://doi.org/10.18653/v1/2022.findings-emnlp.329

Copy in another style

Gutiérrez, B. J., McNeal, N., Washington, C., Chen, Y., Li, L., Sun, H., & Su, Y. (2022). Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again. https://doi.org/10.18653/v1/2022.findings-emnlp.329
Author names are used as deposited, so check their order before you submit.

Metadata is harvested from open-access indexes and can be incomplete. Check a citation against the published version before relying on it — see our open access & attribution policy.