Prompt Engineering of GPT-3.5 for Extracting Signs and Symptoms and Their Assertions in Maternal Electronic Health Records: A Pilot Study

Dylan Montoya-Lora; Andrés F. Giraldo-Forero; Ever A. Torres-Silva; Santiago Rua; Sara Arango-Valencia; Juan G. Barrientos-Gómez; Jose F. Florez-Arango; Andrés Orozco-Duque

doi:10.1007/978-3-031-89514-2_5

Prompt Engineering of GPT-3.5 for Extracting Signs and Symptoms and Their Assertions in Maternal Electronic Health Records: A Pilot Study

Dylan Montoya-Lora
, Andrés F. Giraldo-Forero
, Ever A. Torres-Silva
, Santiago Rua
, Sara Arango-Valencia
, Juan G. Barrientos-Gómez
, Jose F. Florez-Arango
, Andrés Orozco-Duque

Public Health

Research output: Chapter in Book/Conference proceeding › Conference and proceedings › peer-review

2 Scopus citations

Abstract

Extracting Named Entity Recognition (NER) associated with signs and symptoms, diseases and syndromes, or health care activities from raw text in maternal electronic health records (EHR) can help in the management of high maternal risk. In addition, identify if the entity is present, absent, or is only a possibility is important to avoid misinterpretation. In this paper, we present an implementation of an automatic information extraction scheme from text in EHR oriented to detect entities and assertions in maternal electronic health records using GPT-3.5 turbo through a few shot architecture. Additionally, we developed a prompt-based approach to interact with the model and obtain NER responses in JSON format, enabling integration with existing healthcare systems and workflows. Our results show an F1-score of 0.75 for NER and 0.76 for assertions. These results show that large languaje models can be used to extract information from EHRs which can improve patient follow-up.

Original language	English
Title of host publication	X Latin American Conference on Biomedical Engineering - Proceedings of CLAIB 2024
Editors	Fabiola M. Martinez-Licona, Virginia L. Ballarin, Ernesto A. Ibarra-Ramírez, Sandra M. Pérez-Buitrago, Luis R. Berriere
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	53-62
Number of pages	10
ISBN (Print)	9783031895135
DOIs	https://doi.org/10.1007/978-3-031-89514-2_5
State	Published - 2025
Event	10th Latin American Conference on Biomedical Engineering, CLAIB 2024 - Panama City, Panama Duration: 2 Oct 2024 → 5 Oct 2024

Publication series

Name	IFMBE Proceedings
Volume	121
ISSN (Print)	1680-0737
ISSN (Electronic)	1433-9277

Conference

Conference	10th Latin American Conference on Biomedical Engineering, CLAIB 2024
Country/Territory	Panama
City	Panama City
Period	2/10/24 → 5/10/24

Bibliographical note

Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

Keywords

Electronic Health Records
Large Language Models
Named Entity Recognition
Retrieval Augmented Generation

Types Minciencias

Scientific events with a public engagement component

Access to Document

10.1007/978-3-031-89514-2_5

Cite this

Montoya-Lora, D., Giraldo-Forero, A. F., Torres-Silva, E. A., Rua, S., Arango-Valencia, S., Barrientos-Gómez, J. G., Florez-Arango, J. F., & Orozco-Duque, A. (2025). Prompt Engineering of GPT-3.5 for Extracting Signs and Symptoms and Their Assertions in Maternal Electronic Health Records: A Pilot Study. In F. M. Martinez-Licona, V. L. Ballarin, E. A. Ibarra-Ramírez, S. M. Pérez-Buitrago, & L. R. Berriere (Eds.), X Latin American Conference on Biomedical Engineering - Proceedings of CLAIB 2024 (pp. 53-62). (IFMBE Proceedings; Vol. 121). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-89514-2_5

Montoya-Lora, Dylan ; Giraldo-Forero, Andrés F. ; Torres-Silva, Ever A. et al. / Prompt Engineering of GPT-3.5 for Extracting Signs and Symptoms and Their Assertions in Maternal Electronic Health Records : A Pilot Study. X Latin American Conference on Biomedical Engineering - Proceedings of CLAIB 2024. editor / Fabiola M. Martinez-Licona ; Virginia L. Ballarin ; Ernesto A. Ibarra-Ramírez ; Sandra M. Pérez-Buitrago ; Luis R. Berriere. Springer Science and Business Media Deutschland GmbH, 2025. pp. 53-62 (IFMBE Proceedings).

@inproceedings{65ae97529a7a483baa76bbe4166acf26,

title = "Prompt Engineering of GPT-3.5 for Extracting Signs and Symptoms and Their Assertions in Maternal Electronic Health Records: A Pilot Study",

abstract = "Extracting Named Entity Recognition (NER) associated with signs and symptoms, diseases and syndromes, or health care activities from raw text in maternal electronic health records (EHR) can help in the management of high maternal risk. In addition, identify if the entity is present, absent, or is only a possibility is important to avoid misinterpretation. In this paper, we present an implementation of an automatic information extraction scheme from text in EHR oriented to detect entities and assertions in maternal electronic health records using GPT-3.5 turbo through a few shot architecture. Additionally, we developed a prompt-based approach to interact with the model and obtain NER responses in JSON format, enabling integration with existing healthcare systems and workflows. Our results show an F1-score of 0.75 for NER and 0.76 for assertions. These results show that large languaje models can be used to extract information from EHRs which can improve patient follow-up.",

keywords = "Electronic Health Records, Large Language Models, Named Entity Recognition, Retrieval Augmented Generation",

author = "Dylan Montoya-Lora and Giraldo-Forero, \{Andr{\'e}s F.\} and Torres-Silva, \{Ever A.\} and Santiago Rua and Sara Arango-Valencia and Barrientos-G{\'o}mez, \{Juan G.\} and Florez-Arango, \{Jose F.\} and Andr{\'e}s Orozco-Duque",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.; 10th Latin American Conference on Biomedical Engineering, CLAIB 2024 ; Conference date: 02-10-2024 Through 05-10-2024",

year = "2025",

doi = "10.1007/978-3-031-89514-2\_5",

language = "Ingl{\'e}s",

isbn = "9783031895135",

series = "IFMBE Proceedings",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "53--62",

editor = "Martinez-Licona, \{Fabiola M.\} and Ballarin, \{Virginia L.\} and Ibarra-Ram{\'i}rez, \{Ernesto A.\} and P{\'e}rez-Buitrago, \{Sandra M.\} and Berriere, \{Luis R.\}",

booktitle = "X Latin American Conference on Biomedical Engineering - Proceedings of CLAIB 2024",

}

Montoya-Lora, D, Giraldo-Forero, AF, Torres-Silva, EA, Rua, S, Arango-Valencia, S, Barrientos-Gómez, JG, Florez-Arango, JF & Orozco-Duque, A 2025, Prompt Engineering of GPT-3.5 for Extracting Signs and Symptoms and Their Assertions in Maternal Electronic Health Records: A Pilot Study. in FM Martinez-Licona, VL Ballarin, EA Ibarra-Ramírez, SM Pérez-Buitrago & LR Berriere (eds), X Latin American Conference on Biomedical Engineering - Proceedings of CLAIB 2024. IFMBE Proceedings, vol. 121, Springer Science and Business Media Deutschland GmbH, pp. 53-62, 10th Latin American Conference on Biomedical Engineering, CLAIB 2024, Panama City, Panama, 2/10/24. https://doi.org/10.1007/978-3-031-89514-2_5

Prompt Engineering of GPT-3.5 for Extracting Signs and Symptoms and Their Assertions in Maternal Electronic Health Records: A Pilot Study. / Montoya-Lora, Dylan; Giraldo-Forero, Andrés F.; Torres-Silva, Ever A. et al.
X Latin American Conference on Biomedical Engineering - Proceedings of CLAIB 2024. ed. / Fabiola M. Martinez-Licona; Virginia L. Ballarin; Ernesto A. Ibarra-Ramírez; Sandra M. Pérez-Buitrago; Luis R. Berriere. Springer Science and Business Media Deutschland GmbH, 2025. p. 53-62 (IFMBE Proceedings; Vol. 121).

Research output: Chapter in Book/Conference proceeding › Conference and proceedings › peer-review

TY - GEN

T1 - Prompt Engineering of GPT-3.5 for Extracting Signs and Symptoms and Their Assertions in Maternal Electronic Health Records

T2 - 10th Latin American Conference on Biomedical Engineering, CLAIB 2024

AU - Montoya-Lora, Dylan

AU - Giraldo-Forero, Andrés F.

AU - Torres-Silva, Ever A.

AU - Rua, Santiago

AU - Arango-Valencia, Sara

AU - Barrientos-Gómez, Juan G.

AU - Florez-Arango, Jose F.

AU - Orozco-Duque, Andrés

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

PY - 2025

Y1 - 2025

N2 - Extracting Named Entity Recognition (NER) associated with signs and symptoms, diseases and syndromes, or health care activities from raw text in maternal electronic health records (EHR) can help in the management of high maternal risk. In addition, identify if the entity is present, absent, or is only a possibility is important to avoid misinterpretation. In this paper, we present an implementation of an automatic information extraction scheme from text in EHR oriented to detect entities and assertions in maternal electronic health records using GPT-3.5 turbo through a few shot architecture. Additionally, we developed a prompt-based approach to interact with the model and obtain NER responses in JSON format, enabling integration with existing healthcare systems and workflows. Our results show an F1-score of 0.75 for NER and 0.76 for assertions. These results show that large languaje models can be used to extract information from EHRs which can improve patient follow-up.

AB - Extracting Named Entity Recognition (NER) associated with signs and symptoms, diseases and syndromes, or health care activities from raw text in maternal electronic health records (EHR) can help in the management of high maternal risk. In addition, identify if the entity is present, absent, or is only a possibility is important to avoid misinterpretation. In this paper, we present an implementation of an automatic information extraction scheme from text in EHR oriented to detect entities and assertions in maternal electronic health records using GPT-3.5 turbo through a few shot architecture. Additionally, we developed a prompt-based approach to interact with the model and obtain NER responses in JSON format, enabling integration with existing healthcare systems and workflows. Our results show an F1-score of 0.75 for NER and 0.76 for assertions. These results show that large languaje models can be used to extract information from EHRs which can improve patient follow-up.

KW - Electronic Health Records

KW - Large Language Models

KW - Named Entity Recognition

KW - Retrieval Augmented Generation

UR - https://www.scopus.com/pages/publications/105004661625

U2 - 10.1007/978-3-031-89514-2_5

DO - 10.1007/978-3-031-89514-2_5

M3 - Ponencia publicada en las memorias del evento con ISBN

AN - SCOPUS:105004661625

SN - 9783031895135

T3 - IFMBE Proceedings

SP - 53

EP - 62

BT - X Latin American Conference on Biomedical Engineering - Proceedings of CLAIB 2024

A2 - Martinez-Licona, Fabiola M.

A2 - Ballarin, Virginia L.

A2 - Ibarra-Ramírez, Ernesto A.

A2 - Pérez-Buitrago, Sandra M.

A2 - Berriere, Luis R.

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 2 October 2024 through 5 October 2024

ER -

Montoya-Lora D, Giraldo-Forero AF, Torres-Silva EA, Rua S, Arango-Valencia S, Barrientos-Gómez JG et al. Prompt Engineering of GPT-3.5 for Extracting Signs and Symptoms and Their Assertions in Maternal Electronic Health Records: A Pilot Study. In Martinez-Licona FM, Ballarin VL, Ibarra-Ramírez EA, Pérez-Buitrago SM, Berriere LR, editors, X Latin American Conference on Biomedical Engineering - Proceedings of CLAIB 2024. Springer Science and Business Media Deutschland GmbH. 2025. p. 53-62. (IFMBE Proceedings). doi: 10.1007/978-3-031-89514-2_5