Early Study of LLM Implementation in Survey Interviews
DOI:
https://doi.org/10.34123/jurnalasks.v17i1.792Keywords:
ChatGPT, Interview, LLM, Official Statistics, SurveyAbstract
Introduction/Main Objectives: This research aims to conduct a preliminary study into the use of LLMs for extracting information to fill out questionnaires in survey interviews. Background Problems: BPS-Statistics Indonesia used paper-based questionnaires for interviews and is recently utilizing the Computer Assisted Personal Interviewing (CAPI) method. However, the CAPI method has some drawbacks. Enumerators must input data into the device, which can be burdensome and prone to errors. Novelty: This study uses a large language model (LLM) to extract information from survey interviews. Research Methods: This study utilizes a text-to-speech application to translate interview results into text. Translation accuracy is measured by the Word Error Rate (WER). Then the text was extracted using the ChatGPT 3.5 Turbo model. GPT-3.5 Turbo is part of the GPT family of algorithms developed by OpenAI. Finding/Results: The extraction results are formatted into a JSON file, which is intended to be used for automatic filling into the database and then evaluated using precision, recall, and F1-score. Based on research conducted by utilizing the Speech Recognition API by Google and the ChatGPT 3.5 Turbo model, an average WER of 10% was obtained in speech recognition and an average accuracy of 76.16% in automatic data extraction.
Downloads
References
UNECE HLG-MOS, Large Language Models for Official Statistics, United Nations Economic Commission for Europe – High Level Group on Modernisation of Official Statistics, Dec. 2023. https://unece.org/.
R. Peng, K. Liu, P. Yang, Z. Yuan, and S. Li, “Embedding-based retrieval with LLM for effective agriculture information extracting from unstructured data,” arXiv preprint, arXiv:2308.03107 [cs.AI], 2023.
J. Gao, H. Zhao, C. Yu, and R. Xu, “Exploring the feasibility of ChatGPT for event extraction,” arXiv preprint, arXiv:2303.03836, 2023.
Pemerintah Republik Indonesia, Undang-Undang Republik Indonesia Nomor 16 Tahun 1997 tentang Statistik [Law of the Republic of Indonesia Number 16 of 1997 concerning Statistics], 1997
BPS, Pemanfaatan Aplikasi CAPI (Berbasis Android) dalam Pendataan Updating PODES 2020 [Utilization of CAPI (Android-based) Application in Data Collection for Updating PODES 2020], 2020. https://bukittinggikota.bps.go.id/news/2020/07/15/41/pemanfaatan-aplikasi-capi--berbasis-android--dalam-pendataan-updating-podes-2020.html
T. Takdir, “Analisis Kinerja, Kualitas Data, dan Usability pada Penggunaan CAPI untuk Kegiatan Sensus/Survey,” Jurnal Aplikasi Statistika & Komputasi Statistik, vol. 10, no. 1, pp. 9–26, 2018.
J. K. Höhne, K. Gavras, and J. Claassen, “Typing or Speaking? Comparing Text and Voice Answers to Open Questions on Sensitive Topics in Smartphone Surveys,” Social Science Computer Review, vol. 08944393231160961, 2022.
W. Wicara, Meningkatkan Produktivitas dengan Menggunakan STT [Enhancing Productivity by Using STT], 2023. https://widyawicara.com/meningkatkan-produktivitas-dengan-menggunakan-speech-to-text/
Y. Feng, “Intelligent speech recognition algorithm in multimedia visual interaction via BiLSTM and attention mechanism,” Neural Computing and Applications, vol. 36, no. 5, pp. 2371–2383, 2024.
T. Lenzner and J. K. Höhne, “Who is willing to use audio and voice inputs in smartphone surveys, and why?,” International Journal of Market Research, vol. 64, no. 5, pp. 594–610, 2022.
J. Gao, H. Zhao, C. Yu, and R. Xu, “Exploring the feasibility of chatgpt for event extraction,” arXiv preprint, arXiv:2303.03836, 2023.
C. Dinata, D. Puspitaningrum, and E. Erna, “Implementasi Teknik Dynamic Time Warping (DTW) pada Aplikasi Speech To Text [Implementation of Dynamic Time Warping (DTW) Technique in Speech-to-Text Applications],” Jurnal Teknik Informatika, vol. 10, no. 1, pp. 49–58, 2017, doi: 10.15408/jti.v10i1.6816.
I. K. S. Buana, “Implementasi Aplikasi Speech to Text untuk Memudahkan Wartawan Mencatat Wawancara dengan Python [Implementation of Speech-to-Text Application to Facilitate Journalists in Recording Interviews using Python],” Jurnal Sistem Dan Informatika (JSI), vol. 14, no. 2, pp. 135–142, 2020, doi: 10.30864/jsi.v14i2.293.
F. Adnan and I. Amelia, “Implementasi Voice Recognition Berbasis Machine Learning [Implementation of Machine Learning-Based Voice Recognition],” Edu Elektrika Journal, vol. 11, no. 1, pp. 24–29, 2022.
G. Gartlehner et al., “Data Extraction for Evidence Synthesis Using a Large Language Model: A Proof-of-Concept Study,” medRxiv, 2023-10, 2023
Y. Zou et al., “ESGReveal: An LLM-based approach for extracting structured data from ESG reports,”arXiv preprint, arXiv:2312.17264, 2023
T. Ricketts, Speech Recognition Application With Tone Analyzer (Doctoral dissertation). Alabama Agricultural and Mechanical University, 2023
E. Vidal, “Finite-State Speech-to-Speech Translation,” in Proceedings of the International Conference on Acoustics, Speech and Signal Processing, Munich, Germany, 1997.
S. Ozdemir, Quick Start Guide to Large Language Models: Strategies and Best Practices for Using ChatGPT and Other LLMs. Addison-Wesley Professional, October, 2023.
H. Dalianis, “Evaluation Metrics and Evaluation,” in Clinical Text Mining, Springer, Cham, 2018, doi: 10.1007/978-3-319-78503-5_6.
