Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024

Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor 2024-05 ELRA and ICCL Torino, Italia text conference publication humeval-2024-human https://aclanthology.org/2024.humeval-1.0/ Quality and Quantity of Machine Translation References for Automatic Metrics Vilém Zouhar author Ondřej Bojar author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication zouhar-bojar-2024-quality https://aclanthology.org/2024.humeval-1.1/ 2024-05 1 11 Exploratory Study on the Impact of English Bias of Generative Large Language Models in Dutch and French Ayla Rigouts Terryn author Miryam de Lhoneux author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication rigouts-terryn-de-lhoneux-2024-exploratory https://aclanthology.org/2024.humeval-1.2/ 2024-05 12 27 Adding Argumentation into Human Evaluation of Long Document Abstractive Summarization: A Case Study on Legal Opinions Mohamed Elaraby author Huihui Xu author Morgan Gray author Kevin Ashley author Diane Litman author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication elaraby-etal-2024-adding https://aclanthology.org/2024.humeval-1.3/ 2024-05 28 35 A Gold Standard with Silver Linings: Scaling Up Annotation for Distinguishing Bosnian, Croatian, Montenegrin and Serbian Aleksandra Miletić author Filip Miletić author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication miletic-miletic-2024-gold https://aclanthology.org/2024.humeval-1.4/ 2024-05 36 46 Insights of a Usability Study for KBQA Interactive Semantic Parsing: Generation Yields Benefits over Templates but External Validity Remains Challenging Ashley Lewis author Lingbo Mo author Marie-Catherine de Marneffe author Huan Sun author Michael White author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication lewis-etal-2024-insights https://aclanthology.org/2024.humeval-1.5/ 2024-05 47 62 Extrinsic evaluation of question generation methods with user journey logs Elie Antoine author Eléonore Besnehard author Frederic Bechet author Geraldine Damnati author Eric Kergosien author Arnaud Laborderie author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication antoine-etal-2024-extrinsic https://aclanthology.org/2024.humeval-1.6/ 2024-05 63 70 Towards Holistic Human Evaluation of Automatic Text Simplification Luisa Carrer author Andreas Säuberli author Martin Kappus author Sarah Ebling author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication carrer-etal-2024-towards https://aclanthology.org/2024.humeval-1.7/ 2024-05 71 80 Decoding the Metrics Maze: Navigating the Landscape of Conversational Question Answering System Evaluation in Procedural Tasks Alexander Frummet author David Elsweiler author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication frummet-elsweiler-2024-decoding https://aclanthology.org/2024.humeval-1.8/ 2024-05 81 90 The 2024 ReproNLP Shared Task on Reproducibility of Evaluations in NLP: Overview and Results Anya Belz author Craig Thomson author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication belz-thomson-2024-2024 https://aclanthology.org/2024.humeval-1.9/ 2024-05 91 105 Once Upon a Replication: It is Humans’ Turn to Evaluate AI’s Understanding of Children’s Stories for QA Generation Andra-Maria Florescu author Marius Micluta-Campeanu author Liviu P Dinu author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication florescu-etal-2024-upon https://aclanthology.org/2024.humeval-1.10/ 2024-05 106 113 Exploring Reproducibility of Human-Labelled Data for Code-Mixed Sentiment Analysis Sachin Sasidharan Nair author Tanvi Dinkar author Gavin Abercrombie author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication sasidharan-nair-etal-2024-exploring https://aclanthology.org/2024.humeval-1.11/ 2024-05 114 124 Reproducing the Metric-Based Evaluation of a Set of Controllable Text Generation Techniques Michela Lorandi author Anya Belz author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication lorandi-belz-2024-reproducing https://aclanthology.org/2024.humeval-1.12/ 2024-05 125 131 ReproHum: #0033-03: How Reproducible Are Fluency Ratings of Generated Text? A Reproduction of August et al. 2022 Emiel van Miltenburg author Anouck Braggaar author Nadine Braun author Martijn Goudbeek author Emiel Krahmer author Chris van der Lee author Steffen Pauws author Frédéric Tomas author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication van-miltenburg-etal-2024-reprohum https://aclanthology.org/2024.humeval-1.13/ 2024-05 132 144 ReproHum #0927-03: DExpert Evaluation? Reproducing Human Judgements of the Fluency of Generated Text Tanvi Dinkar author Gavin Abercrombie author Verena Rieser author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication dinkar-etal-2024-reprohum https://aclanthology.org/2024.humeval-1.14/ 2024-05 145 152 ReproHum #0927-3: Reproducing The Human Evaluation Of The DExperts Controlled Text Generation Method Javier González-Corbelle author A Vivel-Couso author J M Alonso-Moral author A Bugarín-Diz author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication gonzalez-corbelle-etal-2024-reprohum https://aclanthology.org/2024.humeval-1.15/ 2024-05 153 162 ReproHum #1018-09: Reproducing Human Evaluations of Redundancy Errors in Data-To-Text Systems Filip Klubička author John D Kelleher author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication klubicka-kelleher-2024-reprohum https://aclanthology.org/2024.humeval-1.16/ 2024-05 163 198 ReproHum#0043: Human Evaluation Reproducing Language Model as an Annotator: Exploring Dialogue Summarization on AMI Dataset Vivian Fresen author Mei-Shin Wu-Urbanek author Steffen Eger author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication fresen-etal-2024-reprohum https://aclanthology.org/2024.humeval-1.17/ 2024-05 199 209 ReproHum #0712-01: Human Evaluation Reproduction Report for “Hierarchical Sketch Induction for Paraphrase Generation” Mohammad Arvan author Natalie Parde author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication arvan-parde-2024-reprohum https://aclanthology.org/2024.humeval-1.18/ 2024-05 210 220 ReproHum #0712-01: Reproducing Human Evaluation of Meaning Preservation in Paraphrase Generation Lewis N Watson author Dimitra Gkatzia author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication watson-gkatzia-2024-reprohum https://aclanthology.org/2024.humeval-1.19/ 2024-05 221 228 ReproHum #0043-4: Evaluating Summarization Models: investigating the impact of education and language proficiency on reproducibility Mateusz Lango author Patricia Schmidtova author Simone Balloccu author Ondrej Dusek author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication lango-etal-2024-reprohum https://aclanthology.org/2024.humeval-1.20/ 2024-05 229 237 ReproHum #0033-3: Comparable Relative Results with Lower Absolute Values in a Reproduction Study Yiru Li author Huiyuan Lai author Antonio Toral author Malvina Nissim author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication li-etal-2024-reprohum https://aclanthology.org/2024.humeval-1.21/ 2024-05 238 249 ReproHum #0124-03: Reproducing Human Evaluations of end-to-end approaches for Referring Expression Generation Saad Mahamood author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication mahamood-2024-reprohum https://aclanthology.org/2024.humeval-1.22/ 2024-05 250 254 ReproHum #0087-01: Human Evaluation Reproduction Report for Generating Fact Checking Explanations Tyler Loakman author Chenghua Lin author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication loakman-lin-2024-reprohum https://aclanthology.org/2024.humeval-1.23/ 2024-05 255 260 ReproHum #0892-01: The painful route to consistent results: A reproduction study of human evaluation in NLG Irene Mondella author Huiyuan Lai author Malvina Nissim author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication mondella-etal-2024-reprohum https://aclanthology.org/2024.humeval-1.24/ 2024-05 261 268 ReproHum #0087-01: A Reproduction Study of the Human Evaluation of the Coverage of Fact Checking Explanations Mingqi Gao author Jie Ruan author Xiaojun Wan author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication gao-etal-2024-reprohum https://aclanthology.org/2024.humeval-1.25/ 2024-05 269 273 ReproHum #0866-04: Another Evaluation of Readers’ Reactions to News Headlines Zola Mahlaza author Toky Hajatiana Raboanary author Kyle Seakgwa author C Maria Keet author 2024-05 text Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 Simone Balloccu editor Anya Belz editor Rudali Huidrom editor Ehud Reiter editor Joao Sedoc editor Craig Thomson editor ELRA and ICCL Torino, Italia conference publication mahlaza-etal-2024-reprohum https://aclanthology.org/2024.humeval-1.26/ 2024-05 274 280