Electronic health record data were accessed from ED visits from the University Children’s Hospital Zurich (UCHZ). All injury cases of children and adolescents aged 0 to 18, who were treated for injury between 1.1.2018 and 31.12.2022 at the UCHZ ED and for whom a general consent was obtained were included in the study. Data was anonymized by the UCHZ prior to the data transfer, ensuring that no personally identifiable data or identification codes, allowing data to be traced back to a patient, were included in the dataset. Ethics application was submitted to the Ethics Committee of the canton of Zurich (BASEC-Nr.: 2023-01947) and the project was approved.

Data include: Reason for treatment (injury vs. illness) recorded as a structured variable upon admission in the ED, further variables such as age (in years), sex, and month/year of arrival, as well as unstructured narrative data, such as a diagnosis and medical history text. To ensure data accuracy and consistency, a data cleaning process was conducted prior to analysis. The data cleaning involved exclusion of cases with repeated visits for the same injury (n = 4,270), missing key variables such as age, sex and month/year of treatment (n = 4), fell outside the observation period (n = 7), outside age range (n = 2) or duplicates (n = 6). The final dataset comprised 30,876 injury cases of children and adolescents aged 0–18 years, with a mean age of 6.9 years (SD 4.5).

This study followed a structured iterative process for the model development that involved various steps (see Figure 1) and adopted a cyclical workflow, enabling stepwise refinement through continuous learning and evaluation. The iterative process facilitated dynamic adaption to new insights based on evaluation and testing, addressed challenges as they emerged, and progressively enhanced study outcome.

The internationally established injury coding system EU-IDB was chosen to label injury-relevant information [18]. It has a hierarchical structure, in which each variable can be further specified in up to two or three levels of labels (e.g., variable place of occurrence - Level 1 private living area - Level 2 kitchen) [18]. It covers all age groups including relevant injury labels for children and adolescents. After the initial manual annotation phase and preliminary classification results, the coding system was substantially reduced and adapted to better reflect injury patterns in children and adolescents, such as reducing the number of labels and adding child-specific labels, reducing the labels by a third from a total of 1187 to 574. The final coding system comprised the variables place of occurrence, mechanism of injury, activity when injured, products involved, type of sport and mode of transportation, as well as height, added by the authors to obtain additional information about the accident circumstances.

To create a dataset for the model training and evaluation, a randomly chosen subset of the narrative medical history texts was manually annotated. The variables and their corresponding labels were assigned to identified text spans which provided relevant information for a specific variable of interest (see Table 1). For the annotation process, the tool Prodigy (https://prodi.gy/) was used. Manual annotation was done in two phases by a total of five individuals (annotators): three members of the project team, and two scientific assistants. All annotators received training that included an introduction to the coding tree and coding rules, followed by practical exercises in which randomly selected cases were annotated and discussed in the group.

The degree of agreement between the annotators was analysed twice during the feasibility study. For each test, a sample of 50 cases was randomly selected from the entire dataset and independently manually annotated by each annotator. Cohens κ was used to assess interrater reliability, where agreement is defined as both annotators selecting the same variable and producing overlapping text spans. In the first annotation phase, two annotators manually annotated n = 1,000 cases using the original IDB coding tree. The first interrater test involved three annotators, of which one was considered as the reference annotator because of his established experience in applying the IDB coding tree. The first interrater reliability showed a moderate agreement (κ = 0.45). Given the changes to the coding system, two new annotators were trained and involved in the second manual annotation phase, which comprised the manual annotation of an additional n = 1,000 cases and re-annotation of the initial n = 1,000 cases to reflect updated coding rules. This resulted in a final dataset (n = 2,000) single-annotated according to the same coding system. The new annotators performed a second interrater test using the adapted coding tree, resulting in substantial agreement between annotators (κ = 0.73).

To evaluate the performance of the trained models the manually annotated dataset was randomly divided into a training set (n = 1,800) and a test set (n = 200).

The base model, an instruction fine-tuned version of the Mistral 7B v0.3 model (released May 22, 2024), was chosen because of its multilingual ability, openly available weights, and low hardware requirements for the fine-tuning. It is a transformer model with approximately seven billion parameters [19]. Supervised fine-tuning was performed using Parameter-Efficient Fine-Tuning (PEFT) via the Low-Rank Adaptation (LoRA) method [20]. Details regarding the hyperparameters can be found on the project page (https://github.com/AIPRA-IS/feasibility). Fine-tuning was formulated as a multi-label text-span classification task. During training, the model is provided with ED texts together with a semi-structured prompt that highlights the target variable and instructs the model to extract the corresponding spans. The model then produces a structured output that contains both the extracted text span and its associated hierarchical labels (levels 1–3) in the same structure as the annotated dataset in Table 1. This output format ensures that the model does not only locate the relevant span but also assigns the appropriate variable. Training was conducted on a single NVIDIA Tesla V100 GPU. Three variations of the fine-tuning setup were implemented to compare different prediction targets:

ALL One model to find text spans and assign the correct variable as well as the labels from the hierarchical coding system.

Model performance was assessed in two steps. First, for each ground-truth text span, we checked whether the model produces a text span sharing at least one word with it. Second, for each such overlapping span, we verified whether the predicted Variable and Level 1–3 labels are correct. This enables the generation of a confusion matrix for each label at each level, from which an F1 score is computed that is especially suitable for unbalanced class distributions.

To obtain uncertainty estimates for the performance scores without requiring costly model training, we applied non-parametric bootstrapping by resampling the test documents with replacement over 1,000 iterations, computing the F1 score in each iteration to derive 95% confidence intervals.

To determine the frequency of each code within the entire dataset, the IND model, chosen because it performed best across the hierarchical levels, was applied to estimate the prevalence of all hierarchical labels for each variable separately. However, applying a classifier directly to estimate prevalence via naïve counting is known to produce biased estimates [21], as asymmetric misclassification rates cause systematic over- or underestimation of class frequencies. To correct for this, we calibrated the raw prediction counts using the confusion matrix derived from the annotated subset and the Bayesian Classify and Count (BCC) method proposed by von Däniken et al. (2024) [22], which adjust prevalence estimates to account for the classifier’s error structure. To assess plausibility, we compared the prevalence estimates obtained from the automatic annotations with those from the manually annotated subset.

Our subset of 2,000 manually annotated narrative medical history texts contained a total of 5,326 annotations, each containing a text span with the corresponding variable and labels from the coding system. Table 2 shows the utilization of the labels per levels (coding hierarchy). Level 1 and 2 show a high utilization with 92.6% respectively 71.3%, although usage is lower for the variables product and sport for level 2. In contrast, level 3 shows lower utilization, with 53.9% of the available labels being used.

Table 2 also shows the annotation counts per level grouped by variables. Mechanism of injury is the most often used variable with a total of 1,859 annotations. Of these n = 1,858 have an annotation on level 1, n = 1,213 on level 2 and n = 95 on level 3.

The number of annotations, and the number of labels used indicate that annotations of variables and level 1 are frequent. On level 2 the number of annotations is still high (n = 3,790) but spread across a lot of labels (n = 184). Level 3 has fewer annotations (n = 929) but still comprising many labels (n = 130), resulting in a lower number of annotations per code.

Figure 2 shows the classification performance across the three training variations (ALL, VAR and IND) including the 95% confidence intervals. Prediction performance on mechanism of injury, the variable with the most annotations, was high across all three variations (ALL = 0.65, VAR = 0.795, IND, 0.80) with the smallest confidence intervals. The models VAR and IND showed a good performance on height (VAR = 0.77, IND = 0.78) while ALL exhibited the lowest performance (ALL = 0.4) compared to all other variables. The confidence intervals on height were also the largest and it is the class with the lowest number of annotations. Fine-tuning a model to predict only the variable (VAR) yielded better average performance across most variables (Macro F1 = 0.63), however, it seemed to work worse for the variables sport, mode of transportation, and activity when injured. The best performance on average was achieved by separately fine-tuning individual variable-specific models (IND), with an average Macro F1 score of 0.64.

Table 3 shows the macro F1 scores for variable-level and hierarchical code-level prediction across the three model configurations (ALL, VAR, IND). At the variable level, the IND and VAR models performed comparably, and both outperformed the ALL model (mean F1: IND = 0.64, VAR = 0.63, ALL = 0.57). Gains were most pronounced for specific variables. The IND model substantially improved prediction of height (F1 = 0.78 vs. 0.39 for ALL) and mechanism of injury (F1 = 0.80 vs. 0.65), while the VAR model showed the strongest performance for product (F1 = 0.67) and achieved F1 = 0.78 for height. Across all models, prediction performance declined with each lower level of the label hierarchy. At level 1, the IND model again outperformed ALL (mean F1: 0.40 vs. 0.32). At level 2, performance was considerably lower across all variables (IND mean F = 0.26, All mean F1 = 0.23), with mechanism of injury showing the greatest benefit from the IND configuration (F1 = 0.42 vs. 0.23). Level 3 prediction was largely poor or not applicable. Mechanism of injury achieved F1 = 0.00 under the ALL model and only 0.28 under IND, while mode of transportation returned F1 = 0.00 under both configurations. Across all three model types, mechanism of injury and mode of transportation at the variable level, and height at level 1, showed the strongest absolute performance, whereas level 3 prediction across all variables remained the most challenging aspect of the classification task.

Table 4 shows the prevalence (including the estimated standard deviation) of the most relevant labels (prevalence at least 1%) of level 1 and 2 labels grouped by their variable within the annotated subset (n = 2,000) and compares it to the entire dataset (n = 30,876). The prevalence from the manual annotations is compared to the prevalence derived from the predictions obtained by the IND models on the full dataset. This comparison yields a Spearman correlation coefficient of ρ = 0.948 across the labels listed in Table 3.

For most labels, the difference between the prevalence estimations obtained from the annotated dataset and those obtained from the entire dataset is small. However, notable absolute discrepancies exist for certain labels, such as product (baby equipment or children’s product: 6%), activity when injured (sport and physical activity: 6%), place of occurrence (home: 10%).

Several limitations of this feasibility study should be acknowledged:

First, the complexity of the EU-IDB coding tree posed a substantial challenge for automatic classification. Despite reducing the original coding system, the remaining hierarchical structure proved difficult for the models to navigate, particularly at lower levels where prediction performance declined markedly. Future work should explore whether further reduction of the coding tree, guided by annotation density and surveillance priorities, could improve classification performance without sacrificing the detail need for evidence-based injury prevention.

Second, the small number of manual annotations limited the size of the training and test datasets. To draw robust conclusions about the automated extraction, further evaluation using a larger test dataset with sufficient sample size for all analysed label categories is required. This highlights the necessity of increasing the manually annotated data to provide the basis for further development and enhancements of automated approaches for injury monitoring.

Third, the choice of model architecture and training setup involved trade-offs that may have affected performance. Only a single base model was used for the evaluation. Future work should compare different model sizes and output generation approaches such as reasoning strategies, unavailable at the time of the study. In addition, this baseline should be used to identify which narrative annotations would most improve classification performance. In this way, the manual annotation effort can be optimized so that only as much data is manually annotated as necessary to achieve a useful prediction performance.

Finally, model evaluation was limited in scope. Performance was assessed on a small test set, without cross-validation, which affects the robustness of the reported estimates for labels with a small number of samples in the test set. Further evaluation using human annotations of a blinded sample is needed to distinguish whether limitations stem from insufficient annotations, fine-tuning shortcomings, or inherent text ambiguity.

Our feasibility study provides important first insights on the automatic identification and multi-label classification of text spans containing paediatric injury-relevant information in Switzerland. This approach has the potential to improve currently insufficient injury surveillance and evidence-based prevention in children and adolescents in Switzerland. Prerequisites for the successful application of automated systems include digital access to ED patient records and detailed narrative text input from medical staff.

The results demonstrate that reducing its complexity and adding child-specific labels to the EU-IDB coding tree, enhanced model performance. The remaining hierarchical structure continued to pose challenges. Future work should expand the annotated dataset, systematically evaluate alternative models, integrate most recent methodological advances, and explore continuous active learning strategies to optimise the annotation effort. Model evaluation using larger test sets, evaluation across multiple representative test sets, cross-validation when feasible, and human validation will be required before it can be considered as ready for implementation in child injury surveillance in Switzerland. Data privacy, ethical considerations and transparency are other challenges that must be carefully addressed to ensure a secure and responsible implementation of the approach in using clinical data. Considering these challenges and once the necessary advancements for implementation are achieved, applying automated methods for text span classification while preserving injury context information could enable timely surveillance of injury trends allowing rapid responses to emerging injury patterns and circumstances and thereby supporting evidence-based prevention strategies aimed at improving public health outcomes in children and adolescents.