# Automated fine-tuning of a small language model to handle event extraction Can a general-purpose decoder LLM learn document-level event extraction and match specialized models? (The short answer appears to be: yes.) Why? To have a *lightweight* decoder that can extract events from text. It should perform as well as specialized encoder models and therefore similar to untuned frontier models at the task. Given the small size, that means it would be at least one or two orders of magnitude cheaper and faster than frontier models. And any decoder can be fine-tuned on examples, instead of needing some task-specific encoder architecture. This blog post will - explain what event extraction is about and why it is interesting to fine-tune a decoder model for that task, - outline how I approached the fine-tuning, - describe the MUC-4 challenge dataset, - discuss the results, - highlight what I learned along the way, and - propose next steps. ## A bit of background An *event* can be any state change or occurrence with identifiable participants, properties, time, or place: Legislation or justice events like sanctions and sentences, business organizational events, political or government events, disasters, medical events like diagnoses or treatments, or molecular and genetic interactions. And so forth. Once you have a structured event history, a vast spectrum of opportunities to use or monetize that event ledger opens up. To find out if a small decoder model can readily be trained to extract structured events, I asked Opus 5.5 to fine-tune and evaluate a small language model (Qwen3-4B) on the MUC-4 event dataset. Why? Because training a decoder to “translate” text into events is a much easier task than architecting a bespoke extractor, typically an encoder model with a graph layer for linking the event properties together. If a smaller decoder can be tuned to state-of-the-art performance, that opens up doors for efficiently creating event extraction systems. How well do frontier models perform at event extraction, out of the box? [Wen et al. (published Jul 2026)](https://arxiv.org/html/2607.27654v1) looked at the event analysis performance of GPT 5.2, Opus 4.5, and Gemini 3 Pro. It shows that these pretty frontier models have not made specialized event-extraction systems obsolete. Wen et al. reveal substantial weaknesses in structured event analysis, while direct comparisons show that task-specific supervised models outperform general-purpose LLMs. There is one (common) caveat: You will need to have properly annotated training data. In this exercise, I used a dataset with around 1700 documents. In total, about 1000 of those docs describe about 1500 events. While modern datasets that give you even stronger results might have 10x that size. But you might be able to use frontier models to generate a dataset for you. Or at least make it much easier to create one. ## Approach For training and evaluation, I used the [Message Understanding Conference 4](https://dl.acm.org/doi/10.3115/1072064.1072066) (MUC-4) event extraction [dataset](https://www-nlpir.nist.gov/related_projects/muc/muc_data/muc_data_index.html). I let Claude run the experiments on [Google Colab](https://colab.research.google.com/), tracked them with a free [Weights & Biases](https://wandb.ai/site/) account, and had Opus implement the fine-tuning script with [Unsloth](https://unsloth.ai) and Hugging Face's [Transformers Reinforcement Learning](https://huggingface.co/docs/trl/en/index) library. To keep the setup accessible, I trained a Quantized Low-Rank Adaptation (QLoRA) of a **Qwen3-4B-Instruct** model with rank-16 adapters, running three training epochs over 1,298 documents on a free Colab T4 instance. The final fine-tuning run took 1 hour 48 minutes. [The repository](https://github.com/fnl/fine-tuning-decoder) shows how data preparation, completion-only training, generation and scoring work. You can go back in time to see every step of the process. ## The MUC-4 challenge Warning: The dataset used in this experiment is on the topic of violent terrorist attacks in Latin America in the early 1990s. If that kind of content disturbs you, please skip this section. The MUC-4 dataset is a collection of annotated English news articles. Their annotations describe terrorist attacks and other violent incidents in Latin America. Event extraction experts frequently use MUC-4 to evaluate their extraction systems. Despite the sensitive topic, it is one of the most commonly used event extraction datasets. The language model reads a news article and writes out its events as JSON: Zero or more events, each with its type and the perpetrators (people and organizations), targets, victims and weapons involved. Each entity is a list of its coreferent mentions. For example, document `DEV-MUC3-0221` in the training split reports a bomb that damaged Velox Bank's windows. Thirty minutes later, a second bomb exploded on a downtown corner lined with banks: > it has been learned from police sources that another device has exploded on the downtown corner of sarmiento and san martin streets, where many banking institutions are located. the federal police bomb squad is trying to determine the damage caused by this second bomb, which exploded 30 minutes after the one that damaged the windows of the velox bank. The gold annotations cover the two events in a tree-like structure: ```text Source document ├── Bombing: downtown corner │ ├── Target → ["banking institutions"] │ └── Weapon → ["device", "bomb"] └── Bombing: Velox Bank └── Target → ["velox bank"] ``` The model must keep the events separate, identify the event type, and assign each entity to the right event (group) and role. This example also shows some weaknesses in the gold annotations. The text refers to the first bomb as 'the one', yet the annotators left that event's weapon slot empty. It is also ambiguous whether the second bomb actually targeted any banking institutions. ## Results On 200 held-out test documents, fine-tuning raised micro-F1 from **21.4 zero-shot to 53.9**. Published encoder-based systems score **50.2 (GTT)** and **53.0 (IterX)**. So the fine-tuned decoder matches the performance of published specialized models. A three-shot prompting approach did not improve over zero-shot: it scored 20.0. After fine-tuning, output parse failures dropped from 4.5% (zero-shot) to 0.5%. These are encouraging results for a modest training budget. The [results page](https://github.com/fnl/fine-tuning-decoder/blob/main/docs/RESULTS.md) documents the comparison and the scoring caveats. Overall, the results suggest that fine-tuning a 4B decoder LLM can deliver competitive performance on multi-event extraction from documents. The major caveat is that the event dataset may be in Qwen3's pretraining data. MUC-4 is a public benchmark from 1992, and texts and annotations have been discussed on the Web for years. The weak zero-shot score (21.4) shows the base model does not simply recall the answers, but it does not rule out exposure. Ruling out data leakage would require a private or proprietary dataset. Also, "match specialized models" here means systems scored with the GTT scorer. Newer papers, including those on fine-tuned LLMs, report a new metic, CEAF-RME. I used GTT because the code is available, facilitating reproducibility. CEAF-RME obviously scores the same systems far lower: GTT drops from 50.2 to 32.3 and IterX drops from 53.0 to 35.2. Those systems were developed before that new scorer even existed, so they are not adapted to its matching criteria. The best CEAF-RME result I found is 42.5, from the [ThinkTwice](https://arxiv.org/abs/2601.18395) method. ThinkTwice fine-tunes Qwen3-32B, samples several candidate extractions, and uses a trained reward model to pick the best one. With 4B parameters, the decoder is much larger than the comparison models but much less complex than ThinkTwice: GTT's BERT-base has about 110M parameters, and IterX's T5-large encoder a few hundred million. Cheap adaptation does not mean the inference is as cheap as the published encoder-based approaches, as discussed in a [previous post](https://fnl.es/Blog/Machine+Learning/2026-09-13+A+Review+of+document-level+financial+event+extraction). ## What I learned These results are promising evidence that fine-tuned decoders can be competitive on event extraction tasks. But an important open question is whether that effect holds on private, newly generated, or proprietary data. I had set up the exercise to learn if the copilot could fine-tune a decoder, not to beat the state of the art. That the tuned model still came in range of it was a welcome surprise. The project reinforced a classic lesson: Inspect the training tokens that your setup creates after tokenization. Unsloth's copy of the chat template for Qwen3-4B adds an empty thinking block that the model cannot handle. So at first, the model generated nothing! Encouragingly, Opus was pretty quick to identify the culprit on its own. Where even the latest LLMs fall short is the ability to prioritize what matters. As I iterated on the milestones, my main job was to ensure that Claude would not go meandering down every possible concern. Once a concern surfaces in the documentation or planning, it is very hard for an LLM to “let go” of it again. So I found myself pruning content from the generated work that was not mission-critical, and directing the LLM what **not** to do. (I go deeper on my coding assistant setup in a [follow-on post about Matt Pocock’s skills for ML](2026-09-29%20Tuning%20Pocock%20skills%20for%20ML.md).) Would someone unskilled in event extraction be able to achieve the same result? Possibly, with more time, and as long as you provide the LLM with a firm, measurable goal. Probably with way more tokens wasted on “reasoning” and quite a lot of worthless meandering on side-gigs, too. ## Next steps Auditing missed events on the development data could lead to further improvements. So could stopping repetition loops at generation time: on one development document, the model repeated "bomb" until it hit the output limit, which alone cost about 3 points of performance. And of course, adding a CEAF-RME scorer. More importantly, repeating this recipe on other datasets, such as DocEE, MultiMUC, or BETTER, would turn this result into stronger evidence that it generalizes. Repeating it on novel or private data would test whether the result survives when pretraining exposure can be ruled out.