# Automated fine-tuning of a small language model to handle event extraction Can a general-purpose decoder LLM learn document-level event extraction and match specialized models? To find out, I asked Opus 5.5 to fine-tune and evaluate a small language model (Qwen3-4B) on MUC-4. This blog post describes how I set this up, the results, and what I learned along the way. For the evaluation, I used the [Message Understanding Conference 4](https://dl.acm.org/doi/10.3115/1072064.1072066) event extraction [dataset](https://www-nlpir.nist.gov/related_projects/muc/muc_data/muc_data_index.html). And I ran the experiments on Google Colab, tracked them with a free Weights & Biases account, and did the fine-tuning with Unsloth and Hugging Face's TRL. ## The MUC-4 challenge The MUC-4 dataset is a collection of annotated English news articles. Their annotations describe terrorist attacks and other violent incidents in Latin America. Researchers have often used MUC-4 to evaluate information extraction systems. The language model reads a news article and writes out its events as JSON: zero or more events, each with its type and the perpetrators (people and organizations), targets, victims and weapons involved. Each entity is a list of its coreferent mentions. For example, document `DEV-MUC3-0221` in the training split reports a bomb that damaged Velox Bank's windows. Thirty minutes later, a second bomb exploded on a downtown corner lined with banks: > it has been learned from police sources that another device has exploded on the downtown corner of sarmiento and san martin streets, where many banking institutions are located. the federal police bomb squad is trying to determine the damage caused by this second bomb, which exploded 30 minutes after the one that damaged the windows of the velox bank. The gold annotations cover the two events in a tree-like structure: ```text Source document ├── Bombing: downtown corner │ ├── Target → ["banking institutions"] │ └── Weapon → ["device", "bomb"] └── Bombing: Velox Bank └── Target → ["velox bank"] ``` The model must keep the events separate and assign each entity to the right event and role. This example also shows some weaknesses in the annotations. The text mentions the first bomb as "the one", but the annotators left the Velox Bank event's weapon role empty. It is also unclear whether the second bomb actually targeted any bank. ## Approach For the experiment, I used a widely accessible setup: **Qwen3-4B-Instruct**, Unsloth and Hugging Face's TRL, QLoRA with rank-16 adapters, and three training epochs over 1,298 documents on a free Colab T4 instance. The final fine-tuning run took 1 hour 48 minutes. [The repository](https://github.com/fnl/fine-tuning-decoder) shows how data preparation, completion-only training, generation and scoring work. I built the entire project and produced the results with Claude Code, running the Opus 5.5 model. I was pleasantly surprised by how well [Matt Pocock's skills](https://github.com/mattpocock/skills) support this kind of machine learning work. I started with an interactive Q&A session using Matt's [`grill-me` skill](Tech/Agents/skills/grill-me/SKILL.md), then asked the model to write the [original DESIGN.md document](https://github.com/fnl/fine-tuning-decoder/blob/2f90282701bc573fedd93d24d7226a047506ec9c/docs/DESIGN.md). For each milestone, I used the [`wayfinder` skill](Tech/Agents/skills/wayfinder/SKILL.md) to plan the route, and then the [`implement` skill](Tech/Agents/skills/implement/SKILL.md) to drive the coding. I use [my own light variations of Matt's skills](https://github.com/fnl/agents), tuned to my preferences, but I don't believe the differences matter for reproducing the results. ## Results On 200 held-out test documents, fine-tuning raised micro-F1 from **21.4 zero-shot to 53.9**. Published systems score **50.2 (GTT)** and **53.0 (IterX)**. A three-shot prompting approach did not improve over the zero-shot: it scored 20.0. After fine-tuning, parse failures dropped from 4.5% (zero-shot) to 0.5%. These are encouraging results for a modest training budget. The same adapter scored only 44.5 on the 200 *development* documents. Most of that gap comes from the split itself: the test documents hold more events (812 gold items against 731), and the baselines moved only a point or two between the two splits. The [results page](https://github.com/fnl/fine-tuning-decoder/blob/main/docs/RESULTS.md) documents the comparison and the scoring caveats. Overall, the results suggest that fine-tuning a 4B decoder LLM can deliver competitive performance on multi-event extraction from documents. The major caveat is that the test documents and their answers may be in Qwen3's pretraining data. MUC-4 is a public benchmark from 1992, and the GTT preprocessing, with texts and annotations, has been on GitHub for years. The weak zero-shot score (21.4) shows the base model does not simply recall the answers, but it cannot rule out exposure that fine-tuning then draws on. Fully ruling out this caveat will require using a private or proprietary dataset. Also, "match specialized models" here means the GTT-scored systems. Newer papers, including those on fine-tuned LLMs, report only CEAF-RME, a stricter metric that matches mentions against whole coreferent entities and ignores the event type. It scores the same systems far lower: GTT drops from 50.2 to 32.3 and IterX drops from 53.0 to 35.2. The best CEAF-RME result I found is 42.5, from the [ThinkTwice](https://arxiv.org/abs/2601.18395) method. ThinkTwice fine-tunes Qwen3-32B, samples several candidate extractions, and uses a trained reward model to pick the best one. With 4B parameters, the decoder is much larger than the comparison models: GTT's BERT-base has about 110M parameters, and IterX's T5-large encoder a few hundred million. So cheap adaptation does not mean the inference is as cheap as the published encoder-based approaches, as discussed in a [previous post](https://fnl.es/Blog/Machine+Learning/2026-09-13+A+Review+of+document-level+financial+event+extraction). ## What I learned These results are promising evidence that fine-tuned decoders can be competitive on event extraction tasks. But an important open question is whether that effect holds on private or proprietary data. The project reinforced a classic lesson: Inspect the training tokens that your setup creates after tokenization. Unsloth's copy of the chat template for Qwen3-4B adds an empty thinking block that the model cannot handle. So at first, the model generated nothing! To my pleasant surprise, Opus was pretty quick to identify the culprit on its own. Another takeaway is how well Claude Code was able to set up this evaluation with Pocock's ["wayfinder" skill](Tech/Agents/skills/wayfinder/SKILL.md). The AI assistant did most of the work across the codebase and the browser, accessing Colab and Weights & Biases. The whole project took only about four workdays to develop and complete, including this blog post. Opus 5.5 needed only very occasional directional input from my side: Most of its choices and decisions were reasonable. The project showcases the huge progress autonomous copilots have made since the start of 2026. ## Next steps A follow-up could first fix my scorer port, which is slightly stricter than GTT's on composite event types such as "attack / bombing", and report CEAF-RME. It could then repeat training across several seeds and obtain the published systems' per-document predictions for paired comparisons. Auditing missed events on the development data could lead to further improvements. So could stopping repetition loops at generation time: on one development document, the model repeated "bomb" until it hit the output limit, which alone cost about 3 points of recall. Most importantly, repeating this recipe on other datasets, such as DocEE, MultiMUC, or BETTER, would turn this result into stronger evidence that it generalizes.