Can a language model be trusted to drive particle physics software?

Not “explain the physics” — that part it does fluently. I mean: write the exact commands that MadGraph, Pythia 8, or Delphes will actually accept.

This is harder than it sounds, and the failure modes are nasty. Write   anti-t   instead of   t~   and the run stops. Enter 13000 GeV as the energy of each beam instead of 6500 and the run succeeds — at the wrong collision energy. Apply a cut at the wrong stage and the event sample silently changes. A fluent paragraph about top-quark physics is worth nothing if the artifact below it is malformed.

So we built HEPToolBench: 28 collider-simulation tasks, every one graded by deterministic code rather than a human or a model judge. We ran 42 deployments, from a 270-million-parameter model on a workstation to hosted frontier systems.

The central experiment asks one question: *Does it matter how you ask?*

Five task pairs keep the physics request identical — same process, same beam energy, same event count — and change only the interface. In one version the model writes the native MadGraph syntax. In the other it returns a typed JSON object (model, initial state, final state, beam energy, events…) and ordinary deterministic software renders the card.

How far can we answer now?

• Mean score rises from 0.42 to 0.90; task passes from 21/210 to 159/210.
• 41 of 42 deployments improve.
• Eleven deployments go from passing zero of five native tasks to five of five structured ones — seven of them small open-weight models served locally through Ollama.
• Of the 306 native-syntax failures across the cohort, 302 contain an interface or syntax error. Only four fail on physics content alone.

The lesson: native tool syntax is a poor place to leave model uncertainty. Ask the model for what it means in typed fields, and let deterministic code own the how.

Two honest caveats. Our structured scorers tolerate Markdown fences and our native scorers don’t; an audit on archived responses shows the effect survives symmetric scoring (still 6–10× more passes), but a full-cohort rescoring is the outstanding measurement. And a passing artifact is not a scientifically valid calculation —   p p > h j   at tree level in the SM builds a perfectly well-formed card and then finds no diagrams.

Also: bigger helps but doesn’t decide. Two 14B models score 0.815 and 0.336 on the same suite.

Why this matters beyond physics: the biggest gains went to small open-weight models running on a single workstation. That is exactly the regime you are forced into when data cannot leave the building — hospitals, clinical research, defence, finance, any field where a cloud API is not an option. The recipe is domain-agnostic: benchmark with deterministic scorers, and move syntax out of the model into code. Everything is open source so others can adapt it to their own tools.

Paper: https://arxiv.org/abs/2608.28232

Code, prompts, scorers, and every model response: https://github.com/AadarshSingh0/HEPToolBench

This project was driven by Aadarsh Singh, who designed the benchmark, built the scorers, and ran every one of the 42 deployments on a single workstation. Centre for High Energy Physics, IISc Bengaluru.