Speculative Programmatic Tool Calling | Alex L. Zhang
alexzhang13.github.io · 2,576 words · saved by 2 readers
Speculative programmatic tool calling is a class of techniques for overlapping tool call computation with the code being generated by a harness.
Fig 1. Speculative Programmatic Tool Calling (sPTC) Example the model's turn — streaming WITH SPECULATION DECODE EXECUTE prep sub-calls ×6 judge execute baseline answer · 7/7 claims WITHOUT SPECULATION · SAME PROGRAM answer c₁c₂ c₃c₄ c₅c₆ judge execute collapses — 2.4×, no async Time t The associated code is here: https://github.com/alexzhang13/spec-ptc It is no surprise from my previous writing and work on Recursive Language Models (RLMs) that I believe (1) code in a REPL is the only “tool” a system needs; (2) all other tools should be functions in that code tool. When code becomes…
saved by
related reading
- Speculative Decoding - philkravphilkrav.com
- Looking back at speculative decodingresearch.google
- Recursive Language Models | Alex L. Zhangalexzhang13.github.io
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- Malleable software in the age of LLMsgeoffreylitt.com
- [2510.04371] Speculative Actions: A Lossless Framework for Faster Agentic Systemsarxiv.org
- Decoding Speculative Decoding from First Principlesjwlabs.vercel.app
- ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
- Speculative Decoding: How It Evolved, When It Stays Lossless, and What's Nextneurips2026-speculative-decoding.vercel.app
- Speculative Decoding - Deep Dive — ROCm Blogsrocm.blogs.amd.com
- How speculative decoding makes LLMs go brrr – Leonie Monigattileoniemonigatti.com