Maanav Khaitan

70 followers · 55 following · 3292 views

on the atlas · 183

highlights · 1844

The information is “in the model” already, but it takes a very smart human to pull it out.
LLMs reward expertise
Tao makes several leaps and suggestions himself. He almost never takes the model’s advice about where to go next
LLMs reward expertise
Both funds give investors access to private companies, but they focus on companies at different stages. RVI primarily invests in late-stage private companies—well-established businesses that haven't gone public yet. RVII will invest in early and growth-stage companies, with some recently founded. Earlier entry generally involves greater risk, but the potential for meaningful growth if the company succeeds.
Robinhood Ventures Fund II FAQ | Robinhood
BNY is already the primary custodian for RLUSD’s reserves and provides custody and investment management for OpenEden’s tokenized Treasury fund.
24/7 financial rails: How BNY plans to eliminate the weekend lag in U.S. Treasuries
Unlike stablecoins, whose circulating supply is a direct proxy for demand — one token, one dollar — tokenized stocks move with their underlying equities, so market cap does not cleanly separate the effects of new tokens minted and existing tokens repricing.
3 charts on the tokenized stocks boom - a16z crypto
The evaluation it was extracted from layers self-improvement on the same pieces: a playbook the agent rewrites and re-reads every turn, durable memory files, and an install_tool meta-tool that lets the agent write new ES-module tools composing the emulator primitives. Over two long runs the agent authored 14 tools (dialog mashing, battle recovery, pathing macros) and 2 skills entirely on its own.
Game & Third-Party Tool Integration | exo docs
The payoff: sessions you can stop, resume, fork, and rewind across runs, regardless of which coding agent is driving.
Executors & Harnesses | exo docs
The exoharness stores and orders events durably; executors interpret them to construct message history and higher-level behavior.
Data Model | exo docs
AI products often focus so much on capabilities that they forget fundamentals like stability, speed, and reliability.
Hard Truths From the AI Trenches - Jason Liu
Nobody cares that your RAG system has 78.9% recall. They care that FancyCorp saved $2M using your product.
Hard Truths From the AI Trenches - Jason Liu
You are doing it wrong if you aren’t looking at lots of data.
- Your AI Product Needs Evals
using raw agreement is generally not recommended and can be misleading when classes are imbalanced. Instead, you should typically measure precision and recall separately to get a more accurate picture of your judge’s alignment.
- Your AI Product Needs Evals
After bringing the model-based evaluator in line with the human, you must continue doing periodic exercises to monitor the model and human agreement.
- Your AI Product Needs Evals
Use the most powerful model you can afford. It often takes advanced reasoning capabilities to critique something well.
- Your AI Product Needs Evals
Having humans periodically evaluate at least a sample of traces is a good idea. I often find that “correctness” is somewhat subjective, and you must align the model with a human.
- Your AI Product Needs Evals
Furthermore, there are tools like Lilac which uses AI to search and filter data semantically, which is incredibly handy for finding a set of similar data points while debugging an issue.
- Your AI Product Needs Evals
We decided to make the final output editable by a human so that we could curate & fix data for fine-tuning. These tools can be built with lightweight front-end frameworks like Gradio, Streamlit, Panel, or Shiny in less than a day.
- Your AI Product Needs Evals
you don’t necessarily need a 100% pass rate. Your pass rate is a product decision, depending on the failures you are willing to tolerate.
- Your AI Product Needs Evals
If you streamline your evaluation process, all other activities become easy. This is very similar to how tests in software engineering pay massive dividends in the long term despite requiring up-front investment.
- Your AI Product Needs Evals
That means the routing problem is worth learning.
Training an Agentic Router for Optimal Cost-Performance on SWE Tasks | Applied Compute
This is how multi-model systems should improve: not by treating model selection as a one-time benchmark comparison, but by continuously shaping both the router and the routed models.
Training an Agentic Router for Optimal Cost-Performance on SWE Tasks | Applied Compute
If Nemotron 3 Ultra is already the oracle choice for a large fraction of tasks, it can be post-trained on the regions where it underperforms and move the entire frontier.
Training an Agentic Router for Optimal Cost-Performance on SWE Tasks | Applied Compute
With the right training infrastructure, a dataset that faithfully represents the production task, and a reward function that encodes how the product actually needs to behave, enterprises can post-train a specialized model that occupies the exact point on the cost-latency-performance Parento frontier required.
Unlocking Real-Time Bug Detection at Cognition | Applied Compute
If all the value is accrued by only a few models, the political economy will simply not tolerate it. There is no societal permission for an AI future that hollows out entire industries.
A frontier without an ecosystem is not stable
The companies that build this early will have an advantage that is hard to replicate
A frontier without an ecosystem is not stable
Seen this way, the swarm starts to resemble a compiler. A compiler translates source code down to machine code through a series of intermediate steps. The swarm does something similar with intent.
Agent swarms and the new model economics · Cursor
What was scarce in this experiment, and what we expect to be scarce in software engineering going forward, is the right description of intent.
Agent swarms and the new model economics · Cursor
In the run that used GPT-5.5 for both planners and workers, the workers alone cost $9,373. In the run where Opus 4.8 did the planning and Composer 2.5 did the work, the entire worker fleet cost $411.
Agent swarms and the new model economics · Cursor
In the old run, the biggest files kept growing for the entire run and its single hottest file collected 7,771 conflicts, touched by 1,173 different agents. In the new run, the most contested file in the whole codebase saw 47.
Agent swarms and the new model economics · Cursor
Training models to write for their successors, where better capture leads to better rewards, is an interesting follow-up area of research.
Agent swarms and the new model economics · Cursor
Stigmergy is the mechanism by which swarm organisms like ants and termites coordinate without direct communication. They shape the environment, and the environment shapes the next organism.
Agent swarms and the new model economics · Cursor
The new system peaks at around 1,000 commits per second.
Agent swarms and the new model economics · Cursor
Set benchmarks for repeated work. Do not ask teams to "use a cheaper model" in the abstract. Test which cheaper setup still produces acceptable outcomes.
You're Spending Too Much on AI. You're Also Using Too Little. — Ramp Builders Blog
The companies that manage AI costs best build a culture where efficiency is treated as an engineering achievement, not a constraint.
You're Spending Too Much on AI. You're Also Using Too Little. — Ramp Builders Blog
If a human is waiting for your agent to get back to them, it makes sense to pay for speed.
You're Spending Too Much on AI. You're Also Using Too Little. — Ramp Builders Blog
these graphs are not constructed by benchmarking models against the kind of work your business likely does. These models are benchmarked against brutally hard problems, intentionally.
You're Spending Too Much on AI. You're Also Using Too Little. — Ramp Builders Blog
Artifact Completeness: The model learned to properly use tools and always create an output artifact. This behavior flipped 185 relevant rubric criteria from failing to passing during the course of training. Specificity and Exactness: The base GLM-5.1 model suffers from poor calculations or referring to imprecise numbers, often rounding figures during math (like 1.9 to 2), which is punished by legal graders. This behavior flips 243 criteria from failing to passing. Grounding: Without training, the checkpoint sometimes hallucinates source document items or invents findings from outside the provi…
Training a State-of-the-Art Legal Agent with Harvey | Applied Compute
real memories need to be updated — often in surprisingly subtle ways — as new information comes in and old information becomes stale. A compelling next step would be to understand reward shaping to train for this functionality end to end as well.
Neural Cheat Sheets: Learning to Summarize with Reinforcement Learning | Applied Compute
The same policy improves substantially over baseline and essentially matches the accuracy it reaches when QuALITY is included in the training mix. The model has learned how to take useful notes and amortize reasoning in a token-efficient manner, not domain-specific heuristics or knowledge.
Neural Cheat Sheets: Learning to Summarize with Reinforcement Learning | Applied Compute
It is starkly reminiscent of the kind of frenetic, stream-of-consciousness notes a student might put on a 1-page cheat-sheet before an important exam:
Neural Cheat Sheets: Learning to Summarize with Reinforcement Learning | Applied Compute
latent methods store large tensors for each token whereas summaries store only the token itself, and are thus much smaller and cheaper
Neural Cheat Sheets: Learning to Summarize with Reinforcement Learning | Applied Compute
They close most of the gap toward a full KV cache which is several orders of magnitude larger, and not model-agnostic.
Neural Cheat Sheets: Learning to Summarize with Reinforcement Learning | Applied Compute
A summary is good context if it suffices to answer unseen QA pairs, which ask about fine-grained details, higher-level inferences, and subtext, in the original document. The QA pairs have a verifiable ground truth answer, and are posed as multiple-choice questions.
Neural Cheat Sheets: Learning to Summarize with Reinforcement Learning | Applied Compute
Latent vector stores, eg. learned KV caches. These high-capacity document-specific vectors often yield the best downstream performance when used as context, but are uninterpretable and model-specific.
Neural Cheat Sheets: Learning to Summarize with Reinforcement Learning | Applied Compute
Without a durable specification, one-shot agents risk drifting away from the broader goal. The adversarial loop gives the system a way to pull itself back toward its desired state.
Telos — Declarative software, managed for you
Without a durable specification, one-shot agents risk drifting away from the broader goal. The adversarial loop gives the system a way to pull itself back toward its desired state. Checkpoint 3 One-shot 16% Telos loop 100%
Telos — Declarative software, managed for you
We built a custom instruction proposer that encodes the constraint directly into the reflection prompt.
Optimizing GEPA for production: A test-driven approach to prompt engineering | Decagon
Spending on a frontier reflection model represents only 5-10% of total optimization cost, and a weak reflector means you waste all those task model calls learning nothing.
Optimizing GEPA for production: A test-driven approach to prompt engineering | Decagon
smaller models completely fail at prompt optimization
Optimizing GEPA for production: A test-driven approach to prompt engineering | Decagon
We observed a clear inverted-U relationship between sample size and performance
Optimizing GEPA for production: A test-driven approach to prompt engineering | Decagon