flâneur

Al-Ekram Elahee Hridoy

0 followers · 3 following · 201 views

on the atlas — 36

highlights — 110

  • Excellence requires non-attachment to principles. All mediocre performers are the same: instead of improving on feedback from what happens, they react to what they think is supposed to be happening, or try to reinforce a theory they’ve internalized—perhaps an inaccurate approximation of someone they admire. This attempt at coherence is futile, since the mind of an expert makes no sense. It is filled with exceptions, contradictory principles butting up against each other. Perhaps the knowledge of how your life works is the same.
    Surrender as a non-stupid life strategy
  • The word "agent" has been hijacked by an entire prompt-plumbing industry. Drag-and-drop workflow builders. No-code "AI Agent" platforms. Prompt-chain orchestration libraries. They share a single delusion: that stringing LLM API calls together with if-else branches, node graphs, and hardcoded routing logic constitutes "building an agent." It does not. What they produce are Rube Goldberg machines -- over-engineered, brittle, procedural rule pipelines with an LLM wedged in as a glorified text-completion node. That is not an agent. That is a shell script with grandiose pretensions. You cannot brut…
    shareAI-lab/learn-claude-code: Bash is all you need - A nano claude code–like 「agent harness」, built from 0 to 1 ·
  • Concretely: monitor the meta-signals — is the distribution of benchmark scores changing character? Is the correlation structure between evaluations shifting? Is the model developing capabilities orthogonal to your measurement axes? Track scaling curves for everything — not just loss, but reasoning depth, tool-use sophistication, deceptive capacity — and pay attention when a smooth trend breaks. More ambitiously, build self-evolving evals: evaluation systems that use models to probe other models, automatically generating new test cases as capabilities change, discovering failure modes the origi…
    Your Evals Will Break and You Won't See It Coming - Lun Wang
  • Shan, Li, and Sompolinsky (PNAS, 2026) used statistical mechanics to derive order parameters for deep networks in a continual learning setting, and those order parameters actually predict phase transitions in learning ability. Nanda et al. (2023) used mechanistic interpretability to find "progress measures" that predict grokking before it happens — internal structural changes that precede the visible performance jump.
    Your Evals Will Break and You Won't See It Coming - Lun Wang
  • Training is optimization, and optimization is only as good as its objective. The objective comes from eval. If you know what to measure — if you can predict how those measurements change at scale — then you can design the right training objectives, build the right safety layers, make informed scaling decisions, do RLHF that targets the right behavioral properties instead of proxies that Goodhart at the next phase boundary.
    Your Evals Will Break and You Won't See It Coming - Lun Wang
  • ission. Your safety classifiers wouldn't flag it, because the individual outputs are all technically true. The capability is new, the failure mode is new, and nothing in your evaluation suite was designed to look for it. You'd be monitoring the wrong thing and wouldn't know it.
    Your Evals Will Break and You Won't See It Coming - Lun Wang
  • chaeffer et al. (2023) showed that many apparent "jumps" in LLM capabilities are artifacts of discontinuous metrics like exact-match accuracy. Switch to a continuous metric and the capability often scales smoothly.
    Your Evals Will Break and You Won't See It Coming - Lun Wang
  • Later they added AGENTS.md (describing project architecture and tech stack versions), explicit verification commands (pytest tests/api/v2/ && python -m mypy src/), and architecture decision records. The same model succeeded in all three independent runs, with ~60% better context efficiency.
    Lecture 01. Strong Models Don't Mean Reliable Execution | Learn Harness Engineering
  • s. The engineers' work became: breaking large goals into small building blocks (design, code, review, test), letting the agent assemble them, then using those blocks to unlock more complex tasks. When something failed, the fix was almost never "try harder" — it was "what capability is the agent missing, and how do we make it both understandable and executable?"
    Lecture 01. Strong Models Don't Mean Reliable Execution | Learn Harness Engineering
  • Build a diagnostic loop. Don't treat failures as "the model being dumb again." Treat them as signals that your harness has a defect. Each failure, identify the layer, fix it, never fail that way again. After a few rounds, your harness gets stronger and agent performance stabilizes. Like road repair — every pothole you fill makes the next stretch smoother. Quantify improvements. Keep a simple log: did each task succeed or fail, and which layer caused the failure. After a few rounds you'll see which layer is the bottleneck — focus your energy there.
    Lecture 01. Strong Models Don't Mean Reliable Execution | Learn Harness Engineering
  • Create an AGENTS.md file. Put it in the repo root to tell the agent the project's tech stack, architectural conventions, and verification commands. This is the first step in harness engineering and the highest-ROI step you can take. One AGENTS.md file might be more effective than upgrading to a more expensive model — I'm not joking.
    Lecture 01. Strong Models Don't Mean Reliable Execution | Learn Harness Engineering
  • Write an explicit Definition of Done for every task. Don't say "add a search feature." Say: Completion criteria: - New endpoint GET /api/search?q=xxx - Supports pagination, default 20 items - Results include highlighted snippets - All new code passes pytest - Type checking passes (mypy --strict)
    Lecture 01. Strong Models Don't Mean Reliable Execution | Learn Harness Engineering
  • Verification Gap: The gap between the agent's confidence in its output and actual correctness. The agent says "I'm done" when it's not done — this is the most common failure mode. Diagnostic Loop: Execute, observe failure, attribute to a specific harness layer, fix that layer, re-execute. This is the core methodology of harness engineering. Definition of Done: A set of machine-verifiable conditions — tests pass, lint is clean, type checks pass. Without an explicit definition of done, the agent will invent its own.
    Lecture 01. Strong Models Don't Mean Reliable Execution | Learn Harness Engineering
  • Long tasks spanning sessions are even worse — all discoveries from the previous session are lost, and every new session has to re-explore the project structure and re-understand the code organization. Agents without persistent state see failure rates spike sharply on tasks exceeding 30 minutes.
    Lecture 01. Strong Models Don't Mean Reliable Execution | Learn Harness Engineering
  • Even more common: there's simply no way to verify. No tests, no lint, or verification commands never communicated to the agent. The agent writes code, looks at it, decides it's fine, says "done." It's like asking a student to submit homework with no answer key — they think they got it right, but when you grade it there's a pile of errors. Anthropic also observed an interesting phenomenon: when agents sense context is running low, they rush to finish, skip verification, and choose a simple solution over the optimal one. They call it "context anxiety" — the same thing that happens when you reali…
    Lecture 01. Strong Models Don't Mean Reliable Execution | Learn Harness Engineering
  • he environment is a trap too. Incomplete dev environment, missing dependencies, wrong tool versions. The agent burns precious context window on pip install failures and Node version mismatches instead of solving your actual task. Like hiring a skilled carpenter but forgetting to provide a hammer, nails, or a level workbench — no matter how talented, they can't do the job.
    Lecture 01. Strong Models Don't Mean Reliable Execution | Learn Harness Engineering
  • Even when you do specify, the project has implicit architectural conventions the agent doesn't know. Your team standardized on SQLAlchemy 2.0 syntax, but the agent writes 1.x code by default. All API endpoints must use OAuth 2.0 authentication, but that rule only exists in your head and a Slack message from three months ago. The agent can't see these — it's not that it doesn't want to comply, it literally doesn't know these rules exist.
    Lecture 01. Strong Models Don't Mean Reliable Execution | Learn Harness Engineering
  • nd-to-End Algorithm: Intent-Formalized Code Generation Here is a practical workflow synthesizing TiCoder, specification validation, and the formalization spectrum: Phase 1: Intent Clarification (Interactive) INPUT: Natural language prompt P from user OUTPUT: Ranked code candidates + validated test suite 1. GENERATE CANDIDATES - Use LLM to generate N diverse code implementations {C₁, C₂, ..., Cₙ} from P - Diversity ensures different interpretations are captured 2. GENERATE AMBIGUITY-TARGETED TESTS - For each pair of candidates (Cᵢ, Cⱼ), find input x where Cᵢ(x) ≠ Cⱼ(x) - Generate k test cases T…
    Intent Formalization: A Grand Challenge for Reliable Coding in the Age of AI Agents | alphaXiv
  • iCoder addresses a key problem: when an LLM generates code from a natural language prompt, how do you know if it understood your intent correctly? Instead of making the user read and verify the code, TiCoder asks the user to validate tests that disambiguate different possible interpretations. How It Works Input: Developer provides a natural language prompt (e.g., "find common elements in two lists") Generate candidates: The LLM produces multiple code implementations that might have different behaviors Generate targeted tests: TiCoder creates tests focusing on points of ambiguity—inputs where t…
    Intent Formalization: A Grand Challenge for Reliable Coding in the Age of AI Agents | alphaXiv
  • igure 3 illustrates the TiCoder interactive workflow for test-driven intent formalization. It's a loop where the system uses candidate tests to disambiguate what the user actually wants, rather than making assumptions. The Workflow (Step by Step) Developer LLM generates Generate Approve / Ranked code + NL Prompt → candidate code → candidate tests → reject tests → approved tests ↓ (Yes / No / Undef) ↓ prune & iterate Developer provides a natural language prompt (e.g., "Find the shared elements from two lists") LLM generates multiple candidate code implementations (different interpretations of t…
    Intent Formalization: A Grand Challenge for Reliable Coding in the Age of AI Agents | alphaXiv
  • Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions? This paper provides crucial evidence for the main article's thesis by demonstrating that LLMs can generate formal postconditions from natural language. It is highlighted for showing that these specifications are meaningful enough to catch real-world bugs in the Defects4J benchmark, outperforming prior methods. Madeline Endres, Sarah Fakhoury, Saikat Chakraborty, and Shuvendu K. Lahiri. 2024. Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?. In Proceedings…
    Intent Formalization: A Grand Challenge for Reliable Coding in the Age of AI Agents | alphaXiv
  • A Spectrum of Specifications Intent formalization does not require a single, all-encompassing formal proof for every piece of code. Instead, it offers a spectrum of specifications, ranging from lightweight behavioral examples to mathematically rigorous logical contracts. Input/Output Tests: The most basic form of formalization involves generating concrete examples of intended behavior. For a function designed to remove duplicates from a list, a test would specify that an input like [1, 2, 3, 2, 4] must result in [1, 2, 3, 4] or [1, 3, 4]. These tests are easy to generate and can quickly identi…
    Intent Formalization: A Grand Challenge for Reliable Coding in the Age of AI Agents | alphaXiv
  • Each test case consists of a CLI input and any associated data files. To pass, the AI’s solution must produce exactly the same output as the reference program. T
    MirrorCode: Evidence AI can already do some weeks-long coding tasks | Epoch AI
  • he AI’s solution is evaluated via end-to-end tests derived from the original program’s test suites, real-world data, and LLM-assisted generation.
    MirrorCode: Evidence AI can already do some weeks-long coding tasks | Epoch AI
  • n code-to-index , an LLM reads an existing repository and generates structured indexes that provide a system-level cognitive map. In index-first , developers describe requirements in natural language, and an LLM produces a complete system index as an architectural blueprint for later im- plementation
    [2605.02421] AOCI: Symbolic-Semantic Indexing for Practical Repository-Scale Code Understanding with LLMs
  • Existing software documentation mainly operates at two levels: function-level artifacts, such as comments and API descriptions, and project-level artifacts, such as design documents and architectural overviews. What is largely missing between them is a file-level intent layer — a structured representation that captures each file’s business role, key dependencies, importance, and position within the repository architecture
    [2605.02421] AOCI: Symbolic-Semantic Indexing for Practical Repository-Scale Code Understanding with LLMs
  • he reason is not hard to see. Real-world repositories are mostly low-entropy content — imports, boilerplate, formatting, type an- notations — that carries little architectural meaning but consumes attention. The high-entropy information that actually matters — which module owns which responsibility, how modules depend on each other, what design decisions constrain the implementation — is scattered across files and never stated explicitly. LLMs exhibit position-decay effects in long contexts, making distant but impor- tant information harder to use for cross-file reasoning [ 22 , 28 ], and insu…
    [2605.02421] AOCI: Symbolic-Semantic Indexing for Practical Repository-Scale Code Understanding with LLMs
  • The final key part to making this system work really well is the communication bridge between the coding agent and review agent. Basically, does Devin properly use its broader context of user instructions, decisions, etc. to filter the bugs that come back from Devin Review? This is key to preventing looping, disobeying the user, doing work that is out of scope, and so on. We found that with some dedicated prompting, models today can make reasonable judgment calls here, and you end up getting some very interesting interactions between the two agents and human
    Cognition | Multi-Agents: What's Actually Working
  • You would think that making a model review its own code would not result in any useful findings. But even on PRs written by Devin, Devin Review catches an average of 2 bugs per PR, of which roughly 58% are severe (logic errors, missing edge cases, security vulnerabilities). Often the system will loop through multiple code-review cycles, finding new bugs each time (which isn't always great since it can take a while). Today, we make Devin and Devin Review natively iterate against one another, so that most bugs are already resolved by the time a human opens the PR. The counterintuitive part. Inte…
    Cognition | Multi-Agents: What's Actually Working
  • The complexity inference framework represents a significant technical contribution. It processes Python functions by: Input Expansion: Systematically increasing input sizes using various strategies (random, identity, combined expansion) Isolated Execution: Running code in sandboxed environments using Bubblewrap to prevent side effects Performance Measurement: Capturing runtime with Cprofiler and memory usage with tracemalloc Curve Fitting: Applying non-negative least squares fitting across major complexity classes with simplicity bias
    BigO(Bench) -- Can LLMs Generate Code with Controlled Time and Space Complexity? | alphaXiv
  • The benchmark dataset comprises 3,105 coding problems and 1,190,250 human solutions sourced primarily from competitive programming platforms like Codeforces. Each solution is automatically annotated with complexity classes, curve coefficients, and raw performance measurements. The complexity distribution reflects real-world programming patterns, with linear time O(n) and constant space O(1) being most prevalent, while exhibiting the expected heavy-tailed distribution across 11 time complexity classes and 5 space complexity classes. The evaluation protocol employs multiple metrics: Pass@k: Meas…
    BigO(Bench) -- Can LLMs Generate Code with Controlled Time and Space Complexity? | alphaXiv
  • Modern post-training has a see-saw problem: Math RLVR shortens reasoning traces and hurts open-ended writing. RLHF buys preference alignment at the cost of strict instruction following. Tool-use RL drifts away from STEM benchmarks. When every specialization stage trades against the others, shipping one model that holds onto everything becomes difficult. On-policy distillation (OPD) has emerged as a standard fix. The idea: sample trajectories from the student, then match a teacher's distribution along those rollouts via reverse KL. You get dense, token-level supervision that drops into a GRPO-s…
    Multi-Teacher On-Policy Distillation: A New Post-Training Primitive | Notion
  • is not about “passing on the first try” but rather the probability of a single sample being correct Due to randomness in model outputs, we need estimators to calculate from finite samples The naive estimator is biased and underestimates the true OpenAI’s unbiased estimator provides accurate estimates by properly accounting for sampling without replacement The gap between pass@1 and pass@k reveals opportunities for improving reliability through multi-sampling
    Statistics for AI/ML, Part 4: pass@k and Unbiased Estimator
  • Practical Example Suppose you generate code samples and are correct: Naive estimator: Unbiased estimator: The unbiased estimator gives a significantly higher probability (91.7%) compared to the naive estimator (83.2%). This difference occurs because the naive estimator treats each draw as independent, while the unbiased estimator correctly accounts for the fact that we’re selecting 5 samples from a finite pool of 10 without replacement. The difference becomes more pronounced with smaller sample sizes or when k approaches n. Applications in AI Systems Understanding helps us interpret benchmarks…
    Statistics for AI/ML, Part 4: pass@k and Unbiased Estimator
  • But in reality, when we select k samples from our n generated samples, we’re sampling without replacement. Once we pick a sample, we do not put it back. This means: If our first sample is incorrect, we have one fewer incorrect sample in the pool The probability of selecting another incorrect sample changes from to for . The samples are no longer independent events
    Statistics for AI/ML, Part 4: pass@k and Unbiased Estimator
  • Automated environment generation is an emerging approach to scaling environment diversity. Rather than hand-authoring environments, LLM coding agents write new environment code. AutoEnv (Wang et al., 2025) reports ~$4/env average cost.
    A Taxonomy of RL Environments for LLM Agents
  • Some environments are stateless — each episode starts fresh with no memory of prior runs. A coding agent solving LeetCode problems needs no persistent state. But some environments are stateful: a coding agent that must manipulate a database carries state across actions, and an enterprise agent carries state across episodes. EnterpriseOps-Gym (Zhang et al., 2026) maintains 164 database tables and 512 tools across episodes, where actions in one task affect the state seen by subsequent tasks. That’s a fundamentally different problem for agents to learn.
    A Taxonomy of RL Environments for LLM Agents
  • Verifiable beats judgeable. Programmatic checks such as string match or code execution, are faster, cheaper, and more consistent than LLM-as-judge. Use LLM-as-judge when there’s no other option, not as the default.
    A Taxonomy of RL Environments for LLM Agents
  • Exact match Binary (0/1) Ground truth available Code execution Binary or partial Output can be tested programmatically LLM-as-judge Continuous [0,1] Open-ended quality, no other option Checklist-style Continuous Multi-criteria research tasks Evolving rubric (RLER) Continuous Resistant to reward hacking Process reward model (PRM) Per-N-step continuous Long-horizon credit assignment Pairwise comparison Relative rank Relative quality matters more than absolute Multi-criteria composite Weighted sum Multiple quality dimensions
    A Taxonomy of RL Environments for LLM Agents
  • generating outputs with AI agents is cheap, but verifying their quality becomes progressively harder as tasks grow more open-ended. The goal of the verifier is to map a large, stochastic space of inputs and outcomes into a narrow reward signal, typically between 0 and 1. Designing this mapping is a core challenge in building RL environments.
    A Taxonomy of RL Environments for LLM Agents
  • Harness Type Description When to Use Single-Turn One prompt, one response Math, factual QA Multi-Turn Back-and-forth dialogue Games, structured tasks Tool-Use Model calls tools, receives results Agent benchmarks Stateful Tool-Use Tools modify persistent state Enterprise workflows, SWE-Bench Agentic Full Observation→Orient→Decide→Act (OODA) loop Deep research, complex workflows
    A Taxonomy of RL Environments for LLM Agents
  • Synthetic data for tasks is increasingly a first-class problem. With real-world productivity and research tasks, you rarely have a large labeled dataset. Strategies for generating synthetic tasks include: Back translation: Start from a desired output, reconstruct the task input that would produce it Graph-based synthesis: Build a knowledge graph, generate multi-hop queries over it
    A Taxonomy of RL Environments for LLM Agents
  • n the agent world, a logged record of an agent’s execution — including tool calls, observations, and intermediate outputs — is called a trace. A trajectory is what the trainer sees (state-action-reward tuples); a trace is what the observability system sees (structured execution logs).
    A Taxonomy of RL Environments for LLM Agents
  • In RL, the sequence of states, actions, and rewards that an agent produces while solving a task is called a trajectory. A single run from start to completion is an episode, and the process of executing a policy to generate a trajectory is called a rollout.
    A Taxonomy of RL Environments for LLM Agents
  • Code generation Write code, run it, check outputs SWE-Bench, LiveCodeBench Code review & repair Detect bugs, suggest fixes, verify patches CodeReview-Bench, DebugBench Repository-level coding Navigate large codebases, multi-file edits, resolve issues SWE-Bench Verified, RepoBench
    A Taxonomy of RL Environments for LLM Agents
  • An RL environment for an LLM agent bundles the following objects: a dataset of task inputs, a harness for the model, a reward function to score outputs, the state of the environment, and configurations of the environment. Note that we specifically bundle tasks with the environments as tasks are most often environment dependent. As an example, a coding task is bundled with a coding environment, not with a research environment. With this framing, the training loop looks like this:
    A Taxonomy of RL Environments for LLM Agents
  • Isolation against the model’s mistakes. A coding agent that mounts your repo and has shell access can, and eventually will, delete the wrong directory. Filesystem isolation, copy-on-write snapshots, and per-session ephemerality turn destructive actions into recoverable ones.
    Hidden Technical Debt of AI Systems: Agent Runtime
  • An agent is what you get when you wrap the agentic model with harnesses so it can take actions, observe effects, feed those observation back into the next call. The execution environment where that happens is the runtime.
    Hidden Technical Debt of AI Systems: Agent Runtime
  • Improving the Tester is more complex because a "good" test must be both valid (correctly assessing the problem) and discriminative (able to catch subtle bugs). ZeroCoder defines the Tester reward 𝑟 𝑡 𝑗 r t j ​ ​ as a combination of three terms:
    ZeroCoder: Can LLMs Improve Code Generation Without Ground-Truth Supervision? | alphaXiv
  • A significant challenge in label-free reinforcement learning is the prevalence of uninformative data. If a problem is too simple, all sampled solutions might pass all sampled tests. Conversely, if a problem is too complex, all solutions might fail. In both cases, the passing matrix 𝐸 E is "degenerate"—it lacks diversity in its entries, leading to vanishing gradients that stall the learning proces
    ZeroCoder: Can LLMs Improve Code Generation Without Ground-Truth Supervision? | alphaXiv