MrKTF
0 followers · 4 following · 262 views
on the atlas — 27
- Skills > MCP | Medium1 savers
- Demystifying evals for AI agents \ Anthropic12 savers
- My 7 Takeaways from Deep Work by Cal Newport | by Rational Badger | Medium1 savers
- Productivity - Sam Altman57 savers
- The days are long but the decades are short - Sam Altman72 savers
- Breadcrumbs - by Simon Sarris - The Map is Mostly Water12 savers
- Jeremy Howard - Chinese - YouTube1 savers
- fast.ai – Providing a Good Education in Deep Learning2 savers
- Network School First Impressions - Home1 savers
- ADHD Is Not a Learning Disability (But it Does Affect Learning) - ADDA - Attention Deficit Disorder Association1 savers
- ADHD can cause excessive nail-biting, hair-pulling, and skin-picking1 savers
- Hyperfocus: The Flip Side of ADHD? - Child Mind Institute1 savers
- Overcoming ADHD to Rediscover the Joy of Reading | Teva Stories1 savers
- Writing advice - Alexey Guzey3 savers
- Omens of exceptional talent - Alexey Guzey17 savers
- Intelligence killed genius - Alexey Guzey1 savers
- 7 easy ways of reducing neck pain while studying or working1 savers
- Ergonomics for Prolonged Sitting - Spine Care | UCLA Health1 savers
- 3 ways to avoid back pain when sitting at your desk - The Waldegrave Clinic1 savers
- Mastering Your Dopamine: The Key to Unlocking Hyper-Focus and Flow State | by Med Elassal | Medium1 savers
- How to Stay Productive (Even When You Feel Like Garbage) - Nick Wignall1 savers
- IB English Individual Oral (IO) Explained | LitLearn1 savers
- How to Overcome Perfectionism and Procrastination - Faculty of Health and Behavioural Sciences - University of Queensland1 savers
- The Little-Known Power of Going Slow to Go Fast | by Saleem Rana | Publishous | Medium1 savers
- Theory of Change (Aaron Swartz's Raw Thought)83 savers
- Why we stopped making Einsteins - by Erik Hoel13 savers
- Self-Discipline - Persisting Until You Reach Your Goals1 savers
highlights — 341
Some evaluations have subtle failure modes that result in low scores even with good agent performance, as the agent fails to solve tasks due to grading bugs, agent harness constraints, or ambiguity
Demystifying evals for AI agents \ AnthropicIt can also help to create clear, structured rubrics to grade each dimension of a task, and then grade each dimension with an isolated LLM-as-judge rather than using one to grade all dimensions
Demystifying evals for AI agents \ AnthropicTo avoid hallucinations, give the LLM a way out like providing an instruction to return “Unknown” when it doesn’t have enough information
Demystifying evals for AI agents \ AnthropicModel grading often takes careful iteration to validate accuracy. LLM-as-judge graders should be closely calibrated with human experts to gain confidence that there is little divergence between the human grading and model grading
Demystifying evals for AI agents \ AnthropicFor tasks with multiple components, build in partial credit. A support agent that correctly identifies the problem and verifies the customer but fails to process a refund is meaningfully better than one that fails immediately. It’s important to represent this continuum of success in results.
Demystifying evals for AI agents \ AnthropicThere is a common instinct to check that agents followed very specific steps like a sequence of tool calls in the right order. We’ve found this approach too rigid and results in overly brittle tests, as agents regularly find valid approaches that eval designers didn’t anticipate. So as not to unnecessarily punish creativity, it’s often better to grade what the agent produced, not the path it took.
Demystifying evals for AI agents \ AnthropicWe recommend choosing deterministic graders where possible, LLM graders where necessary or for additional flexibility, and using human graders judiciously for additional validation.
Demystifying evals for AI agents \ Anthropicgreat eval design involves choosing the best graders for the agent and the tasks
Demystifying evals for AI agents \ AnthropicUnnecessary shared state between runs (leftover files, cached data, resource exhaustion) can cause correlated failures due to infrastructure flakiness rather than agent performance
Demystifying evals for AI agents \ AnthropicShared state can also artificially inflate performance.
Demystifying evals for AI agents \ AnthropicEach trial should be “isolated” by starting from a clean environment.
Demystifying evals for AI agents \ AnthropicIt’s essential that the agent in the eval functions roughly the same as the agent used in production, and the environment itself doesn’t introduce further noise.
Demystifying evals for AI agents \ AnthropicFor instance, if you only test whether the agent searches when it should, you might end up with an agent that searches for almost everything. Try to avoid class-imbalanced evals.
Demystifying evals for AI agents \ AnthropicTest both the cases where a behavior should occur and where it shouldn't. One-sided evals create one-sided optimization
Demystifying evals for AI agents \ AnthropicFor each task, it’s useful to create a reference solution: a known-working output that passes all graders. This proves that the task is solvable and verifies graders are correctly configured.
Demystifying evals for AI agents \ AnthropicWith frontier models, a 0% pass rate across many trials (i.e. 0% pass@100) is most often a signal of a broken task, not an incapable agent, and a sign to double-check your task specification and graders
Demystifying evals for AI agents \ AnthropicEverything the grader checks should be clear from the task description; agents shouldn’t fail due to ambiguous specs
Demystifying evals for AI agents \ AnthropicEach task should be passable by an agent that follows instructions correctly
Demystifying evals for AI agents \ Anthropicmodel-based graders: vague rubrics produce inconsistent judgments.
Demystifying evals for AI agents \ AnthropicA good task is one where two domain experts would independently reach the same pass/fail verdict
Demystifying evals for AI agents \ AnthropicGetting task quality right is harder than it seems
Demystifying evals for AI agents \ Anthropicprioritizing by user impact helps you invest effort where it counts.
Demystifying evals for AI agents \ AnthropicBegin with the manual checks you run during development
Demystifying evals for AI agents \ Anthropicit’s best to take the 80/20 approach in the beginning
Demystifying evals for AI agents \ AnthropicWe see teams delay building evals because they think they need hundreds of tasks. In reality, 20-50 simple tasks drawn from real failures is a great start
Demystifying evals for AI agents \ Anthropicpass^k measures the probability that all k trials succeed.
Demystifying evals for AI agents \ Anthropicpass@k measures the likelihood that an agent gets at least one correct solution in k attempts.
Demystifying evals for AI agents \ Anthropicwhat we want to measure is how often (what proportion of the trials) an agent succeeds for a task.
Demystifying evals for AI agents \ AnthropicEach task has its own success rate—maybe 90% on one task, 50% on another—and a task that passed on one eval run might fail on the next
Demystifying evals for AI agents \ Anthropicagent behavior varies between runs, which makes evaluation results harder to interpret
Demystifying evals for AI agents \ AnthropicBrowser use agents require a balance between token efficiency and latency. DOM-based interactions execute quickly but consume many tokens, while screenshot-based interactions are slower but more token-efficient
Demystifying evals for AI agents \ AnthropicWebArena tests browser-based tasks, using URL and page state checks to verify the agent navigated correctly
Demystifying evals for AI agents \ AnthropicOSWorld extends this to full operating system control, with evaluation scripts that inspect diverse artifacts after task completion: file system state, application configs, database contents, and UI element properties.
Demystifying evals for AI agents \ AnthropicEvaluation requires running the agent in a real or sandboxed environment where it can use software applications, and checking whether it achieved the intended outcome.
Demystifying evals for AI agents \ Anthropicsandboxed environment
Demystifying evals for AI agents \ AnthropicThey can use any application with a graphical user interface (GUI), from design tools to legacy enterprise software
Demystifying evals for AI agents \ AnthropicComputer use agents interact with software through the same interface as humans—screenshots, mouse clicks, keyboard input, and scrolling—rather than through APIs or code execution
Demystifying evals for AI agents \ AnthropicTwo benchmarks that incorporate multidimensionality are 𝜏-Bench and its successor, τ2-Bench. These simulate multi-turn interactions across domains like retail support and airline booking, where one model plays a user persona while the agent navigates realistic scenarios.
Demystifying evals for AI agents \ AnthropicUnlike most other evals, they often require a second LLM to simulate the user.
Demystifying evals for AI agents \ AnthropicWhile coding and research agents can also involve many turns of interaction with the user, conversational agents present a distinct challenge: the quality of the interaction itself is part of what you're evaluating.
Demystifying evals for AI agents \ AnthropicGiven the subjective nature of research quality, LLM-based rubrics should be frequently calibrated against expert human judgment to grade these agents effectively.
Demystifying evals for AI agents \ AnthropicFor tasks with objectively correct answers (“What was Company X’s Q3 revenue?”), exact match works. An LLM can flag unsupported claims and gaps in coverage, but also verify the open-ended synthesis for coherence and completeness.
Demystifying evals for AI agents \ AnthropicGroundedness checks verify that claims are supported by retrieved sources, coverage checks define key facts a good answer must include, and source quality checks confirm the consulted sources are authoritative, rather than simply the first retrieved
Demystifying evals for AI agents \ AnthropicComponents of evaluations for agents.
Demystifying evals for AI agents \ AnthropicIn practice, coding evaluations typically rely on unit tests for correctness verification and an LLM rubric for assessing overall code quality
Demystifying evals for AI agents \ Anthropicheuristics-based code quality rules can evaluate the generated code based on more than passing tests, and model-based graders with clear rubrics can assess behaviors like how the agent calls tools or interacts with the user.
Demystifying evals for AI agents \ AnthropicOnce you have a set of pass-or-fail tests for validating the key outcomes of a coding task, it’s often useful to also grade the transcript
Demystifying evals for AI agents \ AnthropicTerminal-Bench takes a different track: it tests end-to-end technical tasks, such as building a Linux kernel from source or training an ML model.
Demystifying evals for AI agents \ Anthropicwidely-used coding agent benchmarks, SWE-bench Verified and Terminal-Bench
Demystifying evals for AI agents \ Anthropicsoftware is generally straightforward to evaluate: Does the code run and do the tests pass?
Demystifying evals for AI agents \ Anthropic