flâneur

Anna Wang

40 followers · 16 following · 1475 views

on the atlas — 80

highlights — 336

  • We believe that this recipe will still hold true where the master weights in the parameter server will be FP32 but the actual calculations will be performed in FP8 or even lower such as MX6
    Multi-Datacenter Training: OpenAI's Ambitious Plan To Beat Google's Infrastructure
  • due to convergence challenges with newer model architectures, everyone just simplified their training by moving back to full synchronous gradient descent
    Multi-Datacenter Training: OpenAI's Ambitious Plan To Beat Google's Infrastructure
  • due to the importance of fault tolerant training, publishing of methods has effectively stopped.
    Multi-Datacenter Training: OpenAI's Ambitious Plan To Beat Google's Infrastructure
  • most clusters will be burnt in at both high temperatures and rapidly fluctuating temperatures for a significant amount of time to ensure that all the infant mortality failures are past and have shifted into the random failure phrase.
    Multi-Datacenter Training: OpenAI's Ambitious Plan To Beat Google's Infrastructure
  • fault tolerance is one of the most important aspects to address in scaling clusters of 100k+ GPUs towards a single workload. Nvidia is way behind Google on reliability of their AI systems
    Multi-Datacenter Training: OpenAI's Ambitious Plan To Beat Google's Infrastructure
  • that merely getting a hold of a dangerous agent is the least of the problems of a theoretical bioweapons program, and that small actors are unlikely to be able to effectively pull this off now.
    Book review: Barriers to Bioweapons | Eukaryote Writes Blog
  • One interesting takeaway is that covertness has a substantial cost – forcing a program to “go underground” is a huge impediment to progress. This suggests that the Biological Weapons Convention, which has been criticized for being toothless and lacking provisions for enforcement, is actually already doing very useful work – by forcing programs to be covert at all.
    Book review: Barriers to Bioweapons | Eukaryote Writes Blog
  • Uncertainty and a lack of coordination ultimately lead the program nowhere
    Book review: Barriers to Bioweapons | Eukaryote Writes Blog
  • The book describes the Department of Energy replacing nuclear weapons parts in the late 1990s, and realizing that they no longer knew how to make a particular foam crucial to thermonuclear warheads, that their documentation for the foam’s production was insufficient, and that anyone who had done it before was long retired. They had to spend nine years and 70 million dollars inventing a substitute for a single component
    Book review: Barriers to Bioweapons | Eukaryote Writes Blog
  • but at the low cost of ending up on every watchlist ever,
    Book review: Barriers to Bioweapons | Eukaryote Writes Blog
  • it’d be completely predictable if they gave up their evil plans right there and started volunteering in soup kitchens instead
    Book review: Barriers to Bioweapons | Eukaryote Writes Blog
  • Subtle cues in our behavior influence what other people are capable of.
    The Pygmalion Effect: Proving Them Right
  • Every week, reflect on what things felt less efficient than they needed to be. Think for yourself how to improve these, then talk to friends, colleagues, or mentors to get additional ideas.
    Film Study for Research - Jacob Steinhardt
  • First try to think about whether there’s a way to modify your thought process to reliably come up with such ideas in the future.
    Film Study for Research - Jacob Steinhardt
  • In summary, film study blogs for off-the-cuff research thinking; watch great presentations and record yourself to learn how to speak; pair program and watch programming streams; and read histories of science for long-term research decisions.
    Film Study for Research - Jacob Steinhardt
  • To solve this problem: We’d want people to make conscious decisions about what the AIs should do when all the normal sources of authority disagree and explicitly train the AIs for the right behavior in those circumstances. Also, we’d want them to institute controls to prevent a small number of individuals from unilaterally retraining the AIs.
    Project ideas: Governance during explosive technological growth
  • One proposal to help rectify this power imbalance is to use the framework of solidarity — the equal and just sharing of prosperity and burdens — to more equitably allocate the gains from AI. In practice, this might look like adequately compensating anyone whose actions provide data to train an AI model and redistributing the wealth AI generates for tech companies and their executives.
    “Democracy-Affirming” AI Could Make Things Even Worse
  • Unlike other countries, the U.S. lacks national data protection legislation that would regulate the inputs to AI systems. The White House’s flagship initiative for making AI companies accountable to the public, the AI Bill of Rights blueprint, is in its own words “non-binding and does not constitute U.S. government policy.”
    “Democracy-Affirming” AI Could Make Things Even Worse
  • Basically, once you’ve made a decision, you’ll need to convince others that you’re right and get them to prioritize what you need from them over the other things on their plate.
    Speed as a Habit | First Round Review
  • ust get their input first. Don’t get your work reversed later on
    Speed as a Habit | First Round Review
  • The idea isn't that we'll ask people what their interest levels and business are: we'll try to infer it from what we can observe. And then we can use this inferred information to do what we actually want to do: guess the paper relevance for a given student.
    Probabilistic Programming
  • Examples here include cyborg technologies, brain-computer interfaces etc.
    How Ai Fails Us - RadXChange
  • As this argument runs, the problem is not the use of reward-maximizing algorithms to optimize engagement but the fact that frms are deploying these techniques to maximize proft.
    How Ai Fails Us - RadXChange
  • Moreover, by focusing on replacing human capacities, AEAI imposes a growth ceiling that coincides with displacement
    How Ai Fails Us - RadXChange
  • Automation in this period did not rely on the tools that are today sometimes labeled “AI,” which are more recent origin, but was heavily premised on a robotics tradition that grew out of the previous generation of AEAI in the 1980s.
    How Ai Fails Us - RadXChange
  • external security testing
    FACT SHEET: Biden-Harris Administration Secures Voluntary Commitments from Leading Artificial Intelligence Companies to Manage the Risks Posed by AI | The White House
  • establish or join a forum or mechanism through which they can develop, advance, and adopt shared standards and best practices for frontier AI safety
    Ensuring-Safe-Secure-and-Trustworthy-AI.pdf
  • a) disincentivizes customers from leaving for a competitor or b) reduces/eliminates competition
    Product Advice for Hardware Founders : YC Startup Library | Y Combinator
  • One thing I've learned about rejections: listen to the 'no' but not the 'why'.
    Product Advice for Hardware Founders : YC Startup Library | Y Combinator
  • If you get a 10x increase in compute, you should make your model 3.1x times bigger and the data you train over 3.1x bigger; if you get a 100x increase in compute, you should make your model 10x bigger and your data 10x bigger.
    New Scaling Laws for Large Language Models - LessWrong
  • The search space is huge. It's the cartesian product of all possible types of work, both known and yet to be discovered, and all possible future versions of you.
    How to Do Great Work
  • ew ideas come from doing this about nontrivial things. Which may help explain why people's reaction to a new idea is often the first half of laughing: Ha!
    How to Do Great Work
  • One of the most powerful kinds of copying is to copy something from one field into another.
    How to Do Great Work
  • Indeed, the features that are easiest to imitate are the most likely to be the flaws.
    How to Do Great Work
  • There are definitely some dangers to copying, though. One is that you'll tend to copy old things — things that were in their day at the frontier of knowledge, but no longer are.
    How to Do Great Work
  • Nor does copying necessarily make your work unoriginal. Originality is the presence of new ideas, not the absence of old ones.
    How to Do Great Work
  • But perhaps the worst thing schools do to you is train you to win by hacking the test. You can't do great work by doing that. You can't trick God. So stop looking for that kind of shortcut. The way to beat the system is to focus on problems and solutions that others have overlooked, not to skimp on the work itself.
    How to Do Great Work
  • So when you're learning about something for the first time, pay attention to things that seem wrong or missing.
    How to Do Great Work
  • There's a big difference between doing something you worry might be a waste of time and doing something you know for sure will be. The former is at least a bet, and possibly a better one than you think.
    How to Do Great Work
  • The biggest is probably time. The young have no idea how rich they are in time. The best way to turn this time to advantage is to use it in slightly frivolous ways: to learn about something you don't need to know about, just out of curiosity, or to try building something just because it would be cool, or to become freakishly good at something.
    How to Do Great Work
  • Inexperience makes them fear risk, but it's when you're young that you can afford the most
    How to Do Great Work
  • If you keep projects small and use flexible media, you don't have to plan as much, and your designs can evolve instea
    How to Do Great Work
  • And it is more organized; it just doesn't work as well.
    How to Do Great Work
  • An early version of a new project will sometimes be dismissed as a toy
    How to Do Great Work
  • Begin by trying the simplest thing that could possibly work. Surprisingly often, it does. If it doesn't, this will at least get you started.
    How to Do Great Work
  • You start with something small and evolve it, and the final version is both cleverer and more ambitious than anything you could have planned
    How to Do Great Work
  • Though it sounds more responsible to begin by studying everything that's been done before, you'll learn faster and have more fun by trying stuff
    How to Do Great Work
  • So start lots of small things. Being prolific is underrated.
    How to Do Great Work
  • The best questions grow in the answering. You notice a thread protruding from the current paradigm and try pulling on it, and it just gets longer and longer. So don't require a question to be obviously big before you try answering it. You can rarely predict that. It's hard enough even to notice the thread, let alone to predict how much will unravel if you pull on it.
    How to Do Great Work
  • But actually the more puzzled you are, the better, so long as (a) the things you're puzzled about matter, and (b) no one else understands them either.
    How to Do Great Work