Stephen Lu
1 followers · 5 following · 901 views
on the atlas · 36
- Hacker News5 savers
- Asimov — Intelligent Design2 savers
- Understanding Convolutions on Graphs4 savers
- How does in-context learning work? A framework for understanding the differences from traditional supervised learning | SAIL Blog3 savers
- Hyena Hierarchy: Towards Larger Convolutional Language Models · Hazy Research2 savers
- Understanding UMAP9 savers
- Home | Petar Veličković1 savers
- The Building Blocks of Interpretability6 savers
- Feature Visualization7 savers
- A Gentle Introduction to Graph Neural Networks27 savers
- Scaling in the service of reasoning & model-based ML - Yoshua Bengio1 savers
- 薛之谦 《你还要我怎样》钢琴谱|学琴记1 savers
- Main Page - Algorithms for Competitive Programming2 savers
- ekzlib: code library1 savers
- Statlect, the digital textbook | Probability, statistics, matrix algebra1 savers
- What is Quantum?1 savers
- Learn quantum computing: a field guide - IBM Quantum1 savers
- https://qiskit.org/documentation/qc_intro.html#qc-intro-interference1 savers
- How to Work Hard22 savers
- How to Start a Startup15 savers
- Hugging Face – The AI community building the future.1 savers
- TermsFeed Generators - TermsFeed1 savers
- Applying for a Québec Perspective Scholarship | Gouvernement du Québec1 savers
- Stanford University CS231n: Deep Learning for Computer Vision1 savers
- Supervised Machine Learning: Regression and Classification | Coursera1 savers
- How To Be Successful96 savers
- 15-410 Coding Style and Doxygen Documentation1 savers
- High Scalability -1 savers
- donnemartin/system-design-primer: Learn how to design large-scale systems. Prep for the system design interview. Includes Anki flashcards.5 savers
- The Art of Computer Programming, Volumes 1-4A Boxed Set: 8580001170602: Computer Science Books @ Amazon.com1 savers
- Software Foundations1 savers
- Justin Glibert - How I learned modern Machine Learning12 savers
- What every computer science major should know11 savers
- Master Plan - Justin Glibert (Foundation) - by David10 savers
- Essays8 savers
- Understanding biology, quickly -8 savers
highlights · 56
in-context learning is mainly about locating latent concepts learned during pretraining. In particular, if terms in a particular instance are exposed many times in the pretraining data, the model is likely to know better about the distribution of the inputs. This will provide better evidence to locate latent concepts to perform a downstream task, according to Bayesian inference
How does in-context learning work? A framework for understanding the differences from traditional supervised learning | SAIL Blogthe model still does Bayesian inference as long as there is still enough signal (such as the correct input distribution, output space, and format). Of course, having the correct input-output mapping can still help by providing more evidence and reducing noise, especially when the input-output mapping doesn’t show up often in pretraining data
How does in-context learning work? A framework for understanding the differences from traditional supervised learning | SAIL Blogconditioning on the correct input distribution is important
How does in-context learning work? A framework for understanding the differences from traditional supervised learning | SAIL Blogconditioning on the correct output space is important
How does in-context learning work? A framework for understanding the differences from traditional supervised learning | SAIL BlogReplacing ground truth outputs with random outputs hurts performance significantly less than previously thought
How does in-context learning work? A framework for understanding the differences from traditional supervised learning | SAIL Blog“locating” learned capabilities can be viewed as Bayesian inference of a prompt concept that every example in the prompt shares
How does in-context learning work? A framework for understanding the differences from traditional supervised learning | SAIL Blogmain assumption on the structure of pretraining documents is that a document is generated by first sampling a latent concept, and then the document is generated by conditioning on the latent concept
How does in-context learning work? A framework for understanding the differences from traditional supervised learning | SAIL Blogdocument-level latent concept creates long-term coherence, and modeling this coherence during pretraining requires learning to infer the latent concept:
How does in-context learning work? A framework for understanding the differences from traditional supervised learning | SAIL BlogIn-context learning is a mysterious emergent behavior in large language models (LMs) where the LM performs a task just by conditioning on input-output examples, without optimizing any parameters.
How does in-context learning work? A framework for understanding the differences from traditional supervised learning | SAIL Blogparticular parametrizations of the long convolutions scale more favorably in sequence length and vocabulary size, especially when paired with shorter explicit filters
Hyena Hierarchy: Towards Larger Convolutional Language Models · Hazy Researchdata-controlled linear operator
Hyena Hierarchy: Towards Larger Convolutional Language Models · Hazy Researchwithout data-control, any kind of in-context learning becomes much more challenging.
Hyena Hierarchy: Towards Larger Convolutional Language Models · Hazy ResearchRecent developments in state-space models for language modeling have shown us that with just a little bit of attention - at most a few layers - we can match the quality of fully-attentional models up to the 2.7 billion parameter scale. However, fully removing attention from these new models (and training with the same tokenizer!) reveals a gap
Hyena Hierarchy: Towards Larger Convolutional Language Models · Hazy ResearchTo understand multiple layers together, we would like each layer’s factorization to be “compatible” — to have the groups of earlier layers naturally compose into the groups of later layers.
The Building Blocks of InterpretabilityMatrix factorization lets us pick what our groupings are optimized for, giving us a better tradeoff than the natural groupings we saw earlier.
The Building Blocks of InterpretabilityOn the one hand, it is very easy to end up with an overwhelming amount of information: it would take hours of human auditing to understand the long-tail of channels that slightly impact the output. On the other hand, both the aggregations we have explored are extremely lossy and can miss important parts of the story. And, while we could avoid lossy aggregation by working with individual neurons, and not aggregating at all, this explodes the first problem combinatorially.
The Building Blocks of Interpretabilityalternate way to slice the cube is by channels instead of spatial locations. Doing so allows us to perform channel attribution: how much did each detector contribute to the final output?
The Building Blocks of InterpretabilityRather than asking whether the color of a particular pixel was important for the “labrador retriever” classification, we instead ask whether the high-level idea detected at that position (such as “floppy ear”) was important.
The Building Blocks of Interpretabilitysaliency map — a simple heatmap that highlights pixels of the input image that most caused the output classification.
The Building Blocks of InterpretabilityFirst, it is not clear that individual pixels should be the primary unit of attribution. The meaning of each pixel is extremely entangled with other pixels, is not robust to simple visual transforms (e.g., brightness, contrast, etc.), and is far-removed from high-level concepts like the output class
The Building Blocks of Interpretabilityforeign abstractions that neural networks learn than native human language.
The Building Blocks of InterpretabilityThere also may exist abstractions which are visually familiar, yet that we lack good natural language descriptions for: for example, take the particular column of shimmering light where sun hits rippling water.
The Building Blocks of InterpretabilityThis is important because when neurons appear to correspond to human ideas, it is tempting to reduce them to words. Doing so, however, is a lossy operation — even for familiar abstractions, the network may have learned a deeper nuance.
The Building Blocks of Interpretabilityfind examples that still activate the optimization target highly even if we slightly transform them
Feature VisualizationExamples like these suggest that neurons are not necessarily the right semantic units for understanding neural nets.
Feature VisualizationThe main idea for batching with graphs is to create subgraphs that preserve essential properties of the larger graph.
A Gentle Introduction to Graph Neural NetworksA common practice for training neural networks is to update network parameters with gradients calculated on randomized constant size (batch size) subsets of the training data (mini-batches). This practice presents a challenge for graphs due to the variability in the number of nodes and edges adjacent to each other, meaning that we cannot have a constant batch size.
A Gentle Introduction to Graph Neural NetworksAs argued below, to reason well, we need to have 1) a good model of the world and 2) a powerful inference machine to generate solutions compatible with the world model.
Scaling in the service of reasoning & model-based ML - Yoshua BengioThese algorithms can use interference effects to make the wrong answers cancel out quickly and give us a high probability of measuring the right answer
What is Quantum?amplitudes of finding the coin (qubit) in the state 1 cancel each other out, and we call this effect interference.
What is Quantum?square our amplitudes to calculate the probabilities
What is Quantum?too far apart to influence each other can nevertheless behave in ways that, though individually random, are somehow strongly correlated
Learn quantum computing: a field guide - IBM QuantumA physical system in a definite state can still behave randomly.
Learn quantum computing: a field guide - IBM QuantumEntangled states are states of the whole computer that do not correspond to any assignment of digital or analog states of the individual qubits.
Learn quantum computing: a field guide - IBM Quantumqubits, can represent a 0, 1, or linear combination of both, which is a property known as superposition
Learn quantum computing: a field guide - IBM QuantumIn this model, all information is reducible to bits, which can take the values of either 0 or 1. Additionally, all processing can be performed via simple logic gates (AND, OR, NOT, XOR, XNOR) acting on one or two bits at a time, or be entirely described by NAND (or NOR)
Learn quantum computing: a field guide - IBM Quantumthe amplitude and phase of qubits are continuous degrees of freedom upon which operations can never be done exactly.
https://qiskit.org/documentation/qc_intro.html#qc-intro-interferenceQuantum gates represent information preserving, reversible transformations on the quantum data stored in qubits.
https://qiskit.org/documentation/qc_intro.html#qc-intro-interferenceoperations like “measurement”, represented by the meter symbol in a box with a line connecting to a “target” wire, extract partial information about a qubit’s state, often losing the phase, to be able to represent it as a classical bit and write that classical bit onto the target wire (often a fully classical wire in some readout device)
https://qiskit.org/documentation/qc_intro.html#qc-intro-interferenceclassical algorithm that generates a description of a quantum circuit
https://qiskit.org/documentation/qc_intro.html#qc-intro-interferenceQuantum circuits enable a quantum computer to take in classical information and output a classical solution, leveraging quantum principles such as interference and entanglement to perform the computation.
https://qiskit.org/documentation/qc_intro.html#qc-intro-interferenceWhat I've learned since I was a kid is how to work toward goals that are neither clearly defined nor externally imposed. You'll probably have to learn both if you want to do really great things.
How to Work HardWhen everyone feels they're getting a slightly bad deal, that they're doing more than they should for the amount of stock they have, the stock is optimally apportioned.
How to Start a StartupIf you build the simple, inexpensive option, you'll not only find it easier to sell at first, but you'll also be in the best position to conquer the rest of the market.
How to Start a StartupSo as a rule you can recognize genuinely smart people by their ability to say things like "I don't know," "Maybe you're right," and "I don't understand x well enough."
How to Start a StartupCould you describe the person as an animal? It might be hard to translate that into another language, but I think everyone in the US knows what it means. It means someone who takes their work a little too seriously; someone who does what they do so well that they pass right through professional and cross over into obsessive.
How to Start a Startup“is this person a force of nature?”
How To Be SuccessfulRemember that you are mostly looking for rate of improvement, and don’t overvalue experience or current accomplishment.
How To Be SuccessfulThe best way to make up for your weaknesses is to hire complementary team members instead of just hiring people who are good at the same things you are.
How To Be SuccessfulMost people do whatever most people they hang out with do. This mimetic behavior is usually a mistake—if you’re doing the same thing everyone else is doing, you will not be hard to compete with.
How To Be Successful