✳flâneur — a map of the web's best reading

Mind the Trust Gap: Fast, Private Local-to-Cloud LLM Chat · Hazy Research

hazyresearch.stanford.edu · 2,280 words · saved by 1 readers

TL;DR: The messages we send to and receive from LLMs can often be quite private. Right now, these messages sit as plain text in the cloud, open to inspection or storage. We show it doesn’t have to be that way: inference can happen securely, with messages decrypted only inside the trusted zone of NVIDIA’s Hopper H100—sealed off even from the cloud provider. Specifically, we implemented a Trusted Execution Environment (TEE) protocol that securely connects a trusted local client with the confidential GPU. Our approach includes: (1) an ephemeral key exchange between the client and GPU enclave to derive a shared secret; (2) attestation, where the enclave proves it’s a genuine H100 running in Confidential Computing mode; (3) encrypted, signed, and nonce-protected message passing; and (4) inference entirely within the enclave, where prompts are decrypted and processed securely—never exposed to the cloud. While traditional security measures often add significant performance overhead, our TEE p

Mind the Trust Gap: Fast, Private Local-to-Cloud LLM Chat · Hazy Research May 12, 2025 · 12 min read Mind the Trust Gap: Fast, Private Local-to-Cloud LLM Chat Dan Biderman* , Avanika Narayan* , and Chris Ré . TL;DR : The messages we send to and receive from LLMs can often be quite private . Right now, these messages sit as plain text in the cloud, open to inspection or storage. Current privacy techniques either degrade quality or are several orders of magnitude slower. We explore an alternative Trusted-Execution-Environment (TEE) protocol with messages decrypted only inside a remote confidenti

Explore this link on the map →

related reading