Comments - The Use of Knowledge in (AGI) Society

This seems too optimistic to me. A lot of the importance of tacit knowledge comes from the need to understand the current state of a system when making changes to it. For instance, it's hard to come in and add a feature to a software system without understanding exactly how it currently works. Trying to interfere without this knowledge risks causing unintended effects through unknown unknowns, and this seems true in central planning more generally. But a superintelligent AI system wouldn't be restricted to modifying existing systems — it could conceivably replace them from scratch once it reaches a certain level of performance. Current AI systems might not be able to replicate Google's search algorithm without the tacit knowledge of its employees, but once AI becomes significantly better than humans at software engineering, it would be able to build its own search algorithm from scratch, making that human tacit knowledge disposable. This reminds me of a strategy I once heard of for bec

This seems too optimistic to me. A lot of the importance of tacit knowledge comes from the need to understand the current state of a system when making changes to it. For instance, it's hard to come in and add a feature to a software system without understanding exactly how it currently works. Trying to interfere without this knowledge risks causing unintended effects through unknown unknowns, and this seems true in central planning more generally. But a superintelligent AI system wouldn't be restricted to modifying existing systems — it could conceivably replace them from scratch once it reac