✳flâneur — a map of the web's best reading

Every service should have a killswitch

seangoedecke.com · 857 words · saved by 1 readers

The more time you spend designing systems, the more paranoid you get about things going wrong. The most experienced and paranoid engineers I know build a killswitch into every single piece of automation. If your company has a feature flagging system - which it should - this can be as simple as adding a return if feature_enabled?(pdf_converter_job_killswitch) to the top of your scheduled job. If the job ever goes out of control (running way too often, or using too many resources) you can turn it off by creating or enabling that feature. Enabling a feature flag is usually many minutes quicker than a code deploy. During an incident, when deploying is difficult, it can be hours quicker. A recent example I noticed of this was this Google incident report, which describes: It doesn’t always have to be a feature flag. I’ve seen “safety files” that automation won’t run without - so you can stop the automation by deleting the file. I’ve also heard of packaged software that needs to “phone home”

The more time you spend designing systems, the more paranoid you get about things going wrong. The most experienced and paranoid engineers I know build a killswitch into every single piece of automation. What a killswitch looks like If your company has a feature flagging system - which it should - this can be as simple as adding a return if feature_enabled?(pdf_converter_job_killswitch) to the top of your scheduled job. If the job ever goes out of control (running way too often, or using too many resources) you can turn it off by creating or enabling that feature. Enabling a feature flag is us

Explore this link on the map →

saved by

related reading