Eliminating Toil and Monitoring Distributed Systems

<div> <div> <div> <p>Join the companion discussion on Thursday at 7 pm Eastern <a href="https://bookclub.dev/thursdays" target="_blank" rel= "noopener">bookclub.dev/thursdays</a></p> <h1>Eliminating Toil</h1> <p>"If a human operator needs to touch your system during normal operations, you have a bug. The definition of normal changes as your systems grow."</p> <p>-Carla Geisser, Google SRE</p> <p><strong>4 types of work</strong></p> <ul> <li>Software engineering</li> <li>Systems engineering</li> <li>Toil</li> <li>Overhead</li> </ul> <p><strong>What makes it toil?</strong></p> <ul> <li>Manual</li> <li>Repetitive</li> <li>Automatable</li> <li>Tactical</li> <li>No enduring value</li> <li>Scales linearly with service growth</li> </ul> <p>None of these individually are enough to make it toil, but the more boxes it checks the more likely it is toil</p> <p>Toil isn't always bad, and it is not possible to completely eliminate</p> <p>Figuring out somethings type of work largely revolves around how much value it creates and on what time scale</p> <p>Toil tends to grow and expand if left unchecked</p> <p>Tracking types of work</p> <p>Google has quarterly surveys to ensure they are meeting or beating their <= 50% toil time</p> <p>"If we all commit to eliminate a bit of toil each week with some good engineering, we’ll steadily clean up our services, and we can shift our collective efforts to engineering for scale, architecting the next generation of services, and building cross-SRE toolchains. Let’s invent more, and toil less."</p> <h1>Monitoring Distributed Systems</h1> <p>Terms around monitoring</p> <p>Not consistent, but a basic idea</p> <ul> <li>Monitoring</li> <li>White-box monitoring</li> <li>Black box monitoring</li> <li>Dashboard</li> <li>Alert</li> <li>Root cause</li> <li>Node and machine</li> <li>Push</li> </ul> <p><strong>What can you get from monitoring?</strong></p> <p>Trends, help debugging, alerts, baselines to compare from, data for the business to analyze, and things to analyze in the event of a security breac