SENTRY Error-Budget-Constrained Causal Canary Orchestrator

Abstract

Error-Budget-Constrained Causal Canary Orchestrator in SENTRY: present a rigorous, evidence-based, formal discussion, prioritizing precise definitions, measurable metrics, and clear causal reasoning; organize content with objective analysis and explicit data supports. Services monitored by SENTRY and similar tools should operate indoors with high quality for significant periods of time while inserting erroneous changes to induce a reaction in the canary. A Canary Orchestrator manages the canary insertion and checking, splitting the operator error budget as required. Fail points can also be triggered and checked automatically to maintain service quality in these cases. The Canary Orchestrator dynamically calculates when to insert canaries based on risk appetite and acceptable budget allocation for the affected operations and is built on definitions of temporal error budgets and successive safety constraints. The implementation uses Canaries for impact checking. There are possible speedups through shared evaluation of canaries on similar beta groupings and through automated rollback after sufficient impact monitoring time has passed.