10 Principles for Robust and Secure Software Systems
10 Principles for Robust and Secure Software Systems
Drew Gutstein 2026-05-18
Throughout my career, I've encountered many situations where software failed in bizarre and complex ways. In response to those failures, I've built solutions so that no such failure can happen again. While seemingly obvious (we learn from our mistakes), the lessons I've absorbed are much more nuanced than the textbook examples, however. With the "systems-of-systems" architectures that evolve at any sufficiently technical organization, hardened (or haunted) by the outages of years past, the textbook is not very useful.
I believe that a fundamental outcome of any sufficiently complex system, be it software, networks, rocketships, or life, is that we must expect the unexpected.
In a sense, any business's IT ecosystem can reach a point of complexity where it becomes impossible to foresee and prevent every possible catastrophe. If you've ever written a post-mortem, then this will sound familiar, but my takeaway has always been to engineer for simplicity: reduce the attack surface, limit configuration, expose as little as possible, and keep apart the things that don't need to be together (isolation).
Recently, I've been going through my many encounters with Murphy's Law and developing a set of principles that system designers should keep in mind to minimize disruptive events and maximize recovery. This document is a reference guide for others trying to mitigate the "black swan" events—the rare, unforeseen consequences that inevitably emerge from complexity.
Load-Bearing Axioms for Modern Infrastructure
The following principles establish the foundational requirements for constructing and maintaining highly secure, resilient, and scalable software environments. These axioms are designed to ensure that security and robustness are not tertiary considerations but are baked in from a foundational perspective.
To achieve resilient security, it is imperative that these principles are woven into the very fabric of the software development lifecycle (the birth) and operational runtime (the life) of every software system.
Trust
The cornerstone from which all else follows is trust. This means delimiting what is trusted and reliable from the untrusted and unreliable. Once upon a time, this meant that all software must be developed in-house to be considered safe. This makes the trust consideration simple: is it first-party or not?
Aside from being inefficient, that approach rests on the fallacy that internally built systems are, in fact, first-party. To be honest, I've never seen a truly first-party system (I wrote this article, not your browser). For the most part, all so-called first-party systems are actually second-party; built using third-party components and customized for your specific organization with first-party configurations and enhancements.
From an identity perspective, that means no code can be fully trusted. The supply chain risk concerns that have been elevated in recent years have always been a problem, and so we must stop thinking of any system, even our own, as trusted.
Principle 1: Trust but Verify
This first Principle comes in many forms:
- Change is introduced only after it is confirmed that the system behaves as expected.
- Access is granted only after identity is proven.
- Code is executed only after its provenance is understood (i.e., reproducible and integrous).
Verification is essential to trust. Whether we are talking about zero-trust networking, electronic communication, or binary authorization, we must ensure we know who or what we are trusting before we place our trust in it.
Without our different systems of verification, we would be blindly reliant on fundamentally unreliable components.
Instead, we gate the activity behind some form of validation, be it testing, hash verification, certificate validation, etc. Validation makes trust an explicit property, not something established through implicit assumptions.
In change management terms, this means ensuring that desired behavior must be explicitly observed before a change can be introduced or rolled out.
In zero-trust terms, that means identity precedes access. Access is granted only to what is explicitly declared in the workload specification after verified attestation.
The ultimate benefit is that there is no single root identity. All identities exist within a bounded context which guarantees the two properties essential to good security: isolation and separation.
Principle 2: Not all Trust is Equal
Trust can be established in many different ways, so we should think of trust as a continuum rather than a binary property. The question to be asked is: "To what degree do I trust this artifact?" Unverified artifacts should not be trusted at all, but we must recognize that each method of verification has different strengths and weaknesses; we must consider them to arrive at a level of trust between 0 and 1.
Security posture is defined along independent axes: trust tiers, capability requirements (Storage, Backup, Identity-rigor, Audit, Egress), asset criticality and exposure, etc… This demands a multidimensional approach which ensures that workloads receive resources and protections proportional to their sensitivity and operational needs.
The level of trust required by a system's components must be commensurate with the criticality of the assets being trusted. If a component is integral to performing a critical business function then it must adhere to the highest tier of trust.
This is not an atomic objective, there is no silver bullet. It is only achievable through multiple forms of verifications and protective controls layered in a way that ensures the reliability and integrity of the systems supporting the business activity.
Environmental Hygiene
Moving on from Trust, the next set of principles have to do with what I like to call environmental hygiene, aka pollution. Everyone who has managed a system over time has experienced the slow erosion of consistency and normalization that occurs as systems age. What was once a homogeneous and predictable environment eventually becomes a quagmire of special configurations and customizations as new requirements are discovered and bolted on.
This is similar to the concept of Technical Debt which is an apt description but doesn't quite capture the effect that the debt has on the assets which live in the operational environment. Like Tech Debt, environment pollution increases the cost of ownership over time. It's useful to consider that the same forces that turn clean code into spaghetti also operate on networks, databases and other infrastructure as well.
Just like in nature, a polluted operational environment impacts the entire ecosystem.
As pollution builds up, your well factored and highly tested systems depend on assumptions that may no longer be true. It is necessary to document and enforce the hygiene of the environments where systems live, ensuring that the necessary and minor changes made to environments are accounted for and will persist across build cycles.
Principle 3: Immutability
The best way to enforce hygiene is to prevent the pollution from building up in the first place. This requires the strict enforcement of immutability. This is a "set it and forget it" approach that ensures that once an environment is deployed, its state never deviates without a subsequent deployment. This doesn't necessarily require 100% immutability but that's a good place to start and create the exceptions from there.
Continuous Integration and Continuous Deployment (CI/CD) are the key controls to maintaining an immutable environment. When the next release paves over the patches, customizations and tweaked configurations, which pollute our environments, we are able to flush out the cruft and re-establish the operational trust needed to protect our critical assets.
This implies the use of Infrastructure as Code models which are well evolved and are now essential for maintaining clean environments.
Principle 4: Roll forward (not back)
In my experience this mostly works, but when it doesn't, it can lead to catastrophic consequences. Rolling back one or more components in an environment is theoretically sound, but it can be extremely hard or impossible to predict what a rollback will do to other changes to the system. This partially rolled back state is often untested.
The general rule is that If you roll back one component you should roll back all of them. But better yet: Roll Forward - not back.
This follows from Principle 3: Immutability. If we have strong CI/CD and sufficient immutability, we can identify the root cause, verify with a test and push the change out proactively. This is often less risky and much quicker than relying on Change Advisory Boards to opine on the safety of a roll back.
Safety is verified with testing while change management risks are reduced when the deployed environment is pristine. Properly managed feature flags also help but only when tested in on/off configurations.
Principle 5: Ephemeral State
Ephemeral assets are ones that are not intended to persist in an environment and are therefore resistant to the types of attack where malicious actors can "live off the land." It also reduces loss frequency since a vulnerability is only exploitable when the exposed asset is running in the environment. There are other benefits as well.
Ephemeral assets, which are created at runtime and disposed once a workload is complete, ensure short build horizons, which allows patches to be deployed on a continuous basis.
From an operational perspective, old outdated configurations are outages just waiting to happen. If something is no longer needed in an environment it should simply not run and any irrelevant configuration should be purged from the operating environment. On the next deployment. This is hard to do with fixed assets but is something that happens for free when environments are disposable.
Continuous Deployment is a great way to achieve disposable architecture. Because everything changes, nothing is permanent. By treating deviations as temporary, we enforce the use of our CI/CD pipelines as first class management tools.
The combination of immutable & ephemeral assets give us the foundation needed to ensure that all of our environmental assumptions are valid and reproducible. Consistency is an important pillar for supporting secure and robust systems and critical for implementing robust Recovery controls.
Principle 6: Everything Expires
Every asset: images, secrets, configs, certs, identities, etc… must have a declared maximum age. In the real world everything has a shelf life and treating assets as permanent makes managing the end-of-life for assets incredibly hard.
It could be a long period such as several years for an LTS operating system or system backups; or very short, on the order of seconds for an attestation certificate. Regardless, we need our systems to understand that every asset has an end-of-life and plan for that eventuality from the get go.
Expiration is an excellent way to enforce CI/CD and strong deployment and recovery practices.
Setting reasonable expiration limits is a great way to handle the inevitable exceptions that will be encountered. It ensures that those decisions are revisited on a regular cadence and encourages the proper integration of exceptions in the many cases where the Exception becomes the rule. This keeps exceptions from becoming pollution.
This means that every system has an explicit "dead-man's-switch." If the operational processes that provision new assets to replace the expiring ones fail for long enough, the system will become unstable. The key here is that it is exposing this existing problem for what it is. Inconsistent operations and configuration issues are what cause the majority of outages that impact large systems. To avoid this, we need to treat end-of-life as a first-class property of every asset we operate, and the dead man's switch is the enforcement.
Governance
So far we've been discussing the aspects of how systems are run and deployed into operational environments which is a huge part of the workload for any team supporting critical services. The next set of principles are what ensures that the constraints discussed above are both enforced and efficient.
Principle 7: Actionable Metrics
It's impossible to implement the principles above without having strong governance to make sure the process is followed correctly. This is largely a data problem; aggregating inventory and configurations in a way that the adherence to the controls can be measured.
In the corporate world, what gets measured gets done. As a governance practitioner one must recognize that most people don't have time to do things that are not directly relevant to their job and role. Using metrics that correlate directly to the behavior that we are trying to enforce bridges the gap between the risks we are reducing and the specific actions (behaviors) that are required from people to mitigate the risk.
In this context, actionable, must be something that is recognizable to the person who is assigned the task. It makes no sense, and is in fact dangerous, to send alerts to people who don't understand what to do with them. It degrades the credibility of the security and ops teams when we route problems to people who can't fix them.
Everything we measure for governance must have an action to be performed by the asset owner for when the metric is below agreed SLAs. That action must measurably impact the metric in a way that brings the measurement closer to or within SLA.
Principle 8: Ownership
Actionable Metrics are opinionated. The "good" and "bad" states are well known and understood. They are also attributable. We must have good data on who is ultimately responsible for the assets and is capable of resolving aberrations that are detected through governance.
Ideally that is mapped to the organization in its present state (orgs change). Assets without ownership are monitored for and actioned as teams change and orgs grow and shrink. Generally, organizations change faster than systems so ensuring accurate ownership is a hard requirement for achieving many of the principles above.
Orphaned (aka legacy) infrastructure is a major source of security and operational risk in any organization. It is best dealt with head on which may require navigating the relevant organizations to identify new owners for abandoned systems. This must occur on a continuous basis given the speed at which organizations are formed and reformed to account for either rapid growth, management changes, and head count cuts.
Constant organization change is a reality that is often ignored and a huge contributor to environmental pollution. This includes non production environments as well which are often overlooked and can slow CI/CD to a crawl if left to fester.
The best way to ensure attributable ownership is through gamification. Use actionable metrics and public score keeping to create an atmosphere of friendly competition. Use rewards, prizes, special titles, public endorsements, etc… to incentivize teams that improve or maintain environmental hygiene (money works too). Then every so often, raise the bar. It's a system that has been used for decades to offset real pollution and it works just as well for maintaining pristine environments.
Principle 9: Observable and Queryable
The last principle is the ultimate goal of continuous control monitoring. Through a control maturity ladder, governance and engineering teams must work together to achieve a fully observable system. This is where real operation excellence is tied to real world outcomes.
An observable system creates transparency into the health of automated control implementations to allow for real time operational risk monitoring. A truly observable environment alleviates the need for constant vulnerability scans because scanning is an inventory problem.
Being able to produce a "digital twin" (full inventory) of a complete environment allows the advanced behavioral analytics that will detect and remediate AI generated attacks. It allows the strength of controls to be measured and modeled for realistic threat assessment, blast ratios analysis, and containment.
A fully observable system can be easily reproduced, redeployed, or retired. BCP plans can be vetted and simulated. Recovery times can be nearly instantaneous with fully declarative infrastructure and zero trust architecture.
As attackers become more and more empowered through advances in GenAI coding models, a quick recovery may be our best defense.
Principle 10: Standards are great. Conventions are also good
There are many ways of achieving any or all of these principles in a given complex system. The best are through the adoption of well known and widely understood standards and the tools that support them. Open standards are best for avoiding both vendor lock in and avoiding work for solved problems. (Not all Open standards are great so choose wisely)
When standards aren't an option, consider conventions. Essentially, conventions are reasonable defaults and patterns to follow. Conventions alleviate cognitive load and help bridge the gap between customization and predictability when moving toward greater standardization. Both human operators and AIs benefit from reliable and understandable conventions which are often easier to master than complex configuration management system logic.
Creating new project templates that support the organization's tech biases are essential to enabling consistent behavior. Ensure the environment is properly established for new projects and systems. Preconfigure access groups and identity schemes so that individual engineering teams don't have to. Add git hooks, build instructions, and test harness to projects "for free" and then use them to enforce consistency with the norm. You may get pushback at first but teams will quickly recognize the benefits of uniform practices.
Operational Excellence In Practice
In real world complex environments running systems-of-systems will most likely not adhere to the principles above but it's important to read them as aspirational as opposed to mandated. This is because you'll encounter significant headwinds from engineering teams that are comfortable with the existing practices and reluctant to change.
Achieving true operational excellence is an incremental process. Starting over is generally not an option, so we need to pick and choose which parts of the existing system we can mature. Choosing targets based on risk is important so we can measure and improve overall control effectiveness and demonstrate the value of the work. However, the actual criteria should not necessarily target the biggest risks but rather to prioritize the biggest risk-reductions.
This means taking into account the cost of migrating the changes since some risks will be easier to mitigate than others. Systems that are easy to change will benefit from hygiene work faster than, perhaps, than the work to migrate the most critical systems.
A productive approach is to plan a set number of cleanup sprints with concrete goals that target parts of the environment that are easiest to change. If the risk is broad (high event frequency) then that's a great way to deliver value early and build experience before taking on the most business critical systems.
Next Steps
I hope this has been a helpful article for laying some clear guidelines for a secure by default target architecture. With the principles laid you might now have a sense of where there may be some low hanging fruit for controls to mature.
My goal with GS.works is to help existing teams move from traditional compliance mandated controls (KYA) to the types of effective and proven solutions that provide real security and strong governance.
Over the next few weeks, I'll be blogging about a reference architecture to put some meat on these bones. We'll then test the architecture against SOC-2 and ISO 27001 to see how we stand up to real security and regulatory requirements. Feel free to follow along if that interests you or reach out to discuss how to apply these principles in your existing systems.
All the best,
Drew Gutstein
drew@gs.works
https://www.linkedin.com/in/drew-gutstein-9332b34
Founder, CISSP, Security Automation Specialist
Gutstein Security Works, LLC
https://gs.works