I Broke My OCI Environment on Purpose
Introduction
Most cloud architecture diagrams show the perfect scenario.
Compute instances are healthy. Networks are available. Databases are responding. Monitoring dashboards are green. Applications are running normally.
But production environments are rarely that predictable.
So instead of asking:
“Is my OCI architecture well designed?”
I wanted to explore a different question:
“What happens when individual components start failing?”
This article presents a controlled OCI Chaos Engineering Lab designed to examine how different infrastructure and application failures affect an enterprise cloud environment.
The objective is not simply to break infrastructure.
The objective is to understand four things:
Can I detect the failure?
Can I diagnose it?
Can I isolate the impact?
Can I recover from it?
The Experiment
For this lab, we will examine six controlled failure scenarios representing problems that operations teams may encounter in production environments.
The Chaos Engineering Principle
Chaos Engineering is not about randomly breaking production systems.
Controlled experiments should have:
1. A defined hypothesis
What do we expect to happen?
2. A controlled failure
Introduce one known failure condition.
3. Observability
Determine whether the environment detects the failure.
4. Impact Analysis
Understand which dependencies and business services are affected.
5. Recovery
Restore normal operations.
6. Learning
Improve architecture, monitoring, automation, or operational procedures based on what was discovered.
This gives us something much more useful than simply saying:
PASS / FAIL
It allows us to measure the operational resilience of the environment.
What We’ll Discover
By the end of this experiment, we should be able to answer a much more important question than whether our OCI resources are running:
If something breaks tomorrow, will we know what failed, understand the impact, and know how to recover?
That is where infrastructure monitoring becomes operational resilience.
Part 2 — Running the Chaos Experiments
Now that the lab architecture, success criteria, and JLTDS Failure Visibility Model have been defined, it is time to introduce controlled failures.
The objective is not simply to confirm that something breaks.
For every experiment, we want to understand the complete operational sequence:
Inject Failure → Detect → Diagnose → Isolate → Recover → Learn
The first three experiments focus on an important distinction: infrastructure availability does not always represent application availability.
Experiment 1 — Compute Failure
Scenario
A Compute instance supporting the application tier suddenly becomes unavailable.
Possible real-world causes could include an operating system failure, application host issue, maintenance event, or another condition that makes the instance unavailable.
Hypothesis
If the Compute instance becomes unavailable, the environment should detect the infrastructure condition and operations teams should be able to determine which application component is affected.
What We Observe
At the infrastructure level, this type of failure is generally easier to identify than many application-level failures because the affected resource itself has changed state or stopped providing expected telemetry.
The investigation should correlate several signals:
Compute State → Metrics → Health Checks → Alarms → Application Availability
But detecting the Compute problem is only the first step.
The real question is:
Did losing the instance actually cause a business outage?
In a resilient architecture with redundant application instances behind a Load Balancer, one failed Compute instance may reduce capacity without making the application unavailable.
Expected Result
Detection: Strong
Diagnosis: Relatively straightforward
Isolation: Depends on architecture
Recovery: Instance restart, replacement, or workload redistribution
Business Impact: Depends heavily on redundancy
JLTDS Visibility Rating
🟢 RECOVERABLE
With appropriate monitoring and documented recovery procedures, this type of failure should be relatively visible and manageable.
Lesson
Infrastructure redundancy changes a failure from an outage into a capacity event.
Experiment 2 — Application Failure
This experiment is more interesting.
Scenario
The Compute instance remains operational, but the application process stops responding.
From an infrastructure perspective:
Compute Instance: 🟢 UP
CPU: 🟢 Normal
Network: 🟢 Available
But:
Application: 🔴 DOWN
Users: 🔴 IMPACTED
Hypothesis
Infrastructure monitoring alone may not reliably identify this condition.
Application-aware monitoring or Health Checks should detect that the service itself is unavailable.
What We Observe
This demonstrates one of the most dangerous assumptions in infrastructure operations:
A healthy server does not guarantee a healthy service.
The VM can continue generating metrics while the application process is frozen, stopped, or unable to serve requests.
This means observability must extend beyond the infrastructure layer.
A useful detection chain becomes:
Endpoint → Health Check → Load Balancer Health → Alarm → Application Logs → Operations
Expected Result
Detection: Depends on application-level monitoring
Diagnosis: Requires logs and service context
Isolation: Application process/service
Recovery: Restart service or application component
Business Impact: Potentially High
JLTDS Visibility Rating
🟡 DIAGNOSABLE
Without application-aware monitoring, however, the same environment could fall to:
🔴 BLIND
or
🟠 DETECTABLE
Lesson
This experiment produces one of the most important findings of the lab:
UP ≠ HEALTHY
Monitoring the infrastructure is necessary.
Monitoring the service delivered by that infrastructure is equally important.
Experiment 3 — Network Failure
Now we introduce a failure where almost every dashboard can initially look healthy.
Scenario
A network configuration issue interrupts communication between the application tier and another required component.
For example, connectivity may be affected by a routing or security configuration change.
The individual resources remain operational:
Compute: 🟢 UP
Application Process: 🟢 UP
Database: 🟢 UP
Storage: 🟢 UP
Yet:
Application Communication: 🔴 FAILED
Business Service: 🔴 IMPACTED
Hypothesis
Traditional resource monitoring may not immediately identify the root cause because none of the individual resources necessarily failed.
Diagnosis requires understanding the communication path between components.
Investigation Path
Instead of asking:
“Which server is down?”
we need to ask:
“Which dependency can no longer communicate?”
The investigation becomes:
Source → Subnet → Route → Security Controls → Destination → Application Logs
Network telemetry and diagnostic capabilities become significantly more valuable in this scenario.
Expected Result
Detection: Application symptoms may appear first
Diagnosis: More difficult than Compute failure
Isolation: Requires network-path analysis
Recovery: Correct affected network configuration
Business Impact: Potentially Critical
JLTDS Visibility Rating
🟡 DIAGNOSABLE
If network visibility is poor, this could easily become:
🟠 DETECTABLE
The organization knows the application is broken but spends significant time determining why.
Lesson
When every component is UP but the service is DOWN, investigate the dependencies between them.
What Have We Learned So Far?
The first three experiments already demonstrate three completely different failure patterns:
This reveals an important pattern.
As failures move away from the infrastructure itself and into applications and dependencies, simple infrastructure monitoring becomes progressively less effective.
That is why mature cloud operations require multiple layers of visibility:
Infrastructure → Network → Application → Dependencies → Business Service
The First Major Finding
Our experiments challenge a common operational assumption:
Green infrastructure does not necessarily mean a healthy business service.
The more complex the environment becomes, the more important it is to monitor not only individual OCI resources, but also the relationships and services that depend on them.
And the next experiments make this problem even more visible.
Part 3 — When Infrastructure Looks Healthy
The first experiments demonstrated that resource availability and service availability are not the same thing.
Now we move into failures that are often harder to diagnose because most infrastructure components may remain completely operational.
The dashboard can be green.
The servers can be running.
The network can be reachable.
And users can still be unable to perform their work.
Experiment 4 — Database Connectivity Failure
Scenario
The application tier remains available, but the application can no longer establish the required connection to its database.
At first glance:
Compute: 🟢 UP
Application Server: 🟢 UP
Network: 🟢 Potentially UP
Database: 🟢 Potentially UP
But:
Application → Database: 🔴 FAILED
Business Service: 🔴 IMPACTED
The database itself does not necessarily have to be down. A connectivity, listener, authentication, configuration, or dependency issue may be enough to break the application.
Hypothesis
Infrastructure monitoring may indicate that both the application server and database are healthy while the application itself reports failures.
Therefore, detecting the real problem requires correlation across multiple layers.
Investigation Path
The troubleshooting sequence should move through the dependency chain:
Application Error → Application Logs → Network Path → Database Endpoint → Database Availability → Authentication → Database Logs
The objective is to determine whether the failure exists in the application, network, database service, or authentication layer.
Expected Result
Detection: Application errors may appear first
Diagnosis: Requires correlation across layers
Isolation: Database dependency
Recovery: Restore the failed connectivity or dependency
Business Impact: High to Critical
JLTDS Visibility Rating
🟡 DIAGNOSABLE
With stronger application monitoring and automated dependency validation, this could progress toward:
🟢 RECOVERABLE
Lesson
A healthy database and a healthy application server do not guarantee a healthy connection between them.
Experiment 5 — Storage Capacity Failure
This experiment behaves differently because the failure develops gradually.
Scenario
Storage utilization begins increasing:
70% → 80% → 90% → 95% → 100%
At 70%, everything may operate normally.
At 80%, there may still be no visible user impact.
At 90%, operational risk becomes significant.
At 95%, application processes may begin experiencing problems.
At 100%, writes can fail and applications may become unavailable depending on the workload.
Hypothesis
A mature observability environment should detect the trend, not merely the final outage.
If the first meaningful alert occurs when storage reaches 100%, monitoring technically detected the problem—but operationally, it failed to prevent it.
What We Should Detect
Instead of a single critical threshold, the environment can use progressive awareness:
70% — Normal / Observe
80% — Warning
90% — High
95% — Critical
100% — Failure
However, thresholds alone are not enough.
Growth rate matters.
A filesystem growing from 70% to 80% over six months represents a very different risk from one growing from 70% to 80% in thirty minutes.
Expected Result
Detection: Strong when capacity metrics exist
Diagnosis: Straightforward
Isolation: Storage/filesystem
Recovery: Cleanup, expansion, or workload correction
Prevention: Capacity forecasting and trend monitoring
Business Impact: Progressive → Potentially Critical
JLTDS Visibility Rating
🟢 RECOVERABLE
With automated expansion or remediation for an appropriate workload, it could potentially become:
🔵 SELF-HEALING
Lesson
The best incident is the one monitoring helps you prevent.
Experiment 6 — Identity / Secret Failure
The final experiment is one of the most deceptive.
Scenario
A required identity, authorization, credential, or secret dependency stops working correctly.
The infrastructure itself remains healthy:
Compute: 🟢
Network: 🟢
Database: 🟢
Storage: 🟢
But the application can no longer authenticate or access a required resource.
Authentication / Authorization: 🔴
Business Service: 🔴
Hypothesis
Traditional infrastructure metrics may provide almost no indication that something is wrong.
CPU may remain normal.
Memory may remain normal.
Network interfaces may remain available.
Instances may remain running.
Yet users cannot complete a critical business process.
Investigation Path
Diagnosis now depends much more heavily on contextual telemetry:
Application Error → Authentication Logs → IAM/Policy Validation → Secret/Credential Validation → Dependency Access → Recovery
This demonstrates why observability must include security and identity-related signals alongside traditional infrastructure metrics.
Expected Result
Detection: Potentially difficult
Diagnosis: Requires contextual logs
Isolation: Identity / authorization / secret dependency
Recovery: Restore valid access or configuration
Business Impact: High
Infrastructure Impact: Potentially none
JLTDS Visibility Rating
🟠 DETECTABLE
With centralized logging, application-aware monitoring, expiration monitoring, and clearly defined identity dependencies:
🟡 DIAGNOSABLE
And with appropriate automated controls, selected scenarios could progress further.
Lesson
Some of the most disruptive cloud failures do not look like infrastructure failures at all.
The most dangerous scenario isn’t always the one with the largest infrastructure failure.
Sometimes it is the one where:
Everything looks green except the business service.
⸻
A New Definition of “Healthy”
After six experiments, we can challenge another assumption:
Healthy infrastructure ≠ Healthy application
And:
Healthy application components ≠ Healthy business service
A more complete operational view should therefore evaluate:
Infrastructure Health
↓
Connectivity Health
↓
Application Health
↓
Dependency Health
↓
Business Service Health
Only when these layers are considered together do we get a meaningful picture of service availability.
⸻
The Second Major Finding
Our first three experiments showed that failures become harder to identify as they move away from infrastructure.
The final three reveal something even more important:
The hardest failures are often not invisible. They are visible in the wrong place.
The signal may already exist in an application log, authentication error, capacity trend, database message, or health check.
The real challenge is connecting those signals before users become the monitoring system.
Part 4 — What Failed First?
After six controlled failure scenarios, we can now compare the results.
The most important discovery is that failure severity and failure visibility are not the same thing.
Some failures are obvious because infrastructure disappears. Others leave almost every OCI resource operational while the business service becomes unavailable.
That distinction fundamentally changes how we should think about cloud resilience.
What Failed First?
1 — Compute Failure
Failure visibility: HIGH
When infrastructure itself disappears, telemetry usually provides obvious signals.
The challenge becomes recovery and redundancy rather than detection.
Result: 🟢 RECOVERABLE
2 — Application Failure
Failure visibility: MEDIUM
The host remains operational while the application fails.
This is where infrastructure-only monitoring begins losing effectiveness.
Result: 🟡 DIAGNOSABLE
3 — Network Failure
Failure visibility: MEDIUM–LOW
Individual components remain healthy, but communication between them fails.
The problem exists between resources rather than inside a resource.
Result: 🟡 DIAGNOSABLE
4 — Database Connectivity Failure
Failure visibility: MEDIUM–LOW
Both application and database infrastructure can appear operational.
The failed dependency becomes the real problem.
Result: 🟡 DIAGNOSABLE
5 — Storage Capacity Failure
Failure visibility: HIGH
Storage provides one of the clearest opportunities for proactive operations because degradation can often be observed before failure.
The important signal isn’t:
“Storage is full.”
It is:
“Storage will become full.”
Result: 🟢 RECOVERABLE
Potential maturity:
🔵 SELF-HEALING
6 — Identity / Secret Failure
Failure visibility: LOW
This produced our largest potential observability blind spot.
Compute works.
Network works.
Database works.
Storage works.
Yet authentication fails and the business service can become unavailable.
Result: 🟠 DETECTABLE
The Failure Visibility Paradox
The experiments expose an interesting pattern:
The closer the failure is to infrastructure…
The easier it tends to be to detect.
But:
The further the failure moves into dependencies…
The more context is required to understand it.
Think about the progression:
Compute Failure
↓
Application Failure
↓
Network Dependency Failure
↓
Database Dependency Failure
↓
Identity Dependency Failure
Infrastructure monitoring becomes progressively less capable of explaining the complete situation.
This is the Failure Visibility Paradox.
Monitoring Coverage vs Service Coverage
Another important distinction emerged from the experiments.
An organization may have:
95% Infrastructure Monitoring Coverage
while having significantly less:
Business Service Visibility
Those are not equivalent measurements.
A mature observability strategy should therefore ask two different questions:
Infrastructure Coverage
Are my OCI resources being monitored?
Service Coverage
Can I determine whether users can successfully consume the service those resources provide?
The second question is ultimately more important.
What Should We Automate?
Not every failure should trigger automatic remediation.
Automation should depend on confidence, impact, and predictability.
Good automation candidates:
- Restarting known failed processes
- Replacing unhealthy stateless Compute instances
- Scaling capacity
- Routing around unhealthy components
- Capacity remediation
- Known repeatable recovery procedures
- Notifications and incident enrichment
More complex failures may still require human decisions.
The objective is not:
Automate everything.
It is:
Automate what is predictable, observable, reversible, and safe.
Therefore:
You don’t truly understand the resilience of a service until you understand its dependencies.
JLTDS Chaos Engineering Result
The experiments started with four questions:
Can I detect it?
Can I diagnose it?
Can I isolate it?
Can I recover it?
We can now add a fifth:
Can I predict or prevent it?
That is the transition from traditional monitoring toward resilience engineering.
Part 5 — From Failure to Resilience
The six experiments were never really about breaking OCI resources.
They were about discovering whether the environment could see, understand, contain, and recover from failure before that failure became a major business incident.
Chaos Engineering becomes valuable when every controlled failure produces an improvement.
The cycle should be:
Experiment → Observe → Learn → Improve → Re-test
Moving from Blind to Self-Healing
The JLTDS Failure Visibility Model gives us a practical way to measure that evolution.
🔴 Blind
The organization discovers failures through users, support tickets, or business impact.
Priority: Establish basic visibility.
Implement monitoring, logging, health checks, and service ownership.
🟠 Detectable
The organization knows something is wrong, but identifying the root cause still requires significant investigation.
Priority: Improve context.
Correlate infrastructure metrics with application logs, dependencies, identity events, and network information.
🟡 Diagnosable
The organization can identify the likely cause relatively quickly.
Priority: Standardize response.
Create runbooks, escalation paths, dependency maps, and repeatable recovery procedures.
🟢 Recoverable
Failures can be detected, diagnosed, isolated, and restored through known procedures.
Priority: Automate predictable recovery.
Identify remediation steps that are safe, repeatable, observable, and reversible.
🔵 Self-Healing
Selected failures can be detected and remediated automatically.
The operational flow becomes:
Detect → Validate → Remediate → Verify → Notify → Learn
But self-healing should never mean uncontrolled automation.
Automation must have clear guardrails.
The Automation Safety Rule
Before automating recovery, ask four questions:
Is the failure condition unambiguous?
Is the remediation predictable?
Can the action be safely reversed?
Can we verify that recovery actually succeeded?
If any answer is No, human validation may still be the safer design.
The goal isn’t maximum automation.
The goal is reliable automation.
Chaos Engineering Should Be Continuous
Testing resilience once does not prove that an environment will remain resilient.
Cloud environments change continuously.
Applications change.
Policies change.
Network routes change.
Certificates expire.
Secrets rotate.
Dependencies evolve.
New integrations appear.
Therefore, resilience testing should become part of the operational lifecycle.
A useful cycle is:
Design
↓
Deploy
↓
Observe
↓
Test Failure
↓
Measure Impact
↓
Improve
↓
Re-test
This creates a continuous feedback loop between architecture and operations.
Start Small
Chaos Engineering does not require starting with a regional outage.
Begin with controlled, low-risk experiments.
For example:
Can we detect an application process failure?
Then:
Can we identify a broken dependency?
Then:
Can we recover without affecting other services?
Then:
Can recovery be automated safely?
As confidence grows, experiments can become more sophisticated.
The maturity journey should be progressive:
Understand → Observe → Test → Recover → Automate
Ten Questions Every Critical OCI Service Should Answer
After completing this lab, I would ask these questions for every production-critical service:
- What are its critical dependencies?
- What happens if each dependency fails?
- How will we detect that failure?
- Will infrastructure monitoring actually see it?
- Which logs or telemetry explain the root cause?
- What is the blast radius?
- How do we restore the service?
- How do we verify recovery from the user’s perspective?
- Can any part of the response be safely automated?
- When was the recovery process last tested?
If several of these questions cannot be answered confidently, there is probably a resilience gap worth investigating.
Final Takeaways
This experiment started with a simple idea:
Break something intentionally and observe what happens.
But the six scenarios revealed something much broader.
Cloud resilience isn’t defined only by redundant infrastructure.
It depends on whether we can understand the complete service.
That means understanding:
Resources.
Applications.
Networks.
Databases.
Identity.
Storage.
Dependencies.
And ultimately:
The user experience.
The biggest observability gaps often appear not when infrastructure goes down, but when infrastructure remains green while the service is already broken.
Conclusion
Chaos Engineering changes the question.
Instead of asking:
“Will this architecture fail?”
We assume that eventually something will.
Then we ask:
“When it fails, will we detect it, understand it, contain it, recover from it, and learn from it?”
That is a much stronger measure of cloud resilience.
The goal is not to build an environment where nothing ever fails.
The goal is to build an environment where failure is visible, understandable, recoverable, and increasingly predictable.
Break safely. Observe everything. Learn continuously. Build resilience.
About the Author
João Teixeira
Oracle ACE Apprentice
Oracle EPM & OCI Infrastructure Specialist
JLTDS Oracle Tech
Sharing practical Oracle Cloud Infrastructure and Oracle EPM content focused on architecture, operations, resilience, security, and real-world enterprise challenges.
Comentários
Postar um comentário