When OCI Is Green but the Application Is Down

Introduction

A production incident does not always begin with alarms, failed Compute instances, or a dashboard full of red indicators.

Sometimes, the situation is much more confusing.

The Compute instances are running. The Load Balancer is available. CPU and memory look normal. Storage has capacity. The database is online. Network resources appear healthy. OCI Monitoring shows no obvious infrastructure failure.

Everything is green.

But users cannot access the application.

This is one of the most challenging situations in production operations because the infrastructure and the user appear to be telling us two completely different stories.

The infrastructure says:

Healthy.

The user says:

The application is down.

That exposes an important distinction:

Infrastructure Health ≠ Service Health

A healthy Compute instance does not guarantee a healthy application.

An available database does not guarantee that the application can connect to it.

A functioning VCN does not guarantee that every required network path is working.

A valid application process does not guarantee that it can authenticate to its dependencies.

And a green OCI dashboard does not necessarily mean that users can successfully complete the business transaction the environment was built to support.

Applications Are Dependency Chains

In an enterprise environment, an application is rarely a single component.

What users see as one service may actually depend on a chain such as:

User → DNS → Load Balancer → Network → Web Tier → Application Tier → Database

And behind that apparently simple path may exist additional dependencies:

IAM

Secrets

Certificates

Storage

Network Security Groups

Security Lists

Route Tables

DNS resolution

Application processes

Database authentication

External APIs

Third-party integrations

Every individual OCI resource can appear operational while one broken relationship somewhere in that chain makes the complete business service unavailable.

That is the problem I want to explore in this article.

A Different Troubleshooting Question

During an infrastructure incident, one of the first questions is often:

“Which server is down?”

But what happens when no server is down?

I prefer to approach this type of incident with a different question:

“Where did the user transaction stop working?”

That small change completely alters the investigation.

Instead of troubleshooting resources independently, we follow the transaction.

User

↓

DNS

↓

Load Balancer

↓

Network

↓

Web Tier

↓

Application

↓

Database

↓

Identity / Secrets

↓

External Dependencies

At every step, we ask:

Can this component successfully communicate with the next component required to complete the transaction?

Now troubleshooting becomes a process of eliminating working layers until we locate the first broken dependency.

The Scenario

Let's imagine a production application running in Oracle Cloud Infrastructure.

At 09:17, users begin reporting that the application is unavailable.

The operations team checks the infrastructure.

Compute: 🟢 Running

Load Balancer: 🟢 Available

Database: 🟢 Available

Storage: 🟢 Available

Network: 🟢 Available

CPU: 🟢 Normal

Memory: 🟢 Normal

There is no obvious OCI infrastructure outage.

Yet:

Application: 🔴 UNAVAILABLE

Users: 🔴 IMPACTED

This is where the investigation begins.

What Could Be Wrong?

Even with all major OCI resources operational, several failures remain possible.

The Load Balancer may be unable to reach its backend.

An application process may be running but not responding.

A required port may be inaccessible.

A network security rule may have changed.

The application may have lost database connectivity.

A secret may have rotated.

A certificate may have expired.

A filesystem may be nearly full.

DNS may be resolving incorrectly.

An external API may be unavailable.

Authentication may be failing.

The challenge is therefore not proving that infrastructure exists.

The challenge is identifying which dependency no longer behaves as expected.

First Mistake: Restart Everything

One of the most tempting responses during an incident is:

“Restart the services.”

Or worse:

“Restart everything.”

Sometimes that works.

But there is a problem.

A restart can restore the service while simultaneously destroying useful evidence about why the failure occurred.

Before restarting components, when business impact and incident severity allow it, I want to capture enough information to understand the failure.

I would first ask:

Did the Load Balancer lose healthy backends?

Is the application process responding?

Can the application reach the database?

Did DNS resolution change?

Did a certificate expire?

Was a secret rotated?

Did an NSG or Security List change?

Is the filesystem approaching capacity?

Did an external dependency fail?

Were there recent changes?

A successful restart proves that the service recovered.

It does not necessarily prove what caused the incident.

And without understanding the cause, we may simply be resetting the countdown until the same failure happens again.

Follow the Transaction, Not the Server

This is the core troubleshooting principle for this scenario.

Instead of asking:

“Which resource looks unhealthy?”

follow the transaction:

USER

↓

DNS

↓

LOAD BALANCER

↓

NETWORK PATH

↓

WEB TIER

↓

APPLICATION TIER

↓

DATABASE

↓

IDENTITY / SECRETS

↓

EXTERNAL DEPENDENCIES

For every transition, validate one thing:

Can A successfully communicate with B in the way the application requires?

If yes:

Continue.

If no:

Stop and investigate that boundary.

This provides a much more structured investigation than checking servers randomly.

Layer 1 — Start Where the User Enters

Begin with the service entry point.

Can the hostname be resolved?

Is DNS returning the expected destination?

Can the endpoint be reached?

Is TLS negotiation successful?

Is the certificate valid?

Is the Load Balancer listener available?

Are the expected backends healthy?

Imagine the result:

DNS: 🟢

↓

Endpoint: 🟢

↓

Load Balancer: 🟢

↓

Listener: 🟢

↓

Backend Health: 🔴

That already tells us something valuable.

The problem is probably not the user's DNS resolution or the external Load Balancer endpoint.

The investigation should move deeper into the application path.

Layer 2 — Is the Network Path Really Working?

A healthy VCN does not prove that the application communication path is healthy.

Likewise, a running Compute instance does not prove that traffic can reach the process running inside it.

The actual flow may depend on:

Network Security Groups

Security Lists

Route Tables

Gateways

DNS

Listening ports

Firewalls

Peering

Private endpoints

On-premises connectivity

Instead of asking:

“Is the network up?”

ask something more specific:

“Can source A reach destination B using the required protocol and port through the expected path?”

That is a question we can actually test.

And during incidents, testable questions are much more useful than broad assumptions.

Layer 3 — Is the Application Actually Healthy?

This is where infrastructure-only monitoring can become misleading.

Imagine:

Compute Instance: 🟢 Running

CPU: 🟢 Normal

Memory: 🟢 Normal

Network Interface: 🟢 Available

But:

Application Process: 🔴 Unresponsive

The infrastructure resource exists and continues producing normal metrics.

The application, however, may be:

Stopped

Hung

Waiting on another dependency

Unable to authenticate

Experiencing connection pool exhaustion

Unable to write files

Listening on the wrong port

Returning application errors

From the OCI infrastructure perspective, the server may still appear healthy.

From the business perspective:

Production is down.

UP Does Not Mean HEALTHY

This distinction is important enough to state explicitly:

UP ≠ HEALTHY

A process can exist but fail to respond correctly.

A port can be open while transactions fail.

A database can be available while the application cannot authenticate.

A Load Balancer can be operational while all of its backends are unhealthy.

A VM can run perfectly while the application inside it is unusable.

Resource state is therefore only one piece of service health.

Layer 4 — Validate the Database From the Application Perspective

Now suppose OCI reports:

Database: 🟢 AVAILABLE

Should we eliminate the database from the investigation?

Not yet.

The better question is:

Can the application successfully consume the database service?

That requires looking at the complete dependency.

Application

↓

DNS / Endpoint

↓

Network Path

↓

Port

↓

Listener

↓

Database Service

↓

Authentication

↓

Session / Connection

A database can be available while the application fails because of:

Incorrect credentials

Expired credentials

Network restrictions

Listener problems

Service-name configuration

Wallet issues

Connection-pool exhaustion

Authentication failures

The important operational principle is:

A healthy dependency that cannot be consumed by the application is effectively unavailable to that service.

Layer 5 — Identity Can Break an Otherwise Healthy Environment

Identity failures are particularly deceptive because they may have almost no visible effect on traditional infrastructure metrics.

Consider:

Compute: 🟢

Network: 🟢

Load Balancer: 🟢

Database: 🟢

Storage: 🟢

But:

Authentication / Authorization: 🔴

The business service may still be unavailable.

Potential causes include:

Expired certificate

Rotated secret

Incorrect IAM policy

Dynamic Group mismatch

Token-generation failure

Credential expiration

Authorization changes

Lost access to an OCI service

CPU may remain normal.

Memory may remain normal.

The VM may continue running.

This is why I consider identity part of the runtime dependency chain, not simply a security configuration performed during deployment.

Layer 6 — Storage Can Fail Quietly

Storage issues can also create misleading symptoms.

Imagine a filesystem at:

70% → 80% → 90% → 95% → 98%

The server is still running.

The application process may still exist.

But the application may suddenly become unable to:

Write logs

Create temporary files

Generate reports

Process exports

Complete batch operations

Write application data

Eventually, users see an application error.

But the root cause may simply be:

Storage capacity.

That is why capacity monitoring should focus not only on current utilization but also on growth trends.

The useful question is not only:

“Is the disk full?”

It is:

“When will this disk become full if the current growth rate continues?”

Layer 7 — Don't Forget External Dependencies

Not every OCI production failure originates inside OCI.

The application may depend on:

SaaS services

On-premises systems

Identity providers

External APIs

SMTP services

File-transfer platforms

Third-party integrations

External DNS

OCI infrastructure may be completely healthy while an external dependency makes the business process unusable.

A mature service model therefore needs to answer:

What does this application depend on to deliver the business service?

not only:

Which OCI resources belong to this application?

Introducing the JLTDS Signal Gap

This brings us to a concept I call the JLTDS Signal Gap.

The Signal Gap represents the distance between:

what infrastructure monitoring reports

and

what the business service is actually experiencing.

Imagine this dashboard:

Infrastructure View

🟢 Compute

🟢 Network

🟢 Load Balancer

🟢 Database

🟢 Storage

But on the other side:

Business View

🔴 Login Failed

🔴 Transaction Failed

🔴 Integration Failed

🔴 Service Unavailable

The space between those two views is the:

SIGNAL GAP

The larger that gap becomes, the greater the chance that:

Users become your monitoring system.

That is precisely what mature observability should try to prevent.

Infrastructure Health vs. Service Health

The difference becomes clearer when we separate the questions each perspective answers.

Infrastructure Health

Infrastructure monitoring tells us:

Is the Compute instance running?

Is CPU utilization abnormal?

Is storage available?

Is the database online?

Is the Load Balancer provisioned?

Are the OCI resources operational?

All of this information matters.

But it doesn't tell the complete story.

Service Health

Service monitoring asks:

Can the user authenticate?

Can the application complete a transaction?

Can the web tier reach the application tier?

Can the application reach the database?

Are critical dependencies responding?

Is response time acceptable?

Can the expected business process complete successfully?

These questions are much closer to the user's actual experience.

Building Monitoring in Layers

For a critical OCI application, I prefer to think about monitoring in layers.

Layer 1 — Resource

Monitor the underlying infrastructure:

Compute

Network

Storage

Database

Load Balancer

This establishes basic infrastructure visibility.

Layer 2 — Process

Monitor the processes actually delivering the service:

Application services

Web services

Listeners

Critical ports

Scheduled services

A running VM with a failed application process should not appear fully healthy.

Layer 3 — Dependency

Monitor the relationships between components:

Application → Database

Application → IAM

Application → Object Storage

Application → DNS

Application → External API

This layer is where many invisible failures become visible.

Layer 4 — Service

Test the actual application endpoint.

Can the service respond correctly?

This may involve health checks, application-level validation, or synthetic monitoring.

Layer 5 — Business Transaction

Finally:

Can a representative user transaction complete?

For example:

Login → Query → Process → Save → Response

If that transaction fails, there is a service problem regardless of how many infrastructure indicators remain green.

The JLTDS Service Health Model

This gives us a five-layer model:

RESOURCE

↓

PROCESS

↓

DEPENDENCY

↓

SERVICE

↓

BUSINESS TRANSACTION

Each level answers a different operational question.

Resource: Is the infrastructure available?

Process: Are critical application components running?

Dependency: Can the required components communicate?

Service: Is the application functioning correctly?

Business Transaction: Can the user accomplish what the application exists to provide?

The further we move down the model, the closer monitoring gets to actual business impact.

A Better Incident Investigation Flow

With this model, troubleshooting becomes much more structured.

1 — Confirm the Impact

What exactly are users unable to do?

↓

2 — Test the Entry Point

Validate DNS, certificate, endpoint, and Load Balancer.

↓

3 — Follow the Transaction

Find the first point where communication or processing fails.

↓

4 — Validate Application Health

Check processes, ports, logs, and errors.

↓

5 — Validate Dependencies

Database, IAM, secrets, storage, APIs, and integrations.

↓

6 — Correlate Telemetry

Metrics, logs, alarms, events, and application errors.

↓

7 — Identify the Failure Domain

Infrastructure, network, process, dependency, identity, application, or external service.

↓

8 — Recover

Apply the safest appropriate remediation.

↓

9 — Validate From the User Perspective

Don't stop when the server turns green.

Confirm the business transaction works.

↓

10 — Learn

Determine what should detect the same condition earlier next time.

Recovery Is Not the End

Suppose restarting an application process restores production.

The service is back.

Incident closed?

Not yet.

We still need to ask:

Why did the process fail?

Was there an alert?

Who discovered the problem first?

Could monitoring have detected it earlier?

Could recovery be automated safely?

Could the same condition happen again?

A recovery without learning simply resets the clock until the next incident.

From Troubleshooting to Observability

Traditional troubleshooting is reactive:

Failure → Investigation → Recovery

A more mature operational model becomes:

Detect → Understand → Correlate → Recover → Learn → Prevent

And for predictable scenarios:

Detect → Validate → Automate → Verify

The objective isn't to eliminate human troubleshooting.

Complex systems will always produce unexpected conditions.

The objective is to reduce the time between:

something failing

and

understanding why it failed.

What I Would Change After This Incident

If users discovered the outage before monitoring did, I would treat that as an observability finding.

I would review:

Application health checks

Load Balancer backend monitoring

Critical process monitoring

Database connectivity validation

Certificate expiration monitoring

Secret lifecycle monitoring

Storage growth trends

IAM and authentication failures

External dependency monitoring

Synthetic transactions

Alert routing

Then I would ask:

What signal existed before the user opened the incident?

Very often, some signal was already present.

It simply wasn't being collected, correlated, or turned into an actionable alert.

The Most Important Dashboard May Not Be an Infrastructure Dashboard

Infrastructure teams naturally organize dashboards around technology.

Compute

Networking

Database

Storage

But business applications don't operate in isolated technology towers.

A service-oriented dashboard might instead show:

CUSTOMER PORTAL

DNS: 🟢

Load Balancer: 🟢

Web Tier: 🟢

Application: 🟢

Database Connectivity: 🟢

Authentication: 🟢

External Integration: 🟢

Synthetic Transaction: 🟢

Now the dashboard answers:

“Is the Customer Portal working?”

instead of simply:

“Are the servers running?”

That is a much more useful operational perspective.

Five Questions I Ask During a Green-Infrastructure Incident

When infrastructure appears healthy but the application does not, these are the five questions I would prioritize.

1. Where Does the User Transaction Stop?

Identify the first failed step.

2. Which Dependency Exists Immediately After That Point?

Follow the service chain.

3. What Changed Recently?

Review deployments, patches, configuration changes, network changes, certificate renewals, secret rotations, and integrations.

4. Which Signal Should Have Detected This?

Identify the monitoring gap.

5. How Do We Prevent Users From Discovering This First Next Time?

Turn the incident into an operational improvement.

Those questions often provide more direction than staring at a wall of green metrics.

Final Takeaways

A green OCI dashboard is useful information.

It is not proof that the application is healthy.

Production availability exists across several layers:

Infrastructure

Processes

Connectivity

Identity

Dependencies

Applications

Business Transactions

A failure in any one of these layers can affect the user.

That means infrastructure teams need to evolve from monitoring individual resources toward understanding services and dependency chains.

The main lessons are:

Follow the transaction, not the server.

Validate dependencies from the consumer's perspective.

Preserve evidence before restarting when circumstances allow it.

Treat identity as a runtime dependency.

Monitor capacity before it becomes failure.

Validate recovery from the user's perspective.

And above all:

Never assume green infrastructure means a green business service.

Conclusion

Users do not consume Compute instances.

They do not consume VCNs.

They do not consume route tables, NSGs, block volumes, or monitoring dashboards.

They consume services.

That changes the way we should think about production health.

The question should not only be:

“Is OCI healthy?”

It should also be:

“Can the user successfully complete the transaction this architecture exists to provide?”

If the answer is no while every infrastructure indicator remains green, the dashboard isn't necessarily wrong.

It is simply showing us only part of the system.

That missing part is the Signal Gap.

The objective of mature observability is to make that gap smaller until infrastructure telemetry, application behavior, and business reality tell the same story.

Monitor the infrastructure. Understand the dependencies. Follow the transaction. Validate the service.

About the Author

João Teixeira
Oracle ACE Apprentice
Oracle EPM & OCI Infrastructure Specialist
JLTDS Oracle Tech

I share practical content around Oracle Cloud Infrastructure and Oracle EPM, with a focus on enterprise architecture, infrastructure operations, troubleshooting, observability, resilience, security, and lessons that can be applied to real production environments.