1. The problem we're solving

A cloud architecture is only complete when the end user never discovers an incident first. In the real world this is rarely true. Consider this illustrative case (based on a real incident, sanitized):

The customer's question afterward was fair:

"Isn't there some way for us to find out before a user does?"

The correct answer is yes, and in this lab we're going to build that answer.

2. Learning objectives

By the end of this lab, students will be able to:

1. Distinguish between the three levels of monitoring every production app should have: external (synthetic), host, and application.
2. Provision OCI Health Checks, OCI Notifications, and OCI Monitoring Alarms via the OCI CLI.
3. Implement instance principal authentication so a VM can publish its own metrics without API keys on disk.
4. Apply defensive practices: idempotent scripts, subscription redundancy, geographic separation of probes.
5. Design an alerting strategy that reduces detection time for an outage from "undefined" to "2–3 minutes."

3. Prerequisites (knowledge)

• OCI compartments (basic IAM)
• How to open and use OCI Cloud Shell
• Basic bash reading (variables, conditionals, pipes)
• "Compute instance" and "Docker container" concepts
• Familiarity with SSH

4. The three-layer model

Each layer covers a different class of failure. One layer alone is never enough.

Layer	What it measures	Failures it catches	Failures it does NOT catch
1. Synthetic (external)	The site responds to HTTP from outside	Total outage, public-network issues, expired cert	Internal slowness; one container down while another serves cache
2. Host	CPU, memory, disk, kernel state	Disk full, OOM, high load	App down with healthy host (the war story above!)
3. Application / container	State of each individual component	"Docker is up but one container died"	Logical defects inside the application

We'll build all three.

5. Lab prerequisites (setup)

Each student needs:

1. An OCI Free Tier account (the services we'll use fall within generous free-tier limits).
2. An OCI VM running Oracle Linux 8 with at least one HTTP service exposed on port 443 (e.g. nginx via docker run -d -p 443:443 nginx). This is what we'll monitor.
3. Access to OCI Cloud Shell from the web console.
4. A valid email address to receive alerts.

6. Placeholder convention

Throughout the lab, anything in <...> must be replaced with values from your environment:

Placeholder	Meaning	How to get it
<TENANCY_OCID>	Your tenancy's root OCID	oci iam compartment list --include-root --all --query "data[?\"compartment-id\"==null].id | [0]" --raw-output
<COMP_OCID>	The compartment you'll work in	oci iam compartment list --query "data[?name=='<YOUR_COMPARTMENT>'].id | [0]" --raw-output
<INSTANCE_OCID>	OCID of the VM to monitor	oci compute instance list --compartment-id "$COMP_OCID" --query "data[?\"display-name\"=='<YOUR_VM>'].id | [0]" --raw-output
<PUBLIC_URL>	Public HTTPS URL of your app	You set this up
<EMAIL>	Alert destination	Your inbox
<REGION>	OCI region	Top-right of the console (e.g. us-ashburn-1, mx-queretaro-1)

Throughout the lab I assume you've exported COMP_OCID and TENANCY as variables. Set them at the start:

# Get tenancy and compartment OCIDs
TENANCY=$(oci iam compartment list --include-root --all \
  --query "data[?\"compartment-id\"==null].id | [0]" --raw-output)

COMP_OCID=$(oci iam compartment list --all --compartment-id-in-subtree true \
  --query "data[?name=='<YOUR_COMPARTMENT>'].id | [0]" --raw-output)

echo "tenancy:     $TENANCY"
echo "compartment: $COMP_OCID"

Goal: have a cluster of geographically distributed probes hit your URL every 30 seconds and fire an alarm if they stop getting a healthy response.

Step 1.1 — Create the notifications topic

OCI Notifications uses a publisher / subscriber pattern. An alarm publishes to a topic; the topic fans out to one or more subscriptions (email, SMS, function, etc.). Create the topic first:

TOPIC_ID=$(oci ons topic create \
  --compartment-id "$COMP_OCID" \
  --name "myapp-alerts" \
  --description "Monitoring alerts for my app" \
  --query 'data."topic-id"' --raw-output)

echo "topic: $TOPIC_ID"

Step 1.2 — Subscribe your email

oci ons subscription create \
  --compartment-id "$COMP_OCID" \
  --topic-id "$TOPIC_ID" \
  --protocol EMAIL \
  --subscription-endpoint "<EMAIL>"

You'll receive a confirmation email. You must click the confirmation link to activate the subscription. Until you do, the alarm will fire correctly but no email will ever arrive.

Step 1.3 — Create the HTTP health monitor

Here we pick the vantage points — the locations OCI will probe from. This is an important design decision: the vantage points should represent the geography of your real users.

First, list the available vantage points:

oci health-checks vantage-point list --all \
  --query 'data[*].name' --output table | head -40

You'll see names like aws-iad (Ashburn, VA), aws-pdx (Portland, OR), azr-sat (San Antonio, TX), aws-cdg (Paris), aws-bom (Mumbai), etc. The three-letter codes are airport codes.

For a US-targeted application, a good selection is:

• aws-cmh (Columbus, OH — East-Central)
• aws-pdx (Portland, OR — West)
• azr-sat (San Antonio, TX — South-Central, and a different cloud provider)

Three locations, two cloud providers — if AWS has a regional issue, the Azure probe still works.

HM_ID=$(oci health-checks http-monitor create \
  --compartment-id "$COMP_OCID" \
  --display-name "myapp-public-https" \
  --targets '["<PUBLIC_URL_WITHOUT_HTTPS_PREFIX>"]' \
  --protocol HTTPS \
  --port 443 \
  --path "/" \
  --method GET \
  --interval-in-seconds 30 \
  --timeout-in-seconds 10 \
  --vantage-point-names '["aws-cmh","aws-pdx","azr-sat"]' \
  --is-enabled true \
  --query 'data.id' --raw-output)

echo "health monitor: $HM_ID"

From now on, OCI will run 6 probes per minute (3 vantage points × 2/min). The state is exposed as metrics in the oci_healthchecks namespace.

Step 1.4 — Create the alarm

An important design decision here: under what condition should it fire?

Two relevant metrics:

• HTTP.NoneAvailable — equals 1 if no vantage point could connect (TCP-level: port closed, host unreachable).
• HTTP.IsHealthy — equals 1 per vantage point when the response was 2xx/3xx; 0 if it was 4xx/5xx or no response.

NoneAvailable > 0 only fires on full TCP-level outages. It does not fire when the server returns a 502 (classic case: reverse proxy is up but the backend died).

IsHealthy.mean() < 0.5 also catches 5xx, which is exactly what we want.

We'll create both alarms, because they serve slightly different purposes:

# Alarm 1: full TCP-level outage
oci monitoring alarm create \
  --compartment-id "$COMP_OCID" \
  --display-name "myapp-tcp-down" \
  --metric-compartment-id "$COMP_OCID" \
  --namespace "oci_healthchecks" \
  --query-text "HTTP.NoneAvailable[1m]{resourceId=\"$HM_ID\"}.mean() > 0" \
  --severity CRITICAL \
  --destinations "[\"$TOPIC_ID\"]" \
  --is-enabled true \
  --pending-duration "PT2M" \
  --message-format ONS_OPTIMIZED \
  --body "The application is not responding at all from any vantage point."

# Alarm 2: site responds, but with an error
oci monitoring alarm create \
  --compartment-id "$COMP_OCID" \
  --display-name "myapp-unhealthy" \
  --metric-compartment-id "$COMP_OCID" \
  --namespace "oci_healthchecks" \
  --query-text "HTTP.IsHealthy[1m]{resourceId=\"$HM_ID\"}.mean() < 0.5" \
  --severity CRITICAL \
  --destinations "[\"$TOPIC_ID\"]" \
  --is-enabled true \
  --pending-duration "PT3M" \
  --message-format ONS_OPTIMIZED \
  --body "The site is responding with non-2xx/3xx codes from a majority of vantage points."

Parameter discussion

• [1m] — evaluation window: average over the last minute.
• mean() — aggregation. For 3 vantage points, < 0.5 implies at least 2 are reporting unhealthy.
• PendingDuration "PT3M" — the condition must hold for 3 minutes before the alarm fires. This filters out transient network blips.
• Severity CRITICAL — this one should wake somebody at 3 AM. In production, severity is the input to your routing system (PagerDuty, Opsgenie, etc.).

Step 1.5 — Verify the pipeline without causing a real outage

Publish a test message directly to the topic:

oci ons message publish --topic-id "$TOPIC_ID" \
  --title "TEST: alerting pipeline verification" \
  --body "If you got this email, the topic -> subscription -> email path is working."

You should receive the email in under 30 seconds. This validates 90% of the pipeline. The remaining piece — "alarm fires → publishes to the topic" — is OCI's internal plumbing and well tested.

OCI has an agent preinstalled on its official images (oracle-cloud-agent) that reports host metrics to the oci_computeagent namespace. Nothing to install; we just create alarms.

First confirm the agent is active on your instance:

oci compute instance list --compartment-id "$COMP_OCID" \
  --query "data[?\"display-name\"=='<YOUR_VM>'].\"agent-config\"" --output json

If pluginsConfig contains "Compute Instance Monitoring": "ENABLED", you're set.

Relevant metrics

• CpuUtilization (%) — CPU usage
• MemoryUtilization (%) — memory usage
• FilesystemUtilization (%) — per-filesystem usage (with filesystem dimension = mountpoint)

Create the alarms

We'll use a helper function for idempotency (more on this in the defensive practices section):

mk_alarm() {
  local name="$1" query="$2" severity="$3" body="$4"
  local existing
  existing=$(oci monitoring alarm list --compartment-id "$COMP_OCID" \
    --query "data[?\"display-name\"=='$name'].id | [0]" --raw-output)
  if [ -z "$existing" ] || [ "$existing" = "null" ]; then
    oci monitoring alarm create \
      --compartment-id "$COMP_OCID" \
      --display-name "$name" \
      --metric-compartment-id "$COMP_OCID" \
      --namespace "oci_computeagent" \
      --query-text "$query" \
      --severity "$severity" \
      --destinations "[\"$TOPIC_ID\"]" \
      --is-enabled true \
      --pending-duration "PT5M" \
      --message-format ONS_OPTIMIZED \
      --body "$body" >/dev/null
    echo "created: $name"
  else
    echo "exists:  $name"
  fi
}

mk_alarm "host-disk-high" \
  "FilesystemUtilization[5m]{resourceId=\"<INSTANCE_OCID>\"}.mean() > 85" \
  "CRITICAL" \
  "Filesystem above 85% for 5+ min."

mk_alarm "host-memory-high" \
  "MemoryUtilization[5m]{resourceId=\"<INSTANCE_OCID>\"}.mean() > 90" \
  "WARNING" \
  "Memory > 90%. OOM-kill risk."

mk_alarm "host-cpu-high" \
  "CpuUtilization[5m]{resourceId=\"<INSTANCE_OCID>\"}.mean() > 90" \
  "WARNING" \
  "CPU > 90%. Investigate runaway process or load spike."

This layer takes more setup but detects failures the other two miss.

Scenario: Your host runs three Docker containers: nginx-rp, app-backend, cache. If app-backend dies but nginx-rp keeps running, external probes see 502 — the unhealthy alarm from Lab 1 fires. Good.

But what if cache dies and the app keeps working (slower, no cache)? External probes see nothing wrong. Only per-container monitoring catches this.

Step 3.1 — Instance principal authentication

The classic problem: a script running inside your VM needs to publish metrics to OCI. How does it authenticate?

To enable this we need two IAM resources:

1. A Dynamic Group that matches the VM
2. A Policy that grants permissions to the Dynamic Group

# Dynamic group (lives in tenancy root, not your compartment)
DG=$(oci iam dynamic-group create \
  --compartment-id "$TENANCY" \
  --name "myapp-vm-principal" \
  --description "MyApp VM publishes metrics using its own identity" \
  --matching-rule "instance.id = '<INSTANCE_OCID>'" \
  --query 'data.id' --raw-output)

# Policy in your compartment
oci iam policy create \
  --compartment-id "$COMP_OCID" \
  --name "myapp-vm-publish-metrics" \
  --description "Lets the VM publish custom metrics" \
  --statements '["Allow dynamic-group myapp-vm-principal to use metrics in compartment <COMPARTMENT_NAME>"]'

Step 3.2 — Publisher script on the VM

SSH to the VM. We'll create a script that runs every minute and publishes metrics.

sudo mkdir -p /opt/monitoring
sudo tee /opt/monitoring/post-container-state.sh >/dev/null << 'EOF'
#!/bin/bash
# Publishes container state as a custom OCI metric.
# Auth: instance principal (no API keys on disk).

set -e

REGION="<REGION>"
COMP_OCID="<COMP_OCID>"
NAMESPACE="myapp_containers"
ENDPOINT="https://telemetry-ingestion.<REGION>.oraclecloud.com"
WATCHLIST="nginx-rp app-backend cache"
OCI=/home/opc/.local/bin/oci

TS=$(date -u +%Y-%m-%dT%H:%M:%S.000Z)
METRICS=""
for c in $WATCHLIST; do
  state=$(sudo docker inspect -f "{{.State.Running}}" "$c" 2>/dev/null || echo "absent")
  case "$state" in true) val=1 ;; *) val=0 ;; esac
  M="{\"namespace\":\"$NAMESPACE\",\"compartmentId\":\"$COMP_OCID\",\"name\":\"container_running\",\"dimensions\":{\"container\":\"$c\"},\"datapoints\":[{\"timestamp\":\"$TS\",\"value\":$val}]}"
  if [ -n "$METRICS" ]; then METRICS="$METRICS,$M"; else METRICS="$M"; fi
done

"$OCI" --auth instance_principal --endpoint "$ENDPOINT" monitoring metric-data post \
  --region "$REGION" --metric-data "[$METRICS]" >/dev/null
EOF

sudo chmod +x /opt/monitoring/post-container-state.sh

Step 3.3 — Test manually

/opt/monitoring/post-container-state.sh && echo OK

If everything works, you get OK with no output. If it fails, the most common causes:

• Permissions: verify the policy from Step 3.1 has propagated (~30–60s) and that the Dynamic Group's matching rule points to the right instance OCID.
• Endpoint: confirm --endpoint https://telemetry-ingestion.<region>.oraclecloud.com is present.
• CLI path: if oci isn't at /home/opc/.local/bin/oci, adjust the OCI variable in the script.

Step 3.4 — Cron

echo "* * * * * opc /opt/monitoring/post-container-state.sh >> /var/log/myapp-monitoring.log 2>&1" \
  | sudo tee /etc/cron.d/myapp-monitoring
sudo chmod 644 /etc/cron.d/myapp-monitoring
sudo touch /var/log/myapp-monitoring.log
sudo chown opc:opc /var/log/myapp-monitoring.log

Step 3.5 — Create per-container alarms (from Cloud Shell)

for c in nginx-rp app-backend cache; do
  oci monitoring alarm create \
    --compartment-id "$COMP_OCID" \
    --display-name "container-$c-down" \
    --metric-compartment-id "$COMP_OCID" \
    --namespace "myapp_containers" \
    --query-text "container_running[2m]{container=\"$c\"}.mean() < 1" \
    --severity CRITICAL \
    --destinations "[\"$TOPIC_ID\"]" \
    --is-enabled true \
    --pending-duration "PT2M" \
    --message-format ONS_OPTIMIZED \
    --body "Container $c has been down for 2+ minutes per the local publisher." >/dev/null
  echo "created: container-$c-down"
done

Defensive practices (summary)

Things I learned the hard way that I now apply every time:

1. Idempotent scripts

Every provisioning script should assume it will be run multiple times (re-run after an error, second attempt after a tweak, etc.). The pattern is always:

EXISTING=$(oci <service> list --query "data[?name=='<x>'].id | [0]" --raw-output)
if [ -z "$EXISTING" ] || [ "$EXISTING" = "null" ]; then
  # create the resource
else
  # nothing to do, already exists
fi

This avoids errors like "a topic with that name already exists" on the second run, and lets you re-run the same script across environments (dev, staging, prod) without modifications.

2. Personal backstop subscription

Always subscribe a named individual email to the topic, in addition to the official DL. One accidental "Unsubscribe" click in an alert email can silently kill all alerts for the entire team until the next incident.

3. Geographic probe separation

Vantage points in your users' region, different from the instance's region. Same provider in the same region is not a real internet test.

4. Severity aligned with response

Every alarm has a severity. That severity should correspond to the real response:

• CRITICAL: somebody must act now (page, call).
• WARNING: notify but not urgent (email or team channel).

If everything is CRITICAL, nothing is CRITICAL (alarm fatigue).

5. PendingDuration vs evaluation window

• Evaluation window ([1m], [5m]): how much time the metric is aggregated over before evaluating the threshold.
• PendingDuration (PT2M, PT5M): how long the condition must hold before firing.

Long windows + short PendingDuration = slow but stable alarms.
Short windows + long PendingDuration = fast alarms with noise filtering.

For real outages (target < 3 min detection): [1m] + PT2M or PT3M.
For cumulative conditions (disk): [5m] + PT5M is fine.

6. Validate the pipeline without causing the outage

An oci ons message publish straight to the topic verifies subscription + delivery without touching the app. Do this before any "real" alarm test.

7. Post-incident documentation

Every time you respond to an incident, write a short post-mortem that includes:

• Root cause (not "the VM went down", but what specifically in the VM failed)
• Detection time vs resolution time (they measure different things)
• What permanent changes were applied
• What remains open

This lab came out of a real post-mortem. Without that practice, the incident would have repeated.

Exercises

Appendices

Appendix A — Post-implementation verification

# List all active alarms
oci monitoring alarm list --compartment-id "$COMP_OCID" \
  --query 'data[].["display-name", severity, "lifecycle-state"]' --output table

# List topic subscriptions (all should be ACTIVE)
oci ons subscription list --compartment-id "$COMP_OCID" \
  --query "data[?\"topic-id\"=='$TOPIC_ID'].[endpoint, \"lifecycle-state\"]" --output table

# See custom datapoints from the last 5 minutes
START=$(date -u -d '5 minutes ago' +'%Y-%m-%dT%H:%M:%SZ')
END=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
oci monitoring metric-data summarize-metrics-data \
  --compartment-id "$COMP_OCID" \
  --namespace "myapp_containers" \
  --query-text "container_running[1m].max()" \
  --start-time "$START" --end-time "$END" \
  --output table

# Show every alarm's current evaluation status (OK / FIRING)
oci monitoring alarm-status list-alarms-status --compartment-id "$COMP_OCID" --output table

Appendix B — References

• OCI Monitoring docs: docs.oracle.com/iaas/Content/Monitoring/home.htm
• OCI Health Checks docs: HealthChecks/Concepts/healthchecks.htm
• OCI Notifications docs: Content/Notification/home.htm
• OCI CLI command reference: tools/oci-cli/latest/oci_cli_docs/
• Monitoring Query Language (MQL): Monitoring/Reference/mql.htm
• Instance Principal docs: Identity/Tasks/callingservicesfrominstances.htm

Closing

The three-layer pattern (synthetic / host / application) is cloud-provider independent. The service names change (AWS CloudWatch Synthetics + Alarms, Azure Monitor + Application Insights), but the architecture is the same. Master this pattern on OCI and you've mastered it for any cloud.

The question "Isn't there some way for us to find out before a user does?" should have a yes and an example email as its answer. That's the standard a mature operations team meets.