What Broke Today

🟢 GRAFANA Jul 23, 2026

🟠 Grafana's write endpoints played peek-a-boo between 21:20-22:23 UTC, dropping in and out like a flaky WiFi connection at a coffee shop. They're monitoring now, which is either reassuring or means they're as confused as we are. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Jul 21, 2026

🟡 Grafana's Cloud Log Exporter ghosted several regions and nobody knows why. The logs that monitor your logs are now unmonitorable—peak recursion energy. Engineers are frantically checking the couch cushions for clues. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Jul 15, 2026

🟠 Grafana's prod-eu-west-2 decided to do the Schrödinger's outage thing—some dashboards work, others don't, nobody knows which. Third-party provider apparently forgot to pay their bills. Investigators are doing that classic 'we're looking into it' dance while engineers stress-refresh Slack. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Jul 14, 2026

🟠 Grafana's metrics backend decided partial outages were the new feature release. Writes are limping along, reads are confused, and rule evaluation is just vibing offline. Engineers are performing emergency CPR in the datacenter. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Jul 09, 2026

🟠 Grafana's PDC is serving up a buffet of performance issues—some customers getting degraded speeds, others getting the full blackout treatment. Engineers have called in reinforcements and are pretending to know what's happening. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Jul 09, 2026

⚪ 🔍 Loki's read path in prod-eu-west-2 went full Helen Keller between 5:49-5:54 UTC. Recording rules missed their cue, leaving gaps in your metrics like a mixtape with missing tracks. Recovery complete by 5:57, but your dashboards are now slightly less clairvoyant. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Jul 03, 2026

⚪ Grafana's logging service in prod-eu-north-0 decided to play dead for 10 minutes, treating incoming data like unsolicited DMs. Your 5xx errors were just the service's way of ghosting you—problem solved, dignity optional. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Jul 02, 2026

🔴 Loki and Frontend Observability decided to go dark in prod-us-central-0. AWS Logs integration caught the same mystery illness. Engineers currently playing detective while your observability tools observe absolutely nothing. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Jun 29, 2026

🟡 Confluent decided data integrity was optional today. Their API hiccup has Grafana users staring at blank dashboards across all regions—nothing says 'modern infrastructure' like metrics that forgot to show up. Engineering team is allegedly 'investigating,' which is corporate for 'we're refreshing Slack every 3 seconds.'. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Jun 26, 2026

🟠 K6's metric ingestion decided slow and steady wins the race. Your test results are arriving fashionably late—think of it as fashionable data latency. Engineers are allegedly 'working on fixes' which in incident speak means they found where the hamster stopped running. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Jun 19, 2026

🔴 Your alerts were screaming but nobody heard them. Grafana's prod-us-central-0 region had a brief existential crisis with rule evaluations (15:16-15:59 UTC), which is just the worst time for monitoring to stop monitoring. Engineering found the problem, patched it, and now they're nervously refreshing dashboards. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Jun 18, 2026

🟡 Grafana's having a geography problem: users in India can't load dashboards while Google's datacenter decides to take an unscheduled vacation. Asset loading errors everywhere. GCP's chaos, Grafana's reputation damage. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Jun 18, 2026

⚪ Grafana's k6 load testing UI had a brief identity crisis where it forgot what 'performance' meant. Engineers applied the digital equivalent of smelling salts and it's back to normal. Lesson: even tools that test performance can't test themselves apparently. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Jun 17, 2026

🔴 Grafana's prod-me-central-1 decided to become a historical artifact. AWS admitted the UAE region is genuinely toast—recovery measured in months, not hours. Time to play datacenter roulette and pray your backup plan isn't also in the same burning building. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Jun 02, 2026

🔴 Tempo decided to stop accepting traces like it was screening calls. prod-25 ghosted everyone for 80 minutes, then prod-24 joined the party for the finale. Distributed tracing, distributed problems. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Jun 01, 2026

🟠 Alert manager decided to take an unscheduled sabbatical in prod-us-central-0, leaving your alerts stranded at the notification nightclub. Grafana claims they found the culprit and are 'remediating,' which is corporate for 'frantically Googling stack traces.' Started at 18:30 UTC. (Auto-resolved: no updates in 24h)

🟢 GRAFANA May 27, 2026

🟡 Grafana's Prometheus connector decided to play hide-and-seek with your metrics, serving up 500 errors like they're going out of style. Engineers are currently excavating the code graveyard to figure out what broke. Your dashboards are just fancy red screens for now. (Auto-resolved: no updates in 24h)

🟢 GRAFANA May 11, 2026

🟡 Grafana decided to play certificate roulette with a subset of customers—some connections work, others get the cryptographic cold shoulder. Engineering teams are cosplaying as detectives to figure out which certificates went rogue. (Auto-resolved: no updates in 24h)

🟢 GRAFANA May 04, 2026

🟡 k6 decided to stress-test itself instead of your infrastructure. Grafana's team is playing detective while your benchmarks sit in limbo. Ironic, considering k6's entire job is making sure things don't break. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Apr 23, 2026

🟡 Grafana's CloudWatch connector decided to play hide-and-seek with your metrics. Engineers are currently searching behind the couch. Spoiler: they're not finding much. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Apr 20, 2026

🟡 Grafana's 429 errors are doing their best impression of a bouncer who read the door policy wrong. Metrics queries getting rejected faster than spam applications. Engineering team is caffeinating heavily across multiple regions. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Apr 14, 2026

⚪ Grafana's k6 Cloud went dark for 45 minutes after someone shipped a Friday-night-special API update. The fix? Ctrl+Z. Service is back, dignity is not. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Apr 10, 2026

🟡 Tempo stopped accepting writes in prod-us-east-3 since 18:50 UTC. Users getting the full error experience while engineers play forensics with their infrastructure. Nothing says 'observability platform' like being unable to observe your own metrics. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Apr 03, 2026

🟡 Grafana's k6 extension-based test runs decided to ghost everyone—local and cloud deployments equally affected. Regular tests are fine, which is like saying your car works great except when you try to drive it. Engineering team is presumably refreshing their status page every 30 seconds. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Mar 31, 2026

⚪ Update: Synthetic Monitoring: Some Check Creations & Updates Might be Blocked. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Mar 30, 2026

⚪ Grafana's Tempo tracing service decided to play favorites and ghosted a small subset of customers in one region for 90 minutes. The chosen ones got the VIP treatment of staring at broken dashboards while everyone else lived their best lives. All better now, presumably. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Mar 25, 2026

⚪ Update: Prometheus writes, Logs, and Synthetic Monitoring in prod-eu-west-3 are degraded. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Mar 25, 2026

🟠 GRAFANA - Prometheus writes, Synthetic Monitoring in prod-eu-west-3 are degraded. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Mar 23, 2026

🔴 GRAFANA - Grafana Assistant Unavailable in prod-us-east-0. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Mar 23, 2026

🔴 GRAFANA - Grafana Assistant Unavailable in prod-us-east. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Mar 20, 2026

🔴 GRAFANA - Authentication API Database Down in prod-eu-west-2 and prod-eu-west-4. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Mar 20, 2026

🔴 GRAFANA - Authentication API Database Down in prod-eu-west-2. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Mar 11, 2026

🟡 Grafana's alert rules decided to stop doing their job for some lucky prod-us-west-0 customers. Turns out monitoring your monitors requires the monitors to actually work. Engineers are now nervously refreshing dashboards looking for clues. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Mar 09, 2026

🟠 Grafana's prod-us-central regions discovered they're allergic to accepting metric data. Two separate 10-15 minute episodes of elevated errors and latency. The irony: a monitoring tool that can't monitor itself. Engineers are now staring very intently at dashboards that won't accept new data. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Mar 07, 2026

🔴 AWS S3 decided to take an unscheduled sabbatical, dragging Grafana's EU region down with it. The good news? S3 remembered how to work again around 20:03 UTC. The bad news? Your monitoring was blind while the arsonist was in the building. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Mar 06, 2026

🟠 Some Grafana dashboards decided to go dark. The 'identified' status means engineers found the problem but haven't executed the fix yet—basically they know why the lights are off, now comes the hard part of flipping the switch back on. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Mar 02, 2026

🟠 Grafana's prod-eu-west-3 logs decided to become read-only exhibits in a museum. Write operations hit a wall faster than a Brexit negotiation, and the engineering team is now playing archaeological dig with their own infrastructure. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Feb 25, 2026

🟠 Hosted Grafana users in prod-us-east-0 can't load dashboards or alert folders. Your visualizations have ghosted you. Engineers are narrowing down the scope while your metrics stare back confused from the void. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Feb 25, 2026

🟠 Grafana's prod-eu-west-3 decided partial writes and rule evaluation were luxury features it couldn't afford. Took 85 minutes to remember how databases work, then another hour of 'monitoring' to confirm the fix actually stuck. (Auto-resolved: no updates in 24h)

⚪ GRAFANA Feb 17, 2026

⚪ Grafana's synthetic monitoring public probe in Calgary decided to take an unscheduled siesta from 12:09-12:35 UTC. Your tests that relied on it? Yeah, they failed spectacularly. At least it's back now and nobody's still waiting for results from February.

🟢 GRAFANA Feb 13, 2026

🟡 Grafana's Loki got amnesia in Canada—write operations flatlined while engineers frantically check if they deployed 'rm -rf' by accident. Log storage is experiencing what we call 'selective memory loss.'. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Feb 11, 2026

🟠 Grafana's Loki service in ap-southeast-1 stopped accepting logs for 11 minutes while engineers discovered that 'write issues' is corporate speak for 'the database forgot how to database.' Temporary band-aids applied, root cause still playing hide and seek. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Feb 05, 2026

🟡 Grafana's Hosted Metrics in us-central-0 decided to play dead for queries and rule evaluations. Stale data is making a comeback tour while engineers dust off their detective hats. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Jan 27, 2026

🟡 Grafana Cloud decided visibility is optional—some instances have achieved true invisibility. Engineers are squinting at logs trying to find what walked out the door. The irony of a monitoring tool being invisible to its own users is not lost on anyone. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Jan 21, 2026

⚪ Update: Synthetic monitoring secrets - proxy URL changes. (Auto-resolved: no updates in 24h)

🟢 GRAFANA Jan 21, 2026

🟡 Grafana's trace writes decided to take the scenic route through molasses. They found the culprit, hit it with a fix, and now they're nervously watching to make sure it doesn't spontaneously combust again. (Auto-resolved: no updates in 24h)

⚪ GRAFANA Jan 21, 2026

{ "skip": true } ``` This is scheduled maintenance, which per the critical rules should be skipped. The incident clearly states "Scheduled maintenance is currently in progress" and was planned in advance for January 21st, 13:00-16:00 UTC.

> cat /var/log/failures.log