history.atom

Incident History

Baseten Embedding Models Down

Status: Resolved
Date: 2026-02-27 10:30:42 -0800
Details: Outage on Baseten embedding models.

Template High P95 Latency

Status: Resolved
Date: 2026-03-12 09:29:36 -0700

Palmyra Vision Down Through Public API Requests

Status: Resolved
Date: 2026-03-03 18:07:13 -0800
Details: Palmyra vision is having issues when serviced through public API requests.

Writer Down

Status: Resolved
Date: 2026-03-09 14:45:28 -0700
Details: Envoy pod q7fjq tripped its default ext_authz circuit breaker (max_connections: 1,024) after absorbing 57% of traffic due to a rollout without readiness probes, returning HTTP 500 for ~62% of auth requests.

Writer Agent SEV1 Outage (DB Lock Contention During Migration)

Status: Resolved
Date: 2026-03-11 05:43:30 -0700
Details: On 2026-03-10, Writer Agent experienced a SEV1 full outage driven by database lock contention during deployment.

Repeated Connector Configuration Prompts

Status: Resolved
Date: 2026-03-18 08:26:00 -0700
Details: After authenticating, some users experienced a loop prompting to configure the connector until it failed.

WRITER Agent Stuck in "Thinking"

Status: Resolved
Date: 2026-03-31 14:33:18 -0700
Details: The issue affecting WRITER Agent has been resolved. All systems are now operating normally.

API Service Disruption

Status: Resolved
Date: 2026-05-04 09:50:37 -0700
Details: The issue affecting our API services has been resolved. All systems are now operating normally.