## Incident History

### Baseten Embedding Models Down
**Status**: Resolved  
**Date**: 2026-02-27 10:30:42 -0800  
Details: Outage on Baseten embedding models.

---

### Template High P95 Latency
**Status**: Resolved  
**Date**: 2026-03-12 09:29:36 -0700

---

### Palmyra Vision Down Through Public API Requests
**Status**: Resolved  
**Date**: 2026-03-03 18:07:13 -0800  
Details: Palmyra vision is having issues when serviced through public API requests.

---

### Writer Down
**Status**: Resolved  
**Date**: 2026-03-09 14:45:28 -0700  
Details: Envoy pod q7fjq tripped its default ext_authz circuit breaker (max_connections: 1,024) after absorbing 57% of traffic due to a rollout without readiness probes, returning HTTP 500 for ~62% of auth requests.

---

### Writer Agent SEV1 Outage (DB Lock Contention During Migration)
**Status**: Resolved  
**Date**: 2026-03-11 05:43:30 -0700  
Details: On 2026-03-10, Writer Agent experienced a SEV1 full outage driven by database lock contention during deployment.

---

### Repeated Connector Configuration Prompts
**Status**: Resolved  
**Date**: 2026-03-18 08:26:00 -0700  
Details: After authenticating, some users experienced a loop prompting to configure the connector until it failed.

---

### WRITER Agent Stuck in "Thinking"
**Status**: Resolved  
**Date**: 2026-03-31 14:33:18 -0700  
Details: The issue affecting WRITER Agent has been resolved. All systems are now operating normally.

---

### API Service Disruption
**Status**: Resolved  
**Date**: 2026-05-04 09:50:37 -0700  
Details: The issue affecting our API services has been resolved. All systems are now operating normally.