voyageai-cli
Version:
CLI for Voyage AI embeddings, reranking, and MongoDB Atlas Vector Search
271 lines (207 loc) • 6.57 kB
Markdown
# Error Handling Strategies
Proper error handling is essential for building reliable applications. This guide covers strategies for graceful degradation, recovery, and debugging.
## Error Classification
Categorize errors to determine appropriate response:
**Retryable Errors** (temporary, safe to retry):
- Network timeouts (connection lost)
- 429 Too Many Requests (rate limited)
- 503 Service Unavailable (temporary outage)
- 500 Internal Server Error (might be transient)
**Non-Retryable Errors** (permanent, don't retry):
- 401 Unauthorized (fix credentials)
- 403 Forbidden (insufficient permissions)
- 404 Not Found (resource doesn't exist)
- 400 Bad Request (invalid parameters)
- 422 Unprocessable Entity (invalid state)
## Retry Strategy
Implement exponential backoff with jitter:
```python
import time
import random
max_retries = 5
base_wait = 0.1 # 100ms
for attempt in range(max_retries):
try:
response = api.request(...)
break
except RetryableError as e:
if attempt == max_retries - 1:
raise # Last attempt
wait = min(base_wait * (2 ** attempt), 60) # Cap at 60s
jitter = random.uniform(0, wait * 0.1)
time.sleep(wait + jitter)
```
This prevents thundering herd and gives services time to recover.
## Circuit Breaker Pattern
Prevent cascading failures by "breaking" the circuit when errors exceed threshold:
```python
class CircuitBreaker:
def __init__(self, failure_threshold=5, timeout=60):
self.failure_count = 0
self.failure_threshold = failure_threshold
self.timeout = timeout
self.last_failure_time = None
self.state = 'CLOSED' # CLOSED, OPEN, HALF_OPEN
def call(self, func, *args):
if self.state == 'OPEN':
if time.time() - self.last_failure_time > self.timeout:
self.state = 'HALF_OPEN'
else:
raise CircuitBreakerOpen()
try:
result = func(*args)
self.on_success()
return result
except Exception as e:
self.on_failure()
raise
breaker = CircuitBreaker()
try:
breaker.call(api.get_user, user_id)
except CircuitBreakerOpen:
# Circuit open; use fallback
user = get_cached_user(user_id)
```
## Fallback Strategies
When primary service fails, use fallback:
```python
def get_user_with_fallback(user_id):
try:
return api.get_user(user_id)
except (Timeout, ServiceUnavailable):
# Fallback to cache
cached = cache.get(f'user:{user_id}')
if cached:
return cached
# Fallback to stale data
return db.get_user_stale(user_id)
```
Provide graceful degradation with stale data rather than failing completely.
## Error Logging
Log errors with sufficient context:
```python
logger.error(
'API request failed',
extra={
'endpoint': '/users/user_123',
'method': 'GET',
'status': 404,
'error_code': 'resource_not_found',
'request_id': 'req_abc123',
'duration_ms': 250,
'retry_attempt': 2
}
)
```
Include:
- Endpoint and method
- HTTP status
- Error code (machine-readable)
- Request ID (for tracing)
- Duration
- Retry count
## Distributed Tracing
Use request IDs to trace errors through microservices:
```python
import uuid
request_id = request.headers.get('X-Request-ID') or str(uuid.uuid4())
logger.info('Processing request', extra={'request_id': request_id})
try:
result = api.call_service(headers={'X-Request-ID': request_id})
except Exception as e:
logger.error('Service call failed', extra={
'request_id': request_id,
'service': 'payment_service',
'error': str(e)
})
```
Propagate `request_id` through all services. Later, you can reconstruct call flow:
```
Request ID: req_abc123
1. Web server receives request
2. Calls payment_service (passes req_abc123)
3. Calls fraud_detection_service (passes req_abc123)
4. fraud_detection_service timeout
5. Payment service retries
...
```
## User-Facing Error Messages
Don't expose internal details; provide helpful messages:
```python
# ✗ Bad: Internal details
if error.code == 'DOCUMENT_VALIDATION_FAILURE':
raise Exception("Document failed validation: required field 'orgId' missing")
# ✓ Good: User-friendly
if error.code == 'INVALID_ORGANIZATION':
raise UserError("Organization not found. Please verify the organization ID.")
```
## Alerting
Set up alerts for critical errors:
```yaml
Alerts:
- name: high_error_rate
condition: error_rate > 5%
severity: critical
action: page_oncall
- name: payment_service_down
condition: response_time > 10s OR status_code = 503
severity: critical
action: page_oncall
- name: slow_api_response
condition: p99_latency > 5s
severity: warning
action: notify_team
```
Monitor error rates and latency; alert on anomalies.
## Error Budgets
Track error budgets to balance reliability and feature velocity:
```
Service SLA: 99.9% uptime = 43.2 minutes of acceptable downtime per month
Used this month: 15 minutes
Remaining budget: 28.2 minutes
High-risk changes (deploy when budget >50%): Deploy approved
Low-risk changes: Deploy anytime
```
This prevents deploying risky changes when error budget is exhausted.
## Debugging
Use structured logging for easier debugging:
```python
logger.debug('User lookup', extra={
'user_id': user_id,
'database': 'replica_1',
'query_time_ms': 45,
'cache_hit': False,
'cache_ttl': 3600
})
```
Later, query logs to understand behavior:
```bash
# Find all slow user lookups
logs | grep 'User lookup' | filter(query_time_ms > 100)
# Analyze cache hit rate
logs | grep 'User lookup' | stats(cache_hit)
```
## Error Recovery Scenarios
**Scenario: Payment Processing Fails**
```
1. Payment request times out
2. Circuit breaker detects repeated timeouts
3. Return "Processing, check status later"
4. Background job retries payment
5. Webhook notifies of outcome
6. User checks payment status
```
**Scenario: Database Connection Lost**
```
1. Connection error detected
2. Failover to replica initiated
3. Queries rerouted to replica (read-only)
4. Write operations queued
5. Primary recovers
6. Switch back to primary
7. Process queued writes
```
## See Also
- [Error Responses](../endpoints/error-responses.md) - HTTP error response format
- [Debugging](debugging.md) - Debugging techniques
- [Monitoring](monitoring.md) - Observability and alerting