@cloudkinetix/bmad-enhanced/expansion-packs/ck-llm-agent-dev/templates/monitoring-dashboard-tmpl.md

Cloud-Kinetix enhanced fork of BMAD-METHOD - Breakthrough Method of Agile AI-driven Development with robust versioning and unified validation.

# {{Agent Name}} LLM Agent Monitoring Dashboard

[[LLM: Initial Setup

1. Replace {{Agent Name}} with the actual agent name throughout
2. Configure monitoring tools (LangSmith, Datadog, Grafana, etc.)
3. Set up data collection pipelines
4. Define SLAs and alert thresholds
5. Create visualization dashboards

Default path: .bmad-workspace/ck-ai-agent-dev/monitoring/{{agent-name}}-dashboard-{{timestamp}}.md]]

## Dashboard Overview

[[LLM: Provide a high-level description of what this dashboard monitors and why it's important for the LLM agent's operations.]]

- **Dashboard Name**: {{agent-name}}-monitoring
- **Refresh Rate**: {{5s/30s/1m/5m}}
- **Data Retention**: {{7d/30d/90d}}
- **Primary Users**: {{ops-team/developers/business-stakeholders}}
- **Alert Channels**: {{slack/pagerduty/email}}

## Key Performance Indicators (KPIs)

### Business Metrics

[[LLM: Define business-relevant metrics that demonstrate agent value.]]

<<REPEAT: business-metric>>

#### {{Metric Name}}

- **Description**: {{what-it-measures}}
- **Target**: {{target-value}}
- **Calculation**: {{formula-or-query}}
- **Visualization**: {{gauge/line-chart/counter}}
- **Alert Threshold**: {{warning/critical-levels}}
  <</REPEAT>>

### Technical Metrics

#### Response Time Metrics

- **P50 Latency**: {{median-response-time}}
- **P95 Latency**: {{95th-percentile}}
- **P99 Latency**: {{99th-percentile}}
- **Timeout Rate**: {{percentage-timeouts}}

#### Throughput Metrics

- **Requests/Second**: {{current-rps}}
- **Success Rate**: {{successful-requests-percentage}}
- **Error Rate**: {{error-percentage}}
- **Queue Depth**: {{pending-requests}}

#### Resource Utilization

- **Token Usage**: {{input/output-tokens-per-minute}}
- **API Quota**: {{percentage-of-quota-used}}
- **Cost per Hour**: {{current-burn-rate}}
- **Model Distribution**: {{usage-by-model}}

## LLM-Specific Monitoring

### Prompt Performance

[[LLM: Track prompt effectiveness and optimization opportunities.]]

````yaml
prompt_metrics:
  - name: Prompt Cache Hit Rate
    query: cache_hits / total_requests
    target: '> 60%'
  - name: Average Prompt Length
    query: avg(prompt_tokens)
    alert: '> 2000 tokens'
  - name: Prompt Injection Attempts
    query: count(security_flags.prompt_injection)
    alert: '> 0'
```text

### Model Behavior Tracking

<<REPEAT: behavior-metric>>
#### {{Behavior Metric}}
- **Metric**: {{metric-name}}
- **Tracking Method**: {{how-measured}}
- **Baseline**: {{normal-range}}
- **Anomaly Detection**: {{detection-method}}
<</REPEAT>>

## Quality Monitoring

### Output Quality Metrics

[[LLM: Define how to measure and track output quality.]]

- **Relevance Score**: {{0-1-scale-or-percentage}}
- **Completeness**: {{response-completeness-measure}}
- **Accuracy**: {{factual-accuracy-tracking}}
- **User Satisfaction**: {{feedback-score}}
- **Hallucination Rate**: {{false-info-percentage}}

### Safety Metrics

- **Content Filter Triggers**: {{harmful-content-blocks}}
- **Bias Detection Events**: {{bias-pattern-alerts}}
- **Privacy Violations**: {{pii-exposure-count}}
- **Compliance Failures**: {{policy-violations}}

## Infrastructure Monitoring

### Service Dependencies

[[LLM: Monitor all external services the agent depends on.]]

<<REPEAT: dependency>>
#### {{Service Name}}
- **Health Check**: {{endpoint-or-method}}
- **Availability Target**: {{sla-percentage}}
- **Latency Budget**: {{max-acceptable-latency}}
- **Circuit Breaker**: {{threshold-settings}}
<</REPEAT>>

### Database/Storage Metrics

- **Vector DB Query Time**: {{embedding-search-latency}}
- **Cache Hit Rate**: {{redis/memcached-hits}}
- **Storage Usage**: {{current-gb-used}}
- **Index Performance**: {{query-efficiency}}

## Observability Setup

### Distributed Tracing

[[LLM: Configure OpenTelemetry or similar for request tracing.]]

```yaml
tracing:
  enabled: true
  sampling_rate:
    '[object Object]': null
  exporters:
    - jaeger:
        '[object Object]': null
    - datadog:
        '[object Object]': null
  spans:
    - llm_request
    - prompt_construction
    - response_parsing
    - tool_calls
    - cache_lookup
````

### Logging Configuration

````yaml
logging:
  level:
    '[object Object]': null
  structured: true
  fields:
    - request_id
    - user_id
    - model_used
    - prompt_tokens
    - completion_tokens
    - total_cost
    - response_time
    - error_type
```text

## Alert Configuration

### Critical Alerts

[[LLM: Define alerts that require immediate attention.]]

<<REPEAT: critical-alert>>
#### {{Alert Name}}
- **Condition**: {{trigger-condition}}
- **Threshold**: {{specific-value}}
- **Duration**: {{time-before-alert}}
- **Notification**: {{who-gets-paged}}
- **Runbook**: {{link-to-response-guide}}
<</REPEAT>>

### Warning Alerts

<<REPEAT: warning-alert>>
#### {{Alert Name}}
- **Condition**: {{trigger-condition}}
- **Threshold**: {{specific-value}}
- **Duration**: {{time-before-alert}}
- **Notification**: {{slack-channel}}
<</REPEAT>>

## Dashboard Panels

### Main Overview Panel

[[LLM: Design the primary dashboard view showing most important metrics.]]

````

+------------------+------------------+------------------+
| Active Users | Success Rate | Avg Response Time|
| {{count}} | {{percentage}}% | {{ms}} |
+------------------+------------------+------------------+
| Token Usage | Cost/Hour | Error Rate |
| {{tokens/min}} | ${{amount}} | {{percentage}}% |
+------------------+------------------+------------------+

````text

### Time Series Panels

1. **Request Volume**: {{line-chart-config}}
2. **Latency Trends**: {{p50/p95/p99-lines}}
3. **Error Patterns**: {{error-types-over-time}}
4. **Cost Tracking**: {{cumulative-spend}}

### Real-time Panels

- **Live Request Feed**: {{recent-10-requests}}
- **Active Sessions**: {{current-conversations}}
- **Queue Status**: {{pending/processing}}
- **Model Usage**: {{distribution-pie-chart}}

## Cost Optimization Insights

### Cost Breakdown

[[LLM: Provide detailed cost analysis and optimization opportunities.]]

- **By Model**: {{gpt-4/gpt-3.5/claude-costs}}
- **By Feature**: {{feature-level-costs}}
- **By User Segment**: {{power-users-vs-casual}}
- **Optimization Opportunities**: {{identified-savings}}

### Token Efficiency

```yaml
efficiency_metrics:
  - prompt_compression_ratio:
      '[object Object]': null
  - cache_effectiveness:
      '[object Object]': null
  - unnecessary_tokens:
      '[object Object]': null
  - optimal_model_selection:
      '[object Object]': null
````

## Historical Analysis

### Trend Reports

[[LLM: Define what historical trends to track.]]

- **Weekly Growth**: {{user/request-growth}}
- **Quality Trends**: {{satisfaction-over-time}}
- **Cost Trends**: {{spend-trajectory}}
- **Performance Trends**: {{latency-improvements}}

### Capacity Planning

- **Projected Load**: {{next-month-estimate}}
- **Resource Needs**: {{scaling-requirements}}
- **Budget Forecast**: {{cost-projection}}
- **Optimization Impact**: {{planned-improvements}}

## Integration with Tools

### LangSmith Configuration

````python
from langsmith import Client

client = Client(
    api_key="{{langsmith-api-key}}",
    project_name="{{agent-name}}"
)

# Auto-trace all LLM calls
```text

### Custom Metrics Export

```python
# Prometheus metrics example
from prometheus_client import Counter, Histogram, Gauge

request_count = Counter('ai_agent_requests_total',
                       'Total requests',
                       ['model', 'status'])
response_time = Histogram('ai_agent_response_seconds',
                         'Response time in seconds')
active_users = Gauge('ai_agent_active_users',
                    'Currently active users')
````

## Maintenance Procedures

### Daily Checks

- [ ] Review error rates and patterns
- [ ] Check cost burn rate
- [ ] Verify all services healthy
- [ ] Review user feedback

### Weekly Reviews

- [ ] Analyze performance trends
- [ ] Update alert thresholds
- [ ] Review optimization opportunities
- [ ] Update documentation

### Monthly Planning

- [ ] Capacity planning review
- [ ] Cost optimization implementation
- [ ] Dashboard improvements
- [ ] SLA compliance report

## Access Control

- **View Access**: {{reader-groups}}
- **Edit Access**: {{admin-groups}}
- **Alert Access**: {{ops-team}}
- **Data Export**: {{compliance-team}}