UNPKG

meld

Version:

Meld: A template language for LLM prompts

github.com/adamavenir/meld

adamavenir/meld

147 lines (118 loc) • 4.47 kB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
# SDK Integration Test Patterns ## Test-Implementation Misalignment We've identified a pattern where SDK integration tests may be making oversimplified assumptions about internal service behavior. This creates potential maintenance challenges and false negatives in our test suite. ### Case Study: Output Service Integration The `api/api.test.ts` integration tests demonstrate this pattern clearly: ```typescript // SDK integration test makes simple assumptions const content = ` Some text content @run [echo test] More text `; // Expects: // - Raw text preservation // - Simple directive handling // - Direct content matching ``` However, the actual `OutputService` implementation and its unit tests reveal more sophisticated behavior: 1. Transformation Modes - Non-transformation mode has specific directive handling rules - Transformation mode replaces directives with results - Mode selection affects entire output pipeline 2. Format-Specific Behavior - Each format (markdown, llm) has unique requirements - LLM XML format has special handling needs - Directive handling varies by format 3. State Management - Service tracks transformation state - Handles state variables differently in different modes - Complex interaction between state and output ### Impact on Test Reliability This misalignment causes: 1. False negatives - tests fail despite correct implementation 2. Maintenance burden - fixing "failing" tests can break actual functionality 3. Documentation gaps - simplified tests don't reflect actual behavior ### Recommendations 1. SDK Integration Tests Should: - Consider transformation modes - Account for format-specific behavior - Match documented interface behavior - Test actual use cases rather than implementation details 2. Documentation Updates: - Clearly document transformation modes - Explain format-specific requirements - Provide SDK usage examples that reflect actual behavior 3. Test Structure: - Move implementation details to unit tests - Keep integration tests focused on real-world usage - Add test cases for different modes and formats - Document expected behavior in test descriptions ## Implementation Plan ### Phase 1: Test Infrastructure Updates (1-2 hours) - [ ] Update TestContext initialization - [ ] Add transformation mode helpers - [ ] Add format-specific test utilities - [ ] Update test documentation patterns ### Phase 2: Basic Transformation Tests (2-3 hours) - [ ] Test transformation mode enabling/disabling - [ ] Test state variable preservation - [ ] Test basic directive handling - [ ] Test content preservation rules ### Phase 3: Format-Specific Tests (2-3 hours) - [ ] Markdown format tests - [ ] Headers and formatting - [ ] Code blocks - [ ] Directive placeholders - [ ] LLM format tests - [ ] XML structure - [ ] Special characters - [ ] State representation ### Phase 4: Integration Scenarios (3-4 hours) - [ ] Full pipeline tests - [ ] Parse -> Transform -> Output - [ ] State management - [ ] Error handling - [ ] Mixed content tests - [ ] Multiple directive types - [ ] Nested transformations - [ ] State inheritance - [ ] Edge cases - [ ] Empty content - [ ] Invalid directives - [ ] State conflicts ### Phase 5: Documentation & Examples (2-3 hours) - [ ] Update test documentation - [ ] Add example test patterns - [ ] Document common pitfalls - [ ] Create test templates ## Action Items 1. Review other SDK integration tests for similar patterns 2. Update test documentation to reflect actual service behavior 3. Consider adding SDK-level transformation mode controls 4. Add integration test examples to SDK documentation ## Risk Assessment ### Low Risk Areas - Test infrastructure changes (good existing patterns) - Basic transformation tests (clear requirements) - Documentation updates (straightforward) ### Medium Risk Areas - Format-specific edge cases - State management complexity - Performance implications ### Mitigation Strategies 1. Incremental implementation 2. Comprehensive test coverage 3. Clear documentation 4. Regular review points ## Timeline - Total estimated time: 10-15 hours - Can be implemented incrementally - Key milestones align with phases - Regular review points after each phase ## Success Criteria 1. All tests pass consistently 2. No false negatives 3. Clear test patterns documented 4. Easy to maintain and extend 5. Matches actual service behavior