@oriolrius/node-red-contrib-kafka/docs/KAFKA_HISTORY_READER_IMPLEMENTATION.md

Node-RED Kafka nodes: Send, Receive, and Schema validation with modern KafkaJS

# Kafka History Reader Node - Implementation Summary

## Overview

Successfully created a new **Kafka History Reader** node (`hm-kafka-history-reader`) that allows retrieving historical messages of specific types from Kafka topics before starting real-time consumption.

## Files Created/Modified

### New Files Created:

1. **`js/kafka-history-reader.js`** - Main node implementation
   - Complete JavaScript implementation with error handling
   - Support for Schema Registry integration
   - Configurable message type filtering
   - Unique consumer group generation to avoid conflicts

2. **`js/kafka-history-reader.html`** - Node configuration UI
   - Complete HTML configuration form
   - Support for all configuration options
   - Dynamic show/hide for Schema Registry settings
   - Input validation and user-friendly interface

3. **`docs/KAFKA_HISTORY_READER_GUIDE.md`** - Comprehensive documentation
   - Usage instructions
   - Configuration examples
   - Best practices
   - Troubleshooting guide

4. **`examples/kafka-history-reader-complete-flow.json`** - Complete example flow
   - Shows integration with inject, function, and consumer nodes
   - Demonstrates how to process historical data before real-time consumption

5. **`tests/test-kafka-history-reader.js`** - Unit tests
   - Node loading verification
   - Configuration validation
   - Message type detection tests

### Modified Files:

1. **`package.json`** - Updated to include the new node
   - Added node registration in `node-red` section
   - Updated keywords to include `kafka-history`

2. **`README.md`** - Added documentation for the new node
   - Updated node list to include History Reader
   - Added feature description and use cases
   - Added link to comprehensive guide

3. **`tests/test-node-loading.js`** - Updated to test new node
   - Added history reader to loading test

## Key Features

### Message Type Detection
The node searches for message type in multiple fields (in order):
- `type`
- `messageType` 
- `eventType`
- `kind`
- `msgType`

### Configuration Options
- **Message Types**: Comma-separated list of types to search for
- **Max Messages per Type**: Limit number of messages retrieved per type
- **From Offset**: Start from earliest or latest messages
- **Encoding**: Support for UTF-8, ASCII, Base64, Hex
- **Schema Registry**: Optional Avro schema validation support
- **Timeout**: Configurable timeout for historical search

### Advanced Features
- **Non-destructive**: Uses temporary consumer groups
- **Early termination**: Stops when all requested types are found
- **Comprehensive output**: Returns detailed summary and metadata
- **Error handling**: Graceful handling of connection and parsing errors

## Usage Pattern

1. **Inject Node** → **History Reader** → **Process Historical Data** → **Start Real-time Consumer**
2. Configure message types you need (e.g., "temperature,humidity,status")
3. Set maximum messages per type (default: 10)
4. Process the historical data to initialize your application state
5. Start your real-time consumer for new messages

## Integration Benefits

- **State Initialization**: Get last known values before real-time starts
- **Catch-up Mechanism**: Handle offline periods gracefully  
- **Debugging**: Analyze historical message patterns
- **Zero Impact**: Doesn't affect existing consumer offsets

## Example Output Structure

```json
{
  "payload": {
    "historicalMessages": {
      "temperature": [...],
      "humidity": [...],
      "status": [...]
    },
    "summary": {
      "totalTypes": 3,
      "totalMessages": 25,
      "typeDetails": {
        "temperature": { "count": 10, "latestTimestamp": 1672531200000 }
      }
    },
    "request": {
      "messageTypes": ["temperature", "humidity", "status"],
      "maxMessages": 10,
      "topic": "sensor-data",
      "totalProcessed": 1500,
      "duration": 2340
    }
  }
}
```

## Node Status Indicators

- **Yellow ring**: "Initializing..."
- **Green ring**: "Ready" 
- **Blue dot**: "Reading history..."
- **Green dot**: "Found X/Y types (Z msgs)"
- **Red ring**: "Error: ..."

## Testing Status

✅ Node loads successfully without syntax errors  
✅ Package.json properly configured  
✅ HTML configuration form complete  
✅ Documentation comprehensive  
✅ Example flows provided  
✅ Unit tests created  

## Next Steps

The node is ready for use. Users can:

1. Deploy the updated package
2. Find the new "kafka history" node in the Network palette
3. Use the provided example flows as starting points
4. Refer to the comprehensive documentation guide

This implementation provides a robust solution for retrieving historical Kafka messages by type, filling the gap between static state and real-time consumption.