logagent-filter-output-sql/README.md

Filter and aggregate parsed logs with SQL

# logagent-filter-output-sql
Filter and aggregate parsed logs with SQL  and [@sematext/logagent](https://sematext.com/logagent)

This applies SQL queries on parsed log events. The result of the query is emitted as new event, while the original events are omitted. 

Using SQL it is very easy to aggregate values, e.g. group HTTP requests by status codes. The SQL WHERE statement is useful to filter events, before they get shipped to Elasticsearch or [Logsene](https://sematext.com/logsene). 

# Installation 

Assuming @sematext/logagent is installed gloabally: 

```
npm i -g @sematext/logagent
npm i -g logagent-filter-output-sql
```

# Configuration 

Add following section 'outputFilter' to @sematext/logagent configuration file. Please note you could use the plugin with multiple configurations for different event sources. 

```
input: 
  files:
    - './access.log'

outputFilter:
  - module: logagent-filter-output-sql
    config:
      source: !!js/regexp /access.log|httpd/
      interval: 1 # every second
      queries:
        - # calculate average page size for different HTTP methods
          SELECT 'apache_stats' AS _type, 
                  AVG(size) AS size_avg, 
                  COUNT(method) AS method_count, 
                  method as http_method
          FROM ? 
          GROUP BY method
        - # log each request to the login page 
          SELECT * 
          FROM ? 
          WHERE path like "/wp-login%" 
output:
  elasticsearch:
    url: http://localhost:9200
    index: mylogs
```

Run logagent with your config: 
```
logagent --config logagent-example-config.yml 
```