New Salt Extension: Salt Analytics (Salt managed data processing pipelines)
The Salt Project team is releasing salt-analytics-framework, a new Salt extension, to allow users to run data processing pipelines alongside Salt.
These pipelines are managed by Salt as an engine and consist of the following steps:
- Collecting (logs, run salt modules, etc…)
- Processing (masking, etc…)
- Forwarding (disk, etc…)
Each of these steps is plugable, meaning you can write your own collectors, processors, and forwarders to suit whatever specific use cases that may arise. You can write a simple Python package with your plugins, and with the correct entrypoints, they will be available to use in your pipelines after your package is installed. With this plugability, salt-analytics-framework opens the door to use machine learning models to process and analyze data.
Simple install instructions: On a salt-master system with targeted minions, run:
salt <targets> pip.install salt-analytics-framework
Example: Dump memory usage and status information to disk every 5 seconds.
engines:
- analytics
beacons:
memusage:
- interval: 5
- percent: 0.01%
status:
- interval: 5
- time:
- all
loadavg:
- all
analytics:
collectors:
beacons-collector:
plugin: beacons
beacons:
- "*"
processors:
noop-processor:
plugin: noop
forwarders:
disk-forwarder:
plugin: disk
path: /var/cache/salt
filename: events-dumped.txt
pretty_print: true
pipelines:
my-pipeline:
collect: beacons-collector
process: noop-processor
forward: disk-forwarder
Links: