How to Fine-Tune Your Salt Reactors

August 30, 2018 - Mike Place

The SaltStack Salt Reactors is at the heart of event-driven workflows. It processes events on a Salt Master that flow in from Minions, evaluates the events for matching patterns, and reacts by spawning job actions in response to the matches.

In very large installations, such as those that could contain many thousands or tens of thousands of Minions connected to a Master, the Reactor may need to process hundreds of events per second (it’s a tough job). In those cases, it’s beneficial to understand some internals to better tune Salt to handle a high volume of events.

In cases where a large number of events are expected, use these settings:

  1. Turn the setting for “reactor_worker_hwm” up.
  2. Set the default value to “10,000.”

This value controls the depth of the number of events that can be enqueued by a Salt reactor while a job is running. To guarantee that events will never be dropped, one may set this value to 0` but at the cost of additional memory consumption and a possible reduction in performance.

If you are in a situation where many long-running jobs are expected and execution concurrency and performance are a concern, one may also increase the value for “reactor_worker_threads.”

This will control the number of concurrent threads that are pulling jobs from the queue and executing them. Notably, this bears a relationship to the speed at which the queue itself will fill up. The price to pay for this value is that each thread will contain a copy of Salt code needed to perform the requested action. (We call these “clients.”)

Tuning this value up too far may result in the Python GIL beginning to show its concurrency weaknesses, so it’s perhaps not advisable not to blindly adjust this.

For those interested in the details on Salt Reactors tuning as well as to follow ongoing work to make the Salt Reactors scale higher than ever, we’re busy on GitHub and progress can be followed here.