Pop Culture: Plugin Oriented Programming for Infrastructure Automation

January 8, 2020 - Thomas Hatch

Brilliant Innovative Geniuses

Linus Torvalds, Walt Disney, Richard Stallman, Guido Van Rossum, Steve Jobs and Bill Gates. These are just a few of the brilliant, innovative geniuses who became so good at what they did, they were no longer able to practice their craft. Walt Disney loved to draw. His passion is what created one of the most successful entertainment companies on the planet. However, he became so busy managing and maintaining his creation that he simply couldn’t afford to animate his own characters. Similarly, how much coding do you think Guido Van Rossum actually does today? For innovators, the unfortunate reality is that most (looking at you engineers, developers and content creators) will eventually become victims of their own creation. Linus Torvalds learned: it’s all about the plugins, or some capability like plugins. This is why I created Plugin Oriented Programming for Infrastructure Automation, aka – POP.

Within three years of creating Salt, it was everywhere. Today, it’s in 147 countries, has thousands of contributors and there are more than 300 Salt-automated jobs or activities happening per second, 24/7, around the globe.

As Salt has become more popular and pervasive, I’ve felt a strong sense of obligation to add more and more capabilities and functions, requiring even more dedication and maintenance. Somehow, creativity and innovation need to be in the mix as well which is…challenging. From the SaltConf19 keynote stage, I announced several new open source projects to help bolster the Salt project and protect it, and hopefully other software projects, against the innovator’s dilemma. I also provide a similar overview of POP and new Salt open-source innovation projects in this webinar.

POP was created specifically to solve this problem. It wasn’t designed to be an interface just to write plugins. POP exists so that programs written within it can be pluggable with each other, saving innovators from becoming slaves to their own creations, and the complexity that results from pervasive adoption. Adoption is the goal and a good problem to have. I simply created POP to make project success less of a problem.

POP seeks to solve the problem we’ve all battled since we started coding. You’ve no doubt heard the old, cliched question, “How do you eat an elephant? Not all at once, you eat it one piece at a time.” This is what POP allows Salt users and contributors to do. It allows human beings and their inventions to become scalable, one piece at a time. The best way to achieve computational scale is to first achieve human scale. POP accomplishes this by taking the approach that everything is pluggable, and through POP I’m excited to tell you about several new pluggable SaltStack projects.

The first new project is called Heist.

Now, I’m well aware that telling your CISO the company needs to install a new program called Heist may raise a few eyebrows. That said, it really is the perfect name for this amazing tech. Salt Heist allows you to develop and distribute dissolvable agents.  With Heist, you can open an SSH tunnel, send a portable agent to a target system and when the job is done, like the perfect heist, all trace of the agent, including the tunnel…disappears. Although Heist was originally designed for Salt, thanks to POP it can be plugged into any other POP project as well.

Next is a project called Umbra.

The definition of Umbra is the darkest part of a shadow. We call this program Umbra because AI (artificial intelligence) and ML (machine learning) decisions are often made in the dark spaces of a program the developer can’t see. Umbra exists to get your data through that darkness. How does it work? Umbra has a plugin system to attach to data streams, then it prepares that data for machine learning and then finally attaches it to an outbound data stream. All possible because Umbra is a POP project and pluggable with any other program.

The last POP-based project I’ll cover is called Idem.

If you’re anything like me, API and cloud management are a frequent source of frustration. Let’s be honest, the entire unified cloud space is a disaster. A lot of software has sought to be the one source to automate them all. During the last 20 years, we’ve become resigned to the conclusion that unified cloud control and security is impossible. But does it need to be?

We have platforms that have produced enough code to handle this problem. So what does that mean? It means we haven’t been looking at this problem the right way.  There are actually many models of developing a unified product at scale. 

  • The Linux kernel
  • Operating systems
  • Programing languages and all of their libraries 

I’ve studied these models extensively and they all have a few common principles I used to build POP. With POP as its foundation, we created Idem. Idem is an idempotent, imperatively executed, declarative programming language. But it’s much more than a programming language.  Idem is a common language runtime and any language can be built on top of it. This takes personal preference out of the mix, making language preference irrelevant. Idem can seamlessly execute code in multiple languages as if they were one. As cool as that is, it still doesn’t solve the problem of unifying API and cloud management. The only real way to solve this problem is to scale people. Remember the question about the elephant? Idem is written in Python and leverages the Python ecosystem so all you need to do to extend Idem is to just make a Python project. There’s no need to reinvent working wheels.

If you’d like more detail about Salt POP and these new Salt open source projects, make sure to listen to this webinar titled, “Salt Innovation Projects Deep Dive: POP, Heist, and Umbra.”

I’m also thrilled that all of these new Salt core projects and components allow us to expand and support the initiatives of our commercial products. Earlier this year we launched SaltStack security operations automation products, SaltStack Comply and SaltStack Protect, built to help IT operations and security teams collaborate better, and automate more, to help businesses realize true security across their production infrastructure.

SaltStack security automation products make the dream of continuous compliance, and vulnerability remediation at scale a very real possibility. Business spends hundreds of billions of dollars every year on security tools that are great at telling us where the risk is, at telling us what’s broken. But these tools do nothing to actually fix issues or security vulnerabilities or to bring infrastructure into compliance.

To say all of us at SaltStack are excited about the future would be an incredible understatement. If you’d like to learn more about what these new Salt projects can offer and how they can impact your organization, please visit the Salt open source projects page here.