Scaling DevOps at Lyft

July 23, 2015 - Chad Garrick

Company culture plays a big role in shaping the direction of Lyft, a ride share app operating in 65 cities across the US with designs on expanding both domestically and internationally. One of the key tenets of its development and engineering culture is based on the idea that, “If you build it, you run it,” meaning developers are primarily responsible for the services they create.

Scaling DevOps has played an important role for Lyft.

As a result, Ryan Lane, Lyft DevOps engineer, and his team of engineers act as consultants to the developers, helping to drive consistency across all of Lyft’s services. One of the biggest goals for the Lyft DevOps team is to create an environment that allows its developers to work autonomously…scaling DevOps.

Ryan Lane gave two talks at SaltConf15. In the first talk, Lane discusses how Lyft is using masterless SaltStack at scale for infrastructure bootstrapping, deployment, service discovery, and server management. In addition, Lane reviews how Lyft organizes its SaltStack infrastructure to allow developers to manage their own operations seamlessly.

“When developers get a new laptop and pull down their new images, everything just starts,” said Lane. “It runs Salt. Everything runs like it would in production. When a developer pulls down a container, all it has to do is start its services, so we can start containers in roughly 10 seconds.”

Before his second talk, Lane posed a couple of rhetorical questions. First Lane asked, “If SaltStack is sequentially ordered by default, then why am I giving this talk.” The answer to that question isn’t that simple. SaltStack technically is sequentially ordered, but that is if you don’t use any of the requisites because basically every requisite that Salt has will modify the order of execution of the states in SaltStack.

He follows that question by asking, “But don’t requisites ensure good behavior?” In a perfect world that would be true, so long as none of your states ever fail, but what they mostly do is mask problems.

So why does Lyft use sequentially ordered states? According to Lane, because they are easier to understand and are more repeatable. Sequentially ordered execution makes it easier for Lyft developers to identify what needs to be fixed if something breaks. In this scenario, Lane and his small team of DevOps engineers act as consultants, putting a base set of frameworks in place for development teams to use that ensure consistency across all services and platforms.

In this talk, Lane reviews how to use the new SaltStack ‘listen’ and ‘listen_in’ requisites and how to use watches effectively. Additionally he demonstrates how to introduce non-ordered executions where necessary or useful, how to use SaltStack grains and pillars effectively, and how to reload grains during a job.

For more details on how Lyft uses SaltStack for scaling DevOps, watch Lane’s SaltConf15 talks embedded below. Also, find more great talks in the SaltConf15 video content blog post.