Open Hour 2020-MAY-07
- Salt news and updates
- Feedback Item of the Week: additional communication channels
- CVE and security patch release
- POCs and API discussion led by Tai Groot
- Open community discussion and questions
- PR Merge Jam and Test Clinic on hold to support CVE
- Ongoing events: Hacks podcast, daily Twitch steam, weekly Open Hours and virtual meetups, and working group meetings. All can be found on the community events calendar.
- The team has recently received requests for RSS feed and security email.
- We will be implementing both of these things and will share when they’re in place
- We’ll make sure that the security updates and emails are focused on security only and that they won’t be connected to sales or marketing
- As of May 7, we have seen an incredible amount of collaboration across the community, folks leveraging their expertise, and helping one another with patches and updates. Thank you!!
- A sizable chunk of people still haven’t applied the CVE patches. We’re going to continue to share messaging and raise awareness
- Blog post content clarification (https://community.saltstack.com/blog/active-saltstack-cve-critical-updates/)
- When we say patch, we mean a patch that’s applicable to all unsupported versions
- When we refer to RPM or package, we mean something for the currently supported versions
- Two waves of patches
- 4/29: All of these patches contain all that is needed to address the CVE. If you picked up and applied the 4/29-4/30 patch, you’re good
- As the week went on, we found 2 issues that were functional regressions causing the master to throw exceptions and errors in the master. They are not part of the security update, but rather a fix for the master. Only applies to those on 2017 and earlier
- For 2019 and 3000, we are going to proceed and update the packages. The new packages will be available next Wed. (2019.2.5 and 3000.3). These are being updated because we’ve identified two issues that introduce functional regressions, not because of security concerns
- Docker images will be updated next week
- Request for updates to document on how to harden Salt deployments and reinforce best practices for security
- The document will be available on Monday, hosted on https://community.saltstack.com/
- There will be another use of @channel for this announcement
- Communications in general
- We appreciate the feedback on best practices and are considering all of the suggestions people make. Community will be updated as we implement these suggestions.
- We’ll also be reinforcing the communication plan so people know what to expect when there are releases or other major announcements
Moe Anderson and Tai Groot
- We’ve been asking those who have found how to exploit to give us a grace period to allow people to patch
- The common ethical method is to allow people time or create a tool that allows people ensure that they’ve been protected
- We’ll be partnering with experts during the blackout period in order to create some utilities to ensure
- We can’t control or force those who are sharing POCs. Thank you to the #salt-store-miner channel for working on this.
- API Tool Overview (Tai Groot)
- He’s been updating saltexploit.com frequently – this is not affiliated with SaltStack in any official capacity. The SaltStack offline checker will be shared on this site as well as SaltStack’s official tool.
- What is the progression of exploits we’ve seen? What malware?
- We can’t guarantee that the first person to use this exploit was the mining application. The attacker used the exploit to grab the master’s key and run a command against all the minions to run a shell script that deleted files, killed processes, and downloaded and run a payload. We have copies of those. That payload was called salt store.
- The initial attack wasn’t too malicious. All that was required to prevent this is to update or patch the salt master and restart
- The next day, a second version was uploaded . We know exact times because bitbucket was used. It was used because it’s whitelisted
- Version 4 was tricky to remove, then with version 5 the advice changed completely. An additional attacker became involved, likely using POCs that were published. The attack added a worm. Current advice:
- Understand that there will be losses. The vulnerability has been shared widely now and made its way into media. It’s the responsibility of admins to update the systems.
- Shut down Salt Master (take a backup & snapshot for forensics state)
- Set up a new Salt Master and make sure it’s updated to the latest version of SaltStack
- Consider using tools for ensuring you’re not vulnerable created by Tai, SaltStack, and another community user
- After verifying you’re not vulnerable, you can attach minions to the master, but that’s risky. If compromised minions are attached to compromised master, another master will need to be spun up
- Once things are up and running, do some forensics and assume that if it was on the box it was taken
- Consider changing API keys
- Can you describe the API tool? How do we ensure it’s not being exploited?
- If you run the API, a new file is created that is called hacked.txt
- He has built and provided this API with some caveats – linked back to part in GitHub when he first announces this. Concern that people are sending their IP addresses to a stranger on the internet. There is valid concern about whether to trust Tai. He’s not going to share exactly how to exploit the POC.
- Motivation: People wouldn’t be interested in this until/unless it affects them. How can he make is easy for people to check whether they’re vulnerable? He considered writing a POC, but acknowledged it would be very easy to weaponize
- He sent this out for a peer review
- When he set this up for his own system, he was going to run something local. He ran into issues importing Salt on Python 3.8. He created an API to support and be helpful
- He’s not logging any of this information
- To confirm, if you’re on 2018 do you need to apply additional packages?
- No. You’re covered by the initial CVE patch
- When is Salt breaking on Ubuntu 20.04?
- Ubuntu 20 wasn’t one of the supported areas. We’re working toward making it supported. When we support it, we will fix it. We’ll see if there’s a way we can accelerate.
- Is the sodium release date changing?
- As of now, we have no plans to change the date
- We may shift some internal milestones
- Release timeline: https://github.com/saltstack/community/wiki
- What’s the latest on the PR Port Jam?
- Community members want to help with PRs that will be ported into master, we’ve gone from 900 to ~350 in the backlog. The intent was to run through prioritized PRs with dedicated blocks of time and support
- After tomorrow’s planning meeting, we’ll be making and sharing a decision on how to run the PR jam and make it a success
Moe Anderson and Cassandra Faris
- Thank you, community for coming together and supporting one another
- Please reach out with suggestions, open hour topics, and how we can support the community
- Submit topic suggestions at: https://docs.google.com/forms/d/1rA5C3fRmOAelNng0jcsPcs6TUeoCJRh-rng-NtDY_eM/edit