How to deal with the "package { gcc: }" problem in Puppet
Posted: Wed, 30 October 2013 | permalink | 4 Comments
In a comment to yesterday’s post on why your Puppet module
sucks, Warren asks
what can be done about the problem of multiple modules needing to include
the same package (gcc being the most common example I’m yet to come
across).
As I stated previously, there is no sane, standardised way for multiple independent modules to cooperate to ensure that certain packages are installed for their cooperative use. If it wasn’t already obvious, I consider this to be a significant failing in Puppet, and one which essentially renders any attempt at public, wide-scale reuse of Puppet modules impossible. Software packages are such a fundamental part of system management today that it is rare for a module not to want to interact with them in some way.
Without strong leadership from Puppet Labs, or someone core to the project who’s willing to be very noisy and obnoxious on the subject, the issue is fundamentally unsolveable, because it requires either deep (and non-backwards-compatible) changes to how Puppet works at a core level, or it needs cooperation from everyone who writes Puppet modules for public consumption (to use a single, coordinated approach to common packages).
The approaches that I’ve seen people advocate using, or that I’ve considered myself, roughly fall into the following camps.
Put your package into a class in your module
Absofrickinglutely useless, because the elementary package resources you
end up creating will still conflict with other modules’ package resources.
Anyone who suggests this can be permanently ignored, as they don’t
understand the problem being solved.
Use a “globally common” module of classes representing common packages
This could work, if you could get everyone who writes modules to use it (see “someone willing to be very noisy and obnoxious”). Practically speaking, anyone who suggests this doesn’t understand human nature. I don’t see this actually happening any time soon.
Use the defined() function everywhere
By wrapping all your package resources in if !defined(Package["foo"])
blocks, you can at least stop your module from causing explosions. What
it doesn’t do is make sure that the various definitions of the package
resource are congruent (imagine if one was package { "gcc": ensure =>
absent }…). In order to safely avoid explosion, everyone would
have to follow this approach, which (in the worst case) reduces the problem
to the “globally common” module of classes.
However, it’s the least-worst approach I can practically consider. At least
your module won’t be the cause of explosions, and realistically no
module’s going to ask for gcc to be removed (or, if they are, you’ll
quickly find them and kill them).
I’ve seen calls from Puppet core dev team members to deprecate and remove
the defined() function entirely. Given the complete lack of alternatives,
this is a disturbing illustration of just how out of touch they are from the
unfortunate realities of practical Puppetry.
Use virtual resources
This is yet another variant of the “common class” technique that would require everyone, everywhere, to dance to the same tune. It has all the same problems, and hence gets a big thumbs-down from me.
Use singleton_packages
This module is a valiant attempt to work around the problem, but practically
speaking is no better than using defined() – because again, if a module
that isn’t using singleton_packages specifies package { "gcc": ensure
=> absent }, you’re going to end up with assplosion. The mandatory
dependency on hiera doesn’t win me over, either (I’m a hiera-skeptic, for
reasons which I might go into some other time).
Use only modules from a single source
This is the solution that I use myself, if that’s any recommendation…
As a result of pretty much every publically-available Puppet module sucking
for one reason or
another, I do not,
currently, have any externally-written Puppet modules in the trees that I
use. Every module (158, at current count) has been written by someone in
the sysadmin team at $DAYJOB, to our common standards. This means that I
can coordinate use of package resources, using our module of common
classes where appropriate, or refactor modules to separate concerns
appropriately.
If you’re wondering why we don’t have all of these 158 modules available publically, well, we have a few of them up, but yes, the vast majority of them aren’t publically available. Some of them suck mightily, while many others are just too intertwined with other modules to be able to use on their own, and we don’t want to release crap code – there’s far too much of that out there already.
4 Comments
From: pierre
2013-10-30 19:21
You should look at ensure_packages in the standard puppet libraries. It is not perfect, but a step in the right direction. It’s better than if !defined(…), because it tests if the package’s defined state is present. And of course it doesn’t work if someone defined the package with ‘ensure => latest’.
Hope this helps
From: Alex
2013-10-30 22:57
It’s a language flaw which needs addressing. I use both Puppet and Ansible for various servers - with Ansible I can tell it to install the same package as many times as I want.
Overall the Puppet language doesn’t feel like it’s something which is easy to extend/enhance, which is why I’m probably in the same situation - all packages are created internally. Sometimes I’ll use another package as inspiration, and create a considerably smaller version which does only the things I need.
From: John Cooper
2013-10-31 01:05
Hey we have written over a hundred modules as well, as have a lot of shops. I see this as a massive waste that needs to be fixed not a solution to the problem!
I think the forge is one of the weakest points of puppet at the moment. There are 53 apache modules there already, to take one example. I think some coordination on these would really benefit. There have been some discussions on the mailing list about this and the creation of a standard module layout repo to explore some ideas. https://github.com/stdmod/puppet-modules
Thanks for posting these, more discussion of ideas is always good.
john
From: Matt Palmer
2013-11-04 08:38
Wow… write a rabble-rousing blog post, go offline for a couple of days, and look what happens!
Pierre: I wasn’t aware of the existence of
ensure_packages. It’s curious that none of the web searches I did on the subject brought it up. That’s a pretty serious concern, really – something that’s invisible to module authors isn’t going to get used by them. Also,ensure_packagesstill suffers from the same problem as the other “third party” solutions – everyone needs to use it in order for it to not be a ticking time-bomb of random fail.John: I’m by no means saying that writing all your modules yourself is a good solution, merely that it is the practical solution. Without a single, centralised solution to this (and a number of other problems I haven’t bothered to rant about yet), you just can’t reliably use other people’s modules.
Post a comment
All comments are held for moderation; markdown formatting accepted.