Posted: Fri, 22 March 2013
[The Standalone Sysadmin](http://www.standalone-sysadmin.com/blog/) asks,
"[How do you manage database server
> Do you have one (or a few) centralized database servers, either standalone
> or clustered, or do you spread the load like we are currently?
His argument for centralisation is one of easing the management burden of
configuration and backups, whereas the distributed approach eliminates a
central point of failure and performance degradation.
I go for distributed, all the way. For a start, we run so many databases
for so many customers that there's no way on earth we could stand up a small
number of database servers and handle all the load (hell, we've got single
customers who consume a cluster of machines with 384GB of RAM and all the
SSDs you can eat). Security and permissions is a whole other kettle of
fish; the contortions we'd have to do to allow customers the level of
management they need with a centralised database system would be immense.
Then there's the need of some customers for MySQL, some for PgSQL, different
performance tuning for different workloads... nope, centralised DBs don't
work for us.
Given this, we've bitten the bullet and solved pretty much all of the
management problems. Installation and configuration is all handled via
Puppet, and backups are trivial -- the same system that installs the DB
server itself also drops a hook script that the backup agent uses to know
that it has to dump a database server. Monitoring that this backup is
taking place successfully is also automatically provisioned, so we know that
we're not missing anything.
Ultimately, this same approach applies to practically anything that you're
tossing up between centralised and distributed. At scale, you can never
rely on centralisation, so you may as well bite the bullet and learn how to
do it distributed pretty much from the start. That saves some serious
system shock when you discover what your hardware vendor wants for the next
step up in big iron hardware...