Version Control

Lack of version control over your infrastructure leads to eventual confusion. We use version control for tracking OS configuration files, OS and application binaries and source code, and tools and administrative scripts. We have managed independent evolution of several infrastructures, and were able to do rollbacks or rebuilds of damaged servers and other components.

It may seem strange to start with version control. Many sysadmins go through their entire careers without it. But infrastructure building is fundamentally a development process, and a great deal of shell, Perl, and other code tends to get generated. We found that once we got good at "doing infrastructures", and started getting more work thrown at us, we had several distinct infrastructures at various stages of development at any given time. These infrastructures were often in different countries, and always varied slightly from each other. Managing code threatened to become a nightmare.

We found that CVS helped immensely in managing many different versions and branches of administrative code trees [cvs] . It took some careful thought and some tool building to be able to cram O/S configuration files and administrative code into a CVS repository and make it come back out okay on all the right machines. It was worth the effort. In later iterations, we began migrating the hundreds of megabytes of vendor-supplied O/S code itself into the CVS repositories, with some success. The latest version of CVS (1.10) has additional features which would have made this much easier, such as managing symbolic links natively.

Since all of our code is mastered from a CVS repository, we can actually destroy entire server farms and rebuild them with relative impunity during the course of development, moves, or disaster recovery. This also makes it much easier to roll back from undesired changes.

In short, based on our experience, we'd strongly advise setting up and using CVS and associated tools as the first step in an infrastructure development program.

We've tried various vendor-supplied version control tools -- everyone has their favorites. While many of these seem to offer better features than CVS, none of them turn out to be flexible, robust, or WAN-centric enough to manage operating system code in-place on live machines scatter all over the world. Because of its Internet heritage and optimized use on far-flung projects [samba] [bsd] , CVS is almost perfect for this. Where it isn't, we've been able to get under the hood and get what we need in a way that we would never have been able to with a proprietary tool.