How good system administration is like new underwear

I can’t remember the last time I bought my own underwear. I may not have EVER bought any for myself. Each Christmas, there’s typically a few new pairs under the tree with my name on them. Well, more precisely, my name is on the wrapping paper that the underwear come in, not actually on the garments hemselves (because I’m too old now or much too young to have people labeling my clothes for me).

But I never have to worry about underwear. There’s an unseen process chugging along that keeps my clothes drawers replenished with fresh undergarments. It’s a constant. The underwear is just there and it just works.

Good quality system administration is like new underwear because it’s a constant, it’s just there, and it just works.

If the development team is worried about placating the system administration team or if the dev team constantly has to work around various odd limitations in the system, then you’re system administration is not like new underwear at all. It’s like a sock that has holes.

You don’t want to worry when you put on your socks in the morning that your big toe might shoot out of your sock into the cold morning air, right? Is there a more uncomfortable feeling? Similarly, you don’t want to launch your new app and have to cross your fingers that the database is going to hold up. (Perhaps I’m starting to mix metaphors here with all the underwear and socks talk)

But my point is this… When your dev team has to worry about the system administration, they’re going to spend more time talking about and coming up with solutions to issues that could probably be addressed by more robust system administration. That makes your programming projects take longer. And that’s costing you money. Real countable money.

So get yourself a good admin. Pay them wisely. Give them new toys to play with. And never worry about your underwear again.

Continue reading

Google Adsense and Google Analytics may be blocked by McAfee

I can’t confirm what’s going on but I believe that at least McAfee and maybe other firewall appliance providers recently starting blocking in the software of their appliances. It started happening at work this week where all of a sudden Google Adsense and parts of Google Analytics were unavailable to us. When I traced it, it was all requests to *, mostly Once we created an entry within our McAfee appliances to allow those addresses through, everything went back to normal. YMMV.

Continue reading

Development Environment layout using Linux, Apache, PHP, and Subversion

Some of the age old questions I face lately are:

  • What’s for dinner?
  • Should I accept that friend request on Facebook for the friend of a friend of a friend that I knew 15 years ago?
  • What’s the best development and test environment layouts for PHP using Apache as a web server with Subversion for version control for multiple developers?

Some of you may be asking yourselves the same questions. The choice of dinner is a personal one. I won’t go into that except to say that everyone loves a good burrito. Spicy! And you probably don’t care about my take on Facebook etiquette since your friends list probably dwarfs mine.

But I do have some definite thoughts on the layout of development environments. And I find that there’s a huge lack of information about this on the interweb, so here you go.

We use Linux, Apache, PHP, and subversion in our development environment and so these instructions will be biased towards these topics but I think you can apply this method using various other technologies.

I like to give each developer their own development web site and development database. I find it’s easier for everyone to have their own individual sandbox to play in. We give them each their own domain using their initials, something like for me, Rich Zygler, and for another developer, Vinny Bagadonuts. We set the Apache directories up on the Linux dev box in a similar fashion:


This has a few benefits. If I need to show Vinny something with my site development, I can just send him the link to I can make changes to code, even major infrastructure code and not break anything for the other developers. We do the same thing with the databases, prefacing them with our initials.

Now, since our dev boxes use Linux, we set up Samba for sharing on these web directories. This means that all the devs can edit files and use source code management on the Linux server itself or on their Windows machines (we use either Eclipse or Zend Studio and create projects on the shares, that’s a whole different posting!).

This dev site layout is closely linked to the way we use Subversion for version control. When we start a new site or application, if we can split out the development evenly enough, we’ll just have everyone work from the trunk version of the code, with each developer working on their own little section. Each developer puts the trunk in their Apache dir and we edit the Apache configs to reflect this:


The root of the dev sites typically look like this:

/var/www/ ( your Apache document root )
/var/www/ ( non-public PHP library code )

When we commit code changes in Subversion, we have a hook that updates our main development site here:


Again, that site can be seen on the web at This way, we can do integration testing on our code to make sure our new code doesn’t break code from someone else within the dev site.

Now, the important thing here is that the Quality Assurance (QA) and Testing people ( if you’re lucky enough to have them ), don’t use any of these previously mentioned sites for their testing. Why not? Well, because if they’re doing a good job and are therefore sufficiently anal, they’re going to complain when code is changing on the site they’re looking at.

So we give them their own test site and database that’s viewed on the web at and setup in apache at:


The developers will meet and create the list of files and database changes that get moved over to the test site. How the actual moving is done doesn’t really matter. If you have the time and energy to set up some Ant or Phing tasks, that works great. But copying/rsyncing files and running some SQL on the test database works just as well. The most important part is that the developers meet to decide which part can go to test. Otherwise, you could have code going to test and eventually production that might not be fully vetted.

When QA finds bugs in, they can send them to the developers. The developers can instantly start working on fixing the bugs in their own dev space at and not affect the other developers or the ongoing testing of the application. Pretty nice right?

Advantages of this approach

  • Uses source code management
  • Developers can unit test their own code
  • Developers can do integration testing between each other’s code
  • Developer A typically doesn’t destroy code or data that developer B is using
  • Developers don’t destroy code or data that QA/testing is looking at
  • Developers can both edit files and use source code management in either Linux or Windows environment
  • Very scalable. Adding new developers into the mix is as simple as adding their respective sub domains and databases (of course, this can also be viewed as a disadvantage, see below)
  • Less bugs make it to production


  • Lots of sysadmin overhead initially and with each subsequent domain added. You have to set up all those developer sites, rzdev, vbdev, etc. Same overhead when using branching within subversion. Plus, you have to setup all those databases and setup the config code to connect to the appropriate database for each developer domain.
  • Lots of file space for all the sites and databases.
  • Confusing for lone wolf and gunslinger developers who are used to overwriting production or each other’s development code (too bad for them!)

So what do you think? How do you setup YOUR PHP development environment?

Continue reading

Using Subversion externals property for WordPress upgrades

I find that upgrading apps like WordPress, Drupal, Symfony and open source PHP apps is simple for less complicated environments, but once you start adding in things like new directories, custom themes/modules, source code control as well as separate development, test, and production systems, the upgrades start to get pretty hairy.

Take WordPress for example. A typical upgrade of WordPress involves copying the new WordPress files over your existing files. Then you have to copy back safe versions of things like wp-config.php, .htaccess (if you’re using it), as well as any custom themes/modules from the wp-content/ directory. Not to mention any of your own directories that should exist alongside your wp-includes/ and wp-content/ directories. After that you can run the upgrade.php file.

These upgrade steps aren’t terrible. They’re quite a bit better then most open source apps out there but they still suffer from a few problems:

  1. If you have your code, including your WordPress install, in Subversion or another source code control system, you have to commit all the files that change with each WordPress version. There may be files added, deleted, etc. You’ll have to keep combing thru “svn status” messages to figure out everything you need to do to get all the WordPress files into your repository. This can be painful. And take a long time.
  2. WordPress is specifically written so that you don’t ever have to muck with the guts of it. You create themes and plugins for added functionality. So, since you’re not maintaining the code that powers WordPress, do you really need all those deltas in your Subversion repository? I think not.
  3. What do you do with your own code in directories that sits alongside wp-includes/? What if it’s in a Subversion repo?

The WordPress site also has instructions for using Subversion with your site. Here, they advocate the use of “svn switch” to update your site. This is much more manageable and solves a few of the above problems. Most svn users can probably can get away with this method. But unfortunately not me.

I have additional directories on some of my sites that I need to add into my WordPress install. So I have to copy/move them into the WordPress dirs which gets tough. And then my “svn status” will get all wonky because my WordPress dir is under one repo and my code is under another. This was endlessly confusing for me.

So I found myself looking for a way to completely wall off my WordPress install from the rest of my files. I was reminded recently of the use of the Subversion externals property and my mind started buzzing with possibilities. With “externals,” I can say:

Pull the stable WordPress code from and put it into this directory named /docs/wp/

Then my other directories, which are under my own local subversion repo can exist at /docs/dir1, /docs/dir2, etc. Of course, some Apache Alias magic is needed to make all this work.

Here’s the way I set it up for my some of my projects. So far so good. This is a bit hairy to set up but subsequent upgrades are a breeze. I use this across development, testing, and production systems (how to get those environments to work with WordPress will be another entry)

First off, the previous Apache document root for domain1 was at /www/domain1/docs, so the WordPress files wound up like this:


But I also have a lot of dirs that sit alongside of wordpress like this:



We’re going to wind up changing that.

Create an subversion external property in /www/domain1/docs for WordPress

vi starts up and you can add the following line:

Save and exit

This downloads the WordPress code from the above address into your wp/ directory. Now we are cooking.

  • Now, /www/domain1/docs/wp is where all your WordPress code lives.
  • Copy wp-config.php to /www/domain1/docs/wp/
  • Copy .htaccess to /www/domain1/docs/wp/
  • Create a link from the stock wp-content/ dir to your personal wp-content dir like this

  • In Apache config, set document root for this domain to /www/domain1/docs/wp
  • Put wp-content/ dir and any other non-WordPress dirs/files into /www/domain1/docs
  • Create an alias for wp-content/ and any other non-WordPress dirs/files in Apache config

This looks like a lot of work, but it’s really only a lot the first time around. Next time WordPress has an upgrade:

Everything after the propedit in this group can and should be scripted which will basically give you a 2 step process for upgrading WordPress, while keeping you wp-content/ dir under local source code control, as well as leaving room for any other directories or files your site might require.

This technique will probably also work with Symfony although I haven’t tried it yet.

Continue reading

“SSL Error: certificate verify failed” for Amazon S3 using s3sync

Amazon has changed their certificates for SSL access to the S3 service.  If you are receiving this error and you are using s3sync, you can look here for new certs for SSL access to S3.

Long term though, this is a problem with my (and possibly your) backup solution.  Look here for more info.  Specifically this bit:

The certificate chain supporting Amazon S3 SSL is an implementation detail of the system that may change from time to time. A robust application should not depend on the Amazon S3 SSL certificate being signed by a particular certification authority. However, you can depend on the fact that we will only use reputable CAs that are widely supported by existing user-agents. The easiest way to select root CAs to bundle with your application is to simply import the set from a modern web browser with a large market share. 

Continue reading

You know you’ve made it when the spammers start using your name.

All of a sudden, I have a bunch of returned email in one of my inboxes. Looking thru it, it appears that some spammer has been sending email thru various open relays around the world and using random names at this domain as the email address of the sender.
At first I freaked out thinking they were coming from my server. I looked around the box and didn’t see any evidence of abuse. And I’m not in any server blacklists either. On a whim, I tried connecting to some of the sending servers specified in the emails and sure enough, I could send email out from a couple, so I don’t believe there’s anything fishy going on with my server. I’ve notified the appropriate people regarding the open relays.

Most normal people will never have heard of my site. It’s not famous or popular. So when they get an email from an address at this site, they should immediately know that it’s spam. It’s not like sending an email out from a site that a lot of people use like the dozen “service at paypal” emails I receive each week. I’m just a rinky-dink site about programming that earns a few dollars a month in advertising, not even enough to cover my costs.

But I can see the spike in traffic this is creating as people either open the spam email or look at the sender and want to know more. (question – exactly who are these people that have time to research every single spam item they receive? I think I want their job ). With traffic going up, bandwidth usage goes up, and eventually costs go up. Traffic going up because more people are reading my site is great. Traffic going up because people are ticked off at me is not so great.

I’m still perplexed as to how to get this spammer to stop using my domain name as the sender’s address.

Continue reading

Gmail 2.0 crashing Firefox with extreme prejudice

It’s mostly when clicking different folders like “spam.” I wasn’t sure what was going as this just started happening the past 3 days when I noticed that sure enough, I’ve been moved over to Gmail 2.0. I tried disabling and then finally uninstalling all my add-ons (plugins) to Firefox to no avail. Selecting “older version” in the top right nav of the gmail page made the problem go away. If you’re listening google, please fix this.

UPDATE:  you may need to select “older version” every day or at least a couple of days a week since gmail doesn’t seem to remember that setting.  And even the latest Firefox with no extensions has this problem.

Continue reading

Unable to find the socket transport “ssl” – did you forget to enable it when you configured PHP?

Ran into this error recently… here’s how to fix it, assuming you have OpenSSL already installed on your system.  For OpenSSL your PHP config values look like this:

–with-openssl[=DIR]    Include OpenSSL support (requires OpenSSL >= 0.9.6)

So, if you’ve compiled from scratch, you can just recompile adding this flag to your configure command.

Continue reading