My Server Blew Up

GeekLeave a Comment

Not my new Exchange server, but my hosted server. I am always fighting disk space issues on the server because it only has 2 GB of disk. However, I keep it cleaned up through crontab. Always at 90+% used though. So I woke up this morning and I had jumped the 2 GB and the server was unusable. When you are out of disk space everything goes bad. The first to die is MySQL. It starts corrupting all the files so the database just goes down.

I ended up finding out that I had 100s of megs of deferred email messages in /var/spool/mqueue. Turns out that the way my server is, it never cleans these up and they just get stuck. Enough of them with big attachments and you start filling up. I had so many files that rm returned too many arguments. Ended up doing some magic with technical support and we got it cleaned up.

I am now way back down on disk space and learned that you need to have a sendmail -q60 in your /etc/rc file in order to keep this from happening. Wish they would have set that up originally for me!