As many of you must have noticed, either by visitng or by my desperate tweets, my sites have been down yesterday for almost 12 hours. This is due to what i’d like to call a mistake of mythical proportions. But why am i writing this post now? Well, for starters i want to let you guys know what happened yesterday and, secondly, because my mistake can help you out and warn you, so you won’t have to deal with the same problems i had. As they say: “It’s a very good thing to learn from your mistakes but even better to learn from others’“. So, here is my story.
All my sites, details of which you can find on my sites index here, are hosted on what we call a Virtual Private Server (VPS) – more on VPS’s here. Yesterday was cleaning day. I opened a console to the machine and started tiding up things here and there, cleaning logs, unneeded backups, folders etc. When i was done i wanted to update my machine to the latest software available. The OS that hosts my sites is a Debian Linux distribution so, all needed, was a simple command “apt-get upgrade“. What this does actually is very simple. It checks online to see what versions are available, crosschecks them with the ones already installed and decides which need update. This way everything is done automatically. When i did that, it told me that some packages shouldn’t be updated. That puzzled me a little and i thought “well if i am to update i shall update” so i forced a total update. Process went smooth. All was done. Literally. When the update ended i wanted to check out that all was fine, and guess what, it wasn’t! My web server was unresponsive, it wouldn’t bring up my virtual hosts (thus my sites where all down) and generally it had a mind of it’s own. I started trying to revive Apache but he was still refusing any help. On the process i discovered that it all started by a serious misconfiguration the update caused to the very basic files of the virtual hosting. The damage was irreversible. All was left was a clean install. Fortunately my hosting company offers a very easy re-install process of the OS. The big thing though was backing up, restoring and reconfiguring the server. I had the opportunity to take all the backup i wanted but since time was pressuring i didn’t have a clear mind to remember everything i wanted plus some of the things i did want were too big to backup at the moment so i decided to sacrifice them. All in all it was a very long process that took almost 7 hours to partly complete. The sites are up, just a few administration tasks are there left to complete.
Now, what is this all worth mentioning? Well one big thing for me. Create a backup plan well before you need it. Because when you do, even if you are given an opportunity to take it (like my case), you are going to have loses. That is because on the time of a crisis you don’t have a clear mind to decide and remember what you need to keep and what doesn’t matter. Backup plans are a product of mature thinking and everyday tasks rather than panic reactions. So, my advice, think that your site or server is going down now, what do you want to save? Thanks for sticking around all!
Recently something similar happened to me. My own Host suspended me.
I know it sounds silly, but it happened and I don’t even have a back up. I have to do a lot of things overnight in order to make my site up, the next day.
I’ve been constantly backing up my database now and hope everything goes well for you and for me.. 🙄
Its good that everything is fine now, I will right away ask my VPS host to see if there is some automated backup process available, if not I must plan for something. Probably one lesson to be learnt here is – if we have an automated process setup to do a job, we must leave the job to that. What say?
That is the reason I stay away from VPS 😕 coz I dont understand a bit and things get worse when I try to understand 🙄
But I seriously need to shift one day or the other and fear what if this type of situation may occur 🙁 Glad you resolved them!
Nice to see you back 😉
Wow, this sounds like quite a bit of a nightmare! I am glad that you are back up. And it sounds like you’ve learned some valuable lessons. 😉
@All Thanks for coming back and supporting 😉
@Sid If you need any help i’ll be glad to!
Good to see that you resolved the problem. One of tha main reasons i am not shifting to VPS is because of such issues that can happen. But when i do, i will sure ask for your help
@Madhur: I’ll be there when you need me 😉
I’m glad you have it all fixed up Stratos. I am in the process of doing automatic updates of all my blogs, just in case something happens.
One of my hosts has moved me to another server because they say that my usage is not suitable for shared servers. The problem is that I don’t know why my usage is so high. I have updated all my plugins and the wordpress is the latest version. I’ve given them admin access so they can check it out and they say all seems fine and I have to wait 24 hours for the next cumulative stats. At least they didn’t suspend me.
PS, I forgot to thank you for your last comment. 🙄
@Sire No need to thank for commenting. As i said before i comment when i feel like it and with your blog among a few others i feel like it a lot 😉
As for your sites… What part is overloaded? CPU? Memory? Do you have any kind of caching enabled? If you want any help you know where to find me… I’m always there for a friend 😉
At their request I have installed Super-cache and have it running on all my blogs. They say I am using too much CPU and Top Processes, but I have no idea what can be causing it.
@Sire: Well since the configuration of the webserver is theirs my bet would be on DB queries. If you want we can take a full mirror of your blog install it on a machine and start hitting it. Disabling plugins one at a time see what is using up all the CPU. It must be the plugins…
I don’t want to put you to any trouble stratos. Are there any plugins that have a known issue?
@Sire: Let me put it this way. If you feel comfortable and trust me with your data in my hands then i’d be delighted to help you. This is the only reason i was hesitating. Else give me a list of the plugins you are using (mail me if you feel better) and i’ll check them out see what happens…
Statos, I have no worries at all about you looking at it. Email me what you need and I would appreciate any help you can give.
Sounds horrible 🙁 A backup plan is SO important. Good thing you’ve found gog.com to distract you from your troubles 😉
@Sire I’m mailing you know…
@Kim Thanks to you i did! I appreciate you stopping by Kim!
Thanks for taking the time to help me out Stratos. 🙂
@Sire: As i said on your blog if i come up with anything more i’ll let you know. Also, if you need anything else you know my mail 😉
Yes, I do stratos, and I assure you I won’t be a pain in the proverbial 😀
@Sire: I’ll be glad to help out… Don’t feel that way 😉
@Sid, Madhur and all those who are saying VPS is troublesome,
you are completely mistakes guys. There are 2 kinds of VPS – managed and unmanaged. What stratos is using is Unmanaged VPS, but what I am using is Managed. I control everything using cPanel & Power Panel… way too easy!!
For those who are not comfortable with linux, i suggest to go for VPS. It rocks!
I meant “managed VPS” in the last line.
@Raju: This is a very nice thing you cleared out man. It is true that there is a difference and managed VPS is very easy to administer…
I can imagine your situation and relief and happiness you got after everything is restored to normal. 🙂
Glad its back up and running. Is it just the database, plugins, .htaccess and wp-config.php you need to backup for a blog, or do you need to download the whole thing to back it up? I have the database backup plugin installed, which emails me a copy of the database each day, not sure if you use it but its a pretty useful plugin.
@Khaled: Well backing up your database is half the job. In order to be 100% sure you need to backup the whole directory structure from your FTP. This is how i do it…
It’s good to see you got it back up though.
A backup plan is important, I don’t do anything that would make huge differences to mine without first making backups and making sure I can fix anything I may screw up.
@Funny Stuff: Well you are absolutely right but there are certain points in history that you do a majorly stupid thing without even noticing…