Tech, Autos, & Gear in Layman's Terms Since 2006

233

March 7, 2008 • Editorials

Surviving the Nuclear Winter after an Atomic Server Crash, Part One

As you may have noticed, Gear Diary suffered a massive server crash on Saturday, March 1st. If you didn’t know it, then it is only a testament to how awesome the Gear Diary Team is, because we all immediately scrambled to get things back to normal as quickly as possible. Things aren’t quite there yet, and for better or worse they will never be quite the same again, but as usual I am getting ahead of myself…

The night before everything happened, I had posted about my MacBook Pro order, and I had already started to receive informative post comments as well as private emails from you all about what I should do to make the transition from Vista smooth. I was excited, on top of the world, and tired…I had stayed up way too late yet again, but I planned on sleeping in the next morning to make up for it.

About 9:30 I finally crawled out of the world’s most comfortable bed, had some cereal, played with Avah, watched CNN for a bit, and then sat in front of my computer to take a look at the morning’s email. I noticed that there were a bunch of comments pending moderation, so instead of doing them individually through the email interface, I logged into our Admin panel. I approved the first comment, and got a database error. Anyone who has ever used WordPress has seen errors before, and many times a quick refresh will right them. I backed up and tried approving comments again, and got the same error.

Huh.


As I always do when something back-end happens that I can’t figure out, I called Ewdi – website code guru extraordinaire – catching him at lunch. He said he would get online as soon as he returned home. So in the meantime, I sat here waiting…hoping against hope that the site would come back online. Hoping that when Ewdi came on I could laughingly tell him "never mind, and I’m truly sorry for disturbing your lunch!"

But instead, I got to go through one of the most frustrating, depressing, and mind-numbing events any web-master could possibly endure.

Hindsight is 20/20 of course.

Were there mistakes made on my part? Absolutely.

Was my host also culpable? I believe so.

And I refuse to think that I am the only one who would make these mistakes, so with that in mind, I am going to lay it out there and let you know what happened.

My hope is that the events surrounding our crash might serve as a learning experience for all of you who are as I was – relatively inexperienced with the intricate back-end knowledge that it takes to run a website.

See, I made a vital and deadly mistake: I believed that I was paying for daily backups by my host. I believed this because parts of the wording for my VPS account read:

Data Integrity:
With a virtual private server you get built in incremental backups on top of a world class RAID 10 array to ensure data integrity.

and also the statement that Backup & Restore VPS was an included feature.

Honest mistake, right? Except I should have never been so complacent and trusting with Gear Diary’s data. What the hell was I thinking?!

I did do occasional FTP and database backups from WordPress’ built-in backup just to "be extra safe", but I had no regular schedule for them. I’m not saying this to make excuses, just to tell you how it was. If you want to call me a naive idiot, I’ll even agree with you.

So grab a cup of coffee, tea, or (in my case) a Diet Coke, and come ride the rollercoaster with me as I relive what happened last weekend in detail…

As I said, Ewdi had been at lunch, so I was waiting at my desk when he popped online:

3/1/2008
12:35:53 PM
Ewdison Then
I’m here

3/1/2008
12:36:00 PM
Judie *naf
hi!! πŸ˜€ sorry to have disturbed your lunch! no clue what happened :-/

3/1/2008
12:36:30 PM
Ewdison Then
no worries
hmm the server load is really high

3/1/2008
12:37:14 PM
Judie *naf
i had upped the capacity with my host, paid for a higher amount. and i have a flash ad on the side that i am pulling – it was messing things up, making pages slow to load

3/1/2008
12:39:25 PM
Ewdison Then
phpMyAdmin – Error Cannot start session without errors, please check errors given in your PHP and/or webserver log file and configure your PHP installation properly.

apparently mysql server is down, let me login to ssh

3/1/2008
12:39:51 PM
Judie *naf
ok

3/1/2008
12:41:14 PM
Ewdison Then
1.66 is the load average, pretty high. how much are you paying a month?

3/1/2008
12:41:41 PM
Judie *naf
it is about $1K a year

3/1/2008
12:43:25 PM
Ewdison Then
is that a dedicated?

3/1/2008
12:43:31 PM
Judie *naf
VPR

3/1/2008
12:43:33 PM
Ewdison Then
Judie *naf
you can get dedicated with that price on theplanet or softlayer

3/1/2008
12:44:05 PM
Judie *naf
hmmm
would it be better than what i have? service wise? i mean, i just upped it and paid…and they do give excellent service when there are problems – except stuff like this

meaning weird WordPress database errors that could usually be solved with a little bit of template tweaking.

3/1/2008
12:45:52 PM
Ewdison Then
sadly VPS is sharing server with all other VPS customers

3/1/2008
12:46:02 PM
Judie *naf
jeeez – i hate this

3/1/2008
12:46:04 PM
Ewdison Then
/etc/init.d/mysql restart Shutting down MySQL.
looks like mysql server is messed up

3/1/2008
12:46:36 PM
Judie *naf
so i need to contact them?
this is making my stomach hurt πŸ™

3/1/2008
12:52:36 PM
Ewdison Then
hold on, let me see if i can do something about it

3/1/2008
12:52:41 PM
Judie *naf
ok. thank you
if you tell me to move hosts i will
i get so stressed about this stuff πŸ™

3/1/2008
12:53:29 PM
Ewdison Then
i just think you should have your own server for the price you paid, 1 machine for geardiary alone

3/1/2008
12:53:40 PM
Ewdison Then
http://www.theplanet.com/dedicated-servers/default.asp?class=ss

3/1/2008
12:53:43 PM
Judie *naf
looking
let me show you what i have
http://www.hostican.com/hosting/index.php
VPS Extreme

3/1/2008
12:54:16 PM
Ewdison Then
i’m with softlayer, they are reliable and big datacenter but the price is higher

3/1/2008
12:54:17 PM
Ewdison Then
http://www.softlayer.com/servers_ses.html
yeah they [meaning HostICan] don’t even show you what processor they are using, and VPS hosting always bad choice for a reliable server

3/1/2008
12:55:02 PM
Judie *naf
i didn’t know

3/1/2008
12:55:03 PM
Ewdison Then
since you are sharing 1 machine with so many people

3/1/2008
12:55:09 PM
Judie *naf
gotcha. so for the same money, i should do this one?
Intel Pentium 4 Series 2.4+ Intel Pentium 4 2.4GHz+ 80GB Hard Drive 512 MB RAM 750 GB 5 $89.00

Obviously at this point I had no idea that this was anything other than a service outage. They happen, and as upsetting as they are, they usually resolve quickly. I figured I should take Ewdi’s advice, move to a dedicated server, and not look back. But things quickly went south…

3/1/2008
12:56:03 PM
Judie *naf
ewdi, i wrote hostican, this is what they said back

Hello Judie,

Thank you for contacting HostICan support.

Now your VPS is undergoing testing. There is no need to worry, it will be ok soon.

Sincerely,

Randy Slone

HostICan Representative

Knowledge Base: http://www.hosticananswers.com

Phone: 1-800-725-9170 (U.S. & Canada – Toll-free)

Phone: 1-804-412-0390 (International)

Ticket Details

===================

Ticket ID: VDN-345233

Department: Support

Priority: Low

Status: Closed

what the heck???
no need to worry???

3/1/2008
12:56:30 PM
Ewdison Then
thats why i hate small hosting company πŸ™

3/1/2008
12:56:32 PM
Judie *naf
then why not tell me it was going through testing before they did it?????
they have never done anything like this

3/1/2008
12:56:52 PM
Ewdison Then
I’m just afraid they lost one of the database tables

3/1/2008
12:56:56 PM
Judie *naf
oh man
what will that mean for me?

3/1/2008
12:57:56 PM
Ewdison Then
losing database πŸ™
i can’t restart the mysql, no access πŸ™

3/1/2008
12:58:14 PM
Judie *naf
you mean, losing my site???
i don’t know what that means

3/1/2008
12:58:24 PM
Ewdison Then
yeah πŸ™
do you have backup?

3/1/2008
12:58:36 PM
Judie *naf
Ewdison Then
holy god
from february 1

3/1/2008
12:58:48 PM
Ewdison Then
let’s hope for the best for now

3/1/2008
12:58:52 PM
Judie *naf
i am doing one right now
i am in there
i need automatic ones
never had them

3/1/2008
12:59:50 PM
Ewdison Then
softlayer is great with backup

Look at Ewdi rubbing it in… πŸ˜‰

3/1/2008
1:00:31 PM
Judie *naf
ok – i’m there
wow – they are expensive. why are they better than planet?

3/1/2008
1:02:23 PM
Ewdison Then
better hardware. i moved from the planet to softlayer because they offer better backbone connection and hardware but the planet is 2nd choice

3/1/2008
1:04:54 PM
Judie *naf
gah – why is it things go so well…until they DON’T
ok, i am backing up everything though FTP – will that include posts though? what file specifically do i need for posts?

3/1/2008
1:07:48 PM
Ewdison Then
did you hear from them?

3/1/2008
1:07:57 PM
Judie *naf
not yet

3/1/2008
1:08:06 PM
Judie *naf
ok – i am backing everything up via ftp
what file specifically has the posts?? i just want to make sure i get it

Notice how it still hadn’t sunk it how screwed I was? Notice how Ewdi was stalling on giving me the straight dope? Notice how I thought I could still salvage things? I think he was hoping that I would hear from my host before he had to break the news to me…

3/1/2008
1:08:23 PM
Ewdison Then
the post are stored in database

3/1/2008
1:08:31 PM
Judie *naf
the database that is down??? omg omg omg

3/1/2008
1:08:43 PM
Ewdison Then
yap!

3/1/2008
1:08:49 PM
Judie *naf
ok – this is what they said

Hello Judie,

Thank you for your reply.

Its updating itself, not something we caused. We’re just waiting for it to finish.

Sincerely,

Randy Slone

HostICan Representative

Knowledge Base: http://www.hosticananswers.com

Phone: 1-800-725-9170 (U.S. & Canada – Toll-free)

Phone: 1-804-412-0390 (International)

Ticket Details

===================

Ticket ID: VDN-345233

Department: Support

Priority: Low

Status: Closed

3/1/2008
1:08:58 PM
Ewdison Then
WHAT?
no way
MySQL doesnt update itself

3/1/2008
1:09:03 PM
Judie *naf
SERIOUSLY?

3/1/2008
1:09:06 PM
Ewdison Then
its not windows

3/1/2008
1:09:10 PM
Judie *naf
i am screwed
omg
omg

3/1/2008
1:09:33 PM
Ewdison Then
can you ask them if they own their data center or they are just reseller?

3/1/2008
1:09:40 PM
Judie *naf
let me see when the last database backup was.
ok
just asked
still no reply

3/1/2008
1:17:07 PM
Ewdison Then
the db is totally down now πŸ™ i cant even get in to fix it

3/1/2008
1:17:28 PM
Judie *naf
i am going to have a heart attack

3/1/2008
1:17:50 PM
Ewdison Then
argh, i feel like smacking those people

3/1/2008
1:17:57 PM
Judie *naf
i want to throw up

3/1/2008
1:18:22 PM
Ewdison Then
let me see if i can roll back

3/1/2008
1:18:29 PM
Judie *naf
ok

3/1/2008
1:20:55 PM
Ewdison Then
3.83, holy cow

3/1/2008
1:21:00 PM
Judie *naf
???

3/1/2008
1:21:02 PM
Ewdison Then
thats how high the machine load is. 0.5 is normal
1.0 is high

3/1/2008
1:21:13 PM
Judie *naf
omg

3/1/2008
1:21:17 PM
Ewdison Then
3.8 is off the chart
what are they doing with the server?

3/1/2008
1:21:26 PM
Judie *naf
ok – i will move gear diary – but i have to get my site back
help :(((

3/1/2008
1:21:30 PM
Ewdison Then
top – 14:21:25 up 154 days, 5:45, 0 users, load average: 3.12, 3.16, 2.63
do they have a phone # or something?

3/1/2008
1:23:13 PM
Judie *naf
yes. will you call?
i don’t even know what to ask
800 725 9170

3/1/2008
1:25:03 PM
Ewdison Then
okay let me try and call them

3/1/2008
1:25:12 PM
Judie *naf
ok – anything you need – ask
and i will find it

3/1/2008
1:29:04 PM
Ewdison Then
no support, sales, and etc no one picks up

3/1/2008
1:29:15 PM
Judie *naf
omg

3/1/2008
1:29:36 PM
Judie *naf
they have always been fast in the past – i think this is armageddon πŸ™

3/1/2008
1:29:52 PM
Judie *naf
i will move this weekend, if we can just get it back

Look at me, still hoping for the best! πŸ˜›

3/1/2008
1:30:15 PM
Ewdison Then
hmm, there is no way for me to access the db right now
the sql server is not even running

3/1/2008
1:30:23 PM
Judie *naf
holy god

3/1/2008
1:33:51 PM
Ewdison Then
i’m trying

3/1/2008
1:34:09 PM
Judie *naf
thank you, i don’t even know what i would do without you

3/1/2008
1:34:25 PM
Judie *naf
if we can get my data back, i will move this weekend

3/1/2008
1:38:37 PM
Ewdison Then
i’m just upset that they responded without solution or know what’s going on, and lied about mysql updating itself
it just doesn’t make sense

3/1/2008
1:38:46 PM
Judie *naf
right, and now they aren’t responding at all

3/1/2008
1:39:40 PM
Ewdison Then
damn it

3/1/2008
1:41:18 PM
Judie *naf
ok – they just wrote back

Hello Judie,

Thank you for your reply.

We are not resellers, we manage our own network.

Please let us know if you have any further questions.

Sincerely,

Alex Hardin

HostICan Representative

Knowledge Base: http://www.hosticananswers.com

Phone: 1-800-725-9170 (U.S. & Canada – Toll-free)

Phone: 1-804-412-0390 (International)

Ticket Details

===================

Ticket ID: VDN-345233

Department: Support

Priority: Low

Status: Closed

3/1/2008
1:42:59 PM
Ewdison Then
what is the hosting company name again

3/1/2008
1:43:50 PM
Judie *naf
HostICan
www.hostican.com

3/1/2008
1:45:51 PM
Judie *naf
ok – i am emailing them this

Alex,

Please tell me what is going on now – it seems as if they server is completely offline; we cannot access it at all. We need to get our data backed up and safe, which is now impossible. I have tried calling and no one is answering the phone. Please respond.

3/1/2008
1:46:20 PM
Ewdison Then
yeah

3/1/2008
1:46:22 PM
Ewdison Then
http://www.webhostingtalk.com/showthread.php?t=667484&highlight=hostican

not looking good, many people think they are bad

3/1/2008
1:46:38 PM
Judie *naf
ok – i will move as soon as this is resolved – today

3/1/2008
1:47:20 PM
Ewdison Then
worse come to worse, i’ll have to grab the FRM file directly and build it over with the data i can repair

3/1/2008
1:47:28 PM
Judie *naf
PLEASE. whatever we can do. i am freaking out

3/1/2008
1:55:14 PM
Ewdison Then
okay, i’m backing up the raw data rather than mysql db.

So now I am feeling hopeful that Ewdi can get my data back, and I’ll just take the data and move Gear Diary to the new host. Bullet Dodged…right?

3/1/2008
1:57:45 PM
Judie *naf
ok – you think i should go with softlayer
that is what i am going to do – which package – and i will start my account now

3/1/2008
1:58:57 PM
Ewdison Then
the reason i pick softlayer is their datacenter is near my house. if something is wrong, i just pop in πŸ™‚

3/1/2008
1:59:27 PM
Judie *naf
which level of service should i do?

3/1/2008
1:59:53 PM
Ewdison Then
you need to buy 20GB iscsi from them, it’s $15/month

3/1/2008
2:00:10 PM
Judie *naf
ok – on top of the hosting?

3/1/2008
2:00:18 PM
Ewdison Then
yes

3/1/2008
2:00:24 PM
Ewdison Then
and add cpanel for easy backup

3/1/2008
2:13:10 PM
Judie *naf
any luck with what you are doing?

3/1/2008
2:13:17 PM
Ewdison Then
not so far πŸ™

3/1/2008
2:20:14 PM
Judie *naf
ewdi, any luck there? my backup is not working via ftp – stalled at 26%
i am soooooo screwed

3/1/2008
2:21:27 PM
Ewdison Then
server is 100% down

3/1/2008
2:21:33 PM
Judie *naf
yeah. great

3/1/2008
2:21:35 PM
Ewdison Then
they took the server down while i’m doing backup rebuild. damn them!

it took me 20 min to write the rebuilt script and they just shut it off!

70-13-129-192:~ ethen$ ssh -l root geardiary.com ssh: connect to host geardiary.com port 22: Connection refused

3/1/2008
2:24:49 PM
Judie *naf
!!!

3/1/2008
2:24:51 PM
Ewdison Then
ok, now i’m officially pissed

3/1/2008
2:25:00 PM
Judie *naf
what does that mean???

3/1/2008
2:25:31 PM
Ewdison Then
connection refused, means server is unplugged πŸ™

3/1/2008
2:25:42 PM
Judie *naf
ok. why would they do that????

3/1/2008
2:26:46 PM
Ewdison Then
can you try call them again and ask why the server is down?

3/1/2008
2:26:49 PM
Judie *naf
yes, doing now

3/1/2008
2:26:55 PM
Ewdison Then
tel them we cant access it at all

3/1/2008
2:28:04 PM
Judie *naf
they want me to leave a message!!! hanging up, sending an email
kill me NOW

3/1/2008
2:28:51 PM
Ewdison Then
yeah it went to voicemail, i tried to call, too

3/1/2008
2:29:08 PM
Judie *naf
emailing. what exactly should i ask

3/1/2008
2:29:18 PM
Ewdison Then
ask them why we cant access the server

3/1/2008
2:29:21 PM
Judie *naf
ok

3/1/2008
2:37:32 PM
Ewdison Then
thx, still offline, server is not online at all πŸ™

3/1/2008
2:37:44 PM
Judie *naf
ok – no response from them yet, either

3/1/2008
2:38:14 PM
Judie *naf
what a fabulous saturday; thank you so much, i know this is NOT how you meant to spend yours

3/1/2008
2:38:44 PM
Ewdison Then
no worries judie, your problem is mine too when it comes to this stuff :p

Okay…say it with me here: Awwww! How awesome is Ewdi?

3/1/2008
2:38:56 PM
Judie *naf
you are a total sweetheart ewdi πŸ™‚

3/1/2008
2:39:28 PM
Ewdison Then
lol, but seriously, don’t hesitate to call me if you need help

3/1/2008
2:39:49 PM
Judie *naf
well, i don’t think i have ever needed help this badly!

oh! they just wrote back

Judie,

Thank you for your reply.

The server isn’t down. Its fully operational. However, at this time its doing a FileSystem Check, and it will be resolved shortly.

All is under control, we will update you shortly.

Kind Regards,

Lucy King

HostICan Representative

Knowledge Base: http://www.hosticananswers.com

Phone: 1-800-725-9170 (U.S. & Canada – Toll-free)

Phone: 1-804-412-0390 (International)

Ticket Details

===================

Ticket ID: VDN-345233

Department: Support

Priority: Low

Status: Open

what the hell????

3/1/2008
2:41:07 PM
Ewdison Then
fully operational my ass, it’s clearly down, please send them this

Last login: Sat Mar 1 13:18:24 on ttys000 70-13-129-192:~ piper$ ssh -l root geardiary.com ssh: connect to host geardiary.com port 22: Connection refused 70-13-129-192:~ piper$

if that’s not down what is it?

3/1/2008
2:41:21 PM
Judie *naf
ok – doing now, will cc you

3/1/2008
3:03:34 PM
Ewdison Then
still down πŸ™ can you access geardiary.com or it just me

3/1/2008
3:03:50 PM
Judie *naf
oh, i am SURE it is still down. but let me try

3/1/2008
3:04:01 PM
Ewdison Then
File System Check – means their harddrive down??

3/1/2008
3:04:04 PM
Judie *naf
totally down

3/1/2008
3:04:17 PM
Judie *naf
this is what my screen says

Unable to connect Firefox can’t establish a connection to the server at www.geardiary.com. * The site could be temporarily unavailable or too busy. Try again in a few moments. * If you are unable to load any pages, check your computer’s network connection. * If your computer or network is protected by a firewall or proxy, make sure that Firefox is permitted to access the Web.

3/1/2008
3:04:53 PM
Ewdison Then
i think their harddrive was corrupted, that’s why it needs file check, and that was the reason your database was down

3/1/2008
3:05:12 PM
Judie *naf
great πŸ™
still can’t ftp my stuff – there is a huge problem behind the scenes
connection totally dead

3/1/2008
3:08:50 PM
Ewdison Then
yeah to me too

3/1/2008
3:08:55 PM
Ewdison Then
i keep getting voicemail

3/1/2008
3:09:05 PM
Judie *naf
no reply to email, either. for this alone, i should switch

3/1/2008
3:09:35 PM
Ewdison Then
true

3/1/2008
3:09:53 PM
Judie *naf
they have lost my confidence – and i was happy with them until today, this is a freaking major thing

3/1/2008
3:10:50 PM
Ewdison Then
i know, big time. still voicemail. i could not get a hold of anyone πŸ™

3/1/2008
3:17:05 PM
Judie *naf
oh god

3/1/2008
3:30:56 PM
Judie *naf
any change?

3/1/2008
3:34:31 PM
Ewdison Then
nope. they really unplugged it

3/1/2008
3:54:57 PM
Ewdison Then
no news from them?

3/1/2008
3:55:48 PM
Judie *naf
nothing!!!

3/1/2008
3:56:01 PM
Ewdison Then
what was the last backup you have? the database

3/1/2008
3:56:11 PM
Judie *naf
i am afraid to say. i think it was december. nothing was automatic. i have a february 1 ftp backup; but the wordpress database was later

3/1/2008
3:56:39 PM
Ewdison Then
but only ftp files?

3/1/2008
3:56:47 PM
Judie *naf
ftp was february

3/1/2008
3:57:17 PM
Ewdison Then
my concern is many 404 error will cause seo problem with google πŸ™

3/1/2008
3:57:28 PM
Judie *naf
i know. i am totally hosed

3/1/2008
3:57:37 PM
Ewdison Then
we need to at least grab the recent db back when the server went online

3/1/2008
3:57:40 PM
Judie *naf
hostican has killed me

3/1/2008
3:58:18 PM
Ewdison Then
can you try call again and speak to sales or something, see if someone pick up

3/1/2008
3:58:25 PM
Judie *naf
yes
ok – about to try again
on call with them now!!!
got someone

3/1/2008
4:02:30 PM
Ewdison Then
oh gosh, thank god, ask them what happened to the server, its offline

3/1/2008
4:02:34 PM
Judie *naf
he thinks maybe an hour or so. he said it is more than halfway through the filecheck, evidently it started acting up.

already off phone

he was nice but not helpful – said they had no warning it was coming. it messed up and had to go into filecheck or something. he said i should lose no data. i hope not

3/1/2008
4:03:26 PM
Ewdison Then
okay that’s good

3/1/2008
4:03:41 PM
Judie *naf
am so stressed

3/1/2008
4:04:49 PM
Ewdison Then
here is what give me piece of mind on our network, i use two harddrives, 1 is main and 1 is mirroring (in case 1 hdd dies, the server is still on) using raid. I have auto rsync backup to tapes weekly, i have auto iscsi backup to nas daily. and each month i back up the sites to my local harddrive which i keep 1 at home and 1 on safety deposit box in my bank

3/1/2008
4:05:16 PM
Judie *naf
holy cow. that is some system; i need to do something similar
i will move hosts as soon as it is back

3/1/2008
4:05:38 PM
Ewdison Then
we were on the same boat as you last time with dlmag πŸ™

3/1/2008
4:05:44 PM
Judie *naf
:(((

3/1/2008
4:05:53 PM
Ewdison Then
a lesson that is very very hard to swallow :p

3/1/2008
4:06:03 PM
Judie *naf
VERY hard
i need automatic everything because i forget πŸ™

3/1/2008
4:06:27 PM
Ewdison Then
the dangerous thing about automatic sync backup is if you get hacked.
when you get hacked, the auto backup might overwrite your good backup with hacked copy

3/1/2008
4:06:43 PM
Judie *naf
oh gosh. i can see that

3/1/2008
4:07:05 PM
Ewdison Then
so it’s still good to keep a weekly, monthly, and daily copy of the site πŸ™‚

i wasnt a server admin, but over the years i’ve learned its best to be able to do backup myself. how about this – I’ll set up iscsi backup and command script. i’ll guide you step by step

3/1/2008
4:09:43 PM
Judie *naf
OK

3/1/2008
4:09:44 PM
Ewdison Then
to do backup and you can do it daily or weekly

3/1/2008
4:09:54 PM
Judie *naf
i would truly appreciate that

3/1/2008
4:09:57 PM
Ewdison Then
it will be a good habit after a while πŸ™‚ so its fairly easy. you can just copy and paste 4 command, and you are done πŸ™‚

3/1/2008
4:10:20 PM
Judie *naf
ok πŸ™‚

3/1/2008
4:10:28 PM
Ewdison Then
so i’ll set everything up for you

3/1/2008
4:10:33 PM
Judie *naf
thank you thank you :))

3/1/2008
4:11:00 PM
Ewdison Then
np

Over the next hour or so, as Ewdi and I waited for news from my host, he guided me through the purchase of a new hosted server…

3/1/2008
5:02:55 PM
Judie *naf
Option Monthly Setup Datacenter Dallas, TX – Central U.S. Server Single Processor Quad Core Xeon 3210 – 2.13GHz (Kentsfield) – 2 x 4MB cache $179.00 $0.00 Ram 2 GB DDR2 667 $0.00 $0.00 Remote Management Reboot / Remote Console $0.00 $0.00 Uplink Port Speeds 100 Mbps Public & Private Networks $10.00 $0.00 Public Network Port 100 Mbps public uplink public_port $0.00 $0.00 Private Network Port 100 Mbps private uplink service_port $0.00 $0.00 Public Bandwidth 1000 GB Bandwidth $0.00 $0.00 Primary IP Addresses 1 IP Address $0.00 $0.00 Secondary IP Addresses 4 IP Addresses $0.00 $0.00 Disk Controller SATA Disk Controller $0.00 $0.00 First Hard Drive 250GB SATA II $0.00 $0.00 Lockbox 1 GB Lockbox $0.00 $0.00 Network Attached Storage None $0.00 $0.00 iSCSI 20 GB iSCSI SAN Storage $15.00 $0.00 EVault None $0.00 $0.00 Operating System CentOS 5 (32 bit) $0.00 $0.00 Control Panel Software cPanel/WHM with Fantastico and RVskin $25.00 $0.00 Database Software None $0.00 $0.00 Web Analytics Software None $0.00 $0.00 Anti-Virus & Spyware Protection None $0.00 $0.00 Insurance None $0.00 $0.00 Monitoring Host Ping $0.00 $0.00 Notification Email and Ticket $0.00 $0.00 Response Automated Notification $0.00 $0.00 Intrusion Detection & Protection Network IDS/IPS Protection $0.00 $0.00 Hardware & Software Firewalls APF Software Firewall for Linux $0.00 $0.00 VPN Management – Private Network Unlimited SSL VPN Users & 1 PPTP VPN User per account $0.00 $0.00 Vulnerability Assessments & Management Nessus Vulnerability Assessment & Reporting $0.00 $0.00 Server Quantity 1 Host and Domain Names Hostname 1: ex: server01 Domain Name1: ex: example.com Promotional Code Pre-Tax Total: $229.00 $0.00 Texas Sales Tax: $15.11 $0.00 Total: $244.11 $0.00 Initial Charge: $244.11

3/1/2008
5:03:17 PM
Ewdison Then
nice quadcore πŸ™‚

3/1/2008
5:05:06 PM
Judie *naf
holy cow – this is going to be $1k every 4 months πŸ˜› they better take care of me πŸ˜›

3/1/2008
5:05:23 PM
Ewdison Then
yeah πŸ™ hefty

3/1/2008
5:05:33 PM
Judie *naf
but worth it if it works

3/1/2008
5:05:48 PM
Ewdison Then
πŸ™‚

3/1/2008
5:10:35 PM
Judie *naf
now if hostican will just come up πŸ™ this is 6 hours πŸ™

3/1/2008
5:28:06 PM
Ewdison Then
i have to go home soon, running out of battery :p

3/1/2008
5:28:13 PM
Judie *naf
thank you SO MUCH!!!!!

Sheesh – Ewdi hadn’t even been home while he was doing all this! Talk about a good friend!

3/1/2008
5:28:39 PM
Ewdison Then
np, i’ll drive home in a bit, and hopefully by 7 or 8 we have geardiary back up

3/1/2008
5:28:46 PM
Judie *naf
gosh, i hope so

3/1/2008
5:28:49 PM
Ewdison Then
and i’ll setup and secure your server then move it in, and set up your backup system and show you how to back it up πŸ™‚

3/1/2008
5:29:01 PM
Judie *naf
YES πŸ˜€

3/1/2008
5:29:12 PM
Ewdison Then
alrighty judie, I’ll talk to you in a bit, call me if something urgent

Of course, nothing urgent came up because the site was still just as dead by the time Ewdi got home…

3/1/2008
9:18:32 PM
Ewdison Then
Any news?

3/1/2008
9:18:46 PM
Judie *naf
nothing – look at this, though:

http://forum.hostican.com/virtual-private-servers-vps/838-vps-problems-seems.html

can we start moving and rebuilding – and if it really comes back then we can add it, but otherwise – something catastrophic happened and they are lying

3/1/2008
9:20:53 PM
Ewdison Then
yeah, i’m checking the new server info, making sure everything is right

3/1/2008
9:26:07 PM
Ewdison Then

Hostican-Denis (lnxcode) 6:33: Their syncing the RAID pairs now for RAID 10 making progress… there isnt an ETA but for sure will be 100% operational in the AM but no eta… I hope it will be in a couple of hours

so their HDD did go down…i knew the harddrive was crashing!

3/1/2008
9:26:18 PM
Judie *naf
YEP. ok – so if the hard drive crashed, and if it is really back up sometime tomorrow – will my data still be there? or is there no way to know?"

3/1/2008
9:27:44 PM
Ewdison Then
if they are telling the truth, yes you can get it back, but if they are stalling for time, then probably no πŸ™

i don’t trust them after today’s email exchange

3/1/2008
9:27:59 PM
Judie *naf
me neither πŸ™

3/1/2008
9:28:18 PM
Ewdison Then
they don’t own their own data center; they are reseller, sad sad

3/1/2008
9:28:30 PM
Judie *naf
so they LIED TO ME??!!!!

3/1/2008
9:28:36 PM
Ewdison Then
i just did research on them

3/1/2008
9:28:38 PM
Judie *naf
you saw that email!!!

3/1/2008
9:28:41 PM
Ewdison Then
yap! there are only 45 datacenter in US. if they own one, i would have known

3/1/2008
9:29:27 PM
Judie *naf
i could cry

3/1/2008
9:29:39 PM
Ewdison Then
HostICan Datacenter Locations: * Equinix, Ashburn, Virginia

http://www.customerhostingreviews.com/hostican

they are reseller for Equinix, they don’t have control over the hardware

3/1/2008
9:29:59 PM
Judie *naf
for goodness sake. why did they lie???

3/1/2008
9:30:04 PM
Ewdison Then
so harddrive went down, they have to open ticket to equinix, they don’t have physical access to the machines, unlike softlayer or the planet

3/1/2008
9:30:36 PM
Judie *naf
this makes me sick. i better get a refund, and i want my damn site back!!!!!!!!!!!!!!!!!!!!

3/1/2008
9:34:54 PM
Ewdison Then
yeah, damn liars!

3/1/2008
9:35:16 PM
Judie *naf
i should have asked if he had physical access to the servers – that is the only question i didn’t ask

…more waiting…

3/1/2008
10:37:46 PM
Ewdison Then
so nothing from hostican?

3/1/2008
10:37:57 PM
Judie *naf
NOTHING. they are hiding

3/1/2008
10:38:50 PM
Ewdison Then
arghhh, i’ll stay up late tonight until we finish things up

3/1/2008
10:39:10 PM
Judie *naf
god ewdi, i am so sorry πŸ™

3/1/2008
10:39:42 PM
Ewdison Then
no worries, i just dont want geardiary to go down too long because the risk of geting 404, dead links is higher

3/1/2008
10:39:49 PM
Judie *naf
i know

3/1/2008
10:39:55 PM
Ewdison Then
harder to fix the sitemap priority after that

3/1/2008
10:39:59 PM
Judie *naf
right

3/1/2008
10:48:28 PM
Ewdison Then
np, well you can go to sleep tonight if you want, we wont be able to move tonight since we cant retrieve the database

3/1/2008
10:48:44 PM
Judie *naf
i am not going to bed til you stop for the night; not fair

3/1/2008
10:49:18 PM
Ewdison Then
nah, don’t worry about me, i need you to wake up fresh tomorrow so i can teach you the backup technique. :p

3/1/2008
10:49:59 PM
Judie *naf
haha – what time tomorrow?

3/1/2008
10:50:24 PM
Ewdison Then
probably around 11am? or 10? so i can get things prepared

3/1/2008
10:50:58 PM
Judie *naf
ok – 10. i will be here with bells on πŸ™‚ well, you know what i mean. and MAYBE the database will be back

3/1/2008
10:51:14 PM
Ewdison Then
hahah okay. i really do hope so

3/1/2008
10:51:21 PM
Judie *naf
doubt it tho

3/1/2008
10:51:35 PM
Ewdison Then
so backup plan is to use december db restore right?

3/1/2008
10:51:45 PM
Judie *naf
god, if we have to. i could just cry

3/1/2008
10:51:57 PM
Ewdison Then
okay, let’s hope for the best

3/1/2008
10:52:14 PM
Judie *naf
ok

Of course, I never went to sleep that night, and the next day things only got worse. All morning long, Ewdi and I transferred my .ftp backup I had over to the new Host, while hoping against hope that HostICan would come through and produce my database. Things were not looking good, however.

3/2/2008
3:14:57 PM
Judie *naf
i am still having huge trouble with that december database backup; it appears to NOT be zipped. <sigh>

3/2/2008
3:26:34 PM
Ewdison Then
you got to blog about this hosting company when GD is up, it annoys the hell out of me that they keep changing their story, now they said it did not crash, http://forum.hostican.com/virtual-private-servers-vps/838-vps-problems-seems-5.html

3/2/2008
3:27:06 PM
Judie *naf
looking…good lord…they are lying liers

3/2/2008
3:28:48 PM
Ewdison Then
yeah the worst thing is they are posting it on their own forum πŸ˜†

3/2/2008
3:28:53 PM
Judie *naf
yep

3/2/2008
3:33:53 PM
Judie *naf
i’m trying to send the dec 15 backup file again

3/2/2008
4:17:38 PM
Judie *naf
http://forum.hostican.com/virtual-private-servers-vps/838-vps-problems-seems-5.html#post6031

they are still buying time

3/2/2008
5:39:16 PM
Ewdison Then
same response πŸ™ yet nothing

3/2/2008
5:39:28 PM
Judie *naf
yep

3/2/2008
6:10:51 PM
Judie *naf
ok – it is done uploading – remember – this is the feb 2 ftp

3/2/2008
6:11:57 PM
Ewdison Then
judie, just received your email πŸ™
the backup data is not a valid one πŸ™

3/2/2008
6:12:06 PM
Judie *naf
oh man

3/2/2008
6:12:15 PM
Ewdison Then
http://www.ilfilosofo.com/forum/topic/51

3/2/2008
6:12:19 PM
Judie *naf
looking

3/2/2008
6:12:37 PM
Judie *naf
oh my god

3/2/2008
6:12:39 PM
Ewdison Then
the backup plugins you use output the wrong encoding πŸ™

FYI – The backup plugin I was using was the aptly named WordPress Database Backup.

3/2/2008
6:12:51 PM
Ewdison Then
the plugins don’t work with 2.2 (the version installed on geardiary)

3/2/2008
6:12:52 PM
Judie *naf
omg omg

ok

then we have to wait and hope that they do come back with the database, because otherwise my last backup with a file attachment is last MAY, i think

3/2/2008
6:15:34 PM
Ewdison Then
oh, the content type was messed up, it doesn’t work well with 2.2 πŸ™

3/2/2008
6:15:36 PM
Judie *naf
i am going to throw up

3/2/2008
6:16:10 PM
Ewdison Then
oh boy, now our only hope is waiting from them :(, we’ve GOT to call them and get the backup in

3/2/2008
6:16:18 PM
Judie *naf
please call them, you know what to say if they answer the phone

3/2/2008
6:16:35 PM
Ewdison Then
okay, calling now

3/2/2008
6:16:41 PM
Judie *naf
thank you ewdi

3/2/2008
6:19:05 PM
Ewdison Then
voicemail again! we are experiencing a high call volume due to our new promotion, leave a message and an associate will call you back, BULL!

3/2/2008
6:19:11 PM
Judie *naf
i know

3/2/2008
6:19:13 PM
Ewdison Then
oh this is outright wrong :*(

3/2/2008
6:19:21 PM
Judie *naf
you get the same thing on the sales line

3/2/2008
6:19:25 PM
Judie *naf
i tried it earlier

3/2/2008
6:19:38 PM
Judie *naf
i just sent you the last email i had with an actual database attachment – it was from freaking MAY

3/2/2008
6:19:59 PM
Ewdison Then
yap, that was the real database

3/2/2008
6:20:13 PM
Judie *naf
wait!!!!!!!!!!!!!!!!!!!!!!

i have database backups

omg

3/2/2008
6:20:26 PM
Ewdison Then
YAY!!!!

3/2/2008
6:20:27 PM
Judie *naf
they are in the ftp!!!!!!!!!!!!!!!!!!

3/2/2008
6:20:32 PM
Ewdison Then
oh thank GOD!

3/2/2008
6:20:35 PM
Judie *naf
ok
listen to me
i am going to upload what i have to the root
please tell me these are right!~~
one sec

3/2/2008
6:21:26 PM
Ewdison Then
what is the filename?

3/2/2008
6:21:33 PM
Judie *naf
_wrdp1_wp_20070618_316.sql.gz

3/2/2008
6:21:44 PM
Ewdison Then
yap, that’s june backup

3/2/2008
6:21:49 PM
Judie *naf
oh. crap. ok – that is better than may, but mannnn. let me keep looking, this going to suck πŸ™

3/2/2008
6:25:43 PM
Ewdison Then
http://forum.hostican.com/6037-post56.html

not good news, backup might be corrupted as well according to him

3/2/2008
6:26:01 PM
Judie *naf
reading

3/2/2008
6:26:13 PM
Ewdison Then

At the moment, we have more than 50% of all the VPS accounts restored in our backup, but we are reviewing the data on the backup, as it is possible some of the data has gotten corrupted.

3/2/2008
6:26:21 PM
Judie *naf
oh god. ok – let me see

3/2/2008
6:26:26 PM
Ewdison Then
this is why i suggest avoiding automated backup

3/2/2008
6:26:32 PM
Judie *naf
i see. i am going to throw up

3/2/2008
6:29:48 PM
Ewdison Then
so what are we going to do now? we need to get in touch with one of them and get a backup, let me find the guy’s aim

3/2/2008
6:29:55 PM
Judie *naf
please

3/2/2008
6:30:15 PM
Ewdison Then
can you also add him, aim lnxcode

3/2/2008
6:30:38 PM
Judie *naf
i don’t have aim, let me install

3/2/2008
6:30:44 PM
Ewdison Then
okay. message him directly and see if getting the database backup is possible

3/2/2008
6:31:11 PM
Judie *naf
ok

3/2/2008
6:31:42 PM
Ewdison Then
tell him the database name is **_wrdp1_wp

3/2/2008
6:32:06 PM
Judie *naf
ok – one sec

installing still

ok – getting account set up

one sec

ok — i am in

did it

waiting

got him

!!

3/2/2008
6:41:01 PM
Judie *naf

geardiary (6:39:56 PM): can you please check and see if **_wrdp1_wp is available?
lnxcode (6:40:23 PM): Hey
geardiary (6:40:38 PM): hi
lnxcode (6:40:56 PM): Sure, I’ll have them check it out. I can’t right this second as I don’t have access to it (the techs do)
geardiary (6:41:20 PM): ok – please. i am having a total heart attack here
lnxcode (6:41:39 PM): Yep. nps – I had one too when I heard.
lnxcode (6:41:53 PM): Let me go see (will take a bit of time like 30 – 45 mins) geardiary (6:42:10 PM): please, is there any way it can be any faster?
lnxcode (6:42:45 PM): I can try… let me push on em see where it gets me. lnxcode (6:42:48 PM): what was ur vps ID?
geardiary (6:42:53 PM): **
lnxcode (6:43:10 PM): ok
lnxcode (6:43:12 PM): perfect.
lnxcode (6:43:13 PM): thanks
geardiary (6:43:35 PM): ok – when will you know?
lnxcode (6:44:19 PM): I should know soon (I need some time – as I’ve gotta get Chris to look it all up)
geardiary (6:44:37 PM): ok. i will be here. please let me know as soon as possible
lnxcode (6:46:08 PM): be glad to
lnxcode (6:46:15 PM): I take it, you have no backup?
geardiary (6:46:34 PM): no – mine is corrupt from a wp 2.2 error
geardiary (6:46:45 PM): as i have only now learned
lnxcode (6:47:01 PM): oh you upgraded or something and u got an error? geardiary (6:47:16 PM): it was sending me backups in an incorrect format geardiary (6:47:22 PM): due to a 2.2 issue
geardiary (6:47:32 PM): i had no idea until my tech guy tried to use what i had geardiary (6:47:39 PM): and he let me know
lnxcode (6:48:17 PM): oh
geardiary (6:48:35 PM): i am totally screwed if you don’t have the backups
lnxcode (6:48:58 PM): yea, I know…
lnxcode (6:49:13 PM): I mean we must have it… just … if its corrupted or not lnxcode (6:49:15 PM): is the question
lnxcode (6:49:28 PM): this whole week – weekend has been ….
geardiary (6:49:38 PM): i can only imagine what you are going through geardiary (6:49:48 PM): i have advertisers jumping down my throat, too

Note – this wasn’t true, but I was trying to make him understand that it was urgent, and he wasn’t the only one with other people to answer to.

lnxcode (6:50:18 PM): yea, I know how that goes. I feel horrid to be honest, that this happened… but I didnt know it was going to …
geardiary (6:50:54 PM): please just find out. the sooner i know if i have to rebuild from scratch or if i have something to work with, the better it will be for me
geardiary (6:50:57 PM): either way
lnxcode (6:51:02 PM): yep
lnxcode (6:51:04 PM): will update you soon
geardiary (6:51:08 PM): i just need to know what i am looking at
lnxcode (6:51:09 PM): ping me back in 20 – 45 mins
geardiary (6:51:12 PM): ok
lnxcode (6:51:14 PM): thanks.

3/2/2008
6:52:15 PM
Ewdison Then
okay, so 8pm central time

3/2/2008
6:52:18 PM
Judie *naf
i have been reduced to begging

3/2/2008
6:52:22 PM
Ewdison Then
we’ll ping him back

3/2/2008
6:52:24 PM
Judie *naf
yes

3/2/2008
7:05:56 PM
Ewdison Then
me < pray real hard for the backup :p

3/2/2008
7:06:04 PM
Judie *naf
the stress is killing me. i want to cry

3/2/2008
7:07:01 PM
Ewdison Then
πŸ™ oh while waiting for him, want me to guide you on how to backup?

And so Ewdi and I killed time for a while longer, as he taught me how to do a scripted backup…

3/2/2008
7:46:39 PM
Judie *naf
it’s been an hour – let me ping that guy

geardiary (7:46:47 PM): hi, any word?
lnxcode (7:51:23 PM): Hey
lnxcode (7:51:32 PM): just checking with them now (awaiting an update myself)
geardiary (7:51:54 PM): ok
lnxcode (7:52:26 PM): they have some of your database
lnxcode (7:52:31 PM): but one file is actually missing
lnxcode (7:52:37 PM): for the db
geardiary (7:52:39 PM): which?
lnxcode (7:52:46 PM): the user table for you db
lnxcode (7:52:54 PM): seems its curropted…

3/2/2008
7:53:09 PM
Judie *naf [to Ewdi]
what does that mean???

lnxcode (7:53:02 PM): which is unrestoreable basically
geardiary (7:53:18 PM): one sec
lnxcode (7:53:27 PM): I’m asking them about more info
lnxcode (7:53:30 PM): to see if they have info…

3/2/2008
7:53:36 PM
Ewdison Then
CRAP!

3/2/2008
7:53:39 PM
Judie *naf
yeah

3/2/2008
7:53:41 PM
Ewdison Then
well just get the post data, i can restore some of the user data myself

3/2/2008
7:53:50 PM
Judie *naf
ok – so posts are safe? i just lose gd users??

3/2/2008
7:54:04 PM
Ewdison Then
yeah seems like it, it will take some work on my end to fix it

3/2/2008
7:54:14 PM
Judie *naf
oh god, i am so sorry

3/2/2008
7:54:27 PM
Ewdison Then
no dont worry, i don’t mind πŸ™‚

3/2/2008
7:54:29 PM
Judie *naf

geardiary (7:53:18 PM): one sec
lnxcode (7:53:27 PM): I’m asking them about more info
lnxcode (7:53:30 PM): to see if they have info…
geardiary (7:54:22 PM): ok – when can i get this from you?
lnxcode (7:54:34 PM): their exaimining it…
lnxcode (7:54:37 PM): seeing what else is missing

3/2/2008
7:58:29 PM
Judie *naf
so, if it’s just my user database – all 1500 names that is corrupt

3/2/2008
7:58:31 PM
Ewdison Then
leaving other files untouched…

3/2/2008
7:58:39 PM
Judie *naf
maybe we can back at least part of that up, right?
right?

3/2/2008
7:58:44 PM
Ewdison Then
yap πŸ™‚

3/2/2008
7:58:57 PM
Judie *naf
ok. could be worse. please let it just be that

3/2/2008
7:59:14 PM
Ewdison Then
yap yap, i really do hope so

3/2/2008
8:20:08 PM
Ewdison Then
alrighty,! so are we getting the database anytime soon? lol

3/2/2008
8:20:17 PM
Judie *naf
god, let me ask

lnxcode (8:20:38 PM): their going over your database
lnxcode (8:20:41 PM): to see if they can export it
geardiary (8:20:43 PM): ok
lnxcode (8:20:45 PM): cause right now its in this format
lnxcode (8:20:49 PM): .MYO
lnxcode (8:20:51 PM): .MYD
lnxcode (8:20:52 PM): etc…
lnxcode (8:20:55 PM): MySQL files…

3/2/2008
8:21:55 PM
Ewdison Then
thats fine! we’ll take it

3/2/2008
8:22:16 PM
Judie *naf
like that??

i’ll tell him

3/2/2008
8:22:21 PM
Ewdison Then
yap, any form and shape. as long as its mysql db, i cant rebuilt it into usable format. i was doing that when they shut down the server. darn them!

3/2/2008
8:22:44 PM
Judie *naf

lnxcode (8:20:52 PM): etc…
lnxcode (8:20:55 PM): MySQL files…
geardiary (8:22:36 PM): can we have it as is – can i collect it just like that?

3/2/2008
8:23:12 PM
Ewdison Then
tell him no need mysql dump file, frm file and etc is fine too

3/2/2008
8:23:16 PM
Judie *naf
waiting for an answer

lnxcode (8:23:16 PM): Basically those files, if their fragmented or messed up in anyway, it will make the db un restoreable… and make it useless basically… we’re working on it for you right this second.
lnxcode (8:23:32 PM): if we can get like 80 – 90% of them intacted… it will be fine.
lnxcode (8:23:39 PM): if not… then its going to suck

3/2/2008
8:23:53 PM
Ewdison Then
he lied!

3/2/2008
8:24:09 PM
Judie *naf
are you on aim? i am going to bring you in

3/2/2008
8:24:19 PM
Ewdison Then
it doesnt make sense, frm file can be fixed easily

3/2/2008
8:24:19 PM
Judie *naf
can i do that?

3/2/2008
8:24:28 PM
Ewdison Then
yeah like conf chat

3/2/2008
8:34:55 PM
Ewdison Then
the lil bastard’s probably hiding right now

3/2/2008
8:35:08 PM
Judie *naf

geardiary (8:32:25 PM): i am trying to add you to a chat – did you get the invitation?
lnxcode (8:34:37 PM): humm
lnxcode (8:34:38 PM): nope
geardiary (8:34:49 PM): ok – one sec – i will try again
lnxcode (8:34:51 PM): but I really dont need to chat with him
geardiary (8:34:59 PM): do you have my db?
lnxcode (8:35:10 PM): its really dependant on if this db restore is sucessful or not

3/2/2008
8:36:36 PM
Judie *naf
i am going to tell him to talk to you. ok?

3/2/2008
8:36:45 PM
Ewdison Then
sure

3/2/2008
8:37:11 PM
Judie *naf

geardiary (8:36:59 PM): ewdi is going to talk to you. he handles my backend stuff
geardiary (8:37:03 PM): please reply to him
lnxcode (8:37:41 PM): ah ok
lnxcode (8:37:42 PM): cool
lnxcode (8:37:43 PM): will do
lnxcode (8:37:51 PM): give me a few mins…. I need to check all this stuff out

3/2/2008
8:38:06 PM
Ewdison Then

Ewdi Then 8:37 Hi, this is ewdi from fedoraforum.org (http://fedoraforum.org/), i’m trying to help judie to put her site back 8:37 What was the problem with her db?

[to me] lol, i told him that I’m one of the developer of the OS he’s using πŸ™‚

3/2/2008
8:38:26 PM
Judie *naf
YEAH!

3/2/2008
8:38:32 PM
Ewdison Then
so he cant fool me :p

3/2/2008
8:38:38 PM
Judie *naf
that’s why he is asking for a minute

3/2/2008
8:38:44 PM
Ewdison Then
i spent 2 years working on redhat’s fedora linux :p

3/2/2008
8:38:56 PM
Judie *naf
πŸ˜€

3/2/2008
8:39:07 PM
Ewdison Then
so you cant tell me that i don’t know about the stuff i coded myself lol

3/2/2008
8:39:11 PM
Judie *naf
no kidding. let me know when he replies – i have nothing

3/2/2008
8:39:55 PM
Ewdison Then
nope, no response :p

3/2/2008
8:40:06 PM
Judie *naf
i guess he is still using those few minutes

3/2/2008
8:49:08 PM
Judie *naf
anything?

3/2/2008
8:49:16 PM
Ewdison Then
NOPE πŸ™

3/2/2008
8:49:37 PM
Judie *naf
ping him again. he asked for a few minutes – we gave it. please. i think we have to stay on him

3/2/2008
8:51:17 PM
Ewdison Then
just did

3/2/2008
8:51:29 PM
Judie *naf
ok

3/2/2008
8:52:01 PM
Ewdison Then

Ewdi Then 8:51 hello?
8:51 I’m also one of the fedora core developers, so if you need assistance, i might be able to offer some so we can get her db out

3/2/2008
8:52:20 PM
Judie *naf
he better accept your help!!!

3/2/2008
8:53:13 PM
Ewdison Then
still no response πŸ™
nope, the guy chickened out

3/2/2008
8:55:51 PM
Judie *naf
he better come back

3/2/2008
9:01:11 PM
Ewdison Then
he just contacted me

3/2/2008
9:01:15 PM
Judie *naf
ok. and?

3/2/2008
9:01:26 PM
Ewdison Then
waiting for his response, he just say hi

3/2/2008
9:01:29 PM
Judie *naf
ok

3/2/2008
9:03:49 PM
Ewdison Then

lnxcode 9:01 at the moment no, its with the data recovery folks we’re trying to patch together your database (but at the moment its a little curropted)
9:01 so we’re trying to see what can be done.

i offer my help, he said no

3/2/2008
9:03:59 PM
Judie *naf
HUH

3/2/2008
9:04:05 PM
Ewdison Then
so hopefully we can get it back

3/2/2008
9:04:09 PM
Judie *naf
god i hope so

3/2/2008
9:04:30 PM
Judie *naf
i have to run sarah to mcdonalds. πŸ˜› i will be back in 20 minutes

3/2/2008
9:04:47 PM
Ewdison Then
okay, me too, i got to grab a sandwich

3/2/2008
9:06:31 PM
Ewdison Then

Ewdi Then 9:02 okay, hope it went well
9:04 oh btw how do you patch together the db? are you rebuilding it thru mysql repair or something else?
lnxcode 9:05 yea, well you can export the files 9:05 in .MYO and MYD format
9:05 thats all we’ve got… so we’re trying to put it together… but it isn’t workign out so well at the moment
Ewdi Then 9:05 well do you think you can just tar MYO and MYD and FRM files for us?
lnxcode 9:06 Im trying there are some that are curropted

so judie, what do we do?

3/2/2008
9:40:53 PM
Judie *naf
ask him to lay it on the line. do they have my databse or not. if they have it we should wait. if not, i start from scratch basically. i just need to know. no bullshit

3/2/2008
9:41:23 PM
Ewdison Then
can you ask him, he’s offline on my list

3/2/2008
9:41:28 PM
Judie *naf
ok. just pinged him…nothing yet. no response. what do you think? do you think there is even a database left?

3/2/2008
9:45:43 PM
Ewdison Then
I don’t know Judie, its your data, i don’t want to make the call :p if we work on it now and suddenly they say, we got your data, it kinda sucks, but if we don’t, weekday is starting soon and geardiary being down also sucks πŸ™ i don’t know, they say this and that, different words each time

3/2/2008
9:46:06 PM
Judie *naf

Default I have an update for you guys… After pressing a support rep for specifics about data recovery, I was told: Quote: I would suggest you start restoring that while, we continue to see what backups, files, etc… we have for you. After pushing for a further update on the time frame, the rep finally admitted. Quote: We’ve sent it to the labs to try to pull as much data out as possible. However, I would say somewhere in the region of say 72 hours? Looks like this is even worse that we have been led to believe.

this is on the BB

3/2/2008
9:47:05 PM
Ewdison Then
what labs? forensic?

3/2/2008
9:47:24 PM
Judie *naf
i have no idea:
http://forum.hostican.com/virtual-private-servers-vps/838-vps-problems-seems-6.html#post6037

3/2/2008
9:47:26 PM
Ewdison Then
i dont believe they are spending money for forensic recovery, because it cost 250/100MB. if the hdd is 10GB that is 25000 recovery fees

3/2/2008
9:47:53 PM
Judie *naf
that ain’t happening

3/2/2008
9:58:22 PM
Judie *naf

geardiary (9:41:42 PM): you still there?
lnxcode (9:56:23 PM): Hey
lnxcode (9:56:24 PM): yes…
lnxcode (9:56:40 PM): at the moment its messed up pretty bad…
geardiary (9:56:45 PM): ok – i need you to lay it to me straight. some guy went on the bb and he is saying 72 hours
lnxcode (9:56:47 PM): their trying a more intensive process.
lnxcode (9:57:16 PM): I’m going to re-create your VPS right now
geardiary (9:57:22 PM): ok
lnxcode (9:57:22 PM): so you can at least upload what you ahve
lnxcode (9:57:25 PM): have**
geardiary (9:57:33 PM): no – i need what you have
lnxcode (9:57:45 PM): we don’t have much at the moment.
lnxcode (9:57:50 PM): we’re still fighting with it
geardiary (9:57:56 PM): ETA?
lnxcode (9:58:09 PM): approx 72 hours – is what im hearing at the moment
lnxcode (9:58:19 PM): im trying more info faster
lnxcode (9:58:26 PM): but… so far …
geardiary (10:01:41 PM): within 72 hours, can you GUARENTEE i will have my data back?
lnxcode (10:02:01 PM): I can’t gurantee anything to be honest, we’re going to try our best
geardiary (10:09:57 PM): you are asking me to accept being down for 5 days
geardiary (10:10:06 PM): with no guarantees
lnxcode (10:11:29 PM): Im not asking you to accept anything. I’m simply saying that if the data is curropt, then its unrestoreable. I’m sorry to say. I can re-create your vps, and would really like to, so you can upload anything you have, surely you must have some sort of backup even if its a week or a month old?
geardiary
(10:12:07 PM): my last database backup – that is not corrupt due to wp2.2 is from JUNE 2007
lnxcode (10:12:47 PM): thats certainly better then nothing
lnxcode (10:13:10 PM): I know thats not what you want to hear… but until I get more data, thats all I can say
lnxcode (10:14:41 PM): that its good to know that at least you have a backup from then.
geardiary (10:14:56 PM): i expect a full refund. i just paid you all almost $800 to up my server. this is crap, you know it is crap, and you have completely lost my confidence
lnxcode (10:16:04 PM): Im sorry. I will be more then happy to ask billing to do this in the AM.
lnxcode (10:16:28 PM): They will contact you upon processing your refund.
lnxcode (10:16:40 PM): just to let you know its being refunded.
geardiary (10:16:49 PM): thank you
lnxcode (10:17:07 PM): Sorry, I couldn’t help.
lnxcode (10:17:11 PM): Have a great evening.
geardiary (10:17:18 PM): whatever

I am not proud of the way I ended that chat. But I was just so exhausted, frustrated, and heartsick, I couldn’t help myself.

3/2/2008
10:28:17 PM
Ewdison Then
OMG, http://www.geardiary.com/index.php

we lost a lot of URLs πŸ™ i got to do 404 drop so you wont get penalized

3/2/2008
10:28:34 PM
Judie *naf
what do you mean?

3/2/2008
10:28:56 PM
Ewdison Then
if there are a lot of 404 errors, google will penalized your ranking

3/2/2008
10:29:01 PM
Judie *naf
ok. i understand

3/2/2008
10:29:11 PM
Ewdison Then
so what we need to do is tell them to drop 404 rather than to retry and poll in

3/2/2008
10:29:17 PM
Judie *naf
ok

3/2/2008
10:30:32 PM
Ewdison Then
I’m really sorry we couldn’t get the db πŸ™

3/2/2008
10:30:37 PM
Judie *naf
me too. i don’t think we were ever going to get it; i think he knew it, too

3/2/2008
10:31:35 PM
Ewdison Then
he just wasted our time πŸ™

3/2/2008
10:31:42 PM
Judie *naf
yes, he did. i just can’t believe that JUNE was the most recent backup that was any good. i am just devastated

3/2/2008
10:51:34 PM
Ewdison Then
i’m sorry πŸ™

3/2/2008
10:51:48 PM
Judie *naf
not your fault. and i need to just swallow and move on, just crushed tho

but the site will be better, and faster, and stronger

3/2/2008
10:52:55 PM
Ewdison Then
yap

the backup did not tell me which version it is, tit shows version 5581 lol

i’m testing it 1 by 1

3/2/2008
11:22:55 PM
Judie *naf
ok

3/2/2008
11:23:42 PM
Ewdison Then
the files are 2.2, but the db is unknown πŸ™

3/2/2008
11:23:55 PM
Judie *naf
are you kidding me?

3/2/2008
11:24:05 PM
Ewdison Then
i’m not. trying to figure out now

3/2/2008
11:24:10 PM
Judie *naf
what does that mean?

3/2/2008
11:24:17 PM
Ewdison Then
i’m uploading each version and see if it match

3/2/2008
11:24:28 PM
Judie *naf
ok

3/2/2008
11:32:40 PM
Ewdison Then
i’m using this, in hope i can find version number

3/2/2008
11:32:40 PM
Ewdison Then
http://web.archive.org/web/20070624224457/http://www.geardiary.com/

3/2/2008
11:32:48 PM
Judie *naf
looking

3/2/2008
11:32:53 PM
Ewdison Then
i tried from 2.0 to 2.2, nothing matched πŸ™

3/2/2008
11:32:57 PM
Judie *naf
wow

3/2/2008
11:33:06 PM
Ewdison Then
getting same db error when upgrading

3/2/2008
11:33:17 PM
Judie *naf
ok

3/2/2008
11:34:42 PM
Ewdison Then
WordPress 2.2! found it

3/2/2008
11:34:56 PM
Judie *naf
ok πŸ˜€

3/2/2008
11:50:13 PM
Ewdison Then
i’m having problem updating it

3/2/2008
11:50:13 PM
Ewdison Then
still on 2.2.3

3/2/2008
11:50:14 PM
Ewdison Then
http://www.geardiary.com/index.php

What came up at this point was a very barren, very messed up version of our home page. I had to laugh, though – our counter at the bottom was still intact.

3/2/2008
11:50:21 PM
Judie *naf
ha! at least our counts are still good πŸ™‚ little things make me smile now

3/2/2008
11:51:25 PM
Ewdison Then
yeah :p

3/2/2008
11:51:29 PM
Judie *naf
thank you for all your help

3/2/2008
11:51:50 PM
Ewdison Then
judie, i have to wake up at 5 tomorrow to send my sister to school, (26 miles away) so i’ll check on things again tomorrow

3/2/2008
11:52:02 PM
Judie *naf
ok

3/2/2008
11:52:07 PM
Ewdison Then
you want to open the site now?

3/2/2008
11:52:14 PM
Judie *naf
yes, we have to

3/2/2008
11:53:29 PM
Ewdison Then
alrighty, i’m going to bed now, just shoot me an email if something comes up.

3/2/2008
11:53:41 PM
Judie *naf
i will. thank you

3/2/2008
11:53:47 PM
Ewdison Then
goodnight judie

3/2/2008
11:53:57 PM
Judie *naf
thank you ewdi, GOOD NIGHT

3/2/2008
11:54:32 PM
Ewdison Then
you welcome, night πŸ™‚

And with that I spent the rest of the night fixing the broken template and correcting all the errors I could manage. The next morning, I was still going.

3/3/2008
7:26:59 AM
Ewdison Then
Morning Judie, I’ll take a look at geardiary later on today, i have to do coverage on cebit

3/3/2008
7:27:11 AM
Judie *naf
ok πŸ™‚ thanks ewdi

3/3/2008
8:40:01 AM
Judie *naf
one more thing before i forget – all of the photopress pictures seem to be gone. i can’t figure out where they are stored, but they should have been in wp-content/photos – that is what they were mapped as – and nothing. if you can find them it would be awesome – but i suspect that they are gone. :-/

3/3/2008
8:40:36 AM
Ewdison Then
ouch could it use the db storage?

3/3/2008
8:40:52 AM
Judie *naf
hmm

3/3/2008
8:40:53 AM
Ewdison Then
it should be on wp-content/photos

3/3/2008
8:40:57 AM
Judie *naf
i don’t think so

3/3/2008
8:40:59 AM
Ewdison Then
does your backup have it there?

3/3/2008
8:41:07 AM
Judie *naf
no – that’s the freaky thing

3/3/2008
8:41:11 AM
Judie *naf
it shows as being there, and it is NOT. weird

3/3/2008
8:41:32 AM
Ewdison Then
hmm got to check on that

3/3/2008
8:41:37 AM
Judie *naf
ok

3/3/2008
8:41:46 AM
Ewdison Then
http://www.geardiary.com/wp-content/photos/IMG_1885.JPG

3/3/2008
8:41:48 AM
Judie *naf
i’ll just deal with it if it is gone

3/3/2008
8:41:48 AM
Ewdison Then
nothing is in there

3/3/2008
8:41:59 AM
Judie *naf
exactly

3/3/2008
8:42:08 AM
Judie *naf
but the file is full of stuff, just not THAT. it’s only photopress stuff affected, best i can tell. weird

3/3/2008
9:22:46 AM
Ewdison Then
some files are not there. IMG_1885.JPG is not on the ftp. does your ftp backup have them in it? can you check for this file IMG_1885.JPG

3/3/2008
9:23:25 AM
Judie *naf
no – that’s why i am freaking out – where were they saved?

one sec

ok – this is where i am looking

public_htmlwp-contentphotos

and i do not see it anywhere! πŸ™

3/3/2008
9:31:56 AM
Ewdison Then
me neither πŸ™ incomplete ftp?

3/3/2008
9:32:05 AM
Judie *naf
i don’t think so. i think they went somewhere weird. i think it is a photopress issue. and i bet they are on my database. the lost database. SIGH. although i don’t see how

3/3/2008
9:32:57 AM
Ewdison Then
πŸ™ gosh, nothing goes right comes form this crash

3/3/2008
9:33:02 AM
Judie *naf
no. nothing. man – i don’t know how i will ever repay you

3/3/2008
9:52:33 AM
Ewdison Then
nah, dont think about that. what are friends for?

3/3/2008
9:53:01 AM
Judie *naf
still. you go above and beyond

3/3/2008
10:34:49 AM
Judie *naf
ewdi – do you know anything about this plugin?

http://www.ilfilosofo.com/blog/wp-db-backup/

3/3/2008
10:35:45 AM
Ewdison Then
the probem with these backups is it output in sql text with different encoding depending on the server

3/3/2008
10:35:57 AM
Judie *naf
ah, ok

3/3/2008
10:35:59 AM
Ewdison Then
you should back it up from cpanel or rsync

3/3/2008
10:36:10 AM
Judie *naf
ok

3/3/2008
10:36:20 AM
Judie *naf
i liked the automatic – figured an EXTRA layer of security

3/3/2008
10:36:37 AM
Ewdison Then
:p

3/3/2008
3:51:45 PM
Judie *naf
i am going to start using windows live writer so i have backups of my posts, just in case, from now on πŸ˜›

3/3/2008
3:52:29 PM
Ewdison Then
:p good idea

3/3/2008
3:52:33 PM
Judie *naf
hehe. i am learning. never again

3/3/2008
3:53:03 PM
Ewdison Then
so no hope from hostican huh? you got to make them pay!

3/3/2008
3:53:26 PM
Judie *naf
oh – you HAVE to read the latest on their BB!!!! and they said they would pay me back today…i have not raised a stink because the OFF chance they get my data back. i want it.

3/3/2008
3:56:53 PM
Ewdison Then
i’m reading their bb

3/3/2008
3:56:58 PM
Judie *naf
k

3/3/2008
3:56:58 PM
Ewdison Then
people are pissed πŸ˜†

3/3/2008
3:57:00 PM
Judie *naf
YES

3/3/2008
4:31:50 PM
Judie *naf
ewdi, we never figured out where the photopress photos went – the ones with the watermarks? i bet they went on the main database. the WP one

3/3/2008
4:32:13 PM
Ewdison Then
i think so, it’s not around, i checked them

3/3/2008
4:32:58 PM
Judie *naf
ok. just making sure πŸ™

3/3/2008
4:38:21 PM
Judie *naf
my guys have been posting all day. working on stuff at work. they make me want to cry. all volunteers, and you!!! they are so sweet

More site rebuilding, more waiting for word from my ex-host…

3/4/2008
10:33:10 AM
Judie *naf
http://forum.hostican.com/virtual-private-servers-vps/838-vps-problems-seems-12.html

i just sent an email and asked them what the status of my database is

3/4/2008
10:35:42 AM
Ewdison Then
hope they can do something about it

***

But of course, they didn’t. I have since sent daily emails to HostICan asking about the status of my database, and the last one they bothered to answer, dated March 3, read:

Judie,

Thank you for contacting HostICan Support.

You’re account has been issued a full refund in the amount of: $794.66 which you paid to the upgrade of the VPS-Extreme.

At this present time, we don’t have any more information, and we will update you as soon as we do.

—————————–

HostICan Newsletter:

Did you know you can learn more about us?

http://www.hosticannewsletter.com

Ticket Details

===================

Ticket ID: SEW-373329

Department: Support

Priority: Low

Status: Open

Obviously all that could be done at this point was to forget about the database, and to move on. Even though, I was still holding out hope that a miracle might occur.

How to do that, though? We were talking about the loss of eight months worth of content.

This morning I saw my HostICan contact on AIM, and I decided to try asking about the database one last time…

geardiary (9:11:22 AM): hi, do you have just a moment?

lnxcode went idle at 10:35:02 AM.

lnxcode returned at 10:59:10 AM.

lnxcode went idle at 12:03:11 PM.

lnxcode returned at 12:11:27 PM.

lnxcode went idle at 1:04:14 PM.

lnxcode returned at 1:10:32 PM.

geardiary (1:27:52 PM): you there?

lnxcode went idle at 1:28:55 PM.

No reply when he is obviously online is a bit insulting, but I guess I understand why now. Because in a bit of kismet,  I received this email right as I was about to publish:

Hello Judie,

Thank you for your reply.

It seems that your VPS ID ended up extremely corrupted, the only way we where able to tell this was your VPS, was from an email that we had found in your VPS’s /mail file.

I’m very sorry that it turned out this way, we did everything we could to restore your data but the corruption was so bad that there was only a few folders that where intacted (one being the mail folder, and the other being the user password file in the /etc/ folder).

Please let us know if you have any further questions.

Sincerely,

Alex Hardin

HostICan Representative

Knowledge Base: http://www.hosticananswers.com

Phone: 1-800-725-9170 (U.S. & Canada – Toll-free)

Phone: 1-804-412-0390 (International)

Ticket Details

===================

Ticket ID: SEW-373329

Department: Support

Priority: Low

Status: Closed

I guess nothing more needs to be said.

If you have read this far, then I know you are probably exhausted; I certainly am after reliving it. Part two will be posted soon, and in it I’ll tell you how the Team in conjunction with Google Cache saved Gear Diary’s past articles and reviews, and is continuing to.

The lessons learned here have been hard, but they have been learned. If my experience helps one person, then I will consider this peek behind the scenes at a few of the worst days of my life worth sharing. Don’t make the same mistake I did; trust no one, and back it up.

31 Responses to " Surviving the Nuclear Winter after an Atomic Server Crash, Part One "

  1. Wait a minute? We’re all volunteers?! I thought I was just waiting for my paycheck in the mail.

    Oh well. I guess I’ll keep on keeping on…

  2. Yeah, that was on page 60 of your contract, hon. πŸ˜‰

  3. […] replies. Nor have I even received the partial refund offered. Come to find out, other customers received full refunds in the midst of what was happening. Here I am almost as week later, and nothing. What is the […]

  4. BkBlue says:

    Judy,
    thanks for posting about our ordeals. I still have yet to receive my refund, I guess the battle continues on.
    http://forum.hostican.com/virtual-private-servers-vps/853-where-refunds-customers-who-suffered-vps-crash.html#post6187

    FYI, the mail you received from Alex towards the end “identified by file in /mail […] only passwd file recovered” is the EXACT same email sent to me. Which basically means they have NO FREAKIN CLUE what happened and have no problem lying through their teeth to every customer.

    ugh, my blood still boils over this one.

  5. fishgal says:

    Wow Judie, what a heartbreak! You are so fortunate to have such a good and knowledgable friend to give you a hand. πŸ˜€

    Lean on Me when data’s gone,
    and i’ll be your friend,
    i’ll help you get logged on,

    So it won’t be long,
    till data returns,
    and users find your blog.

    Just call on the Ewdi when you need friend
    they are there to support
    and repair the backend

  6. FYI, the mail you received from Alex towards the end β€œidentified by file in /mail […] only passwd file recovered” is the EXACT same email sent to me. Which basically means they have NO FREAKIN CLUE what happened and have no problem lying through their teeth to every customer.

    If HostICAN’T would have just been straight with us from the beginning – with none of this false hope and none of this “everything will be back 100%” – I would have been crushed, but moving on and regrouping would have been much easier to do and accept. Dragging it out and leading us on was akin to torture. Their lies and deception were unforgivable.

    Oh, and if HostICan pulls the entire 15 page BB thread, I am going to post it here in its entirety. That is not a threat; it is a promise.

    BkBlue, I don’t think my refund has actually posted yet, so I may not be in the clear. But I do have the email saying they refunded it at least, should I have to file a dispute.

    @fishgal – Ewdi is a prince among men, but then my entire team are extraordinary, too. I am honored to have such wonderful friends – even and especially Chris C., the smartypants. πŸ˜‰

  7. reidme says:

    Fascinating! Thanks for posting this. My stomach was in knots, so I can’t imagine how you were feeling.

    Oh, and it’s cool to know I earn as much as the Gear Diary staff just for commenting here. πŸ˜€

  8. Oh, and it’s cool to know I earn as much as the Gear Diary staff just for commenting here. πŸ˜€

    Well, they do get perks that I hope make it worth their while, but that is neither here nor there. The GD Team is essentially just a wonderful bunch of writers doing what they love to do, and they happen to be doing it with me. :mrgreen:

  9. Joel Mclaughlin says:

    Yep we get perks! πŸ˜€ Thanks for all of yours and Ewdi’s hard work.

  10. Doug Goldring says:

    Sorry, guys, I am pretty sure she was talking about me in that last entry, :mrgreen:

    Seriously, Judie is the best person to work with. I was speaking with her when all of this was happening, and it was gut wrenching for me. I couldn’t even read the whole thing, I have no idea how you got it all down, Judie. πŸ™‚

    And now look, GD is stronger than ever. πŸ™‚

    Doug

  11. And now look, GD is stronger than ever. πŸ™‚

    We are! The amazing thing is that in some ways it’s almost like the crash never occurred, because of the Team’s hard work getting so much back up so quickly.

    I felt like I had to get this article up so that there would be a record of what happened, how things went down, and why it happened. It was getting easy to be complacent over the last day or so because things seemed so “normal” on the site, but all it takes is hitting one dead link to realize that there are gaping holes that may or may not ever be refilled. 😐

  12. reidme says:

    Getting to “unbox” and try out cool new devices seems like a pretty sweet perk! A lot of the fun of getting a new toy is in those first days, and there’s no chance for buyer’s remorse with a review unit.

  13. Doug Goldring says:

    Speaking of which, I just got a new box from Logitech today. πŸ™‚ Watch for a review of the new Logitech One Universal Remote, which was unveiled at CES this year coming soon. πŸ™‚

    DOug

  14. Mitchell Oke says:

    Yeah, but there is returners remorse πŸ˜‰

  15. Peter says:

    Phew, read it all. I would have gone ballistic, and driven to their office if the phone didn’t answer the first time! πŸ‘Ώ

  16. BkBlue says:

    *sigh* – Hostican has now banned me from their support forum for posting the thread I linked above.

    What weasels – they just don’t learn, do they?

  17. reidme says:

    @ BkBlue

    Shazam had a little fit when you called them on their behavior didn’t he.

    HostICan
    LieICan
    CheatICan
    ProtectyourdataICan’t
    :roll:

  18. Speaking of the small print on page 60, that dimwit, Shazam, made a major typo on the Hostican bulletin board. He said, and I quote, “All Virtuozzo, WHM/cPanel systems have tools in place for you to backup your data. It’s our fault if you don’t use them.”

    I believe he meant to include the word “not” in that last sentence. Go
    hammer him!

  19. slocken says:

    Being both a computer programmer, sysadmin, database admin/programmer…yeah it’s hard to believe but my company has that must trust in me 😈 but this story tore at my heart. I CAN imagine how much pain Judie must have been going through. I only thank that the fact that everything I have built for the company I work for resides on OUR servers (8) and WE do REDUNDANT backups. Even with all that, there is the trust in operating systems, hardware and in my case, simple human error. I have been in similar, just not as daunting and critical situations, and believe me, its either the frustration of either relying on someone/something else or kicking yourself in the arse for not doing something you thought you were (or were not) doing.

    My trust in other hosts just as simple as what Microsoft offers leaves me little to be desired in becoming a full-fledged, world- renowned web site that posts cool information and titillating side-bits that peak our interests…aka, GD (Gear Diary).

    It’s been a roller-coaster ride of a story and am relinquishing/looking forward to? part two. Perseverance pays off and in the end, you learn something new…hopefully not to forget it tomorrow. Kudos to Judie and Ewdison.

  20. reidme says:

    I really can’t believe someone would run a web hosting business that left all data protection up to the customer without having prominent, can’t miss, in-your-face disclaimers telling you that. Hiding it in the small print is inexcusable!

  21. Wayne Schulz says:

    This whole ordeal (and I read most of the original BB postings) seems like a big CYA for the host.

    Sadly, what I’ve observed over the years is that many of the web hosts with poor customer service can easily close up shop and re-open under another name with nobody the wiser.

    And I agree wholeheartedly with Judie about how much it counts for these organizations to keep us posted.

    When I first started using Hosted MS Exchange I was with Mailstreet. They were a smallish hosting company specializing in personal attention. Like most of the things I buy, I found them online via a forum posting

    Then Mailstreet gets acquired.

    Suddenly their focus shifts to selling to bigger companies and offering all sorts of doo-dads to get you to buy in addition to Exchange.

    The support techs became increasingly more difficult to reach.

    Then they had a server crash.

    After two days with no update and no email — I switched hosts.

    I think in some instances, these guys keep quiet because they figure to publicize it looks worse (I secretly suspect that RIM does this with their myriad of Blackberry outages that you only read about when some web site notices).

  22. jessica says:

    OMG Judie, I had no IDEA it was so bad. I am very very glad that you switched hosts and had someone as sweet and helpful as Ewdi to help you out! In the meantime, let me know if you need me to cut someone for you. πŸ˜‰

  23. Jessica, I knew Gear Diary readers were loyal – but damn girl! You just took it to a whole new level! πŸ˜‰ Thank you for that. πŸ˜‰

    And Ewdi: he is truly a prince among men. I’ll never be able to repay him for his kindness and patience through the ordeal. πŸ™‚

  24. Chris Davies says:

    Everybody should have an Ewdi πŸ™‚ Did you know he’s a kick-ass cook, too?

  25. […] a horrifying tale of catastrophe from Gear Diary: Surviving the Nuclear Winter after an Atomic Server Crash, Part One About 9:30 I finally crawled out of the world’s most comfortable bed, had some cereal, played […]

  26. […] Have you ever had a computer die due to a bad disk or something else unidentified? All of us probably have or will. Hence the need for backing up your data or in Gear Diary’s case, our server. […]

  27. […] reporting, GearDiary readers deserve the straight scoop. Heck, we’ve even shared the trials and travails of a potentially devastating server crash. We all came out stronger on the other side of the crisis, but it was the big […]

  28. […] about backing up your data? I’ve learned not to think so. Need I remind anyone of the events of March 1? Yeah, I thought […]

  29. […] you all will recall, Ewdi was my knight in shining armor when Gear Diary suffered its own massive server crash on March 1, and the lessons learned that day were very hard, but they were learned well. For Gear […]

Leave a Reply