Update: OSFlash is up and running again but I'm looking for a new hosting provider as I don't trust LiquidWeb following how they handled this incident.
The OSFlash down time and a warning about LiquidWeb article by Aral Balkan, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-Noncommercial 2.0 UK: England License.
Update: Adam emailed me back following their investigation into the matter. Here's what they found:
This seems to be an isolated incident. That being said, I don't think an upgrade at this point is necessary. From the best I can tell, this was cause by an hung apache child process. I'm not able to narrow it down further than that unfortunately.
The big issue here, of course, is they way they handled this incident -- by shutting down the OSFlash server. Instead, they should have (a) restarted the server, (b) noted whether or not the load issue resurfaced to see whether it was an isolated incident or not, (c) investigated the cause of the issue, and only then should they have emailed me with information on the issue and, if necessary, asked me to upgrade.
OSFlash is currently down. OSFlash is hosted on a VPS plan at LiquidWeb and I got an email a few moments ago from someone called Severin at the LiquidWeb support department. Here's that email:
This ticket has been created to inform you that your server load has climbed high enough to begin affecting the rest of the server that you reside on. Our other customers were affected by this and so, as a result, your virtual enviorment [sic] has been disabled to return things to stable. We ask that you either take steps to prevent this from happening, or upgrade to a VPS2 or higher. Overloading the server is in violation of our ToS.
When possible, please respond to the ticket with confirmation that you've read and understand it. Please feel free to ask any questions you have.
Thanks!
--
Severin
Systems Administrator
OSFlash has been running happily on a LiquidWeb's VPS server since October 2005 and this is the first I have heard of it taxing the server. I could go on about how angry I got at how LiquidWeb chose to handle this situation -- i.e., by switching off my server without warning me first and allowing me to upgrade to a larger account -- but I'll instead publish the email I sent back to them as it pretty much sums up how I feel:
Before anything else, I state for the record that I am very disappointed with how you have handled this situation.
Turning off my web site without even warning me is completely unacceptable. For this alone, I will immediately begin searching for a new hosting provider as it is apparent that you have no regard whatsoever for your customers. As someone who runs a popular web site, I expect my site to run without interruptions. If I am going over the allocated load or bandwidth, etc., I expect you to warn me and to give me the option to upgrade *not* to turn off my web site.
Now, to the practical matter at hand: Please upgrade my account to the next tier in your structure and please inform me how much this will cost. I will pay for this for the next month as I research alternatives to your service where they do not behave as rudely and unprofessionally as you have done in this case.
In a related matter, I just tried calling your support number (the International one since I am in the UK) and I get disconnected after selecting Technical Support (1) and VPS support (3). This has happened twice. The recorded voice tells me "exiting the system... goodbye." I was calling through Skype.
Also, please understand that everyone on the Open Source Flash community will hear about how you have handled this once the site is back up as I explain to them why their service was suddenly cut. Also, I will blog about this on my popular Flash blog at http://aralbalkan.com.
Important: Please upgrade my account to VPS2 and get my web site running again ASAP.
Also, please let me know what on my web site was causing the server load increase.
I expect that you will deal with this promptly and keep me updated about the status of my web site.
Sincerely,
Aral Balkan
http://aralbalkan.com
http://osflash.org (now down thanks to LiquidWeb)
Some of my friends are very happy with RackSpace, so I am going to have a talk with them. I'd been meaning to move OSFlash over to a dedicated managed server but guess which company will not be getting my upgrade order for that move.
If you value not having your web server switched off without warning, I would strongly urge you to stay clear of LiquidWeb.
Update: Well, it's been over an hour since I replied to their email and there's still nothing from LiquidWeb and OSFlash is still down. Anger level: Orange.
Update: Almost three hours to the minute that LiquidWeb shut down the OSFlash server, the server is back up and running. I got email from Adam Elkins at LiquidWeb support that reads (emphasis mine):
Sorry for the delay. I've turned your server back on, and apologize for any inconvenience this has caused you.
It has been our policy to suspend accounts that cause a server or network to become unresponsive, or have any other negative effect on other customers. I do understand that prior notice would be greatly appreciated, and we often do our best to do this.
Again, our sincere apologies for the inconvenience.
Please let me know if there are any further questions or concerns.
I'm happy that OSFlash back up and running but, unfortunately, "we often do our best" isn't good enough when it comes to giving users warning before suspending their accounts. I still don't know what it was on the server that resulted in the high loads or what those high loads were.
If you have a host that you're extremely happy with and they do managed server hosting, I'd appreciate it if you refer them to me in the comments as I'm evaluating my options at the moment and looking to move OSFlash away from LiquidWeb as soon as possible before there is a chance that this incident is repeated.
The OSFlash down time and a warning about LiquidWeb article by Aral Balkan, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-Noncommercial 2.0 UK: England License.
Hi Aral,
About 1 month ago almost the same thing happened to me
with http://www.m6.net
it started with “we’re moving your VPS to a new server, expect some delays”
like that without any warning
and become quite a nightmare,
sure they did moved me to a new server,
but they totally forgot to copy the datas, the settings, etc.!
empty directories everywhere
I sent tons of emails (support, complaint, etc.) not one single answer
I call the support and each time I got the message “no one is able to answer your call”
it has been 29 days, and still no answers from them
what I learned:
- f*ck VPS plan with less-than-average admin
- dedicated server rules
Sure I got some backups, but really I hate them just for not answering to this kind of problem
saddly very few hosting companies know the meaning
of 99.9% uptime…
the funny/strange thing is with m6.net in problem like yours,
they do automatic upgrade to preserve the site accessibility,
too bad they don’t know how to copy files from one server to another…
note:
I’m planing to use a big part of that dedicated server
to host oss too, the server is not up yet, but if you
want a mirror for osflash.org I will gladly help
ThePlanet (theplanet.com) have been very good to me since I got a dedicated server from them. Even though my package is one of the cheapest ones, they give you plenty of warning before performing any maintenance that might affect your server, and they also check everything is running ok when they bring it back up.
I dont think my server has been down without me knowing about it, and not for more than 30 minutes.
Hi zwetan: Sorry to hear about your experience, man, and thanks for the offer. Having mirrors for OSFlash sounds like a very good idea.
Ash: Thanks for the referral. I’ve heard of them (I’m back to WebHostingTalk, researching hosting providers) and will look into them.
[...] Either you don’t have the Flash Player installed or SWFObject wasn’t able to detect it. Download the Flash Player. // 0)) { document.getElementById(’flashPlayerInfo’).innerHTML = “You have Flash player “+ version['major'] +”.”+ version['minor'] +”.”+ version['rev'] +” installed. Detected by SWFObject.”; } // ]]> « OSFlash down time and a warning about LiquidWeb [...]
Man what a pain in the arse! I host with Media Temple and love it. I have left once or twice to try and get cheaper deals, but always come back. Nearly all of my clients are hosted on Media Temple as well, and I have had ZERO issues. I’ve had a couple clients go over their bandwidth and Media Temple just automatically bumps them to the next tier for that month and send an email asking/recommending that they move to the next tier.
Good luck!
Yikes. BIT-101 has been on LiquidWeb since day 1 (5 years now). Never had a problem. Something to watch out for, should my traffic ever magically get that heavy.
Loved that name and shame!! Thought about Media Temple?
We definitively need mirrors.
The content of osflash is now unvaluable.
Please tell us what you’ll do next.
Hi Aral,
I am very sorry for your experience. It is not company policy to disable server environments in this manner and it was an unfortunate mistake that you were not first contacted. We are very sorry for this inconvenience and downtime. I would like to issue a complimentary SLA credit for this . Please email me when you get a chance so that I may access your account.
Thank you for bringing this to our attention. We constantly strive to improve our support practices.
Aral, Liquid Web is committed to being your hosting partner, please trust your business to us and we will provide you a VPS2 upgrade at no charge.
Travis Stoliker
Marketing Director
tstoliker (a t) liquidweb.com
http://www.liquidweb.com
I think its pretty naff that because you’ve now made a public post that you get a reply like this from Liquid Web. I doubt that if you were standard customer you would have got a response like this.
Tink,
I’m sorry that you feel that way but I understand your doubt. All I can do is assure you that Liquid Web is committed to our customers. Liquid Web has a very good reputation of reliability and customer satisfaction unfortunately sometimes errors are made. We are committed to our partners and we hope to prove this to Aral again.
I would be happy to provide you a free shared account so that you can test our support services personally. Just drop me an email if you’re interested.
Hi Travis
My comment wasn’t that Liquid Web isn’t commited to its customers. I have no experience with Liquid Web at all and for all I know they could have always provided the most fantatsic service in the past to their customers and this could be your first ever hiccup. Sometimes one bad story is enough though.
My comment was more to make the point that if this had happened to a standard customer of yours it would be extremely unlikley that the Marketing Director would end up getting in touch with them and offering a credit. It’s only because Aral has been able to make enough noise that marketing are involved.
That said, my comment was crap, as the matter has now happened and all you can do it your best to rectify it, which is what u did. Sometimes i wish i’d just keep my mouth shut :).
Tink: As you mention at the end of your comment, Travis is basically doing what he can to make up for it. I have no doubt that the exposure that a popular (and targetted?) blog provides may set me aside from the rest but I have to say that until this incident, LiquidWeb’s support has actually been quite good. This appears to be more of a policy issue than anything else. They really need to implement a policy that says that they will not shut down web sites/servers before investigating the cause. Of course, if something is repeatedly causing problems, they should shut it down. But they shouldn’t if it is an isolated case. But this isn’t the rank-and-file support person’s problem (unless they weren’t following procedure), it’s the fault of the procedure/process itself.
“They really need to implement a policy that says that they will not shut down web sites/servers before investigating the cause…
But this isn’t the rank-and-file support person’s problem (unless they weren’t following procedure), it’s the fault of the procedure/process itself.”
Well the comment below would imply that isn’t tehir policy.
“It is not company policy to disable server environments in this manner and it was an unfortunate mistake that you were not first contacted.”
Aral, I would be happy to host osflash or any mirror. Our servers are busy nose picking the whole day. Maybe OSFlash will make them really doing their job! Let me know.
[...] This reminds me a bit of Aral’s issue with Liquidweb, where they just shut down OSFlash with no warning or notice due to some big load that the site was putting on the server. [...]
Hi Aral,
I’m a person who happened to come across this web page while searching for reviews and such on LiquidWeb. Being a Sysadmin for a hosting company and reading through this I do understand what you have said, but I can also understand their policy of turning you off.
Look at a scenario of another VPS on the same server also went rouge and was affecting your website. You phone them to find out what’s going on and why you getting such poor performance only to be told “Another VPS on the server has gone rouge, we have emailed the client and are awaiting his response”.
I doubt very much you are going to be calm and say “sure, no problem, thanks” you are more than likely going to ask why they can’t just turn it off, so it doesn’t affect your websites. Fact of the matter is that you wouldn’t care if they turned off another rouge VPS to make sure your vps isn’t affected, so I don’t see why other people who are hosted on the same server should care.
Unfortunately the hosting company always gets stuck in the middle of situations like this, where you have 50 clients screaming and shouting because 1 client is affecting their websites. Personally I think LiquidWeb made the right decision, turn off your vps and notify you about it. If you don’t want to be affected nor have others affected by Neighbor-to-Neighbor performance problems, then consider going with a dedicated server rather than a vps solution.
kthx
Dave
[...] It’s so sad because I’ve been with these people for the past five years and they were so amazing (at least at the beginning). I am still in disbelief that they would do this. It’s the worst thing you can do as a web host and it isn’t the first time it has happened to me either (I had LiquidWeb do something similar on a false alarm once). It’s a horrible thing to disable someone’s application without telling them. Instead, you should contact them and propose a solution (”look, you’ve outgrown this account, let’s move you to this one.”) [...]
Hi Aral,
Can I download Red5 from sourceforge or somewhere else? Thanks.
Personally anytime you use a VPS you know its like moving into a frat house. You have 30 guys sharing one house problems will always arise. It is a bad situation for the host and yourself. As other have posted moving to another VPS on another host will cause you same issues down the road. Moving to a dedicated server is really your only option. I have my Server hosted at Server Intellect and if say someone or something goes rouge on my server at least i know it wont be shut down. The server would have to break the TOS agreement in order for my server and services to be shut down, which personally if someone has gained access to my box I want it shut down ASAP anyways. Dedicated server is the way to go, never have to worry about someone bogging down the box.
To the original poster, you were on a shared environment, if your site is affecting others what do you expect? How is it fair to others to NOT suspend your site, and let it continue to affect other people? What you pay for the shared hosting is cheap so stop whining about it considering you are paying pennies. They have a ton of other customers who pay a lot more than you do and you got more than your money’s worth, LiquidWeb is the best hosting out there, and they have the most helpful support staff on their dedicated server team who will do anything you ask them to.
Just wondering how your new web hosting company is working out for you and who you chose?