Virtual Worldwide Crash Takes Down Second Life
by Alphaville Herald on 29/04/10 at 11:05 am
Serious problems apparently started in the Second Life virtual world at about 9 pm PDT Wednesday, and have continued over the last 6 hours, in what appears to have been a grid-wide crash of the world and associated services. As of this writing, Herald staff are unable to login to the world, and the Second Life startup screen shows 5807 players online – about 1/8th of normal concurrency for this time of day.
Metaverse service provider Linden Lab’s grid status report for the incident paints a grim picture:
[3:23am PDT] [UPDATE] We are still working on rectifying the problems and will let you know when that changes.
[2:21am PDT] [UPDATE] Work continues to bring services back online.
[1:23am PDT] [UPDATE] We are continuing our work to bring our services back online and will keep you updated here.
[29th April 2010 - 12:30am PDT] [UPDATE] We are continuing to investigate current issues and will post updates here as we have them.
[11:23pm PDT] [UPDATE] Logins, the Land Store, the LindeX, XstreetSL.com and registrations have been disabled as we continue to work on these issues.
[10:45pm PDT] [UPDATE] Logins are currently still down. The website and support portal have come back online, though they may be slow to load or experience errors. XstreetSL.com is inaccessible as well. We’re working to resolve this as quickly as we can – please continue to check back for updates.
[9:45pm PDT] [UPDATE] Logins and secondlife.com, including the support portal, continue to be inaccessible as we work to resolve this issue. Please check back here for updates.
[9:15pm PDT] [UPDATE] Secondlife.com and access to live chat and support tickets are also currently affected, and unavailable for the time being.
[9pm PDT] We’re currently experiencing login and inworld issues. We’re investigating the cause and scope of this issue and will post here when we have more information. Please do not rez no-copy objects or make transactions for the time being.
While server maintenance was scheduled overnight, the timing of the crash is puzzling — the problems reportedly began an hour before the scheduled maintenance, and the Lab’s XStreetSL storefront is also throwing "unexpected errors".
Did an over-eager system administrator jump the gun – or is this a case of emergent player behavior? For a problem of this magnitude let us hope for some sort of public explanation once the Lab puts it’s world back together.
Rockhound Helstein
Apr 29th, 2010
In fact it is not total crash, I am inworld right now – At least part of the grid appears to be online.
Jezz Enoch
Apr 29th, 2010
My stupid brother Pappy Enoch done did it, when he were drunk IRL…his F-150 went kerblam into the LL fortress o’ sillytude an his load o’ Shine blowed up like a nucular bum.
GreenLantern Excelsior
Apr 29th, 2010
I was unable to log in to Justice Island or my usual backup sim, Durango. The message was “Login failed. Unable to connect to a simulator.” I used Emerald’s “Visit this location” feature and managed to log in to the NuRepublik sim. I tried to TP to Justice Island and the message was “Could not teleport. Routed to wrong region. Please try again.” Sandbox Goguen and Cordova show as completely empty. Can’t TP there either: “Could not teleport. Problem encountered processing your teleport request. You may need to log back in before you can teleport. If you continue to get this message, please check the Tech Support FAQ at: http://www.secondlife.com/support“. SL login time was 0415. Residents inworld when I logged in was 12.490. Nearby sims I can see with people in them: CANTIK, Castawayz Island, Club Atlantic, Faerun, HUMlab, Jack Island, Monash University, PINK CLOVER RESORT, Purrrrfect Kitties, Takaraduka, Titanic,
Unda Xue
Apr 29th, 2010
I wonder if the implementation of 3rd party viewer controls (either something done during maintenance to the software or a backlash against this policy) has anything to do with the crash?
Anyway, I wanted to post my login workaround. I couldn’t log in to my last location. I wanted to use Xstreet to find a SLURL to visit (many vendors post their in-world slurls on their product descriptions) but it was a dead site.
I got in by going to a hunt blog that lists links to hunt locations and trying those. I got lucky. The first one I tried worked.
amerika
Apr 29th, 2010
Woodbury highway in the sunshine,
Where the cats are longer,
And the Lulz are stronger than moonshine,
You’re gonna go I know…
‘Cause the sim wind is blowin through your hair,
And the bans surround no daylight there:
Furries crying in dispair,
Flyin’ monkeys in the air, IN THE AIR !
Tux Winkler
Apr 29th, 2010
Haha, who got PWN’D now?
Danziel Lane
Apr 29th, 2010
Hmmmm…
If a fake world breaks down … maybe the breakdown is a fake too.
Would mean, they keep us outside and Kalel and M are having fun in the empty grid with no griefers?
Baal
Apr 29th, 2010
waited 6 hours to get in, then read Unda’s Comment so thought I would try my other Sim KOS and no problems at all. So why say logins a disabled. Weird
InuYasha Meiji
Apr 29th, 2010
Well the 3rd party viewer controls are not suposed to take place till the 30th. We still have a day. I am wondering if LL pissed off a hacker, after they read the new TOS, and they tried to take em down. Hell, I don’t want to loose 5 years of labor in Butler Sim, to make a nice public Asian sim to explore, because LL got some hacker upset.
;(
Gaara Sandalwood
Apr 29th, 2010
I doubt it. I really doubt it. LL and the JLU couldn’t be this dumb(could they?), so as to do such a mass grid shutdown or something of the sort.
Anyway, after today I’ll be more active in OS than before. Or rather starting today. I just hope the login issues are fixed soon because I have a slight need of SL to back my stuff up into OS.
OS may not be active as SL but it’s fun at least.
ZoHa Boa
Apr 29th, 2010
I got a message in the concierge group that the datacenter in texas is without power.
I had also problems to login to my home postition.
Probably one of the sims that is down.
The sim where our office is is online and it seems that logins via that sim do work. Sim name is “ZoHa Islands”
InuYasha Meiji
Apr 29th, 2010
It is, I have been running my own standalone for a while, getting it ready to eventually invite friends to see it. IT is lots of fun to have your own regions that you can make anyway you wish, and no neighbors ruining your view. And all Free. Right now the only things I have missing is voice and groups.
BTW SL is back, but laggy because of the maint they didn’t do I am assuming. It might be going on now in the background while everyone is getting on.
Nelson Jenkins
Apr 29th, 2010
A check of my crash log shows something went wrong with the XMLRPC parser.
I’m willing to bet this was partly a result of an XMLRPC format change for the upcoming viewer lockdown. Just happened to kill some servers in the process (perhaps they use the same XMLRPC thing that the viewer does for authentication and whatnot) and when everyone tried to get through secondlife.com/status it melted the site as well.
Total speculation though. I have no clue what I’m talking about.
K+M+KY=?
Apr 29th, 2010
You mean like K and M are playing “Switch” or something? Hope they at least went to Zindra. Maybe that explains why a shunned GLE is posting his Situation Reports here instead of the JLU blotter?Hmmm thats gross.
Anyway, earlier, like around 5pm SLT yesterday, wednesday, I noticed that sim crossings were starting to take 4 or 5 times as long as normal while tooling around near Nautilus in the Blake sea. And some teleports were already failing at that time.
This whole twitter rumour thing about fires being set and burning at the Texas and Virginia co-location centers seems like a bunch of braunschweiger to me.
Something evil has been propagating through computers on their lan me thinks. It started slow but expanded geometrically over about 6 or 8 hours. There is the possibility that this new server version that was fully rolled out a few days ago began attacking their own asset servers but I bet it didn’t start eating itself without some extra “help” to get it going. Like self replicating loli cubes, or some kind of new weapon or something.
I still can’t log on. Its great
Danziel Lane
Apr 29th, 2010
“You mean like K and M are playing “Switch” or something?”
My world, my imagination, as they said formerly.
My imagination is Kalel standing there looking over the empty grid.
No griefer anywhere, no Prok, no Tux, no Danziel, no Pixeleen, no Pappy (I need to ask him if I can take classes in his great queens english), no not yet permabanned WU members, no other universities, no lolcubes, no cages … and feeling the latex smoothing his strong body he is fully aware of his superhero powers … and talks to M:
“See, M, let me continue my great work and the grid will always look clean and safe and peaceful as today.”
Bubblesort Triskaidekaphobia
Apr 29th, 2010
The error I get is that the inventory system is unavailable.
Conspiracy theorists everywhere, it’s time to break out your tinfoil hats! Woodbury’s theory that they got the axe because of Jcool’s ability to take out the asset server might be true!
http://bit.ly/bKYxSD
Gaara Sandalwood
Apr 29th, 2010
Seriously? I dunno if that’s true but I hope it hasn’t affected my inventory. I still have a bit of stuff to backup for importing to OS.
A lot of these rumors seem to center around Jcool. Mostly rumors that he has LL convinced he can fuck them up whenever he wants to, forcing them to abide by his wishes more than they would to any common player.
Tux Winkler
Apr 29th, 2010
@ Danziel
Thanks for making me puke in my mouth.
http://4.bp.blogspot.com/_tCcjI6S-GTw/SPjasWhMW0I/AAAAAAAAAVM/oj45fhXUobI/s400/444.+Shadow+puppeteer+copy
Anyways, seems many parts are still down, and many peoples access is still blocked.
Seems some people are remembering a slow down prior to the event. You couldn’t make this up. If the Lab was cleaning up for a sale then it just fell through. Or maybe Plexibaby got told off for adult behaviour with superzeros outside Zindra and decided to flex his stinky fingers – XD
But somehow I don’t think so.
If I were the Lab I would think before fuelling up their steamroller in future. Because maybe, just maybe, this is a warning of what could happen.
Imagine if this was simply one or two computers giving the asset server what for, what if it was one or two hundred? How many residents would stay if this happened every day? Having their inventory reverted to the previous day? Soon become tiresome wouldn’t it – XD
Of course I am only speculating, what would I know!
Gaara Sandalwood
Apr 29th, 2010
Well today’s the last day before their ToS gets kickin’. Plus this whole event is just frustrating(I doubt any events have been held at all today at Club Carnage due to this, they’ve had similar problems before that caused the club to shut down for the whole day, while LL kept saying everything was just fine). Time to get the hell outta Dodge.
techiedavid
Apr 29th, 2010
Is Woodbury hitting back?
Miss J
Apr 29th, 2010
Wow I leave for a few days and a grid wide crash takes place…but I dont care i can log in now. Sucks for the others who can’t get back in. Good luck
Brookston Holiday
Apr 29th, 2010
Wow, reminds me of old times. Used to be pretty much 2 or 3 times a week this would happen. Back in those days we walked to school, both ways, in the snow.
Silverfox Rainbow
Apr 29th, 2010
its the calm before the storm, thats all i can say about this.
sparkle
Apr 29th, 2010
could-not-have-happened to better people.
always nice to see a grid crash of this magnitude.
anything that screws linden, emerald, gemini is a good thing
LOL
Apr 29th, 2010
so, when this happens it is a GREAT EXCUSE to go outside and say hello to the real world.
Serendipity Seraph
Apr 29th, 2010
It didn’t come back for me until 03:40 SLT the following morning. When it did come back on many TPs were still not working. That is over seven hours when the world disappeared. Not good. I expect better from even OpenSim providers. That LL has that low dependability does not bode well. I was in the middle of an event when it started and had other in world meetings planned for after. But no dice.
I am of the semi-immersionist metaverse pioneers camp. I do not take it at all likely when the virtual world disappears. It is not remotely acceptable for a major internet provider, of any kind, to be down for such a lengthy time.
Get Over It
Apr 29th, 2010
Get over it! As mentioned previously, regular down-time was more common place 2 to 3 years ago, so anyone bawwwing about this blip is doing the serious business thing way, way TOO much.
“frustrating” this and “not remotely acceptable” is basement dwelling hoo-ha. Get a life, get over it and move along…
A Furry
Apr 29th, 2010
@LOL. Obvious troll is obvious.
Looks like someone spilled soda on the server.
Tracy Welles
Apr 29th, 2010
Try another grid. The technicians for Linden are not very network friendly and don’ appear to have the experience and education in the network field. You will find you can run 20 or better simulators on a dual core platform with two gigs of ram and fast eide drives just fine with alternative platforms. As long as you have two or three megabit down and a min of 768 up, you can host away the mere few that land on simulators within the linden grid. And with half the bad experience you get with Linden’s grid.
With all the fraud, and the poor service. Get a real grid and move on.
Tracy Welles
Apr 29th, 2010
@Tux Winkler,
“Haha, who got PWN’D now?”
I wouldn’t joke about it, Linden is not making nice with some pretty important developers and others that take great pride in the belief of open computing. I’m not talking about dumping code out there and stating it’s open source and that you believe in the open source model. I’m talking about open computing and the belief in such.
Not that I agree with poor ethics and adverse or negative technical achievement, but you cannot be who Linden is, and do what they do, and not expect the consequences. And if anyone would deserve it… well.. let the ball roll where it may. For the 30th, their whole company should be shut down. For their fraudulent practices and allowing such for all these years, the company should be shut down and ran off the net.
Linden will continue with their illegal TOS, but before this is all over they will be singing a completely different tune.
My advice, find a real grid and people working on the technology, stay away from Linden Lab and Second Life.
tres devin
Apr 29th, 2010
I’m getting a
‘Login Failed
The inventory system is currently unavailable’ message.
Does anyone know what that is about. My alt can log in no problem.
Bubblesort Triskaidekaphobia
Apr 29th, 2010
I’m still offline. I’m taking this time to explore open cobalt, though. It’s like opensim with no linden labs influence at all. In other words SL without the fail.
I’m still exploring it though. Here’s a link: http://www.opencobalt.org/home
Chloe Constantine
Apr 29th, 2010
It has been described as a power failure at one of the facilities with the backup generators not working either. Be warned… while you may be able to login.. you *really* don’t want to rez anything no copy
Bradley Gazoz
Apr 29th, 2010
Sheesh I guess I was a lucky one – 10 minutes left in a gig i was spinning and suddenly FOOOOOOM! out go the lights. I didn’t mind too much being in the dark with the Army of Bunnies!
All in all, it did make for an interesting blog post this morning.
Now it looks like the club is back and the other DJ’s are back in business.
It would appear the days of old have returned with the massive outages. Can’t wait until the survey shows up in my mailbox about how my SL is improving. hehehehhehe
Tux Winkler
Apr 29th, 2010
@ Tracy
“I wouldn’t joke about it, Linden is not making nice with some pretty important developer”
- I would joke about it. They are not making nice for anyone, from their recent email:
“helping to provide you, the Second Life business owner and or creator, with better protection of your inworld creations.”
- Nothing about residents?
BTW their jurisdiction is limited to their network, anything outside is off limits!
Plus after my recent activities I don’t expect to last much longer – XD
pefton
Apr 30th, 2010
It was a simple power outtage in dallas you turds. Blame USA infrastructure
Gaara Sandalwood
Apr 30th, 2010
Me wonders if it be coincidence I just happened to have to change my password on two accounts now, due to being logged out because someone else is supposedly logging into my accounts. Accounts that I keep as far apart from each other as possible(I have several, I keep Toasty openly affiliated with me, the rest I either seldom use or keep unassociated). One of these unassociated accounts, possibly the one I keep the farthest away from my main, just recently had its password stolen.
It was never in WU, it was never in any group similar, nor was it ever in any group remotely related to any I’ve been on Gaara save perhaps two that I was in on Gaara months ago(another reason for several accounts being the group limit).
I will not directly name that account. I have told Pix who it is, and that this event only occured on the Emerald Viewer so far.
At first I thought, maybe it’s some random hacker. But for some reason it only happened on Emerald so far. And only on the accounts I logged into today(Gaara and this other one thus far).
I logged into this unassociated account and the password was fine. About ten minutes later, “You have been logged out of Second Life because you are logging in from another location”, and I became baffled. The same message I got after a few hours of finishing up my inven backup on Gaara.
I’m not gonna just shout “I think it was the Emerald Devs!”. I don’t want to jump to conclusions just yet. And in some ways, I guess I just hope I’m wrong.
But considering it’d have to have been someone who could only do it through me logging into Emerald from what I’ve seen, did it to two consecutive accounts that I logged into Emerald with, only did it to accounts I logged into Emerald within the past few hours with, etcetera…….I dunno what to think of it atm. And if it IS Jcool or the other devs, why would they want my account anyway?
I’m still speculating, gonna run another test and get on Emerald on this unassciated account and see what happens. Been a solid near half hour on normal SL and nothing, so gonna try Emerald again.
Alyx Stoklitsky
Apr 30th, 2010
The funny thing about this is that SL’s stability has clearly improved by leaps and bounds over the past 3 years if people now consider 6 hours login problems to be newsworthy.
Bubblesort Triskaidekaphobia
Apr 30th, 2010
@Alyx: For me it was more like a full 24 hours.
Phoenix Power Outage Cripples Second Life – Blended Frappuccino Blamed | The Alphaville Herald
Apr 30th, 2010
[...] with a taste for blended frappuccinos may have been to blame for the Second Life crash that crippled the virtual world over the last 24 hours, according to Oskar Linden’s comments in the AWGroupies chat tonight. [...]
IntLibber Brautigan
Apr 30th, 2010
Dear Linden Lab,
Due to repeated acts violating your Terms of Service contract with your users, we have put you on 24 hour suspension. This is your first punishment. If you continue to violate the Terms of Service, the length of your suspensions will increase until your users suspend your accounts with them indefinitely.
Regards,
IntLibber Brautigan
Userbase Governance Team
Tux Winkler
Apr 30th, 2010
Hahaha, Int spot on.
Alyx, For me it was about 24 hours also. Yet my wife, same room different pc, was back in in 6 hours. A new account on my pvr 6 hours also.
Seems the duration may have been related to current standing with SL, or could it have been something else – XD.
Alyx Stoklitsky
Apr 30th, 2010
I’m just going off the blog screenshot in this article for my ‘six hours’ figure.
Even so, this sort of thing was bi-weekly in 2007, when the grid would grid to a near halt whenever there were more than 40k concurrent users.
IntLibber Brautigan
Apr 30th, 2010
So now its griding to a halt whenever theres LESS than 40k concurrent users?
Archie
May 13th, 2010
That explains a few things, thanks
NagatoYuki Huntress
Jun 13th, 2010
I was not too happy with LL post to me when a few Objects i had rezzed not but 30 mins prior to show off to a club had got lost in the whole mess. the owner of the land has not seen them to this day and has checked the land for them and so far they have not shown up.
LL response was a typical well if ya had a premium account we would be able to help you. I think wow and not long you had done a press release of how great ya doing. Would seem liek a chance for LL to step up and say hey sorry to hear of ya losses we have checked your transactions and do indead see ya paid for such and such and please accept from us a sorry and a compensate to buy yourself those items again here is 2 k in lindens. We at LL would love to offer you a chance to become a premium member at half off the cost for the first month …
That would have been awesome instead of a FU ya should have been a premium member for us to help ya ass out
kj
Jul 26th, 2011
Someone should just crash that world for good! Too much scamming and fraud goes on and the company turns a blind eye to it and refuses to do anything about it as long as their pockets get full of money. They have no respect for their paying customers. The game should be free altogether. They are NOT above US laws and since they wanna run this biz they better sure a heck grow up and start actually running a legitimate biz rather than screwing around acting like a mafia. I hope the go down.