Virtual Worldwide Crash Takes Down Second Life

by Alphaville Herald on 29/04/10 at 11:05 am

worldwide crash

Serious problems apparently started in the Second Life virtual world at about 9 pm PDT Wednesday, and have continued over the last 6 hours, in what appears to have been a grid-wide crash of the world and associated services. As of this writing, Herald staff are unable to login to the world, and the Second Life startup screen shows 5807 players online – about 1/8th of normal concurrency for this time of day.

Metaverse service provider Linden Lab’s grid status report for the incident paints a grim picture:

[3:23am PDT] [UPDATE] We are still working on rectifying the problems and will let you know when that changes.

[2:21am PDT] [UPDATE] Work continues to bring services back online.

[1:23am PDT] [UPDATE] We are continuing our work to bring our services back online and will keep you updated here.

[29th April 2010 - 12:30am PDT] [UPDATE] We are continuing to investigate current issues and will post updates here as we have them.

[11:23pm PDT] [UPDATE] Logins, the Land Store, the LindeX, XstreetSL.com and registrations have been disabled as we continue to work on these issues.

[10:45pm PDT] [UPDATE] Logins are currently still down. The website and support portal have come back online, though they may be slow to load or experience errors. XstreetSL.com is inaccessible as well. We’re working to resolve this as quickly as we can – please continue to check back for updates.

[9:45pm PDT] [UPDATE] Logins and secondlife.com, including the support portal, continue to be inaccessible as we work to resolve this issue. Please check back here for updates.

[9:15pm PDT] [UPDATE] Secondlife.com and access to live chat and support tickets are also currently affected, and unavailable for the time being.

[9pm PDT] We’re currently experiencing login and inworld issues. We’re investigating the cause and scope of this issue and will post here when we have more information. Please do not rez no-copy objects or make transactions for the time being.
 

XStreetSL down

While server maintenance was scheduled overnight, the timing of the crash is puzzling — the problems reportedly began an hour before the scheduled maintenance, and the Lab’s XStreetSL storefront is also throwing "unexpected errors".

Did an over-eager system administrator jump the gun – or is this a case of emergent player behavior? For a problem of this magnitude let us hope for some sort of public explanation once the Lab puts it’s world back together.

6 hours of trouble

47 Responses to “Virtual Worldwide Crash Takes Down Second Life”

  1. Rockhound Helstein

    Apr 29th, 2010

    In fact it is not total crash, I am inworld right now – At least part of the grid appears to be online.

  2. Jezz Enoch

    Apr 29th, 2010

    My stupid brother Pappy Enoch done did it, when he were drunk IRL…his F-150 went kerblam into the LL fortress o’ sillytude an his load o’ Shine blowed up like a nucular bum.

  3. GreenLantern Excelsior

    Apr 29th, 2010

    I was unable to log in to Justice Island or my usual backup sim, Durango. The message was “Login failed. Unable to connect to a simulator.” I used Emerald’s “Visit this location” feature and managed to log in to the NuRepublik sim. I tried to TP to Justice Island and the message was “Could not teleport. Routed to wrong region. Please try again.” Sandbox Goguen and Cordova show as completely empty. Can’t TP there either: “Could not teleport. Problem encountered processing your teleport request. You may need to log back in before you can teleport. If you continue to get this message, please check the Tech Support FAQ at: http://www.secondlife.com/support“. SL login time was 0415. Residents inworld when I logged in was 12.490. Nearby sims I can see with people in them: CANTIK, Castawayz Island, Club Atlantic, Faerun, HUMlab, Jack Island, Monash University, PINK CLOVER RESORT, Purrrrfect Kitties, Takaraduka, Titanic,

  4. Unda Xue

    Apr 29th, 2010

    I wonder if the implementation of 3rd party viewer controls (either something done during maintenance to the software or a backlash against this policy) has anything to do with the crash?

    Anyway, I wanted to post my login workaround. I couldn’t log in to my last location. I wanted to use Xstreet to find a SLURL to visit (many vendors post their in-world slurls on their product descriptions) but it was a dead site.

    I got in by going to a hunt blog that lists links to hunt locations and trying those. I got lucky. The first one I tried worked.

  5. amerika

    Apr 29th, 2010

    Woodbury highway in the sunshine,
    Where the cats are longer,
    And the Lulz are stronger than moonshine,
    You’re gonna go I know…

    ‘Cause the sim wind is blowin through your hair,
    And the bans surround no daylight there:
    Furries crying in dispair,
    Flyin’ monkeys in the air, IN THE AIR !

  6. Tux Winkler

    Apr 29th, 2010

    Haha, who got PWN’D now?

  7. Danziel Lane

    Apr 29th, 2010

    Hmmmm…

    If a fake world breaks down … maybe the breakdown is a fake too.

    Would mean, they keep us outside and Kalel and M are having fun in the empty grid with no griefers?

  8. Baal

    Apr 29th, 2010

    waited 6 hours to get in, then read Unda’s Comment so thought I would try my other Sim KOS and no problems at all. So why say logins a disabled. Weird

  9. InuYasha Meiji

    Apr 29th, 2010

    Well the 3rd party viewer controls are not suposed to take place till the 30th. We still have a day. I am wondering if LL pissed off a hacker, after they read the new TOS, and they tried to take em down. Hell, I don’t want to loose 5 years of labor in Butler Sim, to make a nice public Asian sim to explore, because LL got some hacker upset.

    ;(

  10. Gaara Sandalwood

    Apr 29th, 2010

    I doubt it. I really doubt it. LL and the JLU couldn’t be this dumb(could they?), so as to do such a mass grid shutdown or something of the sort.

    Anyway, after today I’ll be more active in OS than before. Or rather starting today. I just hope the login issues are fixed soon because I have a slight need of SL to back my stuff up into OS.

    OS may not be active as SL but it’s fun at least.

  11. ZoHa Boa

    Apr 29th, 2010

    I got a message in the concierge group that the datacenter in texas is without power.

    I had also problems to login to my home postition.
    Probably one of the sims that is down.

    The sim where our office is is online and it seems that logins via that sim do work. Sim name is “ZoHa Islands”

  12. InuYasha Meiji

    Apr 29th, 2010

    It is, I have been running my own standalone for a while, getting it ready to eventually invite friends to see it. IT is lots of fun to have your own regions that you can make anyway you wish, and no neighbors ruining your view. And all Free. Right now the only things I have missing is voice and groups.

    BTW SL is back, but laggy because of the maint they didn’t do I am assuming. It might be going on now in the background while everyone is getting on.

  13. Nelson Jenkins

    Apr 29th, 2010

    A check of my crash log shows something went wrong with the XMLRPC parser.

    I’m willing to bet this was partly a result of an XMLRPC format change for the upcoming viewer lockdown. Just happened to kill some servers in the process (perhaps they use the same XMLRPC thing that the viewer does for authentication and whatnot) and when everyone tried to get through secondlife.com/status it melted the site as well.

    Total speculation though. I have no clue what I’m talking about.

  14. K+M+KY=?

    Apr 29th, 2010

    You mean like K and M are playing “Switch” or something? Hope they at least went to Zindra. Maybe that explains why a shunned GLE is posting his Situation Reports here instead of the JLU blotter?Hmmm thats gross.

    Anyway, earlier, like around 5pm SLT yesterday, wednesday, I noticed that sim crossings were starting to take 4 or 5 times as long as normal while tooling around near Nautilus in the Blake sea. And some teleports were already failing at that time.

    This whole twitter rumour thing about fires being set and burning at the Texas and Virginia co-location centers seems like a bunch of braunschweiger to me.

    Something evil has been propagating through computers on their lan me thinks. It started slow but expanded geometrically over about 6 or 8 hours. There is the possibility that this new server version that was fully rolled out a few days ago began attacking their own asset servers but I bet it didn’t start eating itself without some extra “help” to get it going. Like self replicating loli cubes, or some kind of new weapon or something.

    I still can’t log on. Its great :)

  15. Danziel Lane

    Apr 29th, 2010

    “You mean like K and M are playing “Switch” or something?”

    My world, my imagination, as they said formerly.

    My imagination is Kalel standing there looking over the empty grid.
    No griefer anywhere, no Prok, no Tux, no Danziel, no Pixeleen, no Pappy (I need to ask him if I can take classes in his great queens english), no not yet permabanned WU members, no other universities, no lolcubes, no cages … and feeling the latex smoothing his strong body he is fully aware of his superhero powers … and talks to M:

    “See, M, let me continue my great work and the grid will always look clean and safe and peaceful as today.”

  16. Bubblesort Triskaidekaphobia

    Apr 29th, 2010

    The error I get is that the inventory system is unavailable.

    Conspiracy theorists everywhere, it’s time to break out your tinfoil hats! Woodbury’s theory that they got the axe because of Jcool’s ability to take out the asset server might be true!

    http://bit.ly/bKYxSD

  17. Gaara Sandalwood

    Apr 29th, 2010

    Seriously? I dunno if that’s true but I hope it hasn’t affected my inventory. I still have a bit of stuff to backup for importing to OS.

    A lot of these rumors seem to center around Jcool. Mostly rumors that he has LL convinced he can fuck them up whenever he wants to, forcing them to abide by his wishes more than they would to any common player.

  18. Tux Winkler

    Apr 29th, 2010

    @ Danziel
    Thanks for making me puke in my mouth.
    http://4.bp.blogspot.com/_tCcjI6S-GTw/SPjasWhMW0I/AAAAAAAAAVM/oj45fhXUobI/s400/444.+Shadow+puppeteer+copy

    Anyways, seems many parts are still down, and many peoples access is still blocked.

    Seems some people are remembering a slow down prior to the event. You couldn’t make this up. If the Lab was cleaning up for a sale then it just fell through. Or maybe Plexibaby got told off for adult behaviour with superzeros outside Zindra and decided to flex his stinky fingers – XD

    But somehow I don’t think so.

    If I were the Lab I would think before fuelling up their steamroller in future. Because maybe, just maybe, this is a warning of what could happen.

    Imagine if this was simply one or two computers giving the asset server what for, what if it was one or two hundred? How many residents would stay if this happened every day? Having their inventory reverted to the previous day? Soon become tiresome wouldn’t it – XD

    Of course I am only speculating, what would I know!

  19. Gaara Sandalwood

    Apr 29th, 2010

    Well today’s the last day before their ToS gets kickin’. Plus this whole event is just frustrating(I doubt any events have been held at all today at Club Carnage due to this, they’ve had similar problems before that caused the club to shut down for the whole day, while LL kept saying everything was just fine). Time to get the hell outta Dodge.

  20. techiedavid

    Apr 29th, 2010

    Is Woodbury hitting back?

  21. Miss J

    Apr 29th, 2010

    Wow I leave for a few days and a grid wide crash takes place…but I dont care i can log in now. Sucks for the others who can’t get back in. Good luck ;)

  22. Brookston Holiday

    Apr 29th, 2010

    Wow, reminds me of old times. Used to be pretty much 2 or 3 times a week this would happen. Back in those days we walked to school, both ways, in the snow.

  23. Silverfox Rainbow

    Apr 29th, 2010

    its the calm before the storm, thats all i can say about this.

  24. sparkle

    Apr 29th, 2010

    could-not-have-happened to better people.

    always nice to see a grid crash of this magnitude.
    anything that screws linden, emerald, gemini is a good thing

  25. LOL

    Apr 29th, 2010

    so, when this happens it is a GREAT EXCUSE to go outside and say hello to the real world.

  26. Serendipity Seraph

    Apr 29th, 2010

    It didn’t come back for me until 03:40 SLT the following morning. When it did come back on many TPs were still not working. That is over seven hours when the world disappeared. Not good. I expect better from even OpenSim providers. That LL has that low dependability does not bode well. I was in the middle of an event when it started and had other in world meetings planned for after. But no dice.

    I am of the semi-immersionist metaverse pioneers camp. I do not take it at all likely when the virtual world disappears. It is not remotely acceptable for a major internet provider, of any kind, to be down for such a lengthy time.

  27. Get Over It

    Apr 29th, 2010

    Get over it! As mentioned previously, regular down-time was more common place 2 to 3 years ago, so anyone bawwwing about this blip is doing the serious business thing way, way TOO much.
    “frustrating” this and “not remotely acceptable” is basement dwelling hoo-ha. Get a life, get over it and move along…

  28. A Furry

    Apr 29th, 2010

    @LOL. Obvious troll is obvious.

    Looks like someone spilled soda on the server.

  29. Tracy Welles

    Apr 29th, 2010

    Try another grid. The technicians for Linden are not very network friendly and don’ appear to have the experience and education in the network field. You will find you can run 20 or better simulators on a dual core platform with two gigs of ram and fast eide drives just fine with alternative platforms. As long as you have two or three megabit down and a min of 768 up, you can host away the mere few that land on simulators within the linden grid. And with half the bad experience you get with Linden’s grid.

    With all the fraud, and the poor service. Get a real grid and move on.

  30. Tracy Welles

    Apr 29th, 2010

    @Tux Winkler,

    “Haha, who got PWN’D now?”

    I wouldn’t joke about it, Linden is not making nice with some pretty important developers and others that take great pride in the belief of open computing. I’m not talking about dumping code out there and stating it’s open source and that you believe in the open source model. I’m talking about open computing and the belief in such.

    Not that I agree with poor ethics and adverse or negative technical achievement, but you cannot be who Linden is, and do what they do, and not expect the consequences. And if anyone would deserve it… well.. let the ball roll where it may. For the 30th, their whole company should be shut down. For their fraudulent practices and allowing such for all these years, the company should be shut down and ran off the net.

    Linden will continue with their illegal TOS, but before this is all over they will be singing a completely different tune.

    My advice, find a real grid and people working on the technology, stay away from Linden Lab and Second Life.

  31. tres devin

    Apr 29th, 2010

    I’m getting a
    ‘Login Failed
    The inventory system is currently unavailable’ message.

    Does anyone know what that is about. My alt can log in no problem.

  32. Bubblesort Triskaidekaphobia

    Apr 29th, 2010

    I’m still offline. I’m taking this time to explore open cobalt, though. It’s like opensim with no linden labs influence at all. In other words SL without the fail.

    I’m still exploring it though. Here’s a link: http://www.opencobalt.org/home

  33. Chloe Constantine

    Apr 29th, 2010

    It has been described as a power failure at one of the facilities with the backup generators not working either. Be warned… while you may be able to login.. you *really* don’t want to rez anything no copy

  34. Bradley Gazoz

    Apr 29th, 2010

    Sheesh I guess I was a lucky one – 10 minutes left in a gig i was spinning and suddenly FOOOOOOM! out go the lights. I didn’t mind too much being in the dark with the Army of Bunnies! :) All in all, it did make for an interesting blog post this morning.

    Now it looks like the club is back and the other DJ’s are back in business.

    It would appear the days of old have returned with the massive outages. Can’t wait until the survey shows up in my mailbox about how my SL is improving. hehehehhehe

  35. Tux Winkler

    Apr 29th, 2010

    @ Tracy

    “I wouldn’t joke about it, Linden is not making nice with some pretty important developer”
    - I would joke about it. They are not making nice for anyone, from their recent email:

    “helping to provide you, the Second Life business owner and or creator, with better protection of your inworld creations.”
    - Nothing about residents?

    BTW their jurisdiction is limited to their network, anything outside is off limits!

    Plus after my recent activities I don’t expect to last much longer – XD

  36. pefton

    Apr 30th, 2010

    It was a simple power outtage in dallas you turds. Blame USA infrastructure

  37. Gaara Sandalwood

    Apr 30th, 2010

    Me wonders if it be coincidence I just happened to have to change my password on two accounts now, due to being logged out because someone else is supposedly logging into my accounts. Accounts that I keep as far apart from each other as possible(I have several, I keep Toasty openly affiliated with me, the rest I either seldom use or keep unassociated). One of these unassociated accounts, possibly the one I keep the farthest away from my main, just recently had its password stolen.

    It was never in WU, it was never in any group similar, nor was it ever in any group remotely related to any I’ve been on Gaara save perhaps two that I was in on Gaara months ago(another reason for several accounts being the group limit).

    I will not directly name that account. I have told Pix who it is, and that this event only occured on the Emerald Viewer so far.

    At first I thought, maybe it’s some random hacker. But for some reason it only happened on Emerald so far. And only on the accounts I logged into today(Gaara and this other one thus far).

    I logged into this unassociated account and the password was fine. About ten minutes later, “You have been logged out of Second Life because you are logging in from another location”, and I became baffled. The same message I got after a few hours of finishing up my inven backup on Gaara.

    I’m not gonna just shout “I think it was the Emerald Devs!”. I don’t want to jump to conclusions just yet. And in some ways, I guess I just hope I’m wrong.

    But considering it’d have to have been someone who could only do it through me logging into Emerald from what I’ve seen, did it to two consecutive accounts that I logged into Emerald with, only did it to accounts I logged into Emerald within the past few hours with, etcetera…….I dunno what to think of it atm. And if it IS Jcool or the other devs, why would they want my account anyway?

    I’m still speculating, gonna run another test and get on Emerald on this unassciated account and see what happens. Been a solid near half hour on normal SL and nothing, so gonna try Emerald again.

  38. Alyx Stoklitsky

    Apr 30th, 2010

    The funny thing about this is that SL’s stability has clearly improved by leaps and bounds over the past 3 years if people now consider 6 hours login problems to be newsworthy.

  39. Bubblesort Triskaidekaphobia

    Apr 30th, 2010

    @Alyx: For me it was more like a full 24 hours.

  40. [...] with a taste for blended frappuccinos may have been to blame for the Second Life crash that crippled the virtual world over the last 24 hours, according to Oskar Linden’s comments in the AWGroupies chat tonight. [...]

  41. IntLibber Brautigan

    Apr 30th, 2010

    Dear Linden Lab,
    Due to repeated acts violating your Terms of Service contract with your users, we have put you on 24 hour suspension. This is your first punishment. If you continue to violate the Terms of Service, the length of your suspensions will increase until your users suspend your accounts with them indefinitely.

    Regards,
    IntLibber Brautigan
    Userbase Governance Team

  42. Tux Winkler

    Apr 30th, 2010

    Hahaha, Int spot on.

    Alyx, For me it was about 24 hours also. Yet my wife, same room different pc, was back in in 6 hours. A new account on my pvr 6 hours also.

    Seems the duration may have been related to current standing with SL, or could it have been something else – XD.

  43. Alyx Stoklitsky

    Apr 30th, 2010

    I’m just going off the blog screenshot in this article for my ‘six hours’ figure.

    Even so, this sort of thing was bi-weekly in 2007, when the grid would grid to a near halt whenever there were more than 40k concurrent users.

  44. IntLibber Brautigan

    Apr 30th, 2010

    So now its griding to a halt whenever theres LESS than 40k concurrent users?

  45. Archie

    May 13th, 2010

    That explains a few things, thanks

  46. NagatoYuki Huntress

    Jun 13th, 2010

    I was not too happy with LL post to me when a few Objects i had rezzed not but 30 mins prior to show off to a club had got lost in the whole mess. the owner of the land has not seen them to this day and has checked the land for them and so far they have not shown up.

    LL response was a typical well if ya had a premium account we would be able to help you. I think wow and not long you had done a press release of how great ya doing. Would seem liek a chance for LL to step up and say hey sorry to hear of ya losses we have checked your transactions and do indead see ya paid for such and such and please accept from us a sorry and a compensate to buy yourself those items again here is 2 k in lindens. We at LL would love to offer you a chance to become a premium member at half off the cost for the first month …

    That would have been awesome instead of a FU ya should have been a premium member for us to help ya ass out

  47. kj

    Jul 26th, 2011

    Someone should just crash that world for good! Too much scamming and fraud goes on and the company turns a blind eye to it and refuses to do anything about it as long as their pockets get full of money. They have no respect for their paying customers. The game should be free altogether. They are NOT above US laws and since they wanna run this biz they better sure a heck grow up and start actually running a legitimate biz rather than screwing around acting like a mafia. I hope the go down.

Leave a Reply