Hard Crash
by prokofy on 12/02/07 at 2:02 am
By Prokofy Neva, Dept. of Virtual Estate
Second Life went offline completely at about 10:50 pm SLT, after a very rough evening of avatars reporting constant hard crashes, teleport fails, transactions not going through, objects not rezzing out, etc.
The valiant-sounding error message about “our best efforts” just went up, but even the website has gone completely down. Not even GRID OFFLINE works.
The crash was so sudden I went looking to see if RL San Francisco was still functioning. Seems so.
I don’t recall more than 30,000 online.
UPDATE 11:05 pm SLT The “status” page is now up saying that SL is taken offline “temporarily”. The rest of the site is now working as well.
Nacon
Feb 12th, 2007
So does this mean you got a wild theory for this one?
Come on, what the hell did you expect? SL always crashes.
Unless you want to gamble, sure…
Hackers Attack?
Network overload?
Network burnt out?
An very very bad exploit was found?
Someone made an new GodMode Viewer and toyed with the grid?
Torley Linden got too touchy and decided to hug the servers, frying himself?
Zee Linden tripped over their messy network wires?
Prok just brought an sexy outfit from Naughty Design?
Leftover Banned Goons took on a wild adventure trip into Linden Labs?
Two Grid-Monkeys was talking sense of irony about “what-if”?
Philip Linden was trying to logs on?
Philip Linden was trying to logs out?
Aimee Weber came up with an godly cute idea?
All the online users was trying to teleport at the same time?
Avatars starting to have a mind of their own?
FBI Raid?
Prank jokes?
Blew a fuse?
Someone used a super-super waxy floor soap, tripped and took an wild ride into the network room?
An newbie Linden mistakes network wires for old-fashion phone operator machine and switching wires?
Someone died and spiritually lives into SL world, so he can meet his friends a lot closer?
Neo-Pagan spell took in place digitally?
Someone was trying to upload an holy photograph of Jesus?
Kelly Linden was trying to be a hero and fight off some hard cookies spiders in the network room?
Someone took a dump from a 747 airliner, it landed on LL’s building and exploded like shit?
…place your bet, I’m sure anything can happen.
Seola Sassoon
Feb 12th, 2007
Yeah Prok… and? You just made a story out of what everyone knew…. SL went down…
Nacon, I give props for this one:
Torley Linden got too touchy and decided to hug the servers, frying himself?
But that’s not conspiracy theorist enough Nacon. You have to add a few labels!
The FIC was attacked by the W-hats, because they were too liberal, therefore causing light saber duels, ending in a spark in the network causing it to catch fire, which wasn’t put out fast enough due to Sheep standing in the way.
Dale Luna
Feb 12th, 2007
“Grid and website are feeling better now.”
*Hugs watermelon.
Still can’t log in.
Lorrie Leavitt
Feb 12th, 2007
I noticed the online user population go above 31,000, the first time in my short experience. It was ok as long as you laid still preferrably with a friend and did not attempt anything too strenuous.
Lorrie x
hotlips Tornado
Feb 12th, 2007
“Yeah Prok… and? You just made a story out of what everyone knew…. SL went down…”
Well I was in bed at the time, not being in America, so I didn’t know the grid had gone down.
These kinds of things *are* news. You have a business don’t you? In the short-term it’s useful to know whether, for example, the fact that you had no customers during a certain period isn’t because your new vendor system is a bit flaky. In the longer term, decisions about future directions will surely have to take into account grid performance. It’s best to have more than one source for this kind of stuff.
Nacon
Feb 12th, 2007
Lorrie- “… It was ok as long as you laid still preferrably with a friend and did not attempt anything too strenuous.”
that’s one good laugh you got me there and I’m trying to make my car script run more realistic.
So much for my future in car marketing, heheh.
…and I do recall seeing the count at 32,100 before, kinda scary.
janeforyou Barbara
Feb 12th, 2007
I se the gird are still offline-cant loggin-i got 2 isle a mall with 100 rentors, i got 56 shops round in SL to maintane, got no idea if all are ok.
Today SL was on CNN Global Office….i guess we hit 4 mill nicks and 35-40k ppl online in 1 week or so.There are 70 Universery and a lot of new real world big business Office opening in SL…i wonder wat thay say and think now?
Are SL so complex that its not user friendly? If The Lab cant solve this problems i doubt that this new RL business office want to use more time tapping there fingers.
John Lennon
Feb 12th, 2007
Wake up San Francisco!!!
Charli Ivory
Feb 12th, 2007
Right before I hard crashed (and then couldn’t log in, b/c the grid went down) I got two scary messages from my Anti-Virus Software. Two words. “Trojan Attempt”
I *never* have had such a serious threat on my computer, my network, my nada.
Maybe this points to the hacker attack theory … and could be why the Lindens have been so … well … cheerfully cryptic.
Seola Sassoon
Feb 12th, 2007
“”"”Well I was in bed at the time, not being in America, so I didn’t know the grid had gone down.
These kinds of things *are* news. You have a business don’t you? In the short-term it’s useful to know whether, for example, the fact that you had no customers during a certain period isn’t because your new vendor system is a bit flaky. In the longer term, decisions about future directions will surely have to take into account grid performance. It’s best to have more than one source for this kind of stuff.”"”"
If you need a random post to tell you that SL was down and there’s a lag in profits, then you haven’t been in SL all that long. A quick check in the financial stats easily tells you when the grid’s down for a period of time. Just look at the spike. Not that hard.
I’m still waiting on the FIC, W-hattery, Sheep involvement post from Prok though.
Prokofy Neva
Feb 12th, 2007
Normally I think it’s boring to report every crash or reduction of service of SL the way the Blingsider does. You could just cut and paste the same thing a zillion times.
But this one seemed REALLY abrupt and hard, and also followed a day or two of constant individual hard crashing for no reason. Like, usually, there’s a thing whereby if you press on a texture, you crash. Or if you try to cross a sim, you crash. Or if you try to open a notecard and do something else like put in a landmark to it, you crash, or terraform from too far away. These are all sort of “known issues”.
When you crash merely while standing around doing nothing, it’s more worrisome.
I don’t know the mechanics of it, but how could a virus attacking your individual computer be the same one attacking the Lindens’ servers?
Gerome Finucane
Feb 12th, 2007
I do not understand how an organization that has such an incompetent a product as that offered by the Linens can possibly think they can 1)collect revenue from an inconsistent service; 2) can collect fees from anyone who whats to engage enterprise on Second Life.
What are they thinking, people will continue to pay money to watch the Lindens try and figure out how to offer a real time service, on their terms.
Go figure.
Gerome
hotlips Tornado
Feb 12th, 2007
Yeah, you’ve got me there Seola. I haven’t been in SL very long, therefore my opinion has no validity.
Gerome Finucane
Feb 12th, 2007
To my corporate clients I say “don’t waste your time, this is not a delivery system you can rely on”.
To my individual clients I say “your business is better served through another means, you can not rely on this service to provide a presence”.
If I am wrong, please correct my thinking.
Gerome
Prokofy Neva
Feb 12th, 2007
I’m happy to go on paying them money. Because there is nothing else like it. There is no equivalent 3-D streaming virtual world with property and user-made content that has anything at all reachying the level of interest and creativity and capacity of Second Life.
I do think they need to rethink the credit policy though. They should not go by the *consecutive* 24 hour loss of service they have now as a reason for credit, which they’ve invoked perhaps 2ce in recent years, but by *cumulative* loss per month.
And loss of service shouldn’t be defined as “unable to log on” but precisely this type of state it is in right now, where they issue an announcement telling you not to pull things in and out of inventory. If you cannot use inventory, you do not have a service, especially if Group IMs, teleport, and dollar transactions don’t work. Most people then don’t wish to merely sit on their parcel and type in chat, they could do it faster outside in Yahoo Messenger, which has interactive sound and sketching and such now.
Urizenus
Feb 12th, 2007
I think the case for putting clients here goes like this: Yeah the grid will go down a lot and the performance will vary from weak to awful, but these guys are pioneering the development of a new medium so this is to be expected, and the customers they do have are valuable because they are sophisticated early adopters and, obviously, insanely loyal. Mmmmm, who wants the Kool-Aide next?
Gerome Finucane
Feb 12th, 2007
Sure, it goes like this: IF you want to bet at 10:1 odds the put your money with these guys, they may be incapable of signing on clients for a day at a time, but if that does not matter, then do so. IF you want an image of your company to be one of those where we are there when we want to be but are not there when we don’t feel like it, then I would sign up here too. If you want the image of your company to be “you have a problem then see our terms of service”, then definitely sign up here.
Seems to me that the philosophy of the company you are signing up with is “he, dude, let me have the buck, if you got a problem, see the terms”. I may be wrong.
Gerome Finucane
Feb 12th, 2007
BTW,
I paid for the whole day today, I have been on for <60 min. Oh yea, go to the web site and see the terms of service.
Gerome Finucane
Feb 12th, 2007
Now there are 27000+ online. Not me. Please explain to me how this works!
Gerome Finucane
Feb 12th, 2007
CLick, the light bulb went on. They can claim 28137 online, but they can not log on another. I guess what shows up on their intro page can contain whatever they want it to show. I know they can not log on 28138, or I would be logged on.
Gerome Finucane
Feb 12th, 2007
Now 28602 are logged on, according to the screen. Can they not log me on, or is the number logged a a fabrication to convince some that SL is more significant than it is??
Gerome Finucane
Feb 12th, 2007
Nope, Lindens can not log me on. Now there is 28027. OH well. I paid the money this month for precious little, it seems.
Gerome Finucane
Feb 12th, 2007
Now it si at 28840 and still SL is incapable of logging me on.
What is their capacity?
Are they capable of growth beyond about 28000??
If they can’t log me on, how about the next person that joins?
Gerome Finucane
Feb 12th, 2007
NOw at 28668 and still Lindens are not capable of logging in one more person. What gives?
Would a competent software architect explain how this is possible?
Gerome Finucane
Feb 12th, 2007
I really want an explanation to this one: is is first come first serverd?? Why does any one time out? Am I really logging into another server besides the one I paid for? If so, do I have a deal with that owner too?? If not, is the deal with you? If with you, why does it take hours to log on when your numbers show you logging on thousands while I wait??
Gerome Finucane
Feb 12th, 2007
I have now had the terms of service explained to me: we charge you what we said we would, we provide you the service that we are capable of. Sound like a strong proposition to me. Charge what you want, deliver what you feel like.
Talthybius Brevity
Feb 12th, 2007
Actually, the grid didn’t go offline at 10:50pm SLT last night, it just got really ugly. I was online and accomplishing things (intermittently and with great frustration) until 2:15am SLT this morning.
What amazes me is that, judging from the official blog posts, it is 15 hours after the problems began and LL seems to thus far have no clue at all what is wrong with their network.
Few are the networks that are crippled or down for 15 hours straight. Fewer still are the networks that are down and crippled for 15 hours with the Admins still having no clue at all as to what is wrong with their network.
Watch the VCs shiver in fear.
Would be funny if it weren’t so frustrating and sad.
TB
Talthybius Brevity
Feb 12th, 2007
p.s. there were only about 24,000 concurrent logins when the trouble began last night. Doubtful this is the same load problems we’re so used to since the media found SL.
Myrrh Massiel
Feb 12th, 2007
…seriously, herald staff: kool-aid has no ‘e’, and the peoples temple used flavor-aid, not kool-aid…
Prokofy Neva
Feb 12th, 2007
Re: >the grid didn’t go offline at 10:50pm SLT last night, it just got really ugly
The website went off line, and most people crashed. You couldn’t get any signal out of SL, so you couldn’t tell whether it was online or offline.
When you attempted to log on, you got the “despite our best efforts” and NO GRID STATUS AT ALL.
Eventually, after about 30 minutes, you could try to dial in, and see 11,000 logged on, mysteriously. Today, for example, the grid seemed to be down for all practical purpose, as the message said “Log-ins limited to Lindens,” but the sign-on figure was 11,000.
The Lindens describe the status updates here:
http://blog.secondlife.com/2007/02/12/grid-issues-recurred/
They note that the entire website did go offline, but the grid didn’t. However, at the time the story was posted, it was not possible to know this because there was no outside access to SL, even if some people remained logged in.
Gerome, have you never played a game? They often have log-in queues. The more normal games like WoW and The Sims Online or whatever will tell you your log-in-queue wait time. Our game gods just haven’t bothered to do this yet.
Kool-Aide is a very popular spelling of the old trade-mark name “Kool-Aid” and is in the Urban Dictionary and other sources as a variation. And if Jim Jones used Flavor-Aid instead, it’s one of those footnotes in history that is utterly obliterated by later popular accounts (like Jesus’ famous “Sermon on the Mount” which was actually given on flat land).
Talthybius Brevity
Feb 12th, 2007
Myrrh, to me the grid being offline and people not being able to login are not the same thing. Those of us lucky enough not to suffer a client crash were able to stay logged in and working. Some XML RPC communications to sites that rely on it were still working too. If the grid were actually down, offline, hard crashed etc, that would be offline. (IMO) This is not to suggest that the situation didn’t totally f**king suck, it did.
New blog post tells us 15 hours to figure that it was network hardware failing and which particular piece of hardware it was.
I’ve witnessed some difficult network troubleshooting before, but 15 hours to track it down to a failed piece of hardware seems a little extreme, even taking into account relying on upstream providers to do some of the troubleshooting.
Looking forward to getting back to work tonight. There are stoners waiting for bong upgrades and that’s of dire importance to the world!
Prokofy Neva
Feb 12th, 2007
The message from Lindens themselves said that the grid was offline. But if you read their blog later, they said it wasn’t.
If it is inaccessible to more than half the population, can we call it and say it is “offline”?
It still sucks, as you can log on, but not move around or take inventory or do anything but look at SL like a Disney Viewmaster.
Myrrh Massiel
Feb 12th, 2007
(…that was prokofy, talthybius, not me…)