SL Hits, Admits Performance Wall

by Alphaville Herald on 20/01/09 at 7:54 pm

Lab puts down fatties, brings in database experts
How to sell a stable corporate world on an unstable platform

by Pixeleen Mistral, National Affairs desk

January has been a rocky month for metaverse service provider Linden Lab, and the troubles culminated in severe database problems that brought the virtual world to it’s knees Sunday. The normally tight-lipped Lab confessed monday the current infrastructure is not scaling and cannot keep up with the usage peaks of 75-80,000 avatars and bots. This is not news to residents. Still, admitting there is a problem is a good first step.

In a kiss-and-tell blog post monday, FJ Linden boasted that he is no longer a mySQL virgin, and tells a wistful tale his loss of innocence, saying for the “first time since joining Linden Lab that I’ve experienced a full mySQL crash”.

We somehow suspected that all of those little crashes would lead to this, but FJ – did you still love mySQL in the morning? Really?

Ignoring – at least for now – the allure of other database mistresses, FJ explains that when mySQL has problems, his best option is to block logins — sometimes for a very long time. Residents and FJ are in full agreement that this is not acceptable. When the problems will be addressed is unclear, and there is no lack of problems to address.

While FJ’s immediate priority “is to tune and optimize queries to get us back to a position where we can manage our Resident transactions during peak load”, he is calling in a higher power to assist the in-house game gods, including “some of the best mySQL professional services teams, to help us tune and optimize, as well as recommend long term architectural changes”.

How much the consultants will be able to help is an open question. Sardonyx Linden admits to an “intentional lack of architecture” in the database which evidently just sort of grew  – organically – like the fine sticky BC bud that made the architecture-free “one-database-fits-all model” seem like a good idea.

Now that the Lab claims to be moving out of the sticky bud phase, is there hope that main grid stability may arrive at some distant point in the future? This will become increasingly important as OpenSim alternatives to Second Life become viable options for those who do not need 80,000 of their best e-friends sharing a world.


What can the Lab do to counter OpenSim?


Expect to see the Lab offer small independent grids and argue that running a large unstable 70,000 player grid taught the Lab how to run a small stable 1000-10,000 user grid.

This may not be as crazy as it sounds – if the Lab treats the main grid as a sort of freeloader load-test perti dish. If the Lab believes real money is to be made on selling independent enterprise-ready ImmersiveWorkSpaces, all it would take is dropping Philip Linden’s dream of a single world by adopting an independent grid model.

But disconnected grids create a problem – how to move clothes and other content between the independent databases. Lag-tastic in-world stores are can be a less satisfying shopping experience than buying from a web site. What to do?

The final piece of the puzzle could be to incorporate a web-based storefront to sell clothes and other objects into both the main grid and various corporate worlds – with Linden Lab vetting the content for sale to keep out undesirables. This approach would simplify the inter-world export problem – just buy it from the web site – and avoid linking the corporate grid’s database to the unstable main grid database. Without disconnecting the corporate grids from the main database it is hard to see how Linden Lab can promise much in the way of stability anytime soon. Stability will be important because being unfamiliar with accepting game god abuse, corporate buyers will insist on service level agreements.

Given an LL-branded web storefront for clothes and accessories, corporate users would never need visit the main grid, yet would still be able to purchase and import virtual lifestyle accessories – such as the all-important sexy walk. The content creators would have an expanded market, and a viable corporate product might be possible. Maybe. If the rumors of Xstreet sale the Linden Lab are true, this could be exactly where the Lab is headed.

With M Linden now in charge, will Philip be forced stop walking on the wild side and serve the corporate market?

27 Responses to “SL Hits, Admits Performance Wall”

  1. Professor C

    Jan 20th, 2009

    Interestingly enough, this happened in previous years where the database could not handle the number of people coming in.

    To solve this issue and clean up the grid. All the Lab has to do is kill … Just KILL free accounts that have not logged in for 6 months. Get rid of all of them and the servers that are obscenely overloaded with keeping track of accounts will have a major burden of relief.

    Do it and be done with it. If the people that have those accounts want to cry about losing their shit well they can go fuck themselves. They weren’t paying for it. Let them deal with it. If they spent a lot of money in world it couldn’t have meant TOO much to them they they abandon it for half a year.

  2. Dr. Rogan

    Jan 21st, 2009

    Rumour CONFIRMED. Go look at http://www.xstreet.com and read the top bar on the site

  3. barak

    Jan 21st, 2009

    “the time has come to give up childish ways”
    Linden Lab’s way.

    Obama Linden

  4. Prokofy Neva

    Jan 21st, 2009

    Pixeleen shows a remarkable grasp of software, database, and architectural issues.

    I think given this grisly picture, the smart thing to do is for corporate customers to go over to Cobalt.

  5. bc bud guy

    Jan 21st, 2009

    BC Bud is the best fo’sho’. Don’t knock it till you try it. Oh wait, you were working at University of Toronto not too long ago. Doubtless you got some dank weed while you were there, professor. If not, well then you’re just fuckin’ stupid.

  6. Don’t they call themselves the love machine? And they’re based out of Cali where possession of under an ounce is decriminalized? And you think Philip Rosedale himself isn’t smoking the pot? Everyone at LL smokes weed. I got the pictures to prove it. If you want, I’ll send them to you. It’s easily verifiable.

  7. gone and gone

    Jan 21st, 2009

    The rumours are true. SLxchange and ONrez are goners. email addresses that are associated with SL avatar names at those 2 places will now be in the hands of the evil LL. Hope your email there wasn’t associated with some old avatar that got kicked off of SL in the past… because now LL will know and soon your new avatar will be: POOF, gone when they cross reference the data and find out that you are a comebacker sneaking Lindens through the SLX…

    And the forums over there at SLX will be GONE too, you can bet on that. Think LL is going to let the schlubs continue to spout off over there with their usual truths like: “Linden Labs is so lame that they make the Green Lantern Core look smart” or “God I would like to shove a PN device up Linden Lab’s ass and watch it go but their Goatse is so bad it would just fall out and not do anything” stuff like that.

    Ok so thats a slight exageration but not much. There is going to be a forums VOID created however. And if some WELL KNOWN Second Life related site out there were to ADD FORUMS to their operation, they might get the traffic. The traffic will begin to flow even before LL shuts the SLX forums down. People will leave SLX just because its under BORG control now. A new place needs to be found…

    Oh yeah, Second Life is dying. The only hope they have now is to split up SL into separate grids. To get the number of users on each grid lowered. Maybe a Army grid and a Space grid, and a Slut/Sex/Gorean grid, a Shopping and SRSBSNS grid, and way off in a corner surrounded by inpassable mounds of lumpy kitty litter, a Furry grid.

    OR…. DUMP ALL BOTS that would give them about 20,0000 avi worth of breathing room.

    Or do both…

    I think at this point we need more speculation on just exactly WHAT IS HAPPENING AT LINDEN LABS cash flow and business model wise??

    This stuff with them now taking SLX is bad I think…

  8. Melissa Yeuxdoux

    Jan 21st, 2009

    Doesn’t sound like much of a solution–you get to buy everything as many times as there are fiefdoms you wish to visit. There had better be some way that one’s inventory can go with one.

  9. Hagar Qinan

    Jan 21st, 2009

    I received an anouncement by Xstreet just today. Apparently it IS true that LL bought it.

  10. Dr. Horrible

    Jan 21st, 2009

    LL has already got a “behind the firewall” solution working for the corporates, and I expect universities to be next.

    Farewell, Wild West Era and Rosedale’s vision of “one big grid.” Hello, M. Linden Era: pocket universes and “who invited YOU?”

    The college that stupidly hired Dr. Horrible (who speaks of self in Dole-style third person) is too light in its use of SL to consider a linked and restricted grid. But a lot of schools in the edu-archipelago cannot cope with grid crashes as they have dozens of classes using SL on a regular basis.

    They’ll plug along when the main grid is down, and students and faculty will go to the Mainland’s Wild West or elsewhere on the main grid when a project demands it.

    Boring, yes. Inevitable, also yes.

  11. Marc Woebegone

    Jan 21st, 2009

    It’s all about control an suppression. Control what goes in and what is useable and visible. Users created an economy for the exclusive benefit of the owners.

  12. LUL

    Jan 21st, 2009

    Um, Prok…

    She barely said anything technical at all in this piece and what she did say was quoted from someone else. Pix is too stupid to be Mark McCahill.

  13. Sigmund Leominster

    Jan 21st, 2009

    So Linden Lab acquires Xstreet and OnRez. Good idea. Provided they did their “due diligence,” it’s a not-unusual business move. One relatively easy way to grow a company is through mergers and acquisitions, and if you have the capital, go for it. Doubtless some will construe this as more ‘evidence’ for (virtual) world domination by the Evil Empire – but that’s paranoia. Anyone who’s grown their real world companies by M&A’s knows that the decision is less about “empire building” than increasing profits. Nothing wrong with that – it’s called Capitalism and works pretty good (present corrupt banking practices and stupid sub-prime loans excepted).

    That’s enough – no extended essay, just a comment.

  14. Anon-y-mouse

    Jan 21st, 2009

    I was with Xstreet specifically because it wasn’t Linden. Does anybody know of any independent choices still available?

  15. Alazarin

    Jan 21st, 2009

    Yes, X-Street and OnRez have been assimiliated.

    One suggestion that was overlooked is the possible creation of a ‘bot-only’ grid where bot runners could field massive armies of bots, pitch them in combat against other bot armies and sell admission tickets to regular avatars who could log in and watch all the destructive mayhem. A new form of entertainment perhaps?

  16. Murphy Alderson

    Jan 21st, 2009

    With the amount of traffic and data that flows through their database, LL really needs to look at a db model that will support what they have now, and be scalable for the future.

    In addition, I’d certainly not complain if word came out they were dropping mySQL for a commercial and well supported database like Oracle or MSSQL.

    @Anon…yes, there is apez.biz – I don’t know that much about them, but know they’re out there. What concerns me about the take-over, is the Lindens are saying nothing (despite numerous posts in their forums) regarding whether the XStreet currency exchange will remain intact and, in fact, a better deal for the users (than the LindeX).

  17. Alyx Stoklitsky

    Jan 21st, 2009

    lol @ Prok.

    Does it really fucking matter who the hell Pixeleen is? And as for a showing a ‘remarkable grasp’, maybe such basic terms seem ‘remarkable’ to a grumpy old porch-sitter who jumps at shadows, but to me it seemed pretty average for an internet user under 50.

    Now, back to my original point: Does it really fucking matter who the hell Pixeleen is?

    I mean seriously: WHO FUCKING CARES?

    Do you want to know who I think Pixeleen is? Some unimportant middle-aged man on an average wage doing some average vaguely uninteresting job and has a vaguely uninteresting life.

    Woo fucking yay, you neurotic, early-onsett alzheimers suffering, thoroughly unpleasant, grumpy, jaded, cyncical yet tedious, stubborn old bitch.

  18. derp

    Jan 21st, 2009

    Murphy, stfu. Google uses MySQL. Yahoo uses MySQL. Everyone and their goddamn dog uses MySQL. MSSQL is a gigantic, horrid pile of shit that costs an arm and a leg. Oracle is even worse. How do I know? I used them both in an enterprise network. The only good thing going for MSSQL is that it’s easier to integrate with .NET apps, and by easier I mean you don’t have to download an external library in order to get it working.

  19. mootykips

    Jan 21st, 2009

    MR GORBACHEV, TEAR DOWN THIS WALL

  20. Orion Pseudo (Shamroy on SL)

    Jan 21st, 2009

    You mark my words on this one. Yet even more proof that LL will soon go IPO. Next thing to go will be all the adult content as they further try to clean up their image and build themselves up as “marketable” by acquiring more companies that they can pass off as products.

    The SL that we all know and love – that perverse little place where we can all go do discard our real life selves and be what we’re not, that cool place where we could let our creativity run wild will soon be under the control of the yuppy skumwads over on Wall Street. Censored by the Hitler-esque tactics of burning away blog posts and threatening users with the ban hammer for speaking their minds and expressing themselves.

    All those hours we all spent building and creating, making their world for them out of our own pocketbooks for what? So they can sell it off meanwhile Mark Klingon or whatever the fuck his name is will make a fortune off OUR HARD WORK AND CREATIVITY!

    FUCK YOU LINDEN LABS! You can take your crumbling little world that’s busting at the seems with FUCKING BOTS and MINDLESS NOOBS with IQs less then that of a rat. You can take your private little monopoly of content distributors, AND SHOVE IT UP YOUR FUGLY LITTLE PRIM BUTTS!

  21. Orion Pseudo (Shamroy on SL)

    Jan 21st, 2009

    Speaking of content steralization! Now they’re planning on merging the teen grid with the adult grid?

    http://forums.secondlife.com/showthread.php?t=303396&page=1&pp=30&highlight=teen

  22. VirtuallyTaken

    Jan 21st, 2009

    Ok you have to get it by now. If LL is going to do what i expect them to do . why the heck not. As far as a business model. Well it would work. As a world social network no. Unless they continue a national social open network possibly. I see them creating this whole molecular network or grids and selling them off ok.

    The XStreet Exchange ehy bought well will be their avenue to distribute goods to those cross grids and control the currency. The OnRez issue well that to me says all their years of copying their customers content has paid off. They will now be able to sell that along with their individual grids to private institutions like they wanted to and isolate them for all purposes needed in a commercial or educational field. It is key for privacy and protection. i see it.

    so yes everyone we were the lab rats for their beta worlds with out reward. Don’t expect a thank you it isn’t in the paper work. Not needed. We were the experiment and your don’t with your job. they will deploy the third execution of their business plan as well not so scheduled but it came about.

    I wish them well i hope we can all cross grid and it is a success and that we can all use the newly aquired exchange to aquire new goods for which we will need if we r not thrifty as builders.

    If you can afford to pick up on the IPO if that is to happen lol which don’t be to surprised. It just may be a real thing. Dreams do come true lol for those who live in the real world.

    All i can say is that who ever participated these past few years in this phase of second life and help develop so much to their content. I wish you well. Cool Huh!?? Great??? Huh!!

    Nice people they r huh?
    well that’s business folks. Be sure to invest its the future!!!!

    Oh keep your eyes out for your products you may just find them in another world but not your own!

    Sophia~

    The cost of doing business I guess~ I had a a vision lol…but I wont tell anyone just in case~Thanks Mr.M U did a great job.
    Lets hope the future grids can remain stable and secure. I hope so. Great Idea i know takes time but they should have kept it free. As far as I’m concerned I didn’t pay for them to flee.

    sophia~

    and Torley your the best out of the lot of them!!!! Never commented on you but your the best!Hope you move on to better and greater things!

  23. What next????

    Jan 21st, 2009

    ok how many times do i have to write in here. forget it its not important the tank sunk im over it and if it goes ipo i’ll watch. The whole thing stinks to high heaven and I’m now on a mission to planet earth.

  24. Sigmund Leominster

    Jan 22nd, 2009

    Although not an avowed aim of Linden Lab, I note that this more to have more control over the handing of a virtual economy might well be a marketable asset to other virtual worlds. Sticking my neck out a little further, LL may well be positioning itself to operate as a virtual “Central Bank.”

    Imagine you have created a new virtual world and want to encourage commerce (which is also the same as saying “want to make a profit yourself”). What if you could have all the financial transactions handled by a third-party, to whom you pay a percentage on transactions. That would have to be worth thinking about.

    Banking is simply a business like any other. As more virtual worlds appear, they all need to consider how to handle commercial exchange. Rather than have idiosyncratic currencies, imagine having a single currency. That’s a nice thing. You might not call it the Linden but whatever it is called, I’d sure like to be the one making it all happen and taking 0.1% of all transactions.

    And if I’d like that, you can bet Linden Lab would too.

  25. Sophia

    Jan 24th, 2009

    Yesterday I received a call from a dear friend. I’ve known her since I was 5 years old and hated to see her move a few years back to Florida. Last year I suggested she join Second life so we could enjoy some laughs or maybe even help her start a business. She tried to join SL last January but for some reason she never made a touchdown. she couldn’t get on and she could not get in touch with LL.

    she had given them all her information and was going to have a premium account. whoop -t- doo . Well she said the heck with it and gave up. I know she tried because we were on the phone trying to figure out what the heck happened and why she couldn’t get in.

    Just yesterday I got an email from Second Life that i had been given 500L for refering my friend so and so. MY FRIEND!!! HAHA she never got in! and that was a year ago. Well she called yesterday to tell me that the swines tried to bill her credit card for her membership. Hello!!! What membership she never got in and her account has been inactive for a year!

    Id like to say I find this very odd but I can’t. nothing surprises me anymore with them. They couldn’t get her cash lol because she closed the credit card anyway. How many other people are they going after that haven’t been on for over a year? But then again she wasn’t even ever able to ever log in.

    I know things are tuff over there in the land of Linden but this is crossing the line. Thanks for the 500L even though its a little late. I’d like to know just how many people they are trying to bill that haven’t been on and that don’t look at their bills often. Bunch of crooks. And the service stinks.

    Hey also someone mentioned in a reply about the Xstreet purchase and how LL could use it as a way to distribute goods to other grids. I had the same thoughts. It could just possibly be the way to distribute goods to other grids.

    I don’t know what this year holds for me and sl but it isn’t looking to good right now and im sure others feel the same.
    they Beta get going or I’m going to find another playground.

    I worked for companies that were on the cutting edge of technology yesterday lol and they didn’t take this long to launch a decent product. This is getting a bit humiliating I would say to those who work there. Good luck to them lol.

    Why don’t they go thru all the accounts and see who hasn’t logged in in over a year and send a sweet message asking them why? Instead of billing them first. It could be an easy task for one of their programmers to do.

    Why the heck would they try and bill someone who never even made it in? Seems to be a bit of a scam to me. Why don’t they just check the status of the account before billing. Send a friendly reminder asking something like do you still want to be a member or not lol before we bill you. Billing people is just wrong to me because so many people could possibly miss it and LL would get the cash. Sounds like a Luke (you know who lol) email to me asking if they don’t replay to this email he gets to keep their Lindens.

    Dear “My friends avatar name,”

    This email is notification that we were unable to bill your Second Life account on January 22, 2009. Please review your Second Life membership and billing information, and make sure everything is up-to-date. Some of the more common mistakes are:

    * Your credit card may have been declined for insufficient funds.
    * Your credit card may be expired.
    * You may have entered an incorrect expiration date for your credit card.
    * Your billing address may not match the address on your credit card.

    Additionally, your bank or credit card company may have flagged the charges for review. If none of the above are true and you suspect this to be the case, please contact them to authorize the charges from Linden Lab.

    If you are using Paypal, check your Paypal account’s verification and backing. Your Paypal account must be verified and backed by a funding source other than a credit card. (We are unable to accept Paypal accounts backed by credit cards.) Make sure the Merchant Agreement with Linden Lab is current, and make sure you haven’t exceeded your Paypal transfer limits.

    If everything looks correct, you may want to contact your credit card issuer (or Paypal as appropriate) to identify the problem.

    Don’t worry; this billing failure will not prevent you from logging on with your Second Life account. We will simply try to bill your overdue balance again the next day. If we are unable to collect the amount due within seven (7) days, your account will be suspended pending payment for an additional thirty (30) days. During that time, you will not be able to log in to Second Life. Alt accounts may be placed into administrative hold and you may be logged out of active sessions during this time as well. Failure to resolve this billing issue by the end of the probation period may result in the cancellation of your account. If you have verified the accuracy of your billing information, and are not aware of other outstanding issues that would prevent billing, please contact us through the support portal (https://secondlife.com/support ) as soon as possible.

    To review or modify your account details, or to change or cancel your Second Life membership, visit https://secondlife.com/account.

    Best Wishes,

    Linden Lab and the Second Life Team
    http://www.secondlife.com
    Linden Lab
    945 Battery Street
    San Francisco, CA 94111

  26. Sophia~

    Jan 24th, 2009

    I just loved that email they sent. Dot YOUR I’s and cross YOUR T’s to make sure YOU or YOUR bank didn’t do anything wrong that prevented us from billing YOUR credit card. hehe Oh and the…

    “Don’t worry; this billing failure will not prevent you from logging on with your Second Life account.”

    Im laughing because it follows with this statement…

    “We will simply try to bill your overdue balance again the next day. If we are unable to collect the amount due within seven (7) days, your account will be suspended pending payment for an additional thirty (30) days.”

    Wouldn’t it have just been less painful for THEM to have just inquired first as to whether or not THEY /YOU wanted to remain a member? I mean they have to have some kind of database, search engine that spits out some kind of information regarding active and non-active accounts..come on.

    Why don’t they just pull a Luke. Send out a mass mailing to all their customers. A survey of some kind asking how often do you log into second life or are you still or do you want to remain a resident of second life or something like why don’t you like us anymore kind of question.

    I think it would save some time and money for many people involved in the transaction request. The bank or credit card company wouldn’t have to decline payment and the x-resident wouldn’t have to respond to their bank or what ever payment method they chose at the time that forget it thank you.

    I mean isn’t it a good time now for Linden Labs to go out of their way now to just send out a mass email to all their clients / residents and straight out ask them to reply lol (ohhh big long list of emails to read thru hehe maybe not)but then again they could sort them as a yes or no to get a real figure as to home many people really are on Second life. I’d like to know I guess what the real numbers are.

    hehe is it me or is there something really smelly there hehe. Oh boy they can be very entertaining.

  27. Archie

    Jan 28th, 2009

    WE ALL NOTICED

    I’m glad the lab did too, about ruddy time you boys!

Leave a Reply