The Road To Scalability

by Pixeleen Mistral on 11/05/07 at 7:19 pm

Garrulous Gwyneth Llewelyn speaks out for less loquacious Lindens

by Onder Skall

Gwen
Llewelyn threatens luddites with 5000-word Linden love-fest

Project Open Letter, the controversies surrounding it’s writing, and the ensuing fiasco of a town hall, all ended with a giant question-mark. We still didn’t know what the strategy was for making things better, how many of this “69%” of employees were working on existing bugs or bugs in future feature rollouts, or what in the world all of these vague references to “architecture” meant. My best guess was that a “2.0″ version of Second Life was in the works; a fundamental shift in the way the grid operates.

As it turns out, Gwyneth Llewelyn’s most recent blogpost echoes this sentiment and gets into the details we’ve all been asking for. She actually uses the term “SL 2.0″ in the post and reveals a lot about the behind-the-scenes thinking, although it’s hard to tell where she’s getting this information from. This may be from personal relationships with various Lindens, but it’s never really stated.

Gwyneth’s take on these things is pretty specific, characterizing those who’d rather grid stability than more bling as “Luddites”. Her stance is decidedly pro-Linden, and borders on the sycophantic. Still, if she wants to love the Lindens and how they do things, she absolutely has a right to. [quid pro quo for inside info? - the Editrix]. What makes it worth reading for the rest of us is simply this: it’s hands-down the clearest set of details yet on what in the world is going on here. This far outstrips anything the official Second Life blog has offered.

In her post she details an upcoming white paper from Cory Linden on a server-side implementation of “SL 2.0″:

“We, from this side of the screen, will not notice any real difference on how SL operates – the client will not be affected -except that suddenly IMs will work, teleports won’t fail, inventory won’t be lost, and our L$ account will always display the correct amount.”

That’s what we’ve been waiting to hear!

This, largely, has to do with re-thinking the fundamentals of how the servers operate in the first place. It’s not just a question of throwing more hardware at the problem, but in changing the way the hardware processes things in the first place.

While the server simulators are self-contained and perfectly capable of handling the load demanded of them individually, the grid as a whole doesn’t scale well. The solution would be multiple grids, but the experiment with co-location facilities in Texas and California have demonstrated many of the hurdles there. This is where the major shift is going to have to be: the code that moves an avatar (along with everything they “own”) from one grid to another needs a complete overhaul. This overhaul is being done, and is the major focus of design efforts. We’re not talking about killing bugs here. We’re talking about rebuilding the machine.

Gwyneth also makes the point here that nobody has ever attempted something like moving an avatar from one grid to another. She has a point: moving characters from one server to another in a normal MMO is often either forbidden or involves a rather cumbersome process. We’re asking for it to be seamless, which may be possible, but it’s also unprecedented.

She also gets into many upcoming features, quotes a percentage of 72% for developers working on fixing bugs, and – oh yeah – Linden Lab is profitable now. Guess that means Second Life won’t die any time soon.

You know, I have to be honest: I loved this post. I mean, I didn’t always agree with her complete fandom of all things Linden, but this was real content! These were real answers, real information, and the first indication that Linden Lab may have a plan and was preparing to enact it! This is what the Town Hall should have been. I can’t imagine why Cory Linden didn’t get into his upcoming white paper or the details of how they were looking at grid stability, but if he had, I imagine that we would all have taken it a lot easier on LL.

74 Responses to “The Road To Scalability”

  1. Accasbel Barrymore

    May 14th, 2007

    Gwyneth: ]] Sarcasm also seems to have been outdated recently. Really, claiming that “Gwyneth appears to be espousing a metaverse that embodies(?) intolerance.” is either total sarcasm or failing to spot my own sarcasm (your pick). [[

    I pick ‘none of the above’. Writing that Gwyneth *appears* ...... was neither sarcasm nor a failure to detect it.
    The fact that you felt that you had to clarify after the event illustrates that even you accept that some proportion of readers, significant to care about, might take the view that you *appeared* to be .......
    I used the term "appeares" very deliberately in my original comment. You appear to have missed it. ;)

    ]] Halting the flow of progress, preventing researchers and engineers to deploy things for the next stage of SL, not allowing them to attack the *core* issues plaguing SL, and instead making them focus on minor, irritating details, while postponing what *needs* to be changed (and no, it’s not Group IMs…) — that is what in essence I call a “Luddite mindset” [[

    Yes indeed.! Death to the Luddites, and to the immersionist horse avatars on which they rode into town. (humour! humour!)
    However, unless one defines the absence of new features as "*core* issues plaguing SL", then the Open Letter comes nowhere near to the Luddite essence.
    There would be some justification for a luddite-flavour labelling if the Open Letter were produced in an environment where LL's plans and timescales were known.

    I think that the Open Letter was concerned with the disconnect between what SL is advertised to be and what the experience actually is. The disconnect would be particularly noticeable to those who are paying money for the experience.

    ]] Cory has a plan (I haven’t seen it published yet, but it will be), but that plan is very painful for us all. It means *months* — not days, not even weeks — of breaking the core servers apart and redeploy something entirely different. [[
    If this is accurate, then the disconnect is about to become something that the term “disconnect” does not adequately describe.

    If the PR and the SL website change now to say things like “This is a bit broken right now, and it’s going to be very broken and painful for the next few months” that might alter the feeling of disconnect. Admittedly it might alter a few other things as well.
    A clarification of that last bit: – I don’t seriously expect that to happen. In RL I was running funny face animations when I typed that.

    Reengineering the underlying tech is a no-brainer. It’s not a commercial secret.
    If LL were more open about their plans for stability in a growth situation, about the timescales and the pain, then maybe people would be more accepting of short-term problems.
    Unfortunately, what we get is LL throttling the communication paths between users and themselves. At the same time they unveil plans that do not appear to have been well thought out.
    “Trust us”
    “Why?”
    “Because you have no choice. Resistance is futile.”
    “OK. We trust you”.

  2. Accasbel Barrymore

    May 14th, 2007

    ]] Reengineering the underlying tech is a no-brainer. It’s not a commercial secret. [[

    The second sentence should clarify the meaning of the first. But, just in case of anyone reading with an ‘ambush’ mentality —
    The need to reengineer the underlying tech….. etc.

    I have no illusions that the actual process of reengineering is anything other than a brain-burner.

  3. Prokofy Neva

    May 14th, 2007

    No, Gwyn, the FIC is as real as it ever was — even more so. It now consists of companies that have actually signed NDAs (as it always had, by the way). It consists of people on the special internal dev list. It consists of those in SL Views and those who regularly go to the actual offices in SF or the other new offices and hob-nob with Lindens. It consists of the IRC channel or the open source mailing list or certain more inside-than-though SLED members. There is an internal community of FIC that is even more institutionalized than ever between “us” and “them” as the Lindens sequester themselves more and more and disassociate themselves from “the masses”.

    The idea that FICyness is a “skill” involving a “quest” of putting together a “puzzle” of technical, political, and social cues and clues is a quaint MMORPG holdover — but in fact it’s more about who gets what inside info. It’s always about inside info. If you flatter and cultivate Lindens assiduously, they tell you stuff, like any closed government or agency trying to keep secrets. They have their pets, in the media, in the “fan base” and among the corporations. It is more the case than ever as the have more motivation than ever to keep their cards very close to the chest.

    Re: “with full access to chat logs on their sims”

    Do you have evidence of this? This is something I’ve asked about, because I’ve noted that given how the AR system under the Lindens relied on chat logs, and even used to have a system to check off the literal attachment of them (I suppose they can’t have read all of them) it stands to reason that they’d have to do the same thing for island owners.

    But that raises terrible problems. I certainly don’t think it’s right for me, as an island owner, to have all my tenants’ chat served up to me. That opens the door for terrible abuse across the grid. And in fact, when I asked this very question of Robin at her office hour, indicating that they would have to a) decriminalize chat distribution inworld and b) provide island owners with chat if they actually thought they could do the job of handling and adjudicating ARs, she said no, island owners would not get a chat log, i.e. a master chat log of that sim (if such a thing exists).

    Have you heard something different since then? Can we get the facts on this? My advocacy on this amounts to only one thing: decriminalize chat distribution, as it is needed for the resident population to govern and police itself, end of story. I certainly don’t advocate putting master chatlogs into the hands of land barons, however.

    I’m not understanding all the preciousness about Cory’s architecture, like it’s the Taj Mahal and we have to tip toe around on glass. Nobody is stopping him from doing a damn thing, Gwyn. Asking that the Lindens not disrupt and cease operation of the existing service in order to create the better-working one isn’t some sort of excessive Luddite demand as we’ve all said. If you’re in a plane going across the Atlantic and the pilot tells you the de-icing didn’t work so well, you want him either to get that thing de-iced somehow anyway or land in Newfoundland QUICK, not attempt the journey over the pond with a propeller falling off and the plane crashing. So which is it?

    Joe Miller was the one, not Cory, talking about swapping the engine out of the plane while it was at 30,000 feet. Not a pleasant thought. It’s not some pet project of Cory’s that everyone else is allied against, it’s just the Lab’s. They are hive mind, pretty much, Gwyn, like all cults.

    And…will it work? Well, we’re not allowed to ask that, are we?

    At least, I’m not, after being booted from the town hall for pointedly challenging their bogus concept of a “first land shortage” being about “server supply” when we all know it was about “political decisions” to opt to sell islands to many on a waiting list for more money, than give away servers for free to newbies getting their free 512s. Please.

    I’m not, after being booted from the Linden blog after pointedly asking in the pre-town-hall questions to Cory whether his faith in libsecondlife was warranted, given the griefers, gridcrashers, and very mixed reviews among their midst.

    Booted from a town hall, and from a blog, for asking normal customer questions. Not for being a grid-crasher myself. And this is a company that you think can swap an engine out at 30,000 feet? You can never do stuff like that by suppressing critics. If they are capable of suppresing my *legitimate and legitimately-expressed criticism* on their blogs or in their town hall, then what do they do to those who raise objections in their own midst, in private? One has to wonder…

  4. Seb

    May 14th, 2007

    God has a plan and the bible unfolds this wonderfull plan through a message of prophecy and if the lord does return in comin 7 days.

    Just seemed suiting after all these long posts.

  5. Cocoanut Koala

    May 14th, 2007

    “Halting the flow of progress, preventing researchers and engineers to deploy things for the next stage of SL, not allowing them to attack the *core* issues plaguing SL, and instead making them focus on minor, irritating details, while postponing what *needs* to be changed (and no, it’s not Group IMs…) — that is what in essence I call a ‘Luddite mindset’:”

    This makes me really wonder just how much time you are in world, or try to do in world things.

    Forgive me, but I am not familiar with your business, if any, or of your activities in SL.

    But it seems to me that the features you blow off as not worth fixing, such as group IM’s, are ones which are vital (and I do mean VITAL) to the people who run businesses in SL, or even just try to have fun there.

    And it is those people who pay the tier that keeps SL going.

    I wonder what other features, besides group IM’s, you figure are trivial and expendible?

    How many things are we supposed to be crippled by? And for how long?

    And what for?

    —–

    “It’s *because* LL actually worries about their *future* residents that they are so eager to take this next step, even if it means that the *current* residents have to suffer… for a while.”

    —–

    See, this I never bought for a minute, ever. This is the worst line of thinking possible. ANY plan or mode of functioning which always has current people suffering for the sake of future ones is a lousy plan, not least because those future people somehow NEVER DO COME.

    We are ALWAYS suffering for some nebulous future people. Ditto with injustices, or anything which trods over individuals in the name of some future conglomerate greatness. You can’t give people crap today and expect everyone to understand that SOMEDAY they will get something worthwhile. Just doesn’t work that way.

    For one thing, they all leave. And tell their friends not to come.

    For another, if a philosophy allows individuals to be sacrificed now, individuals will always be sacrificed, and a worthwile future never does come. And that is because there is always something to be desired, and since it’s okay to sacrifice people, it keeps happening.

    The race is always won, seems to me, by the conscientious, who don’t trod all over and kill off all their current people in the hopes of getting future, “better” people; you know, the “real” people, as if the ones one they already have never quite count.

    Moreover, you will never convince me that this plan of (presumably) Cory’s is the only way, the only plan possible, and that it simply MUST be implemented full speed ahead, at any and at all costs.

    And even if it were, you shouldn’t be surprised if people don’t feel like sacrificing their money, their businesses, and their fun, to someone else’s cause.

    You know, it’s never the case that there is only one way to accomplish something, and only one person – one super-special visionary type individual – who can achieve the job at hand, and that one must give him full rein.

    Thinking there is only one thing, and/or only one person who can possibly lead us into the future, and we all must sacrifice ourselves for it, is kind of fanatical. And as such, it is not clear thinking or planning.

    In any case, breaking everything and letting it go unfixed is simply not a way to have a future anything.

    You call others Luddites for wanting things to work. But I think your viewpoint is much more dangerous for the future of SL, and I hope more sensible LL minds prevail.

    coco

  6. Gwyneth Llewelyn

    May 14th, 2007

    Prokofy; Again, I can’t comment on what I have no idea if it’s true or not — companies getting privileged information for signing NDAs. I’m pretty sure that Integrity must have signed NDAs and contracts; it would be impossible to establish a business relationship that gets access to crucial elements in LL’s database otherwise. Cory hints on the last Town Hall that some features/bugs are being dealt with “a studio that is handling this area” (he repeats this twice). I have no idea if a “studio” is their internal name for a “developer team” or if is an outsourced company — they have announced often the willingness to outsource some work (see Philip’s last entry on the Official Linden Blog, as well as some information on the lindenlab.com page). In those cases, since at least these companies need access to the internal bug tracking tool (the “internal Jira” as opposed to the public Jira), I can very well imagine that they’ll have signed NDAs with those, too. And finally, the company that developed the integration of voice with SL will have done the same.

    So while obviously none of those companies are publishing what kinds of NDAs they might or might not have signed, it’s pretty clear that *some* companies might, indeed, have forged a close business relationship with LL — like all real world companies do when partnering together on a common project — and it would be no surprise if they would get access to some sensitive data, and/or disclosure of private information. Thus, one cannot wave away the notion that at least some non-Linden employees do, indeed, have access to privileged information.

    However, saying that these companies now “control the feature of SL” is a slight exaggeration; this is like saying that “an inner core of Linden employees influence Philip’s decisions” — which is almost certainly *true* and a tautology: work inside LL, and obviously you’ll have access to privileged information on a “need-to-know” basis. Work *with* Linden Lab on an outsourced project, and the same will happen, too.

    I don’t exactly understand what the point is.

    As for “secret mailing lists”, IRC channels, and the like, well, the ones you’ve heard about (and that others have mentioned) are definitely open to join to *anyone*. IRC, for instance, has no restrictions to join, but to talk to the Lindens there means being online 24h/day for spotting one of them in a talkative mode for a few minutes. There is a huge difference between occasional chit-chat and a conspiracy to change the world; it’s the same thing as imagining that a Linden employee takes a short pause for work and goes for a coffee with a fellow co-worker, and talks about what they’re doing in LL in a quiet corner — but don’t suspect that the coffee shop owner is actually a SL resident and is taking notes of what they say (and blogs on it later, as a “source of privileged information”). I mean, LL’s not a top security defense contractor developing nuclear warheads, but just a smallish software development house. People talk — on the streets, on the bus, on friend’s parties — and sometimes they simply talk too much. That can happen on IRC or on an in-world meeting. Allegedly, Reuben talked too much on a public interview. And there are some cases here and there. LL’s employees are not screened for working on a top secret security installation, but just for a regular, open-minded software house.

    Of course, I can imagine that people really, really paying attention (going for a coffee at the same time that the LL employees; watching the IRC channels; going to the same parties in RL and waiting for the opportunity to get in touch with a drunken Linden…) might get some bits and pieces of what goes inside “the Lab”, but this is hardly a “conspiracy”…

    On the much more polemic issue of giving estate owners the power to go through chat logs occurring on their estates in order to “better deal with abuse reports”, this was mentioned on one of Robin’s Office Hours. I couldn’t find a chat transcript of that session, but Torley and Tao mention it on Tao’s blog: http://taotakashi.wordpress.com/2007/03/22/the-new-linden-lab-communication-practice/

    I don’t have any idea what the “master chatlog” is, but it was hinted that all chat inside a sim is kept in storage for two weeks, mostly to be able to deal with any request from the authorities, and that estate owners would be able to access it (for their own estates) in the future.

    Personally, I dislike that “power”, but I understand that my notion of dislike is going soon to be outdated when everybody will have access to their own sim server software — and obviously be able to make chat logs of whatever is said there — so apparently LL will want to enable the same for Linden-owned servers.

    Mind you, private IMs go through a separate system. For now.

    As for “supression of LL’s desire to do whatever they please”… well… it’s a tricky question, since on one hand, they have all the *legal* means to do it (the ToS), but they also have to manage it well enough on the RL press, who are paying close attention to what LL is or isn’t doing. Among the 4000 or so signatories of the Open Letter, who knows how many journalists are there? A coordinated public campaign in the world-wide media against LL’s policies and attitudes can force them to develop a mediocre strategy to please the masses while suppressing a good plan to deal with all major outstanding issues — simply because the mediocre plan is easy to explain, the Big Change is almost impossible, except for a insignificantly small number of highly specialised network engineers, IT architects, and system administrators.

    And while I certainly ignore and underestimate “The Power of the FIC”, I do fear and overestimate “The Power of the Press” :) The latter have cost me my job in the past ;) while the former are “mostly harmless”.

    Coco, your objection is sadly one that is not easy to answer. Let me give you another counter-example. Someone has a slight pain in the chest and asks the doctor for a simple painkiller that can be bought at any chemist’s. The doctor makes a thorough diagnosis, runs a few tests, asks for more blood samples, and tells you that you have to undergo major surgery, a heart transplant, and replacing half of your arteries, and that all this is critical to do in the next few weeks, or else you’ll die from internal rupturing of all your circulatory system. Shocked, you ask why the doctor can’t just give you a painkiller and have it over; why the fuss? A *good* doctor will explain everything patiently in terms and words that even you can understand, but some will enter into complex jargon and obscure clinical references, show you textbook cases, point you to Medline for more studies on the subject (not Wikipedia, which is usually easy to read), and suggest you attend a few seminars for multiple-MDs to understand your clinical situation — a “rare condition that only affects one in a hundred million people” — and which is only fully understandable by someone who has studied medical science for over a decade.

    Instead, a good doctor will just say: “Trust me. I know what’s best for you and I know what I’m suggesting, even if I can’t explain in *full detail* what I’m going to do.”

    Of course, many patients will have a different opinion than their doctor, buy the painkiller anyway, feel much better for a few weeks or even months, shake their heads at the silly medical profession who don’t have a clue on what they’re saying — and die from internal bleeding on a massive collapse of all circulatory functions a few months later.

    SL is currently at a similar stage. “Fixing Group IMs” is just buying a painkiller for a “minor nuisance”, when the whole system is truly rotten at the core. Forcing LL to spend developer hours to fix a tiny aspect — but ignoring the central issues (which affect Group IMs anyway…) — means postponing a solution, by giving placebo painkillers to please the masses. At some stage, you need to give up on the placebo and focus on treatment — this is was ultimately someone with a cronic heart condition will learn, after taking years of ‘alternative medicine’ to give them comfort and pain relief when “doctors refuse them painkillers”.

    So, back to my previous words… the issue is not fixing Group IMs… he issue is really being able to thoroughly explain, without technical jargon that only a couple of thousand people in the whole world can truly understand, what they’re attempting to do… it is about patiently explaining that *most* of the “critical issues” (teleports failing, attachments disappearing, inventory disappearing, L$ transactions failing, broken Group IMs, prims not rezzing…) are all tied to a centralised system that has grown beyond its age and usefulness, and it needs to be replaced. As quickly as possible.

    Delaying the heart transplant and demanding for simple painkillers right now is just postponing the inevitable. The Open Letter embodies the wisdom of the crowd that prefers painkillers and a short-term solution (which certainly exists) and has no patience to wait (or doesn’t even want) a heart transplant — because with painkillers they seem immediate relief, while open surgery for several hours and removing all your innards… seems overkill… after all, it’s just a simple Group IM… right? They worked two weeks ago… why can’t they work again as they used to?

    I’m sorry if I can’t explain it better myself; I wish I could. All my professional life I was often stuck with the difficult concept that “everybody has opinions” on how things should be run (and all opinions are valid), but only *qualified opinions* will be the ones that are actually *useful*, and the number of qualified opinions is notably scarce in every area that requires intense knowledge and professional training. Everyone can self-diagnose a chest pain and buy a painkiller after consulting an “opinion” from a friend, and enjoy some relief for a while. It takes a professional doctor with a decade of experience to correctly diagnose a chronic heart failure requiring a transplant and intense surgery. It also takes a good doctor, with charisma, confidence, and a good attitude to earn the trust of the patient, so that they will willingly accept the doctor’s *qualified* opinion to submit to major surgery, instead of rushing out of the door and asking the first person on the street on what painkiller they should buy to relieve their pain.

    I agree that there are obviously many different ways of doing the same thing (the analogy is to ask several doctors about their qualified opinion, and many might suggest a different treatment for the same condition). However, it’s not easy to explain that in the context of Second Life’s grid, there are not many textbook cases for study — nobody ever attempted to do something like this at this scale before (several virtual worlds do exist, but none work like SL, and the ones that do, have an order of magnitude of fewer users). A doctor that handles their first new disease/illness, when there is no documentation on how to cure it, will need to rely upon their past experience knowledge to apply the treatment that will probably work best. Having no previous cases to analyse and study, any doctor will give a similar answer: “let’s try this and see if it works, even if it’s not exactly the same thing as what we’re familiar with”. The patient can still die, but they have much higher hopes of surviving “experimental” treatment from a number of qualified doctors, than by attempting self-medication and relying upon miracles.

    So, while I’m sure there are several possible ways of dealing with the decentralisation problem of LL’s asset/user/inventory servers, there are not *many* textbook cases, just a few to pick from, and none have a 100% probability of success — we can only expect that LL’s very qualified IT architects will pick one of the best available choices. But the difference is that any of the choices they’re going to pick are not “placebo” — but attacking and dealing with the core issues, and doing it now, even if it means hurting a lot of residents in the process, and they’re not afraid to do it. I don’t know if there is a “better” way of doing things, short of replicating the whole infrastructure on a second grid and starting there from scratch, which, from a financial point of view, is pure suicide (and would still not fix the short-term issues!).

    If there is some blame here to put on LL, is their inability to fully explain what they are attempting to do, what this requires in terms of patience while they do the attempt, how long it takes, and what the end results will be — in plain, simple, clear, non-technical language, words that everybody will not only understand, but, first and foremost, regain the trust of the residents, specially of those that are full of good intentions but sadly have not enough data (because LL does not disclose it) to understand what this is all about, why it is important to let Linden Lab go ahead with their plan and ignore short-term solutions that will break in the near future, in order to fully focus on the infrastructure that will support 150 million users in 2010.

    I don’t call Luddites “people wanting things to work”. We all want “things to work”, the difference is in *how* to plan for that, and I certainly raise my voice against the ones that don’t believe that major surgery is needed when a simple painkiller seems to be enough, and that it’s better to prevent LL engaging in this major surgery — which will take months without visible results — and make them focus instead on short-term solutions that give immediate relief (and which will fail again after a few weeks).

    Short-term solutions will not work out, specially for those among us that expect to be around in SL for several years or hopefully decades.

    One might question if “all this is really necessary”. Well, I have only a qualified opinion to give: yes, it is. But don’t take my word for it; ideally, it should be LL fully explaining *why* this is necessary, and *why* all short-term solutions are basically worthless after a few weeks, if the major changes are not implemented *now*. And a thorough explanation of these changes can naturally be validated by other sources beyond Linden Lab.

    However, like on most complex technical issues, it’s not a question of “opinions” (even if a lot of people have the same “opinion” and voice them out loudly). You cannot cure a patient by committee — a thousand friends suggesting the same painkiller will not avoid your chronical heart failure — while listening to the qualified opinion of a *doctor* most certainly will. The only thing that “a lot of opinions” can effectively do is force LL to postpone the unavoidable change and throw in some placebo effects to “please the masses”… while praying that things don’t collapse until they’re allowed to work on the issue again.

  7. Hazim Gazov

    May 14th, 2007

    What’s so bad about luddites? I’m a luddite.

  8. Tabatha Dagger

    May 14th, 2007

    I just red Gwyneth Llewelyn’s new blog. It is a very good one. I appologize for misunderstanding her last one.

  9. Accasbel Barrymore

    May 14th, 2007

    ]] If there is some blame here to put on LL, is their inability to fully explain what they are attempting to do, what this requires in terms of patience while they do the attempt, how long it takes, and what the end results will be — in plain, simple, clear, non-technical language, words that everybody will not only understand, but, first and foremost, regain the trust of the residents, specially of those that are full of good intentions but sadly have not enough data (because LL does not disclose it) …. [[

    Bingo! This is the F'ing Inner Core of most of the bad reaction - LLs failure/inability to explain to the users what they are at.

    ]] Instead, a good doctor will just say: “Trust me. I know what’s best for you and I know what I’m suggesting, even if I can’t explain in *full detail* what I’m going to do.” [[
    Possible interpretations of the doctor's words:
    a) He really means "You are not intelligent enough to understand it".
    b) He is being literally correct. He does not have the skill to explain things.
    c) He can't explain things because he does not fully understand it himself.

    Translated into the LL/user context, none of the options are particularly encouraging.
    a) Is downright insulting
    b) Is the kindest option, but in that case LL need to get the skills in. Spin and/or silence only inflame things
    c) Is a frightening option. In the absence of communication from LL, some people might look at the history of updates breaking things and fear the worst.

    ]] ….. regain the trust of the residents ….. [[
    Oh yes!
    Doing that *before* embarking on ‘months, not weeks or days of pain’ would be a good plan.
    “Fixing SL” does not appear to be an readily available method of doing that in the short term. Explaining properly appears to be the best alternative.

    In my years of computer tech sales support when snafu happened, I’ve made it my business to bring the clients own IT function up to speed on all the options and the proposed solution before discussing with the clients senior management. That’s most of the problem solved. The ‘happiness/confidence’ of their own staff is really the only measure that management have other than the bottom line.

    Translating that to SL, the users both individual and corporate are ‘the management’ to be brought on board. An amorphous subset of users, the F’ing-Interested Core, are ‘the IT staff’ that can give encouraging nods to ‘the management’ (and to the press).
    Happiness is viral! Unhappiness is virulent.

    Let LL just put up as much first-hand information as can possibly be published openly.
    Let the F’ing-Interested Core make sense of it.

    One of the outcomes might indeed be a blizzard of unwelcome ‘Opinions’, but that’s already happening anyway. I would suggest that there would also be positive outcomes, and maybe even the odd insightful opinion.

    If we are really heading into months of grief, then LL will need as many supportive friends as possible. I’d suggest that they make friends not via spin and schmoose, but via openness.

  10. shockwave yareach

    May 14th, 2007

    Gweneth:

    You are being a bit unflattering in your description of the letter signers. Truth is, what was asked for was stability before anymore “features” get added. Period. Whether making the current mass of singlepointsoffailure work better, mod it all to SL1.5, or create a whole new SL2.0 was never specified. Only the hardest Geeks in the list know (or care) about what happens in the background to make SL possible. People are simply sick of putting money into a system that functions so poorly, that’s all.

    To use your analogy, we’ve asked our chainsmoking, obese Doctor Linden to stop putting out free donuts and cigarettes in the waiting room and to start looking out for both of our conditions a bit better. Course, the results of this was Doc Linden lit up a new stogie (voice) and blew smoke in our collective faces (sculpties).

  11. Cocoanut Koala

    May 14th, 2007

    1. You say:

    “A coordinated public campaign in the world-wide media against LL’s policies and attitudes can force them to develop a mediocre strategy to please the masses while suppressing a good plan to deal with all major outstanding issues — simply because the mediocre plan is easy to explain, the Big Change is almost impossible, except for a insignificantly small number of highly specialised network engineers, IT architects, and system administrators.”

    This argument assumes – and I think fallaciously – that what the masses want is something bad, something ignorant.

    It also assumes this information is so specialized and arcane that it can’t be explained in a way people can understand, which is true of exactly nothing in the world.

    The only thing we are arguing about here, in fact, is speed.

    The question is: Should the Big Change be accomplished at:

    (a)a slow enough pace that things that are broken are fixed as you go along;

    (b)even slower, so that fewer things are broken TO fix; or

    (c) as fast as possible, ignoring all the broken things for however many months (I assume months, perhaps years?) it takes to put in the new architecture.

    Nothing in what you have said indicates that there is some mandate that the architecture must be put in at a breakneck pace.

    For example, log-ins could be restricted while the work goes on.

    There is also no guarantee that the new architecture itself will even work, in which case you will have thrown out the baby with the bathwater and have no new water for the tub.

    2. Your medical example couldn’t be further off the mark. It is the rare doctor these days who would dare to tell their patients, “Trust me, I know best.”
    That’s because in the past fifty years, doctors learned their lesson about that sort of arrogance, and got their come-uppance for it.

    People are capable of reason, and of acting in their own best-interest.

    No medical condition is impossible for the average person to understand. NOT ONE. (Though some are more complex than others – diabetes, for example – and require more study, in addition to experience with having the disease.)

    And virtually no patient would be so incredibly stupid as to insist on a painkiller without treating the source of the pain. There will not be “many” patients who react as you suggest. People simply are not that stupid.

    SL residents aren’t that stupid, either.

    Anyone of any profession who thinks that he’s so brilliant that the peons just don’t understand him finds himself rapidly supplanted by the professional who realizes that he, though possibly a doctor or a technician, isn’t actually God.

    I understand that you feel helpless to explain things. But that’s why writers exist. There is nothing medical, or technical, that people can’t understand at whatever levels they need to.

    3. Given that this was your own analogy, I will turn it around a bit. The doctor says, “No, I’m afraid you have something much worse, and we are going to have to take more drastic measures than that. Meanwhile, just go ahead and put up with the pain, and don’t worry if you lose a leg or two, or if it costs you your life savings.”

    That would also not happen, because part of managing treatment involves keeping the patient alive and with as much well-being as possible.

    Imagine then that this same doctor went on to add:

    “Actually, though I COULD help you, I want to maximize my income throughout this, and not let the other doctors get ahead of me. I don’t really care about your losses. So I’m not going to really ‘fess up that your pain is largely my own fault, and I could take some time to make you feel more comfortable if I wanted.”

    “After all,” he goes on, “doctors are still few and far between in this neck of the woods, so as long as you are stuck with me, you get to be my guinea pig. And don’t expect it to be pleasant.”

    4. Back to the actual case of SL, even without a technical writer to explain things, tech-savvy SL residents do have a pretty good clue.

    In fact I would say here that people – including those signing the Project Open Letter – understand far more than you think they do, which in turn blinds you to what THEY are saying, see what I mean?

    I hope LL doesn’t make that same mistake.

    5. It is conceivable that if LL continued breaking things and not fixing them, at breakneck pace

    - EVEN if they came clean and carefully explained that it was necessary in order to get the architecture quickly in place without having to throttle log-ins, and in order to make sure none of their competition gets a leg up on them, and that it would all be much better, eventually

    - it is quite likely people would say, “OK, call me when you’re ready; meanwhile I’m tiering down.”

    I don’t think LL wants that.

    6. This patient isn’t dying so much as the doctor is killing it. The patient WAS perfectly healthy.

    Now everyone would like to expand the life of the patient as long as possible; and no one is adverse to making major technological architectural changes involving distributed networks, etc., etc., etc., in order to help the patient eventually live MUCH longer and better.

    But that won’t work if you don’t at least devise a strategy to keep the patient alive while you are experimenting with cures never before needed.

    7. You say:

    “‘Fixing Group IMs’ is just buying a painkiller for a ‘minor nuisance’, when the whole system is truly rotten at the core. Forcing LL to spend developer hours to fix a tiny aspect — but ignoring the central issues (which affect Group IMs anyway…) — means postponing a solution, by giving placebo painkillers to please the masses. At some stage, you need to give up on the placebo and focus on treatment . . .”

    I disagree. Those things are not tiny nuisances. They are the oil that keeps the machine moving at all.

    If LL is headstrong into putting in this new architecture (which itself may not necessarily work) – while at the same time insisting on letting in as many people in as want in at any given time – to such an extent that they can’t provide a working product in the meanwhile, then the plain fact is they are simply not cut out to do the job.

    Because they have no product.

    If they don’t take the time and manpower to fix things like IM’s as they go along, then people will leave; they will have no more money; and the entire project will end. Now that WOULD be fatal, wouldn’t it?

    That would be equivalent to the cure killing the patient.

    8. You say:

    “We all want “things to work”, the difference is in *how* to plan for that, and I certainly raise my voice against the ones that don’t believe that major surgery is needed when a simple painkiller seems to be enough, and that it’s better to prevent LL engaging in this major surgery — which will take months without visible results — and make them focus instead on short-term solutions that give immediate relief (and which will fail again after a few weeks).”

    First off, I think they only fail again after a few weeks because LL won’t throttle sign-ins at a reasonable level they can handle. (Or perhaps because they are making changes too rapidly.)

    Secondly, no one is arguing that major surgery isn’t needed. Personally, I don’t know or care if it is needed or not (and believe it is needed only to achieve even loftier future LL goals), but I’m willing to go along with that assessment and those goals.

    What I am saying – and what the signers of the Project Open Letter are saying – is that LL must provide the painkillers even as they provide the pain. They must keep the patient functioning well enough to ensure their income.

    LL must choose among the three options listed at the beginning of this post (a, b, or c).

    I am for(a),a slow enough pace that things broken are fixed as you go along; and some people are for(b),even slower, so that fewer things are broken TO fix. Even if that requires admitting that the platform can’t handle 40k sign-ons at a time. Even if it means going slower than they might like.

    The difference between you and the people who signed Project Open Letter is that none of them is for c), as fast as possible, and ignoring all the broken things for however many months it takes to put in the new architecture, and you are for that.

    What you call a “mediocre strategy” is vital, or the patient dies and the entire vision is lost, at least in this incarnation of it.

    I think (and hope) LL realizes they simply can’t expect people to keep paying their salaries for a product that doesn’t work. If they keep on presenting a product that doesn’t work, they will face that tier-down scenario, and the whole jig will be up.

    coco

  12. Accasbel Barrymore

    May 14th, 2007

    ]] What you call a “mediocre strategy” is vital, or the patient dies and the entire vision is lost, at least in this incarnation of it.[[

    There’s the other rather disturbing option. Gwyneth sort of referred (no doubt in a sarcastic humorous way) to the ‘so what?’ of losing the odd million or so of current signups.
    What if the big money looked at the dizzy climb of millions of signups, and said
    - Ok it was fun, but it’s not going anywhere
    - Build the new architecture to cater for the Gartner estimate of 80% of Net users being in a 3D interface by 2010(?) e.g. http://www.informationweek.com/blog/main/archives/2007/04/gartner_predict.html
    - Use your current base as lab rats as far as you can

    What if “this incarnation” is a actually dead end?

    It’s this question that has increased my own involvement in RL :)
    I’ve poured an amount of my soul into SL, perhaps an unwise amount of it. The creative possibilities seduced me. I can’t claim to have created earth-shattering art, but I’d like to think that I was on the way to creating a few things that people would have enjoyed greatly.
    Over the past few weeks, I’ve been pulling back more and more.

    It’s not the cycles of technical problems. It’s not a particular worry over verification of age or identity per se.
    What’s happened is that I’ve lost any faith that I had in LL.
    What’s happened is that I’m beginning to suspect that my time and energy is about to be flushed down the toilet.

    That might not matter to someone looking at 80% of Net users, but it matters to me.

    I think it may matter to a lot of other people too.

  13. Prokofy Neva

    May 14th, 2007

    Accasbel Barrymore, you’re making certain assumptions here that are sadly misplaced:

    1. That LL’s failure to communicate its intentions and plans is somehow a lack of people skills, or management skills, or lack of some other thing required of a company. But it’s not. It’s about proprietary information, a game plan that they dare not reveal in a climate of intense competition and media and corporate scrutiny. So they won’t be talking about basic things like how they intend to fix up the search, not because they don’t have a plan, but because they wish to be competitive. Unfortunately for us, making the platform work in a simple way is part of their competitive proprietary information in a climate of intense competition from other virtual worlds.

    2. That we are LL’s customers. We are not. We are their fans, their beta testers. We’re their daughter *and* their sister. You get the idea. But we’re NOT their customers, who are people they have their eye on down the road, some of whom have showed up already, like your IBMs and your SONYs and your Cokes. Once you realize that we are just the boys tagging along as the circus rolls into town, you’ll understand the level at which we can expect “news”.

  14. shockwave yareach

    May 14th, 2007

    Accasbel:

    The same could be said of any effort. Why paint when the painting could be lost in a fire? Why build an airplane when it could crash? Why “waste” time in a game with friends when the computer could crash and never recover? Why buy a car when it can be wrecked? Why spend real money for virtual things?

    It all boils down to what your bottom line is. If you get something out of SL that is worth your investment in time, energy and money, then it is worth staying. But the threat to continued existance exists in all things, not just the game. Nothing is permanent. So there is no point in worrying about it.

  15. HoJo Kilda

    May 14th, 2007

    Just for the record, yes the sarcasm was lost on me but such is the way of the written word sometimes, especially when the reader knows nothing of the writer. After Gwen’s clarification post, I went and took a look at her blog and found it thought provoking, engaging and her writing style entertaining. With that said, I feel like I just arrived in this strange land and I’m curious to see where this will all go.

    Anyway, see you in the immersionist ghetto Gwen! ;)

  16. Cocoanut Koala

    May 14th, 2007

    Well, I am their customer, Prok. I give them my money and I expect my goodies in return.

    It is true, Asscabel, that LL will flush whoever they want to down the toliet at any time without a moment’s notice if they deem it in their best interest.

    I know perfectly well that someday it could be impossible, due to some LL action, for me to make and sell houses to other people on SL.

    If that happens, well then, game over, and I stop paying them!

    But – I get returns on my time and energy now, every day! So it’s okay for me.

    coco

  17. Accasbel Barrymore

    May 14th, 2007

    I’ve got a tremendous amount out of my short time in SL so far. I think it’s mind-blowing.
    I’m also well aware of the impermanence and uncertainty of everything. I live like that in RL, moving from one startup to the next. Like a child picking up Lego on a beach. – I say Lego, to differentiate from ‘shiny’ pebbles. :)

    However, I’m inclined to have my doubts about continuing to paint the walls when the building I’m in appears to be on fire, and my workrate is falling because the paintbrush is starting to stick to the walls – Even if painting is usually fun.
    That’s totally different to worrying about either the probability or the general possibility of a fire starting.

    By the by – the Total Residents display has been static at 6,240,591 for 4(?) days now.
    Did you know that this is 10111110011100101001111 in binary?
    Ha! I thought not! But I can’t explain the significance of this in detail.

    I demand that LL fix that counter thingie before a single sculptie is allowed in.
    Luddites of SL. Join me in this crusade!

  18. Brenda Archer

    May 15th, 2007

    I share Accasbel’s nagging intuition that I may have wasted my precious time in Second Life. Certainly, I have neglected my real life too much. On the other hand, I also did this when Usenet came out. I’m a sucker for new forms of communication and the radical communities and mutual support that they enable. They are transformative in ways that reflect back to real life. It is intoxicating to watch ordinary people change their world. I just need to remember not to trash mine in the process of watching it, to not try to drink from a firehose of information.

    I suspect Prokofy will hate me for praise, but he is closer to the facts than Gwyneth is, if what I have seen the Lindens say very publicly is any indication.

    However, I don’t do conspiracy theory. For a net company, LL is extremely open, if you know how to listen. No company large enough to need a lawyer on its staff can ever be totally open; that’s just reality.

    Of the changes Gwyneth thinks are pivotal, two of them (banlists and voice) are just the GOMing of something that was already being done on a third-party scale. Already, if a private estate owner wants cooperative banlists and chat recording, he can if he wants to do so. It’s just more difficult and less accessible. Many use Skype and Ventrilo.

    I honestly can’t figure out why Gwyneth believes that opensourcing or licensing the server code, or redoing the architecture, is something of which the average user OR the signers of the Letter will disapprove. The signers want stability and effectiveness, and if re-engineering will get us there, so much the better. (I do not think that it could be any worse than what we have now for downtime and inventory loss, so that’s a moot point.) The users want choices and if more developers step up to the plate they will have them, so everyone wins. I have my ear to the ground with many residents and I do not hear a word of objection to open source, but rather, much support.

    Several months ago on the Community Roundtable email list I predicted that sexual ageplay would eventually become a scandal that would destroy Second Life’s moment of liberty as a Temporary Autonomous Zone. Consider Gibson’s idea of a space that for a short while exists outside the system, where outsiders and creative people may live. Bohemia always gets bought out by the wealthy, or invaded by tourists. I’m not worried about this for a simple reason though: unlike what Gwyneth and Prokofy both seem to be saying, an expanding multiverse is not a zero-sum game.

    There are enough cookies and koolaid for everyone! Whether yours is spiked with rum, ganja or poison is totally up to you.

    The multiverse holds up a mirror to us, and laughs at the funhouse reflections.

  19. Prokofy Neva

    May 15th, 2007

    >Bohemia always gets bought out by the wealthy, or invaded by tourists.

    No, Bohemia gets undermined by BDSM, then destroyed by ageplayers.

  20. Accasbel Barrymore

    May 15th, 2007

    From Wikipedia

    ]] The term bohemian was first used in the nineteenth century to describe the non-traditional lifestyles of marginalized and impoverished artists, writers, musicians, and actors in major European cities. The bohemian lifestyle is often associated with cafés, coffeehouses, drug use (particularly opium), alcoholism, and absinthe. Bohemians were associated with unorthodox or antiestablishment political or social viewpoints, which were expressed through extramarital sexual relations and voluntary poverty.

    The term emerged in France in the 1800s when artists and creators began to concentrate in the lower-rent, lower class gypsy neighbourhoods. The term “Bohemian” reflects a belief, widely held in France at the time, that the Gypsies had come from Bohemia [[

    Let’s get our Luddites and Bohemians straight now!
    I don’t think there can be a Metaverse equivalent of Bohemia until *after* the Metaverse has been Disneyfied.
    The true virtual equivalents of Bohemians will live on a cluster of Mature Adult-flagged Mainland servers that LL has lost track of. The whole setup will be so horrendously huge and complex that anyone stumbling across the rack in the colo will be afraid to to anything to them is case it crashes the entire Metaverse. :)

    The rich and the tourists won’t go there because 1) it will be too dangerous and 2) Eros will have built nice safe Bohemia.1, Bohemia.2, etc.
    The ‘real’ new virtual ‘Bohemia’ will probably be more aptly named as something like Toon Town? That would be a far darker version than that in “Who Framed Roger Rabbit”.

    Speaking of Mainland, the future looks “interesting”.
    LL are adding great rafts of sims in the new continents. At the same time they are distancing themselves from the policing of activity. If only island sims have a shot at self (or any) governance, maybe those future wastelands will be populated almost entirely by anthropologists and social scientists busy interviewing and studying each others’ RP.

  21. Anonymous Avatar

    May 18th, 2007

    The voice feature will flop. You read it here first.

    “They cannot “afford” to think on a centralised model, but imagine how the grid of 2010 will look like: a huge mainland probably run by Linden Lab in a dozen co-location facilities across the world; several “licensed” grids run by corporations and universities, all connected together; and individuals using their own sims running from home and allowing a handful of friends to connect. All this is “part of the Metaverse plan”. Only Linden Lab (and, well, the user community around SL) are thinking about this model.”

    Isn’t this called Project Croquet today?

    “It’s very hard to explain to anyone without formal training in complex distributed network architectures why the impact of a slight change is almost impossible to foresee, when something that was thoroughly tested in a “lab” environment suddenly breaks apart when deployed on the “real” grid.”

    LOL! That kind of statement does distract those who don’t know. Been there, done that better than LL.

  22. Gwyneth Llewelyn

    May 19th, 2007

    Coco, what you’ve described are pretty good arguments for a type of company that Linden Lab is not (or does not want to become).

    Your arguments remind me of two attempts of changing the world air traffic control systems — which were designed in the 1960s. Twice there was an attempt to change them, for several reasons, the most important ones the cost of running a 40-year-old technology when many of the original programmers had already died, and the costs of hiring and training new programmers able to deal with the arcane technology was becoming unbearable; the other reason, of course, is that the technology of the 1960s does not scale well to the changes that we need in the 21st century.

    Well, the first attempt in the 1980s failed completely — again, we’re talking about an incredibly complex piece of software which can only be truly tested and understood when deployed on the real environment, not on small-scale test enviornments, which are too “controlled” (pun intended!) to accurately reflect the chaotic behaviour of the real world. The attempt was abandoned, and the old software was used again.

    In the 1990s, there was a new good reason to change the software again: the year 2000 bug, which many feared it could seriously impact the old software, which had never been designed to be around for 40 years! A new effort was done to rewrite it all and deploy it, well in advance of the dreadful date of January 1, 2000. And again, although the effort had been better planned and more extensively modelled and developed than a decade before, this attempt also failed utterly, in this case almost costing some human lives as the system failed to track some of the airplanes en route, or giving them incorrect information on the way they should approach the airports — naturally enough, pilots are highly skilled trained professionals, and in the airports that had adopted the new software, the pilots went back to completely manual landings with radio guidance, like in the old WWII movies.

    The two attempts were finally abandoned, and the old 1960ish software was put back on all systems. It’s still running flawlessly like it did in the past 40 years. It’s full of bugs, problems, issues, and impossible to do anything nifty with it, but it runs. The alternatives of “designing it from scratch” were all abandoned; instead, as new technology becomes available (example: GPS, which did not exist in the 1960s; or automatic landing systems on airports), parts of the code get rewritten and incorporated in the old system. The code gets “renewed” successfully, after years of developing individual lines of code, testing them, validating them, and finally implementing them, ready to fall back to the last version if something goes seriously wrong again. But I think that people learned their lesson during the two attempts of rewriting the code: don’t mess with complex systems that have a chaotic behaviour.

    SL is very close to becoming that type of software. “Fixing” it here and there might lead to impossible-to-predict side effects. The request to “get us a reliable, stable grid” is as hard to accomplish than rewriting the whole traffic control software system, and very likely as dangerous. Yes, it’s truly like the cure that might kill the patient. However, what LL has done mostly so far is to try to fix some of outstanding issues which are really “bugs” and not fundamental design problems.

    Bear with me for a moment to grossly explain how software in general is developed (although LL does not follow the textbook doctrine on it, of course). It is planned first and conceptually explained (sadly, Philip’s and Cory’s original white paper is not publicly available any more from SL’s pages; it used to be, and I wish I had kept a copy). Then there is a design stage — concepts become specifications, which will describe in some detail (often with graphical languages) how each element will interact with others. And then it gets coded; and debugged.

    A “bug” is essentially a bit of code that was done wrongly according to the specs; the specs might have said “this window is red” and the programmer has done a blue window. That’s a bug. It can be harmless (we can live with blue windows) or have serious consequences (the whole code to change the windows’ colour is broken, and we can’t set it back to red). Bugs, in the sense of “not complying with the specs”, can usually be fixed — in essence, what this means is that a bit of code needs to be rewritten to that it complies with the specs again.

    However, the problem is not with “bugs”, but with “flaws”. This is when the specs are *wrong* — because someone has wrongly understood the scope of the design and wrote specs that are not what the design mandated them to be. Now this means that the code is correct, but it doesn’t do what it was expected to do, since the specs are incorrect. Changing specs means that suddenly perfectly debugged and valid code has to be thrown away, so that the specs can be rewritten and comply to the design. This is the stage where people start to complain that things that worked most of the time (say, Group IMs…) suddenly get broken, although we know that LL can indeed develop working Group IMs, since we had them for 3 years without major problems.

    The worst level of difficulty is, of course, if the design is all wrong. A good example was the very optimistic assumption that people would stay and remain on a single sim most of the time, and that this sim would hold all data to make avatars happy. It didn’t account (or just partially addressed the issue) that avatars travel across the grid all the time, and bring their content with them — content is created in one place and deployed on others. Things like inventory, teleport, or simply using textures that are stored on another sim (most will be!) are good examples of a fluid and dynamic movement of content across the grid. LL optimistically believed that a very simple asset server (and the first generation asset server was childish — but genial! — in its simplicity) would be enough to track millions of objects. It worked — until it had to address around one petabyte (that’s a million gigabytes!) of assets. Then it became obvious that this optimistic and simplistic design simply wouldn’t work, and LL has been working, for at least two years, to make it scale better.

    However, the issue remains: the original *design* was not good enough to handle a 10k-sim grid with 40k simultaneous users. There is so much you can do with technology and engineering (and faster machines!) to handle the increased load; the problem, however, was “bad design” from the start.

    Re-designing SL is a *major* task. It’s not about “fixing bugs” at all. It means laying down all assumptions, getting rid of hundreds of thousands of perfectly legitimate code (which worked well and bug-free for years and years), and doing it all from scratch again, under a new design which can handle the load much better (and there are, fortunately, enough textbook cases to handle this, so it will be intriguing to see which one LL will pick…).

    What seems hard to understand here (and I again will agree that the issue is possibly a communications problem) is that fixing the design is not something that can be done “over a weekend” with Philip whipping the developers, and LL being threatened by angry residents in pitchforks that say “we pay for this, we demand it works NOW”. Like in the air traffic control system, it’s not by willing it to work that people will suddenly realise: “oh, right, I better get started then. Sorry for daydreaming!”

    In the software industry, when something comes to the point where fixing bugs will not fix poor design, the answer is usually the same one: develop a new design. Launch version 2.0. The more complex a bit of software is, the more time will pass between two versions, when the design changes. Thus Microsoft needed 6 years to launch a new version of Windows. Yes, they’ve changed the design in 3 years — and then spent
    another 3 years just to make sure older applications still worked (they didn’t). And this is Microsoft, with dozens of thousands of developers. Linden Lab, three years ago, had a dozen.

    Now Linden Lab, so far, has been doing the “traffic control system” approach: fixing bugs when it was clear they were bugs, while keeping the grid running; changing here and there some flaws in the specs (and sometimes breaking things); but now it comes to the point where they need to handle the biggest challenge: fixing the *design*. And, like the zillions of people who developed the world-wide traffic control system, they know this is not a piece of cake. The challenge, of course, is to change the design *and keep those airplanes safe in the air* — since LL has figured out, very correctly, that they can’t afford to close down the grid for 2-3 years and develop SL 2.0, which would be incompatible with 1.X. They have no choice but to keep it going while they still change the design.

    This doesn’t mean they shouldn’t be fixing bugs (which they are), or even correct implementation flaws (which they are as well). It means more than that: that all work done to fig bugs and flaws under the *current* design might be a waste of time, since many things will have to be rewritten anyway when the *new* design is deployed. Now the issue is, what should the priorities be? Fixing bugs and correcting implementation flaws will *not* make the grid more stable or allow it to handle millions of concurrent users. It will just give “momentaneous relief”, but waste away precious developer time that should be better employed to tackle the core of the problem. And this, as said, is definitely not a painless road.

    Back to things like Group IMs and a voice-enabled SL… far better than figuring out why currently the Group IMs don’t work, would be to have implemented a whole IM system that scales well to millions of concurrent users, and that is voice-ready. There are a few dozen around — Jabber (the one used by Google Talk, for instance) being free, available, and well tested. This could be run perhaps on a single server totally independent of the grid, and Jabber was designed from scratch to deal with scalability issues (you can add more Jabber servers on the network without requiring people to do any changes). LL, however, preferred to hold on with their old code instead, and try to fix its bugs and locate implementation flaws, while adding some more engineering magic to keep it operational. But the whole design of it is flawed from the very beginning. It also was never designed to handle well things like tracking down the online status, or signalling voice. All that had to be *added* on top of something that was already obsolete two years ago — *and* with lots of much better alternatives lying around. Why didn’t LL go for those instead of relying upon their own homegrown variety? The only possible explanation is simply resource allocation — fixing bugs as they occur is far cheaper than redeveloping a new design for the whole IM system.

    This, I believe, could be averted by a different way of communicating better their intents. But there is a limit to how much a corporation exposed to the media is able to do. Cory can’t go to a press conference and say: “We have not been ignoring our users; we know that SL sucks; we’re now going to enter a 3 year project to redesign everything from scratch without breaking everything; in the mean time, have patience, folks, there is not really much we can do at this stage but wait until the developers are finished”. That would be corporate suicide. Instead, they’ve gone to a conference somewhere — like they did in 2001 or so when their whitepaper for SL 1.0 was first released — and say: “Guys, we have excellent news — we’ve just invented the grid of the future, which will be able to scale to a billion accounts and have a dozen of million simultaneous uses online. We’re excited to announce the new stage of our project with you all! And the best part of it is that not a single asset will be lost during this transition to the New Grid”.

    After all, unlike others have claimed, LL is not doing anything different from what, say, Microsoft has done in the past 6 years with Windows XP: fixing bugs and security holes, reviewing flawed implmentations, but not touching on the design, and working in parallel with the new design, while making sure everything that could run on XP would run on Vista as well (MS was not *so* good with that, however…). Of course other software houses have different views on how software should be developed and deployed. The Open Letter project is in essence “demanding” that LL adopts a different approach to software development: “freezing” all development around a stable version and build a new system from scratch, in parallel, and release it in, say, 2010 or so, when it’s thoroughly tested and debugged.

    Well, all that is very nice if your customer base is either very small (a few thousand users) or has a slow growth (hundreds of millions of users just growing a few percent per year). LL has neither — 6.5 million accounts is too big a number to ignore, and the growth is exponential — so it means shorter development cycles with dramatic impact. It’s like demanding that a square peg fits in a round hole; simply put, there are development strategies that work when your mode is a round hole, and it’s worthless to attempt to hammer at the peg with all your strength. It simply will not fit. The best you can do, however, is to gauge the size of the round peg and make it fit as best as possible, and this is what LL is (apparently) trying to do.

    And Anonymous, at some point, someone will always claim “others do it better”. That’s fantastic to know, but show me a 6.5-million-accounts grid of OpenCroquet accounts with 40k concurrent users, and I’ll believe you :) I’m not saying it can’t be done; taking the reverse example, did the 300 betatesters of SL truly believe SL could allow 300.000 or even 30 million? (remember, on those days, a dozen avatars just sitting and chatting on the same sim would crash it…) Probably not, and probably they’d have been as sceptic of any such claims as I’m now with OpenCroquet. My reason, I guess, is based on two whitepapers — the one on which SL’s original architecture was based, and the one describing the work done on OpenCroquet. The latter works upon three major assumptions:
    - unlimited uploading and downloading bandwidth for all users (eg. a local 1 GBps Ethernet LAN for instance)
    - people voluntarily donating servers to store enough persistent data, or running companies to do so and charging for it (ie. in effect creating centralised grids just like LL’s, and prone to the same problems, with a single difference: you can always connect to another server hosted by someone else if one of them fails, while on SL you’re limited to a single grid)
    - point-to-point connections to handle partial views of the world for enough time to make a scene viewable by hundreds of people for enough time while not requiring connections to a persistent server. While certainly OpenCroquet has the academic respect it deserves to attempt exactly that — and their model is curious as a thought experiment — it remains to be seen if the model really works so well.

    Right now, OpenCroquet seems ideal for small groups (10-100 people) with high bandwidth to do some collaborative work in a limited environment, and having the ability to disconnect at will from one enviornment and go to another one hosted by another small group. Simple things like content ownership or a grid-wide economy are not part of its design; it’s a great research tool, and probably a reasonably good communications platform (HTML-on-an-object works!… and so does spacialised sound and other nifty features that we don’t get the privilege to have in SL). But… I’d even say it’s not even a “metaverse”, but a collaborative platform/environment for creating 3D content together using low resources (ie. no need for a grid).

    For a lot of people, that’s more than enough. Sadly, not for the likes of me.

  23. Blinders Off

    May 19th, 2007

    “”SL is to become sanitized corporate showcase, and the new platform code will cause everything suddenly and magically work just right… even though this new code is based on something that no one has ever attempted, there is no way to debug such complicated code until it’s deployed live, and the company in charge of this development not only has far from stellar record when it comes to stability of their software, but also operates under belief it doesn’t actually matter if said software performs as intended to begin with.”
    – Joannah Cramer

    LOL I loved this. Right on the button. We have here Linden Lab saying they’re going to fix everything, magically! Didn’t we hear that way back in 1.6 and 1.7? Oh but this is different! This is 2.0!

    In order to believe any of these claims, we have to accept as gospel that LL has the ability to write code that actually works. As interesting as SL is, it’s the only game I’ve ever played that I actually dread updates and wondered if LL were all idiot savants. And the statement that SL is now “profitable” only tells me that there’s a fool born every minute. :D

    (IF it’s profitable. LL’s reputation for overstatement is well known).

  24. Gwyneth Llewelyn

    May 20th, 2007

    Just don’t forget the sheer size of the number of developers they have now, compared to the number they had back in 1.6/1.7… and aye, I know, quantity is not a substitute for quality, but they have managed to interest quite a new class of top-notch programmers to work for them as well. So they have now more and better teams — let’s just see how they’re able to deploy them!

Leave a Reply