LOL I loved this. Right on the button. We have here Linden Lab saying they’re going to fix everything, magically! Didn’t we hear that way back in 1.6 and 1.7? Oh but this is different! This is 2.0!

In order to believe any of these claims, we have to accept as gospel that LL has the ability to write code that actually works. As interesting as SL is, it’s the only game I’ve ever played that I actually dread updates and wondered if LL were all idiot savants. And the statement that SL is now “profitable” only tells me that there’s a fool born every minute.

(IF it’s profitable. LL’s reputation for overstatement is well known).

Your arguments remind me of two attempts of changing the world air traffic control systems — which were designed in the 1960s. Twice there was an attempt to change them, for several reasons, the most important ones the cost of running a 40-year-old technology when many of the original programmers had already died, and the costs of hiring and training new programmers able to deal with the arcane technology was becoming unbearable; the other reason, of course, is that the technology of the 1960s does not scale well to the changes that we need in the 21st century.

Well, the first attempt in the 1980s failed completely — again, we’re talking about an incredibly complex piece of software which can only be truly tested and understood when deployed on the real environment, not on small-scale test enviornments, which are too “controlled” (pun intended!) to accurately reflect the chaotic behaviour of the real world. The attempt was abandoned, and the old software was used again.

In the 1990s, there was a new good reason to change the software again: the year 2000 bug, which many feared it could seriously impact the old software, which had never been designed to be around for 40 years! A new effort was done to rewrite it all and deploy it, well in advance of the dreadful date of January 1, 2000. And again, although the effort had been better planned and more extensively modelled and developed than a decade before, this attempt also failed utterly, in this case almost costing some human lives as the system failed to track some of the airplanes en route, or giving them incorrect information on the way they should approach the airports — naturally enough, pilots are highly skilled trained professionals, and in the airports that had adopted the new software, the pilots went back to completely manual landings with radio guidance, like in the old WWII movies.

The two attempts were finally abandoned, and the old 1960ish software was put back on all systems. It’s still running flawlessly like it did in the past 40 years. It’s full of bugs, problems, issues, and impossible to do anything nifty with it, but it runs. The alternatives of “designing it from scratch” were all abandoned; instead, as new technology becomes available (example: GPS, which did not exist in the 1960s; or automatic landing systems on airports), parts of the code get rewritten and incorporated in the old system. The code gets “renewed” successfully, after years of developing individual lines of code, testing them, validating them, and finally implementing them, ready to fall back to the last version if something goes seriously wrong again. But I think that people learned their lesson during the two attempts of rewriting the code: don’t mess with complex systems that have a chaotic behaviour.

SL is very close to becoming that type of software. “Fixing” it here and there might lead to impossible-to-predict side effects. The request to “get us a reliable, stable grid” is as hard to accomplish than rewriting the whole traffic control software system, and very likely as dangerous. Yes, it’s truly like the cure that might kill the patient. However, what LL has done mostly so far is to try to fix some of outstanding issues which are really “bugs” and not fundamental design problems.

Bear with me for a moment to grossly explain how software in general is developed (although LL does not follow the textbook doctrine on it, of course). It is planned first and conceptually explained (sadly, Philip’s and Cory’s original white paper is not publicly available any more from SL’s pages; it used to be, and I wish I had kept a copy). Then there is a design stage — concepts become specifications, which will describe in some detail (often with graphical languages) how each element will interact with others. And then it gets coded; and debugged.

A “bug” is essentially a bit of code that was done wrongly according to the specs; the specs might have said “this window is red” and the programmer has done a blue window. That’s a bug. It can be harmless (we can live with blue windows) or have serious consequences (the whole code to change the windows’ colour is broken, and we can’t set it back to red). Bugs, in the sense of “not complying with the specs”, can usually be fixed — in essence, what this means is that a bit of code needs to be rewritten to that it complies with the specs again.

However, the problem is not with “bugs”, but with “flaws”. This is when the specs are *wrong* — because someone has wrongly understood the scope of the design and wrote specs that are not what the design mandated them to be. Now this means that the code is correct, but it doesn’t do what it was expected to do, since the specs are incorrect. Changing specs means that suddenly perfectly debugged and valid code has to be thrown away, so that the specs can be rewritten and comply to the design. This is the stage where people start to complain that things that worked most of the time (say, Group IMs…) suddenly get broken, although we know that LL can indeed develop working Group IMs, since we had them for 3 years without major problems.

The worst level of difficulty is, of course, if the design is all wrong. A good example was the very optimistic assumption that people would stay and remain on a single sim most of the time, and that this sim would hold all data to make avatars happy. It didn’t account (or just partially addressed the issue) that avatars travel across the grid all the time, and bring their content with them — content is created in one place and deployed on others. Things like inventory, teleport, or simply using textures that are stored on another sim (most will be!) are good examples of a fluid and dynamic movement of content across the grid. LL optimistically believed that a very simple asset server (and the first generation asset server was childish — but genial! — in its simplicity) would be enough to track millions of objects. It worked — until it had to address around one petabyte (that’s a million gigabytes!) of assets. Then it became obvious that this optimistic and simplistic design simply wouldn’t work, and LL has been working, for at least two years, to make it scale better.

However, the issue remains: the original *design* was not good enough to handle a 10k-sim grid with 40k simultaneous users. There is so much you can do with technology and engineering (and faster machines!) to handle the increased load; the problem, however, was “bad design” from the start.

Re-designing SL is a *major* task. It’s not about “fixing bugs” at all. It means laying down all assumptions, getting rid of hundreds of thousands of perfectly legitimate code (which worked well and bug-free for years and years), and doing it all from scratch again, under a new design which can handle the load much better (and there are, fortunately, enough textbook cases to handle this, so it will be intriguing to see which one LL will pick…).

What seems hard to understand here (and I again will agree that the issue is possibly a communications problem) is that fixing the design is not something that can be done “over a weekend” with Philip whipping the developers, and LL being threatened by angry residents in pitchforks that say “we pay for this, we demand it works NOW”. Like in the air traffic control system, it’s not by willing it to work that people will suddenly realise: “oh, right, I better get started then. Sorry for daydreaming!”

In the software industry, when something comes to the point where fixing bugs will not fix poor design, the answer is usually the same one: develop a new design. Launch version 2.0. The more complex a bit of software is, the more time will pass between two versions, when the design changes. Thus Microsoft needed 6 years to launch a new version of Windows. Yes, they’ve changed the design in 3 years — and then spent
another 3 years just to make sure older applications still worked (they didn’t). And this is Microsoft, with dozens of thousands of developers. Linden Lab, three years ago, had a dozen.

Now Linden Lab, so far, has been doing the “traffic control system” approach: fixing bugs when it was clear they were bugs, while keeping the grid running; changing here and there some flaws in the specs (and sometimes breaking things); but now it comes to the point where they need to handle the biggest challenge: fixing the *design*. And, like the zillions of people who developed the world-wide traffic control system, they know this is not a piece of cake. The challenge, of course, is to change the design *and keep those airplanes safe in the air* — since LL has figured out, very correctly, that they can’t afford to close down the grid for 2-3 years and develop SL 2.0, which would be incompatible with 1.X. They have no choice but to keep it going while they still change the design.

This doesn’t mean they shouldn’t be fixing bugs (which they are), or even correct implementation flaws (which they are as well). It means more than that: that all work done to fig bugs and flaws under the *current* design might be a waste of time, since many things will have to be rewritten anyway when the *new* design is deployed. Now the issue is, what should the priorities be? Fixing bugs and correcting implementation flaws will *not* make the grid more stable or allow it to handle millions of concurrent users. It will just give “momentaneous relief”, but waste away precious developer time that should be better employed to tackle the core of the problem. And this, as said, is definitely not a painless road.

Back to things like Group IMs and a voice-enabled SL… far better than figuring out why currently the Group IMs don’t work, would be to have implemented a whole IM system that scales well to millions of concurrent users, and that is voice-ready. There are a few dozen around — Jabber (the one used by Google Talk, for instance) being free, available, and well tested. This could be run perhaps on a single server totally independent of the grid, and Jabber was designed from scratch to deal with scalability issues (you can add more Jabber servers on the network without requiring people to do any changes). LL, however, preferred to hold on with their old code instead, and try to fix its bugs and locate implementation flaws, while adding some more engineering magic to keep it operational. But the whole design of it is flawed from the very beginning. It also was never designed to handle well things like tracking down the online status, or signalling voice. All that had to be *added* on top of something that was already obsolete two years ago — *and* with lots of much better alternatives lying around. Why didn’t LL go for those instead of relying upon their own homegrown variety? The only possible explanation is simply resource allocation — fixing bugs as they occur is far cheaper than redeveloping a new design for the whole IM system.

This, I believe, could be averted by a different way of communicating better their intents. But there is a limit to how much a corporation exposed to the media is able to do. Cory can’t go to a press conference and say: “We have not been ignoring our users; we know that SL sucks; we’re now going to enter a 3 year project to redesign everything from scratch without breaking everything; in the mean time, have patience, folks, there is not really much we can do at this stage but wait until the developers are finished”. That would be corporate suicide. Instead, they’ve gone to a conference somewhere — like they did in 2001 or so when their whitepaper for SL 1.0 was first released — and say: “Guys, we have excellent news — we’ve just invented the grid of the future, which will be able to scale to a billion accounts and have a dozen of million simultaneous uses online. We’re excited to announce the new stage of our project with you all! And the best part of it is that not a single asset will be lost during this transition to the New Grid”.

After all, unlike others have claimed, LL is not doing anything different from what, say, Microsoft has done in the past 6 years with Windows XP: fixing bugs and security holes, reviewing flawed implmentations, but not touching on the design, and working in parallel with the new design, while making sure everything that could run on XP would run on Vista as well (MS was not *so* good with that, however…). Of course other software houses have different views on how software should be developed and deployed. The Open Letter project is in essence “demanding” that LL adopts a different approach to software development: “freezing” all development around a stable version and build a new system from scratch, in parallel, and release it in, say, 2010 or so, when it’s thoroughly tested and debugged.

Well, all that is very nice if your customer base is either very small (a few thousand users) or has a slow growth (hundreds of millions of users just growing a few percent per year). LL has neither — 6.5 million accounts is too big a number to ignore, and the growth is exponential — so it means shorter development cycles with dramatic impact. It’s like demanding that a square peg fits in a round hole; simply put, there are development strategies that work when your mode is a round hole, and it’s worthless to attempt to hammer at the peg with all your strength. It simply will not fit. The best you can do, however, is to gauge the size of the round peg and make it fit as best as possible, and this is what LL is (apparently) trying to do.

And Anonymous, at some point, someone will always claim “others do it better”. That’s fantastic to know, but show me a 6.5-million-accounts grid of OpenCroquet accounts with 40k concurrent users, and I’ll believe you I’m not saying it can’t be done; taking the reverse example, did the 300 betatesters of SL truly believe SL could allow 300.000 or even 30 million? (remember, on those days, a dozen avatars just sitting and chatting on the same sim would crash it…) Probably not, and probably they’d have been as sceptic of any such claims as I’m now with OpenCroquet. My reason, I guess, is based on two whitepapers — the one on which SL’s original architecture was based, and the one describing the work done on OpenCroquet. The latter works upon three major assumptions:
- unlimited uploading and downloading bandwidth for all users (eg. a local 1 GBps Ethernet LAN for instance)
- people voluntarily donating servers to store enough persistent data, or running companies to do so and charging for it (ie. in effect creating centralised grids just like LL’s, and prone to the same problems, with a single difference: you can always connect to another server hosted by someone else if one of them fails, while on SL you’re limited to a single grid)
- point-to-point connections to handle partial views of the world for enough time to make a scene viewable by hundreds of people for enough time while not requiring connections to a persistent server. While certainly OpenCroquet has the academic respect it deserves to attempt exactly that — and their model is curious as a thought experiment — it remains to be seen if the model really works so well.

Right now, OpenCroquet seems ideal for small groups (10-100 people) with high bandwidth to do some collaborative work in a limited environment, and having the ability to disconnect at will from one enviornment and go to another one hosted by another small group. Simple things like content ownership or a grid-wide economy are not part of its design; it’s a great research tool, and probably a reasonably good communications platform (HTML-on-an-object works!… and so does spacialised sound and other nifty features that we don’t get the privilege to have in SL). But… I’d even say it’s not even a “metaverse”, but a collaborative platform/environment for creating 3D content together using low resources (ie. no need for a grid).

For a lot of people, that’s more than enough. Sadly, not for the likes of me.

“They cannot “afford” to think on a centralised model, but imagine how the grid of 2010 will look like: a huge mainland probably run by Linden Lab in a dozen co-location facilities across the world; several “licensed” grids run by corporations and universities, all connected together; and individuals using their own sims running from home and allowing a handful of friends to connect. All this is “part of the Metaverse plan”. Only Linden Lab (and, well, the user community around SL) are thinking about this model.”

Isn’t this called Project Croquet today?

“It’s very hard to explain to anyone without formal training in complex distributed network architectures why the impact of a slight change is almost impossible to foresee, when something that was thoroughly tested in a “lab” environment suddenly breaks apart when deployed on the “real” grid.”

LOL! That kind of statement does distract those who don’t know. Been there, done that better than LL.

]] The term bohemian was first used in the nineteenth century to describe the non-traditional lifestyles of marginalized and impoverished artists, writers, musicians, and actors in major European cities. The bohemian lifestyle is often associated with cafés, coffeehouses, drug use (particularly opium), alcoholism, and absinthe. Bohemians were associated with unorthodox or antiestablishment political or social viewpoints, which were expressed through extramarital sexual relations and voluntary poverty.

The term emerged in France in the 1800s when artists and creators began to concentrate in the lower-rent, lower class gypsy neighbourhoods. The term “Bohemian” reflects a belief, widely held in France at the time, that the Gypsies had come from Bohemia [[

Let’s get our Luddites and Bohemians straight now!
I don’t think there can be a Metaverse equivalent of Bohemia until *after* the Metaverse has been Disneyfied.
The true virtual equivalents of Bohemians will live on a cluster of Mature Adult-flagged Mainland servers that LL has lost track of. The whole setup will be so horrendously huge and complex that anyone stumbling across the rack in the colo will be afraid to to anything to them is case it crashes the entire Metaverse.

The rich and the tourists won’t go there because 1) it will be too dangerous and 2) Eros will have built nice safe Bohemia.1, Bohemia.2, etc.
The ‘real’ new virtual ‘Bohemia’ will probably be more aptly named as something like Toon Town? That would be a far darker version than that in “Who Framed Roger Rabbit”.

Speaking of Mainland, the future looks “interesting”.
LL are adding great rafts of sims in the new continents. At the same time they are distancing themselves from the policing of activity. If only island sims have a shot at self (or any) governance, maybe those future wastelands will be populated almost entirely by anthropologists and social scientists busy interviewing and studying each others’ RP.

No, Bohemia gets undermined by BDSM, then destroyed by ageplayers.

I suspect Prokofy will hate me for praise, but he is closer to the facts than Gwyneth is, if what I have seen the Lindens say very publicly is any indication.

However, I don’t do conspiracy theory. For a net company, LL is extremely open, if you know how to listen. No company large enough to need a lawyer on its staff can ever be totally open; that’s just reality.

Of the changes Gwyneth thinks are pivotal, two of them (banlists and voice) are just the GOMing of something that was already being done on a third-party scale. Already, if a private estate owner wants cooperative banlists and chat recording, he can if he wants to do so. It’s just more difficult and less accessible. Many use Skype and Ventrilo.

I honestly can’t figure out why Gwyneth believes that opensourcing or licensing the server code, or redoing the architecture, is something of which the average user OR the signers of the Letter will disapprove. The signers want stability and effectiveness, and if re-engineering will get us there, so much the better. (I do not think that it could be any worse than what we have now for downtime and inventory loss, so that’s a moot point.) The users want choices and if more developers step up to the plate they will have them, so everyone wins. I have my ear to the ground with many residents and I do not hear a word of objection to open source, but rather, much support.

Several months ago on the Community Roundtable email list I predicted that sexual ageplay would eventually become a scandal that would destroy Second Life’s moment of liberty as a Temporary Autonomous Zone. Consider Gibson’s idea of a space that for a short while exists outside the system, where outsiders and creative people may live. Bohemia always gets bought out by the wealthy, or invaded by tourists. I’m not worried about this for a simple reason though: unlike what Gwyneth and Prokofy both seem to be saying, an expanding multiverse is not a zero-sum game.

There are enough cookies and koolaid for everyone! Whether yours is spiked with rum, ganja or poison is totally up to you.

The multiverse holds up a mirror to us, and laughs at the funhouse reflections.

However, I’m inclined to have my doubts about continuing to paint the walls when the building I’m in appears to be on fire, and my workrate is falling because the paintbrush is starting to stick to the walls – Even if painting is usually fun.
That’s totally different to worrying about either the probability or the general possibility of a fire starting.

By the by – the Total Residents display has been static at 6,240,591 for 4(?) days now.
Did you know that this is 10111110011100101001111 in binary?
Ha! I thought not! But I can’t explain the significance of this in detail.

I demand that LL fix that counter thingie before a single sculptie is allowed in.
Luddites of SL. Join me in this crusade!

It is true, Asscabel, that LL will flush whoever they want to down the toliet at any time without a moment’s notice if they deem it in their best interest.

I know perfectly well that someday it could be impossible, due to some LL action, for me to make and sell houses to other people on SL.

If that happens, well then, game over, and I stop paying them!

But – I get returns on my time and energy now, every day! So it’s okay for me.

coco

Anyway, see you in the immersionist ghetto Gwen!