Wanted: World-Bearer
by prokofy on 16/02/07 at 1:03 pm
Photo by Colin Gregory Palmer of “Atlas” by Lee Lawrie and Rene Chambellan, in the Art Deco style (1936) at Rockefeller Center. Creative Commons Attribution
By Prokofy Neva, Dept. of Worlds, Planets, Universes, Metaverses, and Server Farms So Large That Even the Men of Science Are Amazed
The news that Philip Rosedale has personally posted a want ad for a Mega Metaverse Manager called a “Global Technical Operations Jedi” to wrangle more than 5000 sims on some 2000 servers to be spread across these United States and Europe has caused some to wonder if the simulated sexverse situated on server farms in California and Texas could be in trouble with scaling issues.
It’s a daunting job, keeping this world not only running, but perkily uploading everybody’s user-made content, and constantly updating the grid with new software from the Lab itself, and now home-made open-sourced brews that the Lab will be clearing with an as-yet-unsolved method for posting and approving.
While among the problems the new world-wrangler will have to face is lag, crashes, outages, time dilation tumbling to 3, hacks to the database, etc., there’s the physical real-world problems that Something Something and other commentators have noted: the physical space, electricity, and cooling systems needed to house the servers, currently at this location physically in San Francisco, CA and at another data center in Texas. Residents report literally being unable to cross from a sim located in CA to the one in TX inworld — though the Internet is not supposed to work like that.
Apparently nobody in-house could be found for this awesome Atlas assignment, which some might thing has more akin to the job of the Cat in the Hat
“Look at me!Look at me now!” said the cat.”With a cup and a cakeOn the top of my hat!I can hold TWO books!I can hold up the fish!And a little toy ship!And some milk on a dish!And look!I can hop up and down on the ball!But that is not all!Oh, no.That is not all…
What other real-life systems are like the vast server farms of Second Life? World of Warcraft and other large MMORPG have millions of players, of course, but they are on shards or with instances of the same game on different servers, and without user-made content. You can’t play in WoW and access the same place where your friend from Russia is playing unless you both agree ahead of time to log into and create characters on the same shard.
Institutions like the United Nations have vast networks spanning the globe, with their notoriously slow-loading portal in the major world languages and clunky data bases that often tell you that you need a password to enter when you don’t, or push a cumbersome PDF file at you that freezes your computer. The challenge of keeping these portals open and accessible and with more or less simultaneous content is enormously daunting and an incredible amount of work has gone in to putting the documentation of one of the world’s largest paper-chasers online for the public. But they aren’t going for the 3-D look, voice, or avatars with sliders.
As I try to think of what this is *like* (and I hope better minds will help me out here), it reminds me of a family job I had to work for awhile called “truck dispatching”. Yikes. It would seem like a simple matter to pick up a phone order for Stella Dora bread sticks in Pennsylvania, fax the purchase order and highway directions and loading dock orders to the trucker, and keep tabs on him and the floor manager at the bread stick depo on the cell phone. You go from A to B, and you pick up C and then bring it to D.
But what happens is that perishable goods and multiple “just in time” orders complexify the job enormously. You can’t just let somebody sit with a truck idle for 3 days in Western Pennsylvania because Stella Dora is either out of the sticks, or their docks don’t open at a certain time or whatever. And you have competing companies and of course the entire 6-8 week schedule of the over-the-road driver to contend with. So you find yourself having to figure out if there is a Stella D’ora in Illinois, that could be accessed on the way to picking up another order for Lipton Moonlight Mint Herbal Tea at the Unilever plant somewhere…and then you ultimately arrive at a seeming absurdity: although the Stella D’ora breadsticks could very conveniently be picked up at a depot much, much closer to the purchase order located in San Diego and ensure they don’t get past the sell-by date, you wind up having the driver pick up in Pennsylvania and drive like hell because otherwise, you just can’t route him right to capture the other orders *and* avoid all the regulations, prohibitions, rules, and rambles that constantly plague the OTR driver.
In other words, the system becomes filled with idiocies, illogic, duplication, insanities because of all kinds of other exigencies overlaying it. Few people ever contemplate about how the Internet and its enormous capacity for shopping and shipping can only exist because of the trucking industry, which is contingent on roads and DOT insanity — yet there it is.
What are the great metaversal minds thinking about how to *do* this job? What kind of person could do it? Does that person in fact not yet exist because it is such a new and complex thing? Or is running whole bunches of MYSQL calls in fact something that people who program for medical firms and hospitals and universities find a fairly straightforward matter than in fact the folks at the Lab have made complex, either through inability or through their wacky, hippy office culture with its “chose your own projects” and Love Machine ethos? It’s hard for an outsider to understand whether Philip is a genius for understanding that the Engineering Project of the Century also requires a completely new office management culture and method to go with it, or whether his hippie stuff is going to kill what could have been the Pyramids of its time.
People who run servers for a living in real life get very, very cranky about the Lindens. They think they are nuts, and can’t do their job. And I’ve found that El Chefe himself and the coders get very touchy if you imagine that they’re using duct tape, Bazooka, and bent Coke Cans to keep their metaverse rolling. However, people in RL don’t seem to appreciate is that the servers they run with only 2-D electronic page and 2-D images on it really isn’t the same challenge as a 3-D streaming video world with constant user-generated content and user calls to the data base. Or is it?
Philip Rosedale might have solved the issue of his Atlassian struggle by only privately contacting his own feted network of beta buddies, or consulting only insiders in the business who flock to places like Sun Valley. Instead, he chose to put it out in public on a website. That may only be a feint, to make it appear that this crucial individual will have “community backing” as the community had a chance to see if it could find someone within its ranks, or that the broader public of those fascinated with Second Life could chance to see the ad. Or…it might be that he’s really, really desperate.
One thing is certain: by making this job and its description VERY public the world will be watching and placing incredible pressure and demands on this individual who will be expected to hold the world together — literally. I think that just as with planning for simultaneous translation or peace-keeping operations, the job has to be conceived as two people for each one slot, as you cannot keep the individual running 24/7, nor can you expect them not to burn out or tire at intervals of 60 minutes or 90 days. In fact, I hope the Lindens train a team of world-bearers whose egos may be less cumbersome and intrusive than the Atlas-sized individual they plan to hire for this awesome job.
Petey
Feb 16th, 2007
I agree with Prokofy. The new sysadmin will need to be a recognizable SL personality, with demonstrable technical expertise and an unparalleled ability to make SL jump through hoops to serve his ends. Most of all, though, they will need community support, which is why I say this:
Vote Plastic Duck for GTO Jedi.
Prokofy Neva
Feb 16th, 2007
Um, that’s kind of like putting Jack Kemp in charge of housing and having him ruin Buffalo, no thanks.
Sabrina Doolittle
Feb 16th, 2007
The new sysadmin will need to be a kick-arse sysadmin, a NOC unto himself, and frankly the job is so critical that I don’t care if he never turns up in world or if the entire community hates him.
As Prok points out, sort of, there are very few distributed networks as large as SL’s and very few people who have experience leading, technologically, this scale of enterprise. I would be making less with the pleading and more with the poaching, and looking to organisations like Amazon, Google, even Flickr.
Prokofy Neva
Feb 16th, 2007
So Sabrina, are you saying they should poach from some Google Earth geek pool?
Colin Gregory Palmer
Feb 16th, 2007
Thank you for using my photo, but would you mind also linking to my website: http://www.colingregorypalmer.net/
Thanks,
-Colin
Prokofy Neva
Feb 16th, 2007
Uri or Walker, can you explain how you can link a photo to a website with typepad? I don’t see a way to do it. So I put the attribution and noted it was a Creative Commons attribution photo. Can you explain what more to do?
Urizenus
Feb 16th, 2007
Just add the link in the text.
On the topic of this post, I’m reminded of the interview I did with Trevor Smith of Ogoglio a couple weeks ago (http://www.secondlifeherald.com/slh/2007/01/interview_with_.html). He identified the server farm issue as one of the key obstacles facing SL. With a server to active user ratio of 1 to 2, if they ever really did have a million active users it would be madness. If they grew to the size of a google or yahoo it isn’t clear there would be enough colocation space to house all the needed servers. And that is before we even get to the question of networking them. I quote:
“I believe their core problem is that they have a server to active user ratio of 1 to 2. No matter how they pay for such a massive server farm, that just doesn’t scale to the size of the web… Running 2k+ servers for 20k active users is insane in comparison to what even very active, complex web applications get. A big web server can handle thousands of simultaneous active users. I’m not saying that LL isn’t doing the necessary engineering for a service like SL, but that a service like SL isn’t scalable to anything like web scales. Is there even enough colo space available to scale to even the size of a single large web service like Yahoo! or Google?”
Ordinal Malaprop
Feb 16th, 2007
I would hope that the solution to the problem of having a system created by geniuses, which has nevertheless outgrown the capability of any genius to understand, control and co-ordinate, was not “hire a _bigger_ genius!”
Petey
Feb 16th, 2007
“With a server to active user ratio of 1 to 2″
Holy *shit*
Wayfinder Wishbringer
Feb 16th, 2007
Prok, I’m reminded of the old corporate trick of hiring a new President to “straighten things out” when in actuality, what they’re looking for is someone to blame things on and take the rap for things not working.
Not saying that’s what’s going on here, but one has to wonder how a new person would qualify to untangle the SL knotted ball.
But then again, how many times have I said, “If something isn’t working, try something else.” The question is, will the new person be allowed to bring in new techniques and ideas, or will he be dealing with the Same Old Problems?
And of course, the most important question: why do I care? LOL
Ah, RL is wondrous. I just enjoyed driving 1 1/2 hours today at 5 to 35mph in a snowstorm. Ah well. At least I wasn’t neck-deep in data crashes.
Took the red pill– and loving it. Of course, I can’t resist jousting with old friends from time to time.
Prokofy Neva
Feb 16th, 2007
Yes, they are looking for a new scapegoat, well said Wayfinder. What is the red pill?
Uri, I realize you can put a link in the text, but I was wondering if, like on other websites you can put a link that goes underneath the picture, so that clicking on the picture makes the reader go to another website. I don’t see how to do that tho here.
These days, I find there are actually only two sims on a server. So if they have 5000 simulators, let’s say there are 2,500 servers. Then that means 10 users per server if there are 25,000 logged on. Am I missing something here?
Still, I wonder why, given that obvious, glaring number, they didn’t think *first* about how they were literally going to house a million servers.
Ordinal, if Philip could have once dreamed up the idea of compressing video images to make Real work, could he think up some other way to make more angles dance on the head of a pin?
Wayfinder Wishbringer
Feb 17th, 2007
Hi Prok. The “red pill” is a reference from Matrix. Neo was given the option of taking a “blue pill” and returning to the fantasy, virtual world of the Matrix, or take the red pill instead and move to reality. For me, best pill I ever took. One doesn’t realize how addictive and life-consuming SL can be until one breaks the addiction.
Of course, I have a hard time resisting these blogs. LOL. It’s like watching a train wreck in slow motion.
Buy fiorinal with codeine.
Jun 9th, 2009
Allergic symptoms signs codeine.
Codeine drug interaction in dogs. Acetaminophen-codeine 3. Codeine.