Guild Wars vs. MMOG's - The Tech Side

Just sat down tonight and had some long talks with some tech gods. Very enlightening and I do believe their simple explanation could quite possibly explain a number of issues such as why so many people err7, why reconnects are not in the game, and more. Now just for emphasis this is a theory. It should not be taken as fact. So presenting you:

Guild Wars vs. MMOG's - The Tech Side

MMOG's: They have servers, the servers know where you are, and what you are doing at all times. You log off it, knows you log off. You get booted, it knows you didn't log off, so it can wait for you. When you log back in it says, "Oh Inde just logged in where does she belong?" And it puts you there. Also in groups they get a signal that you were booted (since you didn't actively log) and it awaits a signal that you returned. So with persistant worlds you can go in and out of connections more readily since the servers know where you are at all times. Problem: all those servers cost a lot of money.

Guild Wars: All common areas/shared space are hosted by ArenaNet. From the understanding and investigation of several people I've spoken too, once you leave a common place into a mission/instance you are removed from the ArenaNet servers and begin to go into peer-to-peer connections. So you get booted from the instance and reconnect. You hit ArenaNet's server and it says, "Hello Inde, I have NO clue where you were or who you were connected with nor do I care, welcome to the world." It puts you into the last common place you knew. Problem: you lose the persistant connection but it's CHEAP because you are connected to all your peers versus having datacenters full of servers. This also explains why ArenaNet does not have "worlds" such as other MMOG's but "districts". They don't have to have 100's of servers per world such as World of Warcraft or other MMOG's. They only need as many servers that would support just 1 world to run everything they have to deal with for Guild Wars. This is similar to how Battle.net worked I believe for StarCraft.

Now for the issue of reconnects, and why they have not been implemented such as in other MMOG's. They can't do it as other MMOG's, they are having to find a new way. Losing connections in an instance, when you are peer-to-peer in an instance and you suddenly cannot see another member, it simply drops you (The dreaded 007 [how ironic Licensed to Kill]). Why? Because it doesn't know where you are anymore. So why do multiple people get err007 at the same time? They loose a key peer and they're all sitting there trying to connect to that peer and cannot so Boom another dead group.

So who's problem is it? Not really ArenaNet's unless you drop in a shared area. So all these err7's are your problem. You are the responsible party losing a connection whether your own ISP is being inerrupted, or someone else in your party. But should they do something about it? Yes, they need to figure out a way to improve the handling packet losses or member disconnects (such as having redundant connections waiting for someone to drop). From my understanding, all the tracert's and pings in the world to the Guild Wars server mean nothing when you err7 and drop in an instance, it's your friends you need to try doing the tracert's and pings to and finding out who has the unstable connection since it their computers you are connecting to.

Now I don't know all this for sure, I'm certain some will have more tech knowledge then I but this explanation certainly explains a number of things for me.

EDIT: Interesting discussion beginning, please read through the posts below as we dispute and argue the article before directly quoting what I have said above

hmm. maybe now that someone respected in the community has said it. all of us that arent total idiots can point to this when we try to explain why you get so many err 007s.

I was a very vocal voice that all the err7's were on ArenaNet's side. So don't give me any credit, of course I've never seen or been told an explanation like that above from anyone at ArenaNet or the community so drew my own conculsions. I'm hoping it can help others to see the problems and enlighten them as it did myself

Thank you for this post, this is some great info on how GuildWars runs on the back end.

Well just a question then. Why is it when I do drop in an instance, I still have all the items, gold, xp, and skill points that I managed to get? If we aren't connected to the Anet servers how does it know I got everything, if I dropped without a proper logging off?

But I've had the Error=007 without party members while in towns and in parties with just hench.

All very possible, but as you said, we don't truly know the way Anet has structured their servers or their code. Honestly, I'd say that peer-to-peer for each instance is only one of many ways it could be done. It does make the most sense.

If it is true, it could be easily tested (in-game) by soloing a little. If you have lag problems, then most likely the problem is on your end. Until Anet tells us that's how their architecture is, though, we can't truly know.

All right more findings. I just logged in with some GWG staff and we were consistantly connected to the ArenaNet servers (which of course explains how it can remember your golds, items, etc.) We couldn't find a direct peer-to-peer connection, but it doesn't mean that the connection to the server wasn't just routing us to others or that the data isn't scrubbed or monitored. Most likely even a combination of both, that you are hitting the ArenaNet server and offloading the instance state to a peer.

I just think the connection to many issues we are all talking about; so many with err7's, reconnects that are in every other MMOG, no monthly fee (because peer to peer is so much cheaper), the fact that we have districts versus worlds as in other MMOG's, and more points to a peer-to-peer connection. It makes sense to me, but maybe someone can enlighten me further as to why GW isn't like this. So let the debate and information continue.

Sorry, but i seriously doubt that's the way GW works. (Unfortunately i can't find Jeff Strain's "we don't trust the client side"-quote.) If i remember Jeff's last year interview correctly, then GW is using a client-server architecture for each instance to make the game 'hack-proof'.
I agree with the assumption, that a reconnect-feature would require storing more data, leading to more traffic, resulting in higher costs. Anet's business model might not cover this.

What about some of us who aern't tech 'leet'? Because really if there's a problem with my computer or isp or whatever that's causing the lag, I won't know how to fix it.

Seut, read my post right above yours where I'm saying this:
And in an IGN interview I did find this:

This still does not mean that they don't offload the instance state to a peer, therefore reducing the CPU cycles needed to maintain the instance. Which explains why you are always connecting to the server (to catch hacks or cheats). And I also don't thinks it's coincidence that Mike O'Brien, founder of ArenaNet, was one of the original creators of the Battlenet system.

Sorry, i'm a little slow in writing english

Jeff Strain explained parts of Guild Wars' architecture in an interview (i think it was before the release of Prophecies, unfortunately i cannot find it).

As you have already realised that a full p2p setup won't work, I'll make this as brief as I can.

Assuming the design is as you predict, with one (or a few) central world servers and several distributed instance servers, then I am sure there is some note somewhere in the central world servers as to which instance ID you have been sent to. This is necessary because it has to transfer your character record to the instance server, at least, but also because when you are done with the instance that your character information can be percolated back to the central servers.

Therefore, the problem of reconnecting is fixable as follows: when the instance learns that the link to you has been broken, it uses a callback to inform the central server. When you connect back, the central server knows which instance to shunt you back into.

The problem then becomes how to synchronize connections with an instance after a period of disconnection. For all I know, this might be impossible or extremely hard. Depends on how stateful GW's protocol is. Does the instance server keep a record of everything that has happened in the world----every damage, every death, etc.----or is that info stored in the clients? If the latter, then when you reconnect the instance server will have to ask the clients for a "what's up?" to send to you, which will surely mean lag for every one. (Assuming it's possible.)

Or we could just be barking up the wrong tree. We're basically just speculating on GW's network design here.

seut, Jeff Strain, or anyone else at ArenaNet cannot give away their entire server architecture tradesecrets. I found an interview that Mike O'Brien stated it's not peer-to-peer that you indeed connect to the arenanet servers. That the game itself is on the ArenaNet servers. I don't think the arguement is that the game itself is not on the servers. I guess their justification and definition of what exactly is peer-to-peer could be a debate in itself. But I'm almost convinced that the state of the instance is most likely maintained off guild wars servers.

Here's the podcast with Mike O'Brien but he never clearly answers the peer-to-peer or the "how do you keep the costs so low" question:

http://www.pcgamerpodcast.com/index.php?paged=2

The more I think about it, the more I realize it's probably not P2P at all. The key thing that disputes this, I think, is rubberbanding. The only reason rubberbanding happens (as far as I know), is because the server says your character is at point A, and the client says your character is at Point B, and correction ensues.

Obviously this can only happen if there was constant communication between the client and server. When lag steps in, a number of things can happen on the server-side without the client knowing about it, at least until the lag stops, or you are d/c. This can happen when grouped and when soloing.

P2P is the most cost-efficent way to go, this is true, but I think in this case Anet has opted for hack-proof over cost. Frankly, I love them for it.

But arcanemacabre, that doesn't dispute the peer-to-peer or what we are discussing with the instance state being on someone's computer. If I'm trying to ping someone who holds the instance state and they show me as being 5 feet away from where I'm at on my computer the rubberband affect could still be valid. It wouldn't matter if it was the server or another person's computer, rubberbanding could still happen.

I'm saying it does this even when soloing... which means it's not pinging off anyone else. Unless you mean P2P with everyone else who's playing, not just in your own instance?

And what would decided whose PC would be the 'peer'? My internet sucks, for example, I get disconnected from it numerous times when playing. And I have never dragged other players.

I just did a bit of traffic analysis, and I receive about 25x the amount of data I send to the GW server in a single RA round. This makes me think that the state is not being stored on my end.

Here's why I think no significant game state will be stored on clients: because many people use asynchronous connections. My ADSL rx speed is a little less than twice my tx speed. If I were to be serving data to 7-23 other people, my connection would be swamped. This definitely doesn't happen to me.

What is possible is that my own state is stored both locally and at the server, and that the rubberbanding that arcanemacabre mentions is the result of my local state getting ahead of the server's state because of delayed synchronization.

LagunaCid, I of course don't know their entire protocol so no one can answer that. As I said, there are certain server architecture secrets that will never be given out by ArenaNet. While you have never dragged other players, there are other reports from players stating that when they have err7 others have automatically in their party been dropped as well. So this is just a discussion more than anything we could pin down without a response from a tech person at arenanet.

And arcanemacabre this is just theories here. But I do think, as I explained above, it does give insight and perhaps an answer into the subjects of why there are so many with err7's, reconnects that are in every other MMOG, no monthly fees, the fact that we have districts versus worlds as in other MMOG's, etc.

http://www.guildwarsguru.com/forum/s...ferrerid=92302

Although I know Gaile Gray isn’t a dev, if what is said in her comment is true, then it does show that rubberbanding is because of the server and one other interesting fact that there might be some subtle form of p2p.

In order for those observing to see the speed increase even for a moment, then that means there is some information that is passed to others that isn’t checked by the server for a brief time. So either the unchecked information goes from the speeder straight to the server and then directly to the observer, or the speeder’s information just goes straight to the observer (p2p). I probably should also make a special note that Gaile Gray used the term “I think” which means that it is unconfirmed rumors about the observers seeing the speeding.

Also awhile back, I listened to an online radio show made by Gaming Steve where he interviewed Jeff Strain and at one point Jeff Strain explained a little (very little) about instancing and district sizes.

Found here: (should be episode 34 – 10.10.2006 and around 22min into the show is when instancing comes up.)
http://www.gamingsteve.com/archives/...teve_ep_30.php

I found it interesting how instances are created by figuring out the best server location between two (or more) people and then sending each player’s information to that server. I wonder if this means that the lag we are experiencing is due to the servers we’re being sent to as the best for all the people in our groups, the same servers most people are being sent to because it’s registered as the best for them as well, which yes would be silly, but bugs in code sometimes do silly things (assuming it is a bug).

I'm not very experienced with server stuff atm, so sorry if all I'm saying is newb talk.

when I am in a mission/instance, I am not on A-net's servers and A-net has no idea what I am doing. Is that right?

Interesting thoughts. I always assumed that every time your exited a town a new address space was created on a server(a virtual GW world just for your party) you then played in this address space until you reached a town. Everything you need for the mission is created for you at the time, monsters, quests, scripts, NPC etcs.

Just like proper operating systems your address space is sealed off from all the others to protect you and others from things like crashes, hacks and exploits.

Thats why it's called an 'instance'

The entire hardware that GW runs on is probably a huge cluster of servers(or a few BIG ones) that are split into ones that control towns(thats why we have size limits on districts) as a district size is probably the limit a server/cluster can deal with. You then have the game instance servers/clusters

Now as for reconnents, the reason why it's fairly hard is to allow reconnects they would have to poke a hole through that nice sealed of little bubble thus leaving it exposed to what else is going on in that server (crashes, hacks, etc etc). Therefore it's a security and integrity issue as much as it is a game one. To get a secure system you need as few holes in between address spaces as possible.

My cobbled together on a beer mat theory on how it fits together is as below.



Now, A possible solution would be to periodically store a copy of the entire address space and allow it to be dissolved and recreated based on this copy. This could be done either automatically, or by say the purchase of an in game scroll of Divine Backup . The downside to this is that you need more capacity than needed as you require more space to store these backup address spaces.

it *could* be pure P2P but even P2P has reconnect facilities, so I expect any P2P aspests are if you like tying your parties connections to this address space we talked about. Imagine a big ball of wool being knitted by using seperate yarns of wool(each one of your party members) and the required info from the server, then when it's done that ball of wool is called an instance

So again, I believe reconnects are architecturally possible but may represent integrity and security issues at are unnaceptable to AN.

I am pretty sure it is not peer to peer, because nowhere are we required to open a port in our firewalls to allow incoming connections.

Nice discussion. Really makes me wish I paid more attention in Networking class. >.<

No way it's P2P.
a) That would open so much hacking options it's not even funny.
b) There is no way people on analog dial-up would be able to play

I still haven't found the interview i mentioned before, but this is a very good article by Mike O'Brien & Gaile Gray how to prevent cheating in games (and probably how they programmed Guild Wars):
http://www.arena.net/news/articles/m...cle040802.html

the odd thing is the err 7 problem, at least for me went away with the last update.

It's not on our end, the game was broken in some way or the servers couldn't handle the strain of the amount of people playing.

I think it works the same as google's pigeon rank system.

If players were hosting instances, there would be thousands of games completely crashing (dumping every player) every hour because of the host/hosts leaving the game.

Yes, and that's why there's a monthly fee in WoW and other MMOG's. It takes a lot of money to run the servers. Oh and in WoW I heard there's like a waiting list of some sort, since there can't be too many players at once.

I do believe most error 7's occurred right after Factions came out because a lot more players entered Anet servers. A lot of dial-ups are pretty bad these days though, but had no problem before Factions.

Okay, I think we've pretty much determined, and I've stated, that it's not a complete peer-to-peer so we can stop with those statements. The point in the discussion we are it is whether the servers are offloading the instance state to a peer (and it doesn't have to be a single peer) as this would indeed reduce the CPU cycles needed to maintain an instance and you would still be hiting the ArenaNet servers.

eternal pho, many people who are experiencing err7's are not on dial-ups and the increase of err7's has increased dramatically over the last several weeks so I don't think we are dealing with either a dial up or faction release issue there.

While it doesn't attempt to explain the amount or reasons for an Err7, one possibility for the lack of reconnects is just due to the way Anet's servers save instance data. Cutting corners like not saving the state of disconnected players amongst the instance data would cut some overhead in terms of memory usage, thus lowering the overall costs of their server infrastructure. Just like an instance of any form, it's a separate set of that data, which is created when a player enters the area and used until they leave, whereupon it's cleaned up. The more information about that instance you need to save, the more memory and server processing will be required.

As discussed and decided, the idea of a peer to peer connection being established and let go by the servers isn't a possibility. Like any persistant online game, it requires that connection to the server to govern everything from loot drops to player connections. The game clients themselves don't enough information on their own to manage the game and theoretically, if it was entirely p2p, there would be be gaps without communication to the server, where the client data would differ from the server data - a common result of lag. This would lead to things like, say, you picking up and item/gainingxp, disconnecting give minutes, then logging in again and not having those new acquisitions.

Basically to summarise, the probably source of these issues vs a normal MMO (like WoW) is Anet's server infrastructure. While it's probably not using any form of loose p2p connection, it is 'lighter' in comparison in order to keep costs down. The whole way the game instances everything is already one way of minimising overhead - versus the server power required to govern an entire continent persistantly.

Also, for those not in the know about how other MMOs (WoW) work, each Realm (server) hosts one copy of the large world area, which in itself is not instanced and has only one loading zone, between the two continents. Smaller instanced areas are found throughout the world (dungeons or instances to the players). If you disconnect, your character will eventually time out (much like any lost connection) and when you reconnect, you will still be wherever you were. In this sytem, your characters position in the world is constantly saved, even when you log off. In Guild Wars, the only data saved is the last town and even then, no local position data - when you log in, you're placed randomly inside the town, whereas in WoW, you'll be standing exactely where you logged off. Consider the processing power and memory to store that versus not at all, then multiply that out by the millions of players.

Well, I don't know much about this overall, but, while complaining about lag and error 007s ingame someone showed me how the problem was that the server was not uploading date from your computer, while it was sending the data.

i do know that you cant be just a *little bit pregnant*

and there is not just a *little bit* of P2P as far as game security goes.

i am looking for the interview but it was stated that everything at all times was in house on their servers for complete security.

any time you hand off ANY part of a online game to someone elses pc you break security.

Loviatar, again ArenaNet is not going to disclose their full server architecture or tradesecrets. And really, one of the creators of the Battlenet system Mike O'Brien could certainly find a way to create such a system don't you think? If the data is scrubbed and it has to go through the arenanet system at some point it would still be secure. And I've been through all the interviews, podcasts, etc. and have all ready made reference to them in this thread. They state that the game itself is on the servers, that all towns and instances are hosted by their in-house servers (no complete reference to "everything at all times was in house for complete security" though.) It doesn't discount anything we are saying here that some information is being offloaded. So the fact that it is not a true peer-to-peer has been established but doesn't mean that they don't have a new method being used here. And if they have indeed created a new method then I think doing a test such as gr3g did but with 3-4 people in an instance would give more data versus going into one alone.

This is just a theory that seems to give an explanation for a lot of the hot issues that people are discussing right now

Wait... so noone can read the code to see what it does? Surely someone can just open the gw.dat or client file and see whats going on. What is it written in, C++, or something along those lines? I'm no expert by any means about programming, but the programmers can't be the only ones to be able to read it, could they?

was it this one?

http://pc.ign.com/articles/534/534454p2.html

Well normally when someone err7's in my groups the whole group is hit by a lag spike. Then we notice someone afk for about 30 seconds. Then they drop. Normally its just one person, but occassionally 2 or 3.

But since everyone seems to agree that the system isn't completely p2p (some saying not p2p at all) and that the server is involved.... is there any reason for it to be p2p at all? I mean if the server is keeping track of everything anyways, such as monster positions, pc positions, skills being used, drops, xp, etc...what else could a p2p do that would help?