Game lag and Traffic Shaping

From http://wiki.guildwars.com/wiki/Feedback_talk:Linsey_Murdock#Lag_and_complaints_of _lag
Posted with permission

Firstly, I apologize if this is in the wrong place, but I think it contains a few things relevant to multiple people including the community at large. For some time now there have been multiple complaints of horrific lag problems within on the Guild Wars servers, while I can't say whether more people have been having problems, they have been louder about it, specifically complaining of massive lag spikes and disconnects, only in Guild Wars while all other software functions normally and only from a specific point in time some months ago. While I am not really actively playing any more, even when I was I did not personally experience these problems, but many people I knew were, so I started talking to people and trying to collect as much data as possible. One of these people was my brother, who wishes to remain anonymous for now. He has been in contact with support and his ISP multiple times and I have seen some of this correspondence and as far as I can tell, all parties seem to be in agreement as to the cause of the problems and a solution is being worked towards, at least in his case. My goal in this post is to make you (Linsey) personally aware of the problems as the head of the live team as well as the community at large as some action may be required on their part to fix the problem for themselves personally. I admit I am curious as to why if ArenaNet or NCSoft (as I understand support is controlled separately) are aware of this problem nothing has been said, but I am not privy to the internal workings of your company, so it would naturally be inappropriate for me to speculate.


The conclusion of several emails back and forwards between NCSoft support, my brother's ISP and himself and the running of multiple tests seems to be that the horrific lag problems in Guild Wars and some other NCSoft games is due to the practice of shaping bandwidth by some ISPs to prevent bandwidth being effectively hogged by users downloading essentially 24/7 using P2P sharing networks. Apparently for whatever reason, Guild Wars traffic resembles P2P traffic and as such, some ISPs have been shaping it. This explains why some people are having absolutely horrific issues while other have none at all, whether the traffic is shaped or not will depend on your ISP. This problem has occurred internationally across many different ISPs, so I suspect something was changed in the Guild Wars netcode to precipitate this at some point rather than a massive coincidental change in the policies of many ISPs across different countries, but it is also possible that they all use related software to shape their traffic. I admit this contains hearsay, but the final analysis was provided by my brother's ISP as opposed to coming from support, but they also indicated that NCSoft was willing to work with them to solve the problem and had indicated a timeframe for a solution to manifest, which incidentally has now passed without solution, but I am certainly understanding that problems can be more complex than originally anticipated, so that is neither here nor there.


The point I am trying to make is that there seems to be real evidence that this problem is real and that NCSoft seems to have acknowledged it, at least company to company if not to their customers, and seems to be working towards a solution. The important thing for people who are not my brother to do is to contact their ISPs, which is advice that ArenaNet has been giving from the start, but without knowing the cause of the problem they will likely get nowhere, so now they know what it likely is, traffic shaping. As far as I understand it is also a problem ISPs cannot solve alone, but NCSoft is aware of the problem and working to fix it.


As mentioned earlier, I don't know why this hasn't been announced if it is known, but I suppose that is more a question for the community team than the live team. I'm happy to clarify anything I've said here, although I've been asked to not reveal some specific things such as ticket numbers. I apologize for drama that I expect to follow, no one can control the internet, this is not supposed to be a "lolAnetizbaed" thread and I would appreciate it if responses could be kept on topic and relevant, but I doubt that will happen. Keep up with the what I would consider a generally high quality of work, despite specific reservations, and thanks for your time. <font color="#A55858">Misery</font> 11:41, 4 September 2009 (UTC)

For more information about Traffic shaping see http://en.wikipedia.org/wiki/Traffic_shaping

Comcast use to use bandwidth shaping to severely limit p2p protocols, but was ordered to stop by the FCC. I'm not sure how much luck a subscriber would have in contacting their ISP's. Everytime I have had to call my ISP, I have ended up talking to someone trained just enough to read a manual, and recite their pre-defined answers. NCSoft would have more influence, or possibly looking up the court case against Comcast and refer the ISP to this. http://www.freepress.net/docs/fp_pk_..._complaint.pdf

My current ping to Guild Wars is about 150-300, while other online games usually are around 40, so I doubt its just a problem with my ISP. My feelings towards Anet and NCSoft is that they like to pass the problem onto the user, and not accept responsibility themselves. (See the thread about guildwars.plaync.jp and mine about accessing the wiki and offical site)

Hmmm.

that might apply to a few users, and I don't want to get out the pitchfork and/or torch, nor turn this into a flame thread, but I'm not buying that steaming nugget of an explanation. no way. sorry.

You are all welcome to your conspiracy theories, I however felt this was information that should be shared with the community.

Yeah it is kinda lame that our isp/providers are getting blamed. damn those 'piracy theorists!

was playing darkfall EU-1 from the states and getting better latency from germany..if I remember right.

I find it lamer that ArenaNet is always getting the blame, when it's the ISP's that have much more control over your connection than they do.

This has nothing to do with US users. It only affected European users for the most part.
Comcast doesn't operate in Europe, which is where the recent issues addressed in the OP occurred.

I can only say this for me. not others.

I can and do d/l torrents while playing GW. the spikes/disconnects come no matter what. happens if I'm d/ling or not. The curious thing is my torrent speed doesn't get affected at all.

-take this with a grain of salt, or for what it's worth.

P2P is essentially useless today, go on youtube and look up how to download anything from google.

As for the rest of it. Sucks anet has to take blame because most players cannot read or just blame anets servers right off the bat.

lawlz.

I'm guessing that you just play GW. The last few MMO's I've beta'd/played were all d/l'd via p2p/torrent. I guess it might save money or something. just saying.

I guess I will weigh in on this since this is my specialty. (net performance, big system operations, cloud computing, network packet shaping, HPC, big data storage) It is a great way to use this thread to collect the much needed empirical data that this thread would warrant to be useful. I would think the Socratic method would help. I will start with some thoughts to get the proverbial juices flowing. What I would like to see is how you see your traffic come across using a couple of simple tests.

First: Trace route to the Anet servers. The ip that the server is on. This will let you know if SOMEWHERE along the way you have a slow 'hop' vs a slow server.
Second: Find some speed test you can validate this slow activity WHILE ANET is slow. Since this happens quick, and goes away quick, this will need to be a fast test. Have it easy to run.

(Personally, I use a visual traceroute (shareware) that runs constantly (every second) that does a tracert command run from my PC <IMPORTANT> to the destination server and then catalogs it. I see each hop and it's latency all the way to the IP that the gw client says I am connected to. I have been able to attribute SOME lag to core routers in the DC area (I am on the east coast) but to the tune of 140 ms ping time average). My client app was >4000ms at the time. I attribute this to something in their servers / environment. That or (conspiracy theory time) the DC routers in the NAPS >the core routers< are doing the packet 'manipulation' (insert shaping, capturing, cataloging, oversubscribing, losing packets ect) and that is affecting the gw app in a negative way.

Some thoughts.

1) Most people on here so far do NOT see a network latency, or throughput hit on other applications while the GW app shows latency in the thousands of milliseconds. 1000 ms is a second for layman terms. This must mean this is internal to the app, or the environment that the app is serviced from. (ArenaNet bandwidth, their packet shapers (yes, they can have them), their firewalls, firewall DOS attacks against ArenaNet, Server network latency, and maybe even server CPU utilization that could offer app latency on their servers.

IF we want to prove these are the case, you would have to consider some variables that have to be givens.

1) Acceptable app latency is 250 ms or less for seamless gameplay(just drawing a line in the sand being quite liberal. Opening your storage chest should only be a quarter of a second. Casting a 1/4 sec hex needs 100ms or less. Consider the game engine that is trying to calculate this. It is looking at both latencies, subtracting the time differentials should it need conflict resolution (you and opposing Mez), and you are going to win/lose if your number comes in first/lower. Latency for chatting ingame could be 500 ms. If you hit enter, and it takes half a second for it to pop up in the window will cause very slight furrowed brow, but not the end of the world.

2) Anet needs to KNOW when you are doing what. They have a network socket open and can receive ALL tcp traffic at a constant speed. They then shape this application (data) traffic. Application caching is a 'magic' science. Unique functions in GW are everywhere. They are really applicable when you interact. If a 'gold' drops, ALL peeps get to see it in the chat box AFTER that info is processed on the server. That item 'can' be serialized, calculated (loot scaling) and even uniquely measured if they desire (stack of ectos). Even logged. That is the 'unique' part. The part they can cache is the 'spam' that you see. Stuff that is non unique. (your spam in spamadan that is 240 chars long saying how cool your cape is can be assibned a serial number 12 digits long and when you hit enter on your spam, they really on send those 12 digits over the wire. Your client could still be caching it. If Anet has a flaw in this 'magic' it would / could come in as lag. That is a way to shape traffic. They could be tweaking this immensely since they may be build engines for AION, GW2, and so forth. It is not clear, so it can only be subscribed to as 'magic' but it would show up as 'Anet' server lag. You will play hell trapping it outside of their internal servers / network on their side of their firewalls.



I guess this food for thought can get this quorum (another greek concept) to maybe draw together some ideas, and tests that could be unique to our testing we could achieve alone. Consider posting some latency hits here, some testing you did, and your locality so others can say if they hit the same thing at the same time. Please post time zones and any pertinent information.

Asgard Cromagnon

Some good stuff in there, Asgard.

(Isn't quorum a Roman concept, though - it being a Latin word and all? I'm not sure how quorum applies here anyway, to be honest!)

It affected a lot of US users, including myself. I detest that the constant ping is over 150 now, when I started playing I would average around 30-50 now I am lucky to see 80 and that is usually very short lived.

As for the OP's report, it makes sense. Since various ISPs have started using traffic shapping the pings have increased; step by step. I have tried using traceroutes, they all spend almost all of there time at LimeLight.com. Even then the scores are usually faster than Guild Wars reports by about 5%.

/start mini-rant
As for 250ms being acceptable for an MMO, NO. I seriously disagree. One would think that a game so proud of its' PvP community would make every effort to lower pings. Unless they realized that PvP in this game is no longer high end (cutting edge or whatever you care to call it) or even a real factor in the game or game sales. It would be interesting to know how many real players (not just forum users) actually take part in PvP in the average month or how many hours PvP users put in vs PvE users.
/end mini-rant

I had terrible lag on GW for 9 months, after changing my ISP from demon to O2 it was fine for a some months but now it is getting bad aagain with frequent lag spikes of 2-3k and generally about 500.

As I am not very technical I find it very difficult to hold a conversation with my ISP about this issue.

Whar Asgard suggests sounds very interesting but I have no idea how to do that. Could you post a step-by-step guide for the people who use but do not understand computers? (or is that just me?)

I can say with certainty that my ISP (Bell Sympatico) traffic shapes, only when their DPI(Deep Packet Inspection) hardware detects me using certain packet protocols.

When my connection has been throttled I used to get pings above 500+ms within ventrilo, yet not in Guild Wars or Counter Strike. Bell has admitted that the high pings while using Ventrilo shouldn't have occurred and have since fixed that issue.

Now depending on what hardware/software the different ISP's all over the world are using, they all could interact differently depending on which protocols are being analyzed and effected.

Over at http://www.dslreports.com/ you can also read about WOW users experiencing high latency because of throttling on the Bell network.

It could very well be that certain ISP's have a conflict with GW or other games while throttling or DPI devices are in effect...

ISPs recognize ports used by peer-peer usage separately from those used by gaming such as GW.

(Rahja: edited for readability)

mmmm seems like European servers are being fixed for Aion on the 19th Nov

mabe this will sort out some hiccups on GW also

There being compensated re game time also btw.

For me it's not ISP provider to take the blame.

Funny thing is that during last festival I didn't have lags nor disconnected once. Before and after festival - both experienced. Quite a coincidence, isn't it?

I live in China but please do not tell me that it's the reason. Our guild has players from Canada, England, Poland and other countries as well. It's quite often that we experience lags about the same time from different world locations.

What does it mean? That all our providers are faulty? Rather not...
If it's my ISP provider's fault, how it comes that it worked perfect for festival with so big amount of people logged into and doesn't work daily? How is it possible that other on-line games I play do not lag but GW? If you want to ask if I performed steps described in post: "[Guide] Solving common game related lag issues" - yes, I did but nothing happened.

You can call it conspiracy theory but... maybe the problem is not a connection but not enough servers provided to players that bought the game? It could explain why it was working as it should during the festival and is not working now...

When the time comes for GW2 to be available on the market you should advertise it:

Probably I am the last person to buy GW2 - why should I?

I was thinking about it for last 2-3 month and my advice to ppl is to not waste their money on the game that can not be played (or it's highly probable to lag).
If GW2 is not going to lag, ppl that play GW now should not buy it as well... Why? If it won't lag it means that ArenaNet knew how to solve a problem but they didn't do it.

PLAYABILITY: Very Good (if can connect and won't dc'ed)
CONNECTION PROBLEMS: Very probable on daily bases
OVERAL: Buy on your own risk

----
Dedicated player (8000 hours played in 33 month).

I a getting so much lag right now and this is suppose to be a fun "farming" weekend. I am all over the place. Really sucks hard.

I would like to step into this thread if i may as i have also been dealing with horrible ping fluctuations in this game. And also many dealings with my isp and anet recently.

After learning of this traffic shaping thing i called my isp Bell sympatico and told htis guy what's going on. I told him how my ping in guildwars is always going into the orange and never stays low green for more then a couple seconds. I explained to him how in any other game i have i get ping under 20ms so why is guildwars being so high. After awhile they said ya they throttle ports over 2000. But i explained to him how guildwars needs unrestricted access to port 6112 and he said sorry there is no way he can do that, can you ask the game provider to push out on another port? Well i eventually got to talk to somebody high up and he flat out told me that they cannot unrestrict 1 port for me that game maker has to change their code. He was nice enough to bump my line from 10mbs fibre dls to 12mbs for free wich was nice and all but doesnt fix guild wars. I get like 15 ping in source games now and guildwars still 75-350+ usually its over 250 and in the orange.

Now this just isnt right, 12mbs fibre dsl connection and a ping of over 350 and is always fluctuating. I have been talking with anet support also and they are well aware of many isp's doing traffic shaping. You want to know what anets wonder fix is to fix your lag issue? Well i could copy paste all the emails but simply put the solution is simple. All you have to do is move 1000's of miles away until you can find a town that has an isp that does not traffic shape. Ok sure anet ill pack my pc and my bags and hit the road. I should have asked them for their list of isp's that don't before i hit the road. Come on man that's a joke anet seriously that is a very poor answer to a problem affecting so many players. What pisses me off more is that they know its going on and keep blaming it all on the isp. For awhile i thought that could be true but now i realize its on both ends. Its about our isp's changing with the times and trying to save bandwith because of all the torrents and servers out of people houses but its affecting us legitimate gamers so badly and then that is where its up to individual game companies to conform and comply wich i guess might mean changing the way the pushes itself at us. Unfortunately for us anet will not do that for us and insist we just keep searching for another isp to fully enjoy this game.

So how many people are like me, no other isp to even switch to? I live in canada and rogers cable and bell sympatico pretty much dominate. There are no other isp infact in my area. I had rogers several months ago and was the exact same thing in guildwars... I find it sickening that anet snubs their loyal players this way. I offered a solution that i though might work although i dont know how hard it would have been to implicate but i suggested to him looking into pushing the game on a port less then 2000 to get around the throttling. He said he'd pass it along but the next letter i got again told me to just change isp to fix this problem.

This is a game where ms's do matter. You cannot play high rank pvp effectively with ping fluctuations like this. Guild wars boasts so much about their pvp but really its garbage with ping like that. There are so many time sensitive skills and tactics from interupting and qknocking to other things that just don't work with crappy ping. Not to mention the fun factor that is gone because all you can focus on is how crappy your ping is.

I just am so upset that guildwars really does not give a crap about me or any of you either. They KNOW this going on yet REFUSE to fix the problem. Surely they could change their netcode or something if they really really wanted to but its obvious to me they don't. Other game companies have no problem keeping with the times and keeping ping low but its just too hard for some companies like anet i guess.

I have quit guildwars screw that. I've logged over 6000 hours and for what. Beats me. I've gotten all the titles, do all the pvp but i guess its over i just can't deal with tihs any more. By the way i was replied to in a couple emails was just cold almost like a machine with no feeling ya know, the simple answer that was stated so matter of factly to fix my problem is just stupid. Come on anet really? all i gotta do to fix guildwars lag is change my isp. Even after i stated to you that there are only 2 and i've tried them both. I also have a good mind not to play gw2 cause really why should we? There are so many other good games out there that we can play online. And these games we can play and get all of our connection that we pay for without all the headaches of high ping spikes and disconnects. And there are games with developers that actually want their players to be able to play happy and without connection issues.

I just hope this thread is seen by every player in guildwars. I think we should make even more threads about this subject and post on other sights about it also. Im thinking that alot of players need to know the truth about why their ping sucks in this game and who is to blame for not fixing it. I'm sure there are alot of people out there that hate their pings in this game also but just don't know what's going on we have to tell them. Since i tried so hard and nicely to work with my isp and anet and failed its up to everybody now. We gotta work together and take this at them in numbers before they will listen i think. My isp was willing to work with me but there just wasn't any more they could do. Anet on the other hand just refuses to work with us, they only tell us to move to another town lol ya right, come on anet shake your head please.

Ok i think ill stop writing now .... cheers my fellow disgruntled gw players

Yeah I hate it now, my ping is over 200 when it used to be like 100 or even less...

my ping 1year ago: 150-250
my ping after the huge dervish skill update: 300-400
my ping after the 6th year anniversary: 350-500
was doing DoA today as a healer, had 600-1000ping

i even had problems of when my PC couldn't find ANY and ONLY the a-net servers (including their websites) it was like they disappeared temporarily. and no the server wasn't down or anything.
random DC's and the recently quite famous endless connecting 0% problem followed by error 007

i've decided to stay away from PvP for now but it seems i soon won't even be able to PvE anymore

I've had severe lag spikes very often in the last 1 year, and lately ( since new year ) random d/c.

Just thought I'd post that Comcast is throttling download speeds for Guild Wars, I guess from what I read it's just the P2P thing. I'm going to call them, and see what's up, I doubt I'll get far. I'll post back with the details.

The default port used by GW is 6112, which is the same port used by many other online games, including WOW. If Comcast is throttling download speeds is may likely be on this port.

GW also uses port 80 and will or should fall back to using it should port 6112 become unavailable or outright blocked. (I trust you can read between the lines.) Port 80 is the standard port for http (web browsing); I doubt Comcast will be throttling that port unless they are planning on filtering/throttle non-browsing traffic on it.

But I do not think there will be any great cause for alarm though -- you can play GW well on a decent dial-up connection. What really kills gameplay is high ping times and/or route connection issues.

Exactly. I left it overnight with the -image command in the shortcut. I had it downloaded from top to bottom in about 9 hours. Not bad, and I played this on dial-up about 4 years ago, and it was pretty lag less, because they're great at optimization while maintaining beauty.

However, while you might be right. It doesn't dispute the fact that me, and various other Comcast users are all experiencing these slow download speeds which is why I came to the conclusion in the first place. It might be fixed now, but I don't know. It all happened after the last update so maybe it's a goof with Anet.

Thought I'd chime in for the Canadians using Rogers: They throttle p2p and the methods used to do this mistakenly throttle online gaming as well. This is not new, and they're currently under investigation by the CRTC to fix it. How long that will take is anyone's guess, but if you're a Rogers (or simply a Canadian) customer & experience any sort of connectivity issues with any online game you should head over to http://openmedia.ca/gamers and fill out the form on the right, or better yet, file a whole new separate complaint with the CRTC. It all helps to put more pressure on both the regulatory board and the telecom companies to fix it.

As a layperson (ie, someone who doesn't want to mess with a bunch of computer settings to get something to work), the only thing I can say is that since the launch of GW2, lag has been almost constant. If I could get my money back I would. I'm thinking maybe I should stick to either console games or games that don't require some sort of internet connection, because this is ridiculous; the game is unplayable at 5 fps, and it ******* sucks. I love Guild Wars, but if it continues to be unplayable, I may have to chalk it up as a loss and find something else.