ACE Timeouts caused by my ISP?

Discussions about UT99
User avatar
Sp0ngeb0b
Adept
Posts: 376
Joined: Wed Feb 13, 2008 9:16 pm
Location: Cologne
Contact:

ACE Timeouts caused by my ISP?

Post by Sp0ngeb0b »

Hey people,

I didn't expect something like this happen to me personally. It's nothing server related, but only to my client side - or well, more my ISP connection side. After having no real issues with ACE for years, I'm having huge troubles playing on any ACE server for the last 4 days. The symptoms are always the same; I get kicked for "Timeouts during the checks" always 7 minutes into the game. Alright, and now comes the strange and unfortunate part:

I was able to locate the issue down to it's root. I'm almost 100% sure that it's my ISP that's causing the timeouts. I've tried it with multiple installs from my network, also via different machines, always the same result. Now, when I connect via my university's VPN, I'm not having any timeouts at all. Only problem is, that I get huge lag spikes and packet loss due to the VPN tunnelling. My regular 50Mbit coax connection gives me a perfect, smooth ping, but always timeouts.

I'm pretty clueless at the moment; especially since it looks like it's nothing I can actually do something about it. So, right now as I see it, my only options are:

1) Quit playing
2) Wait for the ISP to undo whatever they did 5 days ago
3) Remove ACE from my servers and only play on non ACE-servers
4) Hoping on anth to read this message and hop in with additional support

What I don't understand is that my own network connection is way more stable and powerfull than the VPN one; still I get timeouts, but no timeouts on the shitty VPN connection. So my guess is that my ISP implemented some new technical methods which ACE is simply not compatible with.

Here are two winMTR logs to my gameserver in sweden (which is running ACE), the first with my own connection and timeouts and the second via my university's VPN and no Timeouts:

Code: Select all

|------------------------------------------------------------------------------------------|

|                                      WinMTR statistics                                   |

|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |

|------------------------------------------------|------|------|------|------|------|------|

| 1313a-mx960-02-ae17.aah.unity-media.net -    0 |  617 |  617 |    5 |   11 |   94 |    6 |

| 1313a-mx960-02-ae17.aah.unity-media.net -    0 |  617 |  617 |    8 |   13 |   36 |   12 |

|                          84.116.132.162 -    1 |  609 |  607 |   11 |   14 |   47 |   13 |

|                   ae0.fra44.ip4.gtt.net -    1 |  613 |  612 |   11 |   14 |   36 |   13 |

|              xe-5-3-0.stk30.ip4.gtt.net -    0 |  617 |  617 |   39 |   44 |  119 |   41 |

|              bredband2-gw.ip4.tinet.net -    0 |  617 |  617 |   37 |   42 |   67 |   40 |

| cr-se-sto-kn7-2-gi0-0-0-6.bredband2.net -    0 |  617 |  617 |   38 |   41 |   64 |   39 |

|      c-82-209-160-75.cust.bredband2.com -    1 |  613 |  612 |   38 |   43 |   63 |   41 |

|                   as51701.bredband2.net -    0 |  617 |  617 |   39 |   44 |  227 |   41 |

|                           46.253.192.33 -    0 |  617 |  617 |   39 |   44 |  173 |   39 |

|                          46.253.196.193 -    0 |  617 |  617 |   38 |   41 |   75 |   39 |

|________________________________________________|______|______|______|______|______|______|

   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

Code: Select all

|------------------------------------------------------------------------------------------|

|                                      WinMTR statistics                                   |

|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |

|------------------------------------------------|------|------|------|------|------|------|

|     n7k-ww10-3-vl120.noc.rwth-aachen.de -    3 |  837 |  816 |   17 |   50 |  931 |   42 |

|  n7k-lssnord-1-et2-6.noc.rwth-aachen.de -    3 |  833 |  811 |   16 |   50 |  930 |  106 |

|      fw-xwin-1-vl158.noc.rwth-aachen.de -    3 |  849 |  831 |   16 |   48 |  905 |   25 |

|n7k-ww10-1-xwin-vl106.noc.rwth-aachen.de -    3 |  845 |  826 |   17 |   50 |  906 |   25 |

|                cr-han1-be4.x-win.dfn.de -    3 |  845 |  826 |   24 |   61 | 1000 |  452 |

|cr-tub1-hundredgige0-6-0-0-7.x-win.dfn.de -    3 |  844 |  826 |   29 |   66 |  917 |   37 |

|be4193.rcr11.b015814-1.ham01.atlas.cogentco.com -    2 |  853 |  836 |   33 |   69 |  984 |  452 |

|   be2198.ccr41.ham01.atlas.cogentco.com -    2 |  865 |  851 |   33 |   68 |  951 |  444 |

|   be2281.ccr21.sto03.atlas.cogentco.com -    2 |  853 |  836 |   51 |   87 | 1002 |  458 |

|                           149.11.76.158 -    3 |  829 |  806 |   51 |   91 | 1005 |  106 |

|          po11-40ge.sto4.se.portlane.net -    4 |  817 |  791 |   48 |   87 |  967 |  101 |

|vl-3305-adminor-demarc.sto1.se.portlane.net -    3 |  828 |  806 |   49 |   86 |  938 |   52 |

|                          46.253.196.193 -    3 |  845 |  826 |   48 |   84 |  998 |  459 |

|________________________________________________|______|______|______|______|______|______|

   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider


I would appreciate any input from you guys on this matter, as I don't see any real answer to the problem right now.
Website, Forum & UTStats

Image
******************************************************************************
Nexgen Server Controller || My plugins & mods on GitHub
******************************************************************************
ShaiHulud
Adept
Posts: 459
Joined: Sat Dec 22, 2012 6:37 am

Re: ACE Timeouts caused by my ISP?

Post by ShaiHulud »

That's a really frustrating problem Sponge. My knowledge is pretty limited, but perhaps (as a starting point) you might try telnetting to the port upon which ACE is listening for connections. The ACE help file suggests this would be Serverport + 1 (7778 by default). You'd see a blank console screen if this was successful (https://kb.acronis.com/content/7503).

I don't know why your ISP might have blocked some outgoing ports, that would have the potential to arbitrarily upset a lot of network software. If you can't connect to the ACE listening port, you might want to either give your router a thorough examination (do these things ever update themselves?), or pass on that information to your ISP and ask for an explanation.
Higor
Godlike
Posts: 1866
Joined: Sun Mar 04, 2012 6:47 pm

Re: ACE Timeouts caused by my ISP?

Post by Higor »

You either can't connect to the datalink or the packets of said connections are being mangled.
Yeah, your ISP is at fault.

Also:
5) Run UT on a Linux based environment.

PD:
Get a packet sniffer and try catching the data related to the server's ACE port, if there is data going thru, then your ISP is doing something nasty.
It once happened to @Chamberly that one of the game servers she had, the redirect would mangle the data and all redirected files would have additional unwanted bytes.
User avatar
Sp0ngeb0b
Adept
Posts: 376
Joined: Wed Feb 13, 2008 9:16 pm
Location: Cologne
Contact:

Re: ACE Timeouts caused by my ISP?

Post by Sp0ngeb0b »

Thanks for your replies. I'm perfectly able to connect to the datalink, initial checks are no problems either.

Code: Select all

ScriptLog: [ACEv08h]: Player: <//UrS//>Sp0ngeb0b is now connected to the datalink at port: 7778
Log: ACE: DataLink initialization complete.
Log: ACE: Ident  : Player:<//UrS//>Sp0ngeb0b
.
.
.
ScriptLog: [ACEv08g]: [<//UrS//>Sp0ngeb0b]: [TIME] 05-12-2014 / 13:29:27
It seems like the first few periodic Checks by ACE are not an issue either, as I'm always kicked after 7 minutes (with a periodic check intervall of 45). Anyway, I will check with WireShark what the communication between me and the DataLink is really like.

Now, assuming that it's really my ISP who's doing something "nasty". Any ideas what I can do?
And what if it's not only my ISP, but more and more ISPs start manipulating the packets? It should be possible for anth to adjust ACE to be compatible with these new methods, shouldn't it?


5) Run UT on a Linux based environment.
Because they are not checked by ACE? :wink:
Website, Forum & UTStats

Image
******************************************************************************
Nexgen Server Controller || My plugins & mods on GitHub
******************************************************************************
Higor
Godlike
Posts: 1866
Joined: Sun Mar 04, 2012 6:47 pm

Re: ACE Timeouts caused by my ISP?

Post by Higor »

We'd need a necromancer to raise anth, or have someone capable of taking over the project.
ShaiHulud
Adept
Posts: 459
Joined: Sat Dec 22, 2012 6:37 am

Re: ACE Timeouts caused by my ISP?

Post by ShaiHulud »

Anthrax did chime in on the ACE thread over at unrealadmin a few months back (or perhaps a bit more than a few months ago, but it was earlier this year to be sure) to say that work on the next version was on-going. He hasn't indicated that this has changed. I'm curious about whether EPIC plans to bake some kind of cheat protection into the Unreal Tournament 4 product given the history of cheating in earlier versions. If not, then something similar to ACE would obviously be required, and I would hope that much of Anthax's work might be transferable to that version - which might given him added motivation to continue with it.

It seems such a curious idea that the ISP would tamper with packet content. I don't want to sound like I'm dismissing the notion, just thinking out loud. Wouldn't it manifest itself in other ways too? Would TCP checksums continuously fail resulting in massive numbers of resent packets? Would garbled email arrive in your inbox? Would web pages render in strange and amusing ways?

I don't know, maybe it's some kind of traffic shaping scheme? But then why would it only affect ACE? Surely you'd notice added latency on neighbouring ports too, so a higher ping when playing UT or something.
RocketJedi
Inhuman
Posts: 850
Joined: Wed Mar 12, 2008 7:14 pm
Personal rank: I.T Master
Location: New York
Contact:

Re: ACE Timeouts caused by my ISP?

Post by RocketJedi »

you have packet loss that is most likely the issue. the second one is even worse.
https://www.vulpinemission.com
Image ROCKET-X8 Server
Image MONSTERHUNT w/ NALI WEAPONS 3 + RX8
Image BUNNYTRACK NY
Image SNIPER DEATHMATCH
Image InstaGib + ComboGib + Jailbreak
Image ROSEBUM ROCKET-X RB
User avatar
Sp0ngeb0b
Adept
Posts: 376
Joined: Wed Feb 13, 2008 9:16 pm
Location: Cologne
Contact:

Re: ACE Timeouts caused by my ISP?

Post by Sp0ngeb0b »

That's not the reason Qwerty. As I said, I'm having no timeouts at all with that shitty VPN connection (with constant packetloss).
Website, Forum & UTStats

Image
******************************************************************************
Nexgen Server Controller || My plugins & mods on GitHub
******************************************************************************
User avatar
Chamberly
Godlike
Posts: 1963
Joined: Sat Sep 17, 2011 4:32 pm
Personal rank: Dame. Vandora
Location: TN, USA
Contact:

Re: ACE Timeouts caused by my ISP?

Post by Chamberly »

I somehow can't remember that Higor, lol but that does remind me of Wildcard's recent problem about redirect (er months ago?)

Sponge,
Was there any change with the VPN since your new connection? Do you think using another VPN/route would do any different?
How about increasing the ACE check to 10 seconds if it make any difference? I don't remember how long does it take to check on default but if increasing ACE check timeout don't make any difference, it's more likely the ISP... new IP or such? If so, maybe change to another IP and see if it helps?

Some different ideas here. Hope it all get sorted out.
Image
Image
Image Edit: Why does my sig not work anymore?
RocketJedi
Inhuman
Posts: 850
Joined: Wed Mar 12, 2008 7:14 pm
Personal rank: I.T Master
Location: New York
Contact:

Re: ACE Timeouts caused by my ISP?

Post by RocketJedi »

why are you using a VPN??? and imo any packet loss is bad period
https://www.vulpinemission.com
Image ROCKET-X8 Server
Image MONSTERHUNT w/ NALI WEAPONS 3 + RX8
Image BUNNYTRACK NY
Image SNIPER DEATHMATCH
Image InstaGib + ComboGib + Jailbreak
Image ROSEBUM ROCKET-X RB
User avatar
Chamberly
Godlike
Posts: 1963
Joined: Sat Sep 17, 2011 4:32 pm
Personal rank: Dame. Vandora
Location: TN, USA
Contact:

Re: ACE Timeouts caused by my ISP?

Post by Chamberly »

Found this but not sure if it will be any use because of 1 feature I hate that it let you have: different IP.

https://www.kickstarter.com/projects/ro ... escription

It seem to help a few peoples? Dunno.
Image
Image
Image Edit: Why does my sig not work anymore?
User avatar
Sp0ngeb0b
Adept
Posts: 376
Joined: Wed Feb 13, 2008 9:16 pm
Location: Cologne
Contact:

Re: ACE Timeouts caused by my ISP?

Post by Sp0ngeb0b »

Chamberly wrote: Was there any change with the VPN since your new connection? Do you think using another VPN/route would do any different?
How about increasing the ACE check to 10 seconds if it make any difference? I don't remember how long does it take to check on default but if increasing ACE check timeout don't make any difference, it's more likely the ISP... new IP or such? If so, maybe change to another IP and see if it helps?
The situation is like this: I moved into my new home 2 months ago; I never had any troubles playing via my own connection from then on. Until 2 weeks ago. I'm not using the VPN at all for gaming; it's provided by my university for science purpose. It should be noted though that I'm living close to my university, so my packets probably won't go into the central networking knot of my ISP (which I believe is causing the troubles) but as straight as possible into the VPN. I haven't tried touching any ACE settings yet; but I believe there's some general problem here which needs to be solved in a different way.

Qwerty wrote:why are you using a VPN??? and imo any packet loss is bad period
If u read my opening post again you will see why; it's just for showing purpose to clearify that it's my ISP causing the troubles, cause i get timeouts without using a VPN and no timeouts while using one.


You see, unfortunately, nothing changed so far. Still getting timeouts on all ACE servers after 7 minutes. My money is still on some dubious stuff happening at my ISP. Just for your information, I'm with unitymedia, a german cable TV provider. They are offering coax internet solutions. I also noticed some odd stuff regarding UT with this connection; it looks like the ping requests to all UT Servers provided by the masterservers get blocked sometimes. For example, if I reload the masterserverlist multiple times (smashing F5), it will simply stop pinging at all. All favorite servers report as offline from then on, and the UT Server list remains empty (or only 1% filled). This aswell indicates some kind of ISP intern blocking on some conenctions (DOS/flooding protection?), as the UDP ping requests to the servers simply wont be transfered anymore. I'm not really a network expert, so I really hope there are some of you guys knowing more about it.
In the meamtime, I've saved one complete timeout process (7mins) via WireShark, here are the corresponding TCP packets to ACE's DataLink:
ACETimeoutsWireShark.jpg
I've attached a .zip file to this post containing all important data for this try (including the whole WhireShark file).

The connection to the server was made about 10 secs after the Test start, so all the communication from 16 - 21 secs is the initial ACE initialization. Then there's silence until second 72, which makes absolutely sense, because that's the first periodic ACE Check Intervall (every 45 seconds). And now comes the odd thing (in my eyes): There is not a little bit of communication between my client and ACE's DataLink for over 3 minutes. And then it's only one packet that's being sent and received and that's it (at second 277). There is no more at all. I'm still not kicked at this point though, as my client hangs in there for about 135 seconds until it's getting kicked for timeouts. The last UDP packet from the server reaches my client at second 413.
I have not a real explanation for this behaviour. ACE is configured with it's default variables,

Code: Select all

InitialCheckLinger=3.000000
InitialCheckTimeout=70.000000
PeriodicCheckInterval=45.000000
PeriodicCheckTimeout=60.000000
SShotTimeOut=40.000000
I just don't understand why my client is connected for over 7 minutes while the communication (and the checks) kinda died in between.

Unfortunately, I can't provide a WireShark log right now which shows the communication if everything is fine. Cause using my VPN is messing around with the packet routes to much. I would appreciate it a lot if one of you gentlemen (and women) out there would produce a WireShark log file while playing on an ACE server (~10 minutes should be more than enough). This would help a lot to understand what's actually happening. Next step would be to create such a log on the actual gameserver, to see what packets reach the server and then see how they differ from the client side.

Anyway, I still really hope that anth might throw an eye on this thread. I'm really dedicated to fix this (as you can see), but hearing from the author himself on this matter would be very interesting.
Attachments
ACETimeoutsWireShark.zip
(3.95 MiB) Downloaded 56 times
Website, Forum & UTStats

Image
******************************************************************************
Nexgen Server Controller || My plugins & mods on GitHub
******************************************************************************
User avatar
Sp0ngeb0b
Adept
Posts: 376
Joined: Wed Feb 13, 2008 9:16 pm
Location: Cologne
Contact:

Re: ACE Timeouts caused by my ISP?

Post by Sp0ngeb0b »

I got pointed to try out CyberGhost VPN services. It's the same as using my university's VPN, there are not timeouts at all. This is the communication between my client and ACE's DataLink:
ACETimeoutsWireSharkVPN.jpg
You see, the checking process is pretty much the same; except for the fact that the 2nd periodic checks starting at second 270 obviously doesn't fail. Question still remains why there are 200 seconds between the first periodic check and the 2nd, and about the same between the 2nd and the 3rd (although the Periodic Checks Interval is set to 45). But this probably isn't to important anyway. I've again attached the specific packets as a .txt file to this post.

I'm pretty much at the point right now to just buy a CyberGhost Premium VPN account, as this is really the only option for me at the moment to enjoy gaming on my own servers.
Attachments
ACETimeoutsWireSharkVPN.zip
(57.87 KiB) Downloaded 48 times
Website, Forum & UTStats

Image
******************************************************************************
Nexgen Server Controller || My plugins & mods on GitHub
******************************************************************************
User avatar
Sp0ngeb0b
Adept
Posts: 376
Joined: Wed Feb 13, 2008 9:16 pm
Location: Cologne
Contact:

Re: ACE Timeouts caused by my ISP?

Post by Sp0ngeb0b »

A friend of mine who lives in a different city is with Unitymedia aswell. He's having the exact same troubles: ACE is simply not compatible with Unitymedia's coax connection. If it wasn't only a hand full of people, I would remove ACE from my servers until there is an update fixing this. Unfortunately, I haven't heard anything about it all. Have some of you news from Anth or heard of people getting ACE timeouts due to their ISP aswell?
Website, Forum & UTStats

Image
******************************************************************************
Nexgen Server Controller || My plugins & mods on GitHub
******************************************************************************
User avatar
Chamberly
Godlike
Posts: 1963
Joined: Sat Sep 17, 2011 4:32 pm
Personal rank: Dame. Vandora
Location: TN, USA
Contact:

Re: ACE Timeouts caused by my ISP?

Post by Chamberly »

I don't think I've heard of anything but Anth being busy in RL. He did come on sometime last year and made a few post but not much about.
Image
Image
Image Edit: Why does my sig not work anymore?
Post Reply