ACE Timeouts caused by my ISP?

[papercoffee]

A friend of mine who lives in a different city is with Unitymedia aswell. He's having the exact same troubles: ACE is simply not compatible with Unitymedia's coax connection. If it wasn't only a hand full of people, I would remove ACE from my servers until there is an update fixing this. Unfortunately, I haven't heard anything about it all. Have some of you news from Anth or heard of people getting ACE timeouts due to their ISP aswell?

Strange ...I'm at Netcologne and I have also timeouts and server disconnects.
The provider said, everything is fine but they register periodic ups and downs caused by my cable connection from the wall receptacle to my modem ...I'm connected over their fibreglass multi-cable network (internet+TV+phone). I checked it twice and trice and ...nothing ...I still getting timeouts.
Can you send me the IP of one of the server you try to connect? I will check if I'm also unable to play on this server.

[noccer]

Never had this problem when my ISP was unitymedia, or better said: not more or less ACE problems, it´s same for me with all providers.

Somewhere i found the tip, if you start with linux put your server IP to the startcommand, or when using ucc.init put it there under MYIP=your.server.ip.here
Maybe i am mixing it but i think it was about ACE timeout.

[Sp0ngeb0b]

Strange ...I'm at Netcologne and I have also timeouts and server disconnects.
The provider said, everything is fine but they register periodic ups and downs caused by my cable connection from the wall receptacle to my modem ...I'm connected over their fibreglass multi-cable network (internet+TV+phone). I checked it twice and trice and ...nothing ...I still getting timeouts.
Can you send me the IP of one of the server you try to connect? I will check if I'm also unable to play on this server.

Interesting to hear. It sounds more like a general connection issue though if I understood you correctly. Netcologne says that the issues are somewhere inside your house (receptable/cable/modem)? In my case, I can not really believe it's caused by that, as it worked for several months like a charm and then the timeouts suddenly started without any local changes. As well as the fact that using a VPN is solving it makes me believe that it's something more global at my ISP side. The timeouts happen at any ACE server for me, always after 7 minutes. You might try out my main server to see if you have something similiar: 185.62.204.11:7777

Never had this problem when my ISP was unitymedia, or better said: not more or less ACE problems, it´s same for me with all providers.

Somewhere i found the tip, if you start with linux put your server IP to the startcommand, or when using ucc.init put it there under MYIP=your.server.ip.here
Maybe i am mixing it but i think it was about ACE timeout.

In the first months i didn't have any issues with Unitymedia either, but then the timeouts suddenly started and would not disappear at all. It's the same for all ACE servers out there, so I doubt it is a server-side setting.


I appreciate your answers guys, makes me feel not completely alone and hopefully alerts Anth sooner or later

[Higor]

Hah, your ISP's are sniffing/throttling your traffic, of course they're not going to tell you.

Regarding those practices, FCC's voting on net neutrality on February, this is a thing that affects most of US based servers on the east coast so we should pay closer attention to that as well.
I say that because NetFlix's already been forced to pay Comcast up for better traffic, and Riot is about to make a deal to improve the routes to their servers with these ISP's (who are blatantly throttling).
A big issue in the east coast lines where most of UT servers are.

[UT99.org]

could be. re: contacting Anth

Maybe he has a BNC set up

irc.quakenet.org
#unrealadmin
nickname [anth]

[Sp0ngeb0b]

I can't believe how blind I've been all the time. I just realized I can easily modify ACE's EventHandler to whitelist me for timeouts. Damn. I've been struggling around with VPNs and stuff for months while I completely forgot that ACE offers access to its behaviour in it's public Interface.

Sure, this is not a global solution. But if there's a ServerAdmin out there who is blocked from his own server aswell, it can help him get back into action right away haha...

[UT99.org]

The NAT version of ACE? I thought allowing them bypass being timed out was akin to allowing them to cheat? Just sayin'

[Sp0ngeb0b]

No, I've modified the ACEEventHandler to compare the Nexgen ID of players who timed out with a configurable whitelist. If they match, the timeout is simply ignored and the player (in this case me) can continue playing on the server. Fortunately, ACE offers such freedom to programmers/serveradmins.

[anth]

I've never been able to fully figure this one out, despite months and months of debugging this issue. Regardless of whatever issues your ISP might have, there are certainly problems with ACE too. I can't pin them down to an exact cause though. The situation is like this.

A UT client and server "talk" to eachother using a UDP-based protocol. Let's call that connection the game connection. Regular mods can send data back and forth between the client and server using that same game connection. The problem though is that this game connection is subject to bandwidth caps (configurable using netspeed) and priority rules (mod authors that have ever done replication are familiar with these). ACE periodically sends big chunks of data from the client to the server and back. If it were to simply use the game connection, then I would have to deal with those same issues. That means that while ACE is sending data, it could hog up the entire connection with no guarantee that the data would be delivered in time at all (even though the player's physical connection could definitely handle the amount of data being sent).

The obvious choice here was to work around the caps/rules by establishing a second link between the player and the server. I believe I called this the data link. In my infinite wisdom, I implemented this data link using a (mostly) native class that uses a TCP socket. I could've just used UT's built-in TcpLink class but that one is broken beyond repair. ACE uses this link to send _some_ (but not all) of its data. Either way, this system has always worked great for all of my testing machines. One of those testing machines is a horrible pentium III laptop with 128Mb of RAM and a first generation Wi-Fi chipset. That is one of the main problems: I have never ever found a way to reproduce the problem myself.

That being said, there are some things that I have been able to confirm:
* In some of my tests I have discovered a really nasty bug in the Linux kernel's TCP stack. This has been fixed since and I did also work around it at the time.
* There were data races and deadlocks in earlier versions of ACE that caused the checks to simply hang. Those have also been fixed.
* Using TCP was an all-round stupid idea. I still have no idea why I ever went for TCP. I guess I liked the acronym better than UDP. ACE v0.9's networking layer is based on a reliable UDP protocol instead. Unfortunately, that didn't fix the problem for everyone.
* There's a series of AMD CPUs that have a confirmed hardware bug. It doesn't affect normal people but again, in my infinite wisdom I decided to do something fancy that triggers the bug. The result is that some checks would randomly not start. I only recently discovered this one so it hasn't been fixed. This particular bug only affects a very small fraction of the players that see these timeouts on a regular basis though.

One last issue that I have not confirmed is that NAT servers might simply be deleting the DataLink connection due to inactivity. I could've solved this one pretty easily but I don't think I ever did. I believe this is the bug you're seeing in the WireShark log. The packet in red is a RST packet. It indicates that the packet that preceded it (3 seconds earlier) could not be associated with an active connection. At this point, the TCP link is effectively dead but unfortunately, this death isn't reported to ACE's data link. I believe that it might be possible to solve this particular bug without actually updating ACE. It should be sufficient to just send regular keepalive packets over every ACEDataLink connection. That might in fact even be possible using plain UnrealScript.

[ShaiHulud]

That's some great info, thanks for following this up Anthrax.

[anth]

Anyone who can reproduce this problem and who has access to a server can PM me. I have a possible fix that might solve the problem.