Replication bandwidth limits

Discussions about Coding and Scripting
Post Reply
ShaiHulud
Adept
Posts: 353
Joined: Sat Dec 22, 2012 6:37 am

Replication bandwidth limits

Post by ShaiHulud » Wed Aug 16, 2017 3:27 am

Hey guys,

I've spent a couple of months (on and off) working on some server updates but found when I uploaded the files today that players aren't initializing correctly.

Sp0ngeB0b creating a handy on-screen loading-bar (which he graciously allowed us to use) that indicates the replication state internal to Nexgen (using checksums to determine whether server configuration values have been received on the client side) and it never reaches the "replication complete" stage.

Things work OK when I start a server on my local machine and connect to this from UT - although I had noticed a perceptible delay in the animation state change when a bot grabbed a flag (it would take a couple of seconds before it was visibly apparent that the bot was carrying the flag).

I'm not sure that there's any way to determine how "over the limit" I might be with replicated data (assuming that this is the issue - and since all files and configuration settings are identical between my local PC server configuration, and my net based server, it looks likely). So currently I'm faced with the incredibly tedious and potentially painful process of looking through every class where items are replicated, and trying to choose some for elimination. After the effort involved in getting things to work in the first place, it's almost unbearable to contemplate going through dozens of scripts to choose what I might try and excise to work around the problem.

I'm wondering whether there might be a simpler shortcut first. Some kind of convenient magic bullet. For the record I've tried adjust these settings in the net server UnrealTournament.ini file:

Code: Select all

[Engine.Player]
ConfiguredInternetSpeed=2600 -> to 20000!
ConfiguredLanSpeed=20000
Though I'd supposed this to be a client-side setting, but I was feeling desperate and caught a mention in this thread, even if it does relate to UE4. I'm running Higor's XC_IpDrv->XC_TcpNetDriver with:

Code: Select all

MaxClientRate=20000
Any ideas? Thanks

User avatar
sektor2111
Godlike
Posts: 4135
Joined: Sun May 09, 2010 6:15 pm
Location: On the roof.

Re: Replication bandwidth limits

Post by sektor2111 » Wed Aug 16, 2017 6:07 am

All I have to say is we have some limitations for sure. However, if these limitations are... very low in some ugly form, a MapVote system would never work properly - there replication is probably more aggressive than everywhere. It will be better to check what sort of replication bunches do you want to send in player because if you are going over bandwidth and channels boundaries you can forget some data - some actors will not be updated. Have you seen F6 ? How many channels are used ? Something is wrong coded there for sure if replication is incomplete...
If you are tracking something and you do updates to a value very often, you will cause a sort of "flood" which can be useless. One large issue which initial MH had, was updating monsters for each killed creature, imagine a redeemer fired killing 100 pupae type, and how was used MonstersTotal aggressively computed/updated in ScoreKill and iterating - already killed Pawn does iterations... :barf: . Not only new gibing data has been send to player but number of monsters was pretty much "bunched-updated" for no reason in a sort of mentally retarded formula. In next versions when I was figuring this useless load (heavy to figure everything), I went to a constant and SLOW update rather than not updating as needed and then doing later 20-30 updates per second. This is just a sample... heck knows how much bandwidth I can still save while others are happily involved to send new things from server which are entirely player-side...
Else by bringing a new value in replication to player for MH2 type, I went to a slow update without testing and sending data each tick. Keep replication at minimum.

Related to above link, UT99 is probably more limited because it was not supposed to use a lot of bandwidth - it works on a Modem connection so what are your expectations then ? This is not a P2P application for sending files in the highest speed as possible.

nogardilaref
Masterful
Posts: 579
Joined: Tue Jun 20, 2017 1:00 pm
Personal rank: ⚋⚊⚌☰⚞⌖⚟☰⚌⚊⚋

Re: Replication bandwidth limits

Post by nogardilaref » Wed Aug 16, 2017 11:16 am

There are limits: total number of open channels and maximum amount of replicated data (in bytes).
I think the former is hard-coded in the engine (I have already hit it myself on accident, which made me correct how I was spawning and replicating stuff to be far more efficient), while the latter can be configured but I think it also has an upper bound, but I am not 100% sure of this.

However, it's easy to test this: in each server tick, replicate some data to the client, and increase the amount of data in each tick (you can use a simple string var for this).
Log the "event" in both the server and the client, with an iteration number, and check at which point the client stops receiving data.

It would be best to try first with a default configuration, which is used by most players, and then do the same test with another configuration to check if it changes anything.


Adding to this, if the whole thing is hitting a limit:

1 - If it only hits the limit on the first replication: the player enters the server and there's a bootstrap which runs and replicates a ton of data only at the beginning, this is highly likely to be data which can get replicated over time rather than all at once.
In other words, if you have 30kB worth of data to replicate, and there's a limit of 5kB for example, then you could replicate these 30kB over time in such a way that you would limit yourself to replicate 5kB per second (the client could even have a progress bar of some sort in this case while they wait).

2 - If it hits the limit on every tick or during some other events during the game, then there's probably something seriously wrong with the code itself, it shouldn't ever need that much bandwidth in the first place during the game itself, as there shouldn't be any moment at all during the course of the game which needs such a high amount of data to be replicated.
There are probably exceptions to this, but generally these exceptions would be better served in other kinds of games in other engines (recent UEngine versions or Unity or other).
Here's there's nothing which can be truly done other than optimizing or even refactoring the code so it makes proper use of the resources.

User avatar
Barbie
Godlike
Posts: 1722
Joined: Fri Sep 25, 2015 9:01 pm
Location: moved without proper hashing

Re: Replication bandwidth limits

Post by Barbie » Wed Aug 16, 2017 12:09 pm

ShaiHulud wrote:a handy on-screen loading-bar
Are you sure this tool works correct? Do your doubts base on this tool only or on other observations, too?
"Multiple exclamation marks," he went on, shaking his head, "are a sure sign of a diseased mind." --Terry Pratchett

Hank
Novice
Posts: 21
Joined: Wed Oct 21, 2015 4:17 pm

Re: Replication bandwidth limits

Post by Hank » Wed Aug 16, 2017 2:27 pm

I see your server is down for a day or so, but Maybe the fake client mod ( bots with pings ) you run on your server :thudown: has something to do with it? ( just say'n)

http://www.gametracker.com/server_info/ ... .150:7777/
Image

Please remove this FAKE CLIENT mod, it hurts the community rather then help.

User avatar
sektor2111
Godlike
Posts: 4135
Joined: Sun May 09, 2010 6:15 pm
Location: On the roof.

Re: Replication bandwidth limits

Post by sektor2111 » Wed Aug 16, 2017 5:13 pm

That's why in some topic I could see a question how to do Bots like busy to do something out of their normal reactions, probably for looking more like humans when visitors will join to the "populated" server.
I suggest to make them playing backgammon or chess :ironic: :sleep: or making love with some Nali having a NaliRabbit dropped deeply in ass ? At least real player would be amused and it will not complain... Where is TammT right now... ?

Aside I think you should follow Hank's advice else player might be confusing Bot with Other Player and... someday Alys might go pregnant... :lol2:

User avatar
rjmno1
Masterful
Posts: 599
Joined: Fri Aug 12, 2011 9:38 pm
Personal rank: masterfull

Re: Replication bandwidth limits

Post by rjmno1 » Wed Aug 16, 2017 6:15 pm

as in the past the netspeed was set to 20000, but as in now days the internet speed is much higher lets say configure the netspeed to 30 maby 35 and 40000 netspeed.
Depending on wich contract you have with your isp.And how fast your internet is set to.
Also depending when running a server how much clients, in other words how much people you set the server to play.
So to say if you select lets say 40000 netspeed the max client rate is the total netspeed devided by 8 if you wanna have that 8 players can play on your server.
So the cientrate can be 5000 Theoretical.practical you must play with the settings of the clientrate.
unreal tournament 99
®
Image
Image
ImageImage

User avatar
sektor2111
Godlike
Posts: 4135
Joined: Sun May 09, 2010 6:15 pm
Location: On the roof.

Re: Replication bandwidth limits

Post by sektor2111 » Wed Aug 16, 2017 7:30 pm

Or we do need an ISP able to get over hard-coded limits from UnrealEngine :loool: . I though that you read some post from a few weeks ago toward trick-rate. It's only loading network with no purpose and this is exactly the problem here. If I'm dictating a DM using 7500 for sure player will not get over that rate, server clamps it - you probably did not do any server, and if you did, you have never read console when player wants high speed but server says "AND STAY DOWN". Replication issues might have as cause a bad coded mod and even if God of UT is doing settings to your server, if a mod sucks server does the same.

ShaiHulud
Adept
Posts: 353
Joined: Sat Dec 22, 2012 6:37 am

Re: Replication bandwidth limits

Post by ShaiHulud » Thu Aug 17, 2017 3:25 am

Thank you all very much for your helpful responses, it's enormously reassuring that there are so many helpful hands on deck in a situation like this. Before returning to the forum I went on a debugging spree, and it turns out that the cause of the problem was... entirely unrelated to replication failure. That's the short version. But I'll describe what happened next in more detail in case it aids someone else in future.

Besides anything else, the number of additional variables that my recent activities have added to the replication overhead was comparatively minuscule - probably something like 30 booleans, maybe 10 or 15 strings, and half a dozen integers. When stacked against the hundreds of variables (including multiple 512 entry string arrays and the like) that Nexgen replicates from server to client, it's a bit of a drop in the ocean. Trying to prune off a few of these extra replicated variables looked like an exercise in futility.

I inserted an enormous number of Log entries into the guts of the parts of Nexgen concerned (the calcDynamicChecksum function of NexgenConfig.uc specifically), because it became obvious that the problem was in a mismatch in one of the checksums here. The function is simulated, so I was able to collect a nice print-out of what was inside of the arrays referenced in that function on both client and server. I could see clearly from this that the client contained an extra entry that did not exist on the server within two of these arrays.

This was extremely puzzling. The server replicates the content of the arrays to the client from a single file. The array content should be identical on each machine, once time enough has passed for the data to be replicated. It made no sense that there was extra data - not empty elements, but actual additional values -inside of the same arrays on the client side. The checksum routine loops indefinitely until both client and server agree on what is in the arrays. The assumption here, of course, is that the only disagreement that might occur would happen before values have been replicated, and the arrays are empty on the client side.

So I pondered this for a while, and started haphazardly inserting more Log statements into other places where those same arrays are referenced elsewhere, thinking that somehow they were being manipulated on the client side between checksum loops. I found nothing, but I did stumble on the answer during that exercise.

The class which stores all of those arrays is called NexgenConfigExt.uc. When I opened this file, I immediately spotted that the defaultproperties list contained exactly the two extra values that I was seeing in the replicated arrays on the client side. The client was filling in those array elements with data that it was lifting from defaultproperties. I deleted these values, recompiled, re-uploaded, tested, and now everything works.

At this point I'm too elated to explore the unanswered questions that this leaves, e.g.,:
- why this has never caused a problem before
- how the defaultproperty list came to be populated when that class has never 'decompiled'

But just at the moment I don't care. Now client and server agree on the content of all variables, and Nexgen initialises correctly. At least in limited testing so far (*crosses fingers*).

Thank you again for your kind input - I'll be putting nogardilaref progressive replication test in my back-pocket for future use.

Hank, I realise that this has become a bit of a crusade for you, and frankly I sympathise with your perspective. One of my intentions with this update was to make it easier to switch things over and remove that particular "feature". When the server was first set up, it was (in part) a reaction to another server that has a similar feature set. But I was bothered by it because it advertises extremely inflated player numbers in the server browser, and is almost invariably empty (it's always at the top of the server list for "populated servers"). Not only this, but to maintain the charade, several modern server features have been excluded which would make it obvious that some of the players are bots. I wanted to have a server to play on with all of the commonly available server tools, and not have to compromise on features to accommodate bots. I wanted to offer players an alternative playing experience that wasn't cut down or contrived. And if I was going to go down the "include bots in the game" path, then I wanted to do something above and beyond what was already available (Ferbotz and other projects notwithstanding).
So I invested a great deal of time in modifying the bots so that they can do things like: use the grappling hook to get around maps; vote for maps; talk to players in the server (using Say and Nexgen PM) - and you can actually have conversations with them; dodge as they move (this is a bit primitive, but it makes the experience somewhat more immersive); taunt creatively in response to events (kills, being killed, scoring, game end); double jump in combat; kneel, wave and taunt in combat; and various other bits and pieces. I'm not boasting in any of this, I'm an absolutely elementary-level UnrealScripter. But my point is, these aren't stock bots, they're supposed to be entertaining.
But in a phased transition over the next couple of weeks (assuming no more hiccups), I will make it obvious that they aren't players.

nogardilaref
Masterful
Posts: 579
Joined: Tue Jun 20, 2017 1:00 pm
Personal rank: ⚋⚊⚌☰⚞⌖⚟☰⚌⚊⚋

Re: Replication bandwidth limits

Post by nogardilaref » Thu Aug 17, 2017 12:43 pm

Quick question: do you happen to be the owner of a certain instagib server with such bots? :o

User avatar
Sp0ngeb0b
Adept
Posts: 289
Joined: Wed Feb 13, 2008 9:16 pm
Location: Cologne

Re: Replication bandwidth limits

Post by Sp0ngeb0b » Thu Aug 17, 2017 4:19 pm

That loading bar you mentioned belongs to the earlier versions of my NexgenPlayerLookup Plugin, which required the complete config file to be replicated to the clients (which is basically a complete database) - the larger that database grew, the heavier the impacts of this sudden mass replication became for the client. Ending in the online game stopping for 30 seconds and longer (respectively actors not beeing updated) while the data never arrived completely. That's why I dropped that whole concept and came up with the reliable method in version 2 of the plugin, which performs decent communication with the clients (only sending what is needed).
However, the same procedure of the mass replication is present Nexgen's core config. I wrote some sentences about it here; just as a reminder as you have seen it before. There are two major flaws:
1) Data not neccessary for a specific client being replicated to him nonetheless
2) The variables are send by the default replication scheme "in one piece"
I must admit that point 2) is probably an imprecise statement, since I've never looked into the UnrealEngine netcode and that in general any data being sent is always split up into packages. However, the improved replication system introduced by Defrost in Nexgen 1.12 concentrates on one string variable which is replicated via the default way, but with an additional queue layer on top of it splitting up each command and variable which should be replicated. Please note that I haven't looked into this for years; it's what I remember and might not be 100% correct. But since you appear to be programming explicitly for Nexgen, I highly recommend to use that 'new' way to replicate your data - just have a look at the existing plugins using it. If you have a question or need a hand somewhere don't hesitate to PM/Mail me ;)
******************************************************************************
NexgenAdvancedBanManager || NexgenPlayerLookupSystem || NexgenTeamspeak3Connector
******************************************************************************

ShaiHulud
Adept
Posts: 353
Joined: Sat Dec 22, 2012 6:37 am

Re: Replication bandwidth limits

Post by ShaiHulud » Thu Aug 17, 2017 11:01 pm

Sp0nge I always appreciate your input, you've had so much experience with these things, and I value your insight. I Nexgen-ified a few mutators this time around, and exploited that capability in a couple of them (like the Quake 3 sound effects mutator, which now transmits the playerID of the 'victim' to the 'killer' when the mutatorTakeDamage event fires on the server - this mostly so that players can disable the sound effects on the client if they choose to), so I'm already benefiting from it, albeit in a fairly small-scale way. The way PlayerLookups handles these things is very cool to see in action. I'd love to go in and retro-fit that to some of the Nexgen core at some stage. I've got other projects in mind, but time constraints and UnrealScript burn-out will postpone these modifications for the time being. In salute to the UrS server I included a small tribute in the "intro" screen on player join. Actually I'd intended it to be larger, but I was doing all of my testing on a separate UT install running in a window at relatively low resolution, and (stupidly) forgot to factor-in relative sizes at higher resolution. But it's something.

nogardilaref, hmm. Perhaps this is actually the server you had in mind, though it's not instagib but an instagib/normal-weapons hybrid.

User avatar
Barbie
Godlike
Posts: 1722
Joined: Fri Sep 25, 2015 9:01 pm
Location: moved without proper hashing

Re: Replication bandwidth limits

Post by Barbie » Sat Aug 19, 2017 9:23 pm

ShaiHulud wrote:The server replicates the content of the arrays to the client
And does NOT transmit the number of valid elements? I'd call that a bug then.
"Multiple exclamation marks," he went on, shaking his head, "are a sure sign of a diseased mind." --Terry Pratchett

Post Reply