Thank you all very much for your helpful responses, it's enormously reassuring that there are so many helpful hands on deck in a situation like this. Before returning to the forum I went on a debugging spree, and it turns out that the cause of the problem was... entirely unrelated to replication failure. That's the short version. But I'll describe what happened next in more detail in case it aids someone else in future.
Besides anything else, the number of additional variables that my recent activities have added to the replication overhead was comparatively minuscule - probably something like 30 booleans, maybe 10 or 15 strings, and half a dozen integers. When stacked against the hundreds of variables (including multiple 512 entry string arrays and the like) that Nexgen replicates from server to client, it's a bit of a drop in the ocean. Trying to prune off a few of these extra replicated variables looked like an exercise in futility.
I inserted an enormous number of Log entries into the guts of the parts of Nexgen concerned (the calcDynamicChecksum function of NexgenConfig.uc specifically), because it became obvious that the problem was in a mismatch in one of the checksums here. The function is simulated, so I was able to collect a nice print-out of what was inside of the arrays referenced in that function on both client and server. I could see clearly from this that the client contained an extra entry that did not exist on the server within two of these arrays.
This was extremely puzzling. The server replicates the content of the arrays to the client from a single file. The array content should be identical on each machine, once time enough has passed for the data to be replicated. It made no sense that there was extra data - not empty elements, but actual additional values -inside of the same arrays on the client side. The checksum routine loops indefinitely until both client and server agree on what is in the arrays. The assumption here, of course, is that the only disagreement that might occur would happen before values have been replicated, and the arrays are empty on the client side.
So I pondered this for a while, and started haphazardly inserting more Log statements into other places where those same arrays are referenced elsewhere, thinking that somehow they were being manipulated on the client side between checksum loops. I found nothing, but I did stumble on the answer during that exercise.
The class which stores all of those arrays is called NexgenConfigExt.uc. When I opened this file, I immediately spotted that the defaultproperties list contained exactly the two extra values that I was seeing in the replicated arrays on the client side. The client was filling in those array elements with data that it was lifting from defaultproperties. I deleted these values, recompiled, re-uploaded, tested, and now everything works.
At this point I'm too elated to explore the unanswered questions that this leaves, e.g.,:
- why this has never caused a problem before
- how the defaultproperty list came to be populated when that class has never 'decompiled'
But just at the moment I don't care. Now client and server agree on the content of all variables, and Nexgen initialises correctly. At least in limited testing so far (*crosses fingers*).
Thank you again for your kind input - I'll be putting nogardilaref progressive replication test in my back-pocket for future use.
Hank, I realise that this has become a bit of a crusade for you, and frankly I sympathise with your perspective. One of my intentions with this update was to make it easier to switch things over and remove that particular "feature". When the server was first set up, it was (in part) a reaction to another server that has a similar feature set. But I was bothered by it because it advertises extremely inflated player numbers in the server browser, and is almost invariably empty (it's always at the top of the server list for "populated servers"). Not only this, but to maintain the charade, several modern server features have been excluded which would make it obvious that some of the players are bots. I wanted to have a server to play on with all of the commonly available server tools, and not have to compromise on features to accommodate bots. I wanted to offer players an alternative playing experience that wasn't cut down or contrived. And if I was going to go down the "include bots in the game" path, then I wanted to do something above and beyond what was already available (Ferbotz and other projects notwithstanding).
So I invested a great deal of time in modifying the bots so that they can do things like: use the grappling hook to get around maps; vote for maps; talk to players in the server (using Say and Nexgen PM) - and you can actually have conversations with them; dodge as they move (this is a bit primitive, but it makes the experience somewhat more immersive); taunt creatively in response to events (kills, being killed, scoring, game end); double jump in combat; kneel, wave and taunt in combat; and various other bits and pieces. I'm not boasting in any of this, I'm an absolutely elementary-level UnrealScripter. But my point is, these aren't stock bots, they're supposed to be entertaining.
But in a phased transition over the next couple of weeks (assuming no more hiccups), I will make it obvious that they aren't players.