Ensure if your server not affected by Lag spikes issue

Discussions about Servers
Post Reply
Buggie
Godlike
Posts: 2749
Joined: Sat Mar 21, 2020 5:32 am

Ensure if your server not affected by Lag spikes issue

Post by Buggie »

Recently discover very weird UT bug, which able produced unplayable lags on server if there too many .int files.

https://github.com/OldUnreal/UnrealTour ... ssues/1218

What happens:
If your server not able build list of mutators, then server try do it again and again on each request. Which waste CPU and make ping jumps.

How determine if is your server affected:
Check rules grid for your server in Server Browser.
You can use in-game version, or web version: servers
Or any other tool which query server and show rules.
If there no lines with mutators list - server affected. For web version there be "None" in "Mutators" line.

OFC if not use custom GameType, where this fixed, or some fancy QueryAnswer code. But this highly unlikely.
You can check it on "EL DORADO" servers.

Proper solution is clear not used .int files and make one were listed used mutators with name which alphabetically appear first. Like start with "AAA_" or "000_".

As solution you can use such mutator:
FixLagSpikes.zip
(817 Bytes) Downloaded 13 times
It need install it and add to ServerActors. Not need add to ServerPackages.
ServerActors=FixLagSpikes.FixLagSpikes

If this mutator fix things, you must see in "mutators" line "-".
Last edited by Buggie on Fri Mar 31, 2023 7:18 pm, edited 3 times in total.
User avatar
Berserker
Experienced
Posts: 128
Joined: Fri Sep 27, 2019 5:08 pm

Re: Ensure if your server not affected by Lag spikes issue

Post by Berserker »

Might be a dumb question, but mutators should be able to be loaded in the server without int files anyway, right? Cant I just delete every int file instead?
Visit us on Discord:
https://discord.gg/fcRakgNCjR Image
User avatar
sektor2111
Godlike
Posts: 6412
Joined: Sun May 09, 2010 6:15 pm
Location: On the roof.

Re: Ensure if your server not affected by Lag spikes issue

Post by sektor2111 »

I'm not using any INT file that server doesn't need (out of stock), neither "WEB" control of mutators...
I believe these will allocate some memory for said "objects" which won't help...
User avatar
_21
Average
Posts: 70
Joined: Mon Aug 30, 2021 10:51 am

Re: Ensure if your server not affected by Lag spikes issue

Post by _21 »

I think around December last year we had huge lagspikes on the FFN server. Every few seconds there was a 300ms lag spikes which you could notice by players jumping across the screen. @TaglesMalsto coined the term "TeleportPing" to describe this phenomenon. This basically made the game unplayable for many and left the servers. After week of poking around disabling and re-enabling any recent mutator changes finally with utrace we managed to find out that it was indeed server queries taking a lot of time for the server to respond to.

Our solution was to delete all *.int files (except for a few necessary ones without which the game does not boot) and we also uninstalled XServerQuery since it seems to do a lot more work than a basic query response. There were no more lagspikes ever since, gameplay has been buttery smooth. Except when the actual gametype or gameplay mutator is the cuprit.

So don't install int files on server, delete as many as you can. I think only Core.int Engine.int and IpDrv.int are mandatory. (good practice to back up int files before deleting them, instead deleting you can move them to a subfolder or something, you know in case you maybe need them)
User avatar
sektor2111
Godlike
Posts: 6412
Joined: Sun May 09, 2010 6:15 pm
Location: On the roof.

Re: Ensure if your server not affected by Lag spikes issue

Post by sektor2111 »

My logic goes this way:
Server is asked for mutators by some external queries. Mutators are taken because of INT files. And then if they are incomplete, bad declarations, server will have a heavy task to find them and will attempt to process things that are not loaded and neither can be loaded. Frankly these lousy INT files are useless.
Resources in a game server must be stable and valid, simple as that.
Buggie
Godlike
Posts: 2749
Joined: Sat Mar 21, 2020 5:32 am

Re: Ensure if your server not affected by Lag spikes issue

Post by Buggie »

Berserker wrote: Fri Mar 24, 2023 10:38 am Might be a dumb question, but mutators should be able to be loaded in the server without int files anyway, right? Cant I just delete every int file instead?
Mutators works. But some stuff can be broken. Like webadmin, Nexgen and other server admin tools, which read info about listed gametypes, mutators and other, from .int files.

If you delete all .int files Possible it can hurt game.

If you delete all .int files with mutators, this can lead to spikes, because variable not filled, which cause do parse .int on each request.

So you need .int file for at least one used mutator for any server run (any config).   
Auto merged new post submitted 13 minutes later
_21 wrote: Fri Mar 24, 2023 11:36 am I think around December last year we had huge lagspikes on the FFN server.
It still issue. You need use at least one .int file for any gametype.

servers#/utserv/85.215.164.168:53000
I see here None for mutators.
User avatar
Berserker
Experienced
Posts: 128
Joined: Fri Sep 27, 2019 5:08 pm

Re: Ensure if your server not affected by Lag spikes issue

Post by Berserker »

_21 wrote: Fri Mar 24, 2023 11:36 am I think around December last year we had huge lagspikes on the FFN server.
We are having the same issue where we get a lagspike which last couple of seconds, on every 30 minutes or so. I have also suspected that XServerQuery could be the culprit, because I have noticed a lot of query requests in our server console. I think the only reason we used XServerQuery, was to get country flags info, which we could display it on Discord for our query bot.


Also, does having too many master servers like this a problem for server performance?
Image
Visit us on Discord:
https://discord.gg/fcRakgNCjR Image
Buggie
Godlike
Posts: 2749
Joined: Sat Mar 21, 2020 5:32 am

Re: Ensure if your server not affected by Lag spikes issue

Post by Buggie »

Berserker wrote: Fri Mar 24, 2023 1:02 pm Also, does having too many master servers like this a problem for server performance?
No.
User avatar
_21
Average
Posts: 70
Joined: Mon Aug 30, 2021 10:51 am

Re: Ensure if your server not affected by Lag spikes issue

Post by _21 »

After a longer discussion with Buggie (and some testing too) I think I finally understand the problem, but please correct me if I got it wrong.

There is this one specific field in server query which is populated by listing all the mutators you have enabled. But to get the display names, they are taken from .int files.
At the first query the server will attempt to build this text value to populate this field and then on second query and on it will reuse the field value.
ut99-server-mutator-query.jpg
So ensure this field is populated and you're good to go! @Buggie's fix in the in the first post ensures that this happens.

No need to delete int files as long as you ensure that this text field exists. If it does the server doesn't need to search through int files on every query at all.

But that said, I, me and myself will still reduce the ini files to a minimum, but will keep some mutators listed, I think since mapvote is always on it's a good candidate.
Buggie
Godlike
Posts: 2749
Joined: Sat Mar 21, 2020 5:32 am

Re: Ensure if your server not affected by Lag spikes issue

Post by Buggie »

Yes. Exactly. If there no field then this mean server not able make such list. And server will do this again and again on every query. And this action CPU expensive.

So main goal make this field appear not empty. If you see this, then cache work properly, and this value build only once, on first request and later reused.   
Auto merged new post submitted 16 hours 52 minutes later
Update solution mutator.

Now it use next code:

Code: Select all

function PostBeginPlay() {
	Level.Game.GetRules();
	if (Level.Game.EnabledMutators == "") {
		Log("Server not able make mutators list - fixed", Class.Name);
		Level.Game.EnabledMutators = "-";
	}
	Destroy();
}
Post Reply