(Linux) server hangs at unloading a map

[Barbie]

Sometimes I've noticed that my Linux UT99 server (v451) hangs at unloading a map, but only if this map was running for several hours (without players).

Usually the last lines at map switch in the log are

[...]
-0.0ms Unloading: Package WhatEver
-0.0ms Unloading: Package WhatEverAnother
Garbage: objects: 38711->21577; refs: 300300

But on these accidents the last log entry is about unloading a package, the "Garbage" line is missing and the server consumes about 40% CPU. I gave it 10 or 20 minutes, but nothing changed, and so I terminated and restarted the server manually.
Maybe the long lifetime of the map has lead to a huge amount of allocated memory chunks and/or the garbage collector is slow? Any other hints?

lshw

description: Computer
width: 64 bits
capabilities: vsyscall64 vsyscall32
*-core
description: Motherboard
physical id: 0
*-memory
description: System memory
physical id: 0
size: 4GiB
*-cpu
product: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
vendor: Intel Corp.
physical id: 1
bus info: cpu@0
width: 64 bits
capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf cpuid_faulting pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt

[Higor]

Maybe the long lifetime of the map has lead to a huge amount of allocated memory chunks and/or the garbage collector is slow? Any other hints?

See why crap like PrecipitationGenerator, SmokeGenerator and other never-ending actor spawning stuff are cancerous?

Make an actor that calls the garbage collector every 30 or so minutes and spawn it via ServerActors.

Code: Select all

var int TimeLeft;
event PostBeginPlay()
{
   TimeLeft = 30 * 60;
   SetTimer( Level.TimeDilation, true);
}
event Timer()
{
    TimeLeft--;
    if ( Level.Game.NumPlayers == 0 )
        TimeLeft -= 5; //Accelerate when empty
    if ( TimeLeft <= 0 )
    {
         ConsoleCommand("OBJ GARBAGE");
         ConsoleCommand("FLUSH");
         TimeLeft = 30*60;
    }
}

[Aldebaran]

One thing I have recognized is that Mapvote LA2 (perhaps Mapvote LA13 too) in my case does not switch map automatically although a max time for empty server is entered in config.

[Barbie]

[...] Mapvote LA13 [...] so when the server is empty for some time it will change to a new map automatically

[MapVoteLA13.BDBMapVote]
bEnableEmptyServerLevelSwitch=False
EmptyServerLevelSwitchTimeMins=15

Oh, I have not recalled this feature, thanks.

Make an actor that calls the garbage collector every 30 or so minutes

Thanks for this also. For now I've implemented this in my base mutator: 60 min after the last player left or the last garbage collection was done, the garbage collector is called.

[sektor2111]

Duh, some Map-Vote can be featured with this "collector-type". I did a separate Actor for this job but I guess I can move it.
__
I recall someone speaking about an endless factory spawning over and over craps (sbCreatureFactory like a recall through a mind-fog), boomerang returned at owner .

[Barbie]

sbCreatureFactory

Yep, the first version didn't keep track of the total amount of living created prototypes and so it has produced more and more and more... But now it has a upper limit.

[sektor2111]

Then is good...
Other thing which I could figure. Some times an OS seems to develop a lot of fragmented data in memory. Also I'm guessing at a moment is unable to manage those blocks any more because they might be badly spread. In the past, when I was hosting servers at home I was scheduling a "power-down" at 4:48 AM and a restart VIA Bios at 5:00 AM. In this way things went a bit more stable due to such "refresh". All these are available when things are in your yard, for other rented services I cannot bet that you'll have the best ever memory management.

[Barbie]

didn't keep track of the total amount of living created prototypes

I just played "MH-Tarmation2+fix2" and wondered why monster count was increasing without end... Asgard2.u has the issue of creating ScriptedPawns without an upper limit:

Code: Select all

class swarm extends hatchling;

function TweenToRunning(float tweentime) {
local hatchling G;
local vector X,Y,Z, projStart;
	GetAxes(Rotation,X,Y,Z);
	projStart = Location + 0.5 * CollisionRadius * X - 0.3 * CollisionHeight * Z;
	if ( (numChildren > 1) || (FRand() > 0.5) )
	{
		G = spawn(class 'hatchling' ,,'',projStart + (4.2 * CollisionRadius + class'hatchling'.Default.CollisionRadius) * X);
		if ( G != None )
		{
			G.Parentcarrier = self;
			numChildren++;
			if (AnimSequence != 'Crawl' || !bAnimLoop)
				TweenAnim('Crawl', tweentime);
		}
	}
}

[MrLoathsome]

This is a very interesting thread for me at this point, as I have been migrating all my servers from a Linux box
to a Windows server this month. Had some hardware issues with the antique machine that has been running
4 UT UTPG 440 servers for well over 10 years.

For some reasons I cannot recall, I was under the impression that 451 was best for windows servers, and 440 was
the best to use for linux servers. I am starting to think this is not right. I am starting to think the UTPG stuff sucks.

Even before I was forced to move the servers due to hardware issues, I had been considering downgrading
my servers from 440 to 436 ever since Leo pointed out to me that spider physics was BROKEN in 440....
What else is broken in 440 that I don't know about yet?

Over the years I have always had to keep an eye on the Linux box, as it would seemingly lock up on a random
basis with the UT process pinning one of the 2 cpu's at 100%.
I would just have to restart that UT server again, and it would run again.
Sometimes for weeks or months with no issue, other times it would lock up the same way the next day.
Guess I just got used to it, assumed it was because of something extra I had running on the server.
It made no difference if the linux server itself had been restarted that same day, or had uptime > 200 days.

Have 2 coop servers and a DOM server all running custom gametypes on the new windows 436 server box now.
And a test server running default UT DeathMatchPlus, with a few custom weapons and monsters on it.
Been up 2 weeks on that, and none of the dedicated servers have locked up. No issues at all.

I was considering upgrading them to 451 at some point, but I am not sure if I will bother. Why?
Do spider physics work online with 451? Did it break anything else?

[sektor2111]

I believe I did not rammed anything in connection (not a single OS install with the same results), I could see a lot of PlayerCanSeeMe() crashes in Win 451 any a, b,... x version, in 436 they were a bit rare but still bullshitting. This function is not sanitized and neither works properly - see Higor's description + I logged something in a map. Is utterly trash even in that awesome 451. When player somehow develop another false connection slot, if that slot is closing by Engine and this query happens by some actor server will crash. Owner of Connection being None and/or Connection being also None later, make that C++ crap to ruin everything - it's my deduction and observations - Some day I'll look at it... for memories... Higors's one is 100 times better, even more.

The rest of fixes did not fascinated me. I'm currently happy with 436 and XC_v16.
Visible things in 451:
- 255 ban locations = WHAT FOR ? In the past I had a dynamic IP even 4 years on a DSL line;
- Screwed ShowPath on-line = Not fixed the mostly it was ruined correctly;
- Collision crash is not fixed - still to speak about it;
- String issues ? Not surprised - first lines of log shows whatever Processor (maybe an alien one) in how looks written.
I'm not interested about more issues since the first is really nasty.
What I wanna point out is that 451 supposed better is not better.

For the rest of Linux experiments I'm not curious, I could see enough crashes posted in forums and some of them are coming from Engine.u + dll/so.

[Barbie]

For now I've implemented this in my base mutator: 60 min after the last player left or the last garbage collection was done, the garbage collector is called.

Interim report: some maps produce no garbage while inactive (always same count of objects and refs), some are dynamics.

Example for Garbage collector on map MH-CrystalMine3

2016-04-01 08:36:56 Garbage: objects: 38623->38597; refs: 838441
2016-04-01 09:31:30 Garbage: objects: 38680->38680; refs: 841761
2016-04-01 10:26:04 Garbage: objects: 38571->38571; refs: 837402
2016-04-01 11:20:38 Garbage: objects: 38668->38668; refs: 841282
2016-04-01 12:15:12 Garbage: objects: 38588->38588; refs: 838082
2016-04-01 13:09:47 Garbage: objects: 38636->38636; refs: 840001
2016-04-01 14:46:05 Garbage: objects: 38766->38647; refs: 839624
2016-04-01 15:40:39 Garbage: objects: 38689->38689; refs: 841304
2016-04-01 16:35:13 Garbage: objects: 38694->38694; refs: 841504
2016-04-01 17:29:47 Garbage: objects: 38696->38696; refs: 841586
2016-04-01 18:24:21 Garbage: objects: 38719->38719; refs: 842506
2016-04-01 19:18:55 Garbage: objects: 38687->38687; refs: 841224
2016-04-01 20:13:29 Garbage: objects: 38624->38624; refs: 838704
2016-04-01 21:08:03 Garbage: objects: 38598->38598; refs: 837664
2016-04-01 22:02:37 Garbage: objects: 38627->38627; refs: 838825
2016-04-01 22:57:12 Garbage: objects: 38614->38614; refs: 838304

No hungs detected so far.

[sektor2111]

Empty server won't develop garbage for collecting it unless, like Higor said we have smokes, precipitation particles, drips generated, etc. These are stacking in a big quantity and later they become a hard management.

[Chamberly]

Thinking about these generators... I guess I would've made an actor that automatically kill those generator just to prevent any much of an issue. (Log spam is one of them lol)

[sektor2111]

(Log spam is one of them lol)

Yes, it's both fault modder creating stuff and mapper who configured that. Best example old "PrecipitationGenerator".
It spawns particles which aren't checked if they have been launched, but mapper never understood how to setup range for this lousy thing and then... generator try to spawn them outside of playable zones.

Here we speak about lacks of explanations for new stuff - the same as old messed up MonsterArena where mainly all "mappers" did not get how to deal with MonsterArenaEnd trigger, and generally actors with a custom size (NOT PAWNS).

[Barbie]

smokes, precipitation particles, drips generated, etc.

All these visual effects (=not interacting with anything else) should run on clients only, they even don't have to be synchronous on different clients. Isn't there a setting to achieve this?