(Linux) server hangs at unloading a map

Discussions about Servers
User avatar
Barbie
Godlike
Posts: 2770
Joined: Fri Sep 25, 2015 9:01 pm
Location: moved without proper hashing

(Linux) server hangs at unloading a map

Post by Barbie »

Sometimes I've noticed that my Linux UT99 server (v451) hangs at unloading a map, but only if this map was running for several hours (without players).

Usually the last lines at map switch in the log are
[...]
-0.0ms Unloading: Package WhatEver
-0.0ms Unloading: Package WhatEverAnother
Garbage: objects: 38711->21577; refs: 300300
But on these accidents the last log entry is about unloading a package, the "Garbage" line is missing and the server consumes about 40% CPU. I gave it 10 or 20 minutes, but nothing changed, and so I terminated and restarted the server manually.
Maybe the long lifetime of the map has lead to a huge amount of allocated memory chunks and/or the garbage collector is slow? Any other hints?
lshw
description: Computer
width: 64 bits
capabilities: vsyscall64 vsyscall32
*-core
description: Motherboard
physical id: 0
*-memory
description: System memory
physical id: 0
size: 4GiB
*-cpu
product: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
vendor: Intel Corp.
physical id: 1
bus info: cpu@0
width: 64 bits
capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf cpuid_faulting pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt
"Multiple exclamation marks," he went on, shaking his head, "are a sure sign of a diseased mind." --Terry Pratchett
Higor
Godlike
Posts: 1865
Joined: Sun Mar 04, 2012 6:47 pm

Re: (Linux) server hangs at unloading a map

Post by Higor »

Barbie wrote:Maybe the long lifetime of the map has lead to a huge amount of allocated memory chunks and/or the garbage collector is slow? Any other hints?
See why crap like PrecipitationGenerator, SmokeGenerator and other never-ending actor spawning stuff are cancerous?

Make an actor that calls the garbage collector every 30 or so minutes and spawn it via ServerActors.

Code: Select all

var int TimeLeft;
event PostBeginPlay()
{
   TimeLeft = 30 * 60;
   SetTimer( Level.TimeDilation, true);
}
event Timer()
{
    TimeLeft--;
    if ( Level.Game.NumPlayers == 0 )
        TimeLeft -= 5; //Accelerate when empty
    if ( TimeLeft <= 0 )
    {
         ConsoleCommand("OBJ GARBAGE");
         ConsoleCommand("FLUSH");
         TimeLeft = 30*60;
    }
}
Aldebaran
Masterful
Posts: 672
Joined: Thu Jan 28, 2016 7:30 pm

Re: (Linux) server hangs at unloading a map

Post by Aldebaran »

One thing I have recognized is that Mapvote LA2 (perhaps Mapvote LA13 too) in my case does not switch map automatically although a max time for empty server is entered in config.
Last edited by Aldebaran on Fri Nov 18, 2016 5:03 pm, edited 1 time in total.
User avatar
Barbie
Godlike
Posts: 2770
Joined: Fri Sep 25, 2015 9:01 pm
Location: moved without proper hashing

Re: (Linux) server hangs at unloading a map

Post by Barbie »

Aldebaran wrote:[...] Mapvote LA13 [...] so when the server is empty for some time it will change to a new map automatically
MapVoteLA.ini wrote:[MapVoteLA13.BDBMapVote]
bEnableEmptyServerLevelSwitch=False
EmptyServerLevelSwitchTimeMins=15
Oh, I have not recalled this feature, thanks.
Higor wrote:Make an actor that calls the garbage collector every 30 or so minutes
Thanks for this also. For now I've implemented this in my base mutator: 60 min after the last player left or the last garbage collection was done, the garbage collector is called.
"Multiple exclamation marks," he went on, shaking his head, "are a sure sign of a diseased mind." --Terry Pratchett
User avatar
sektor2111
Godlike
Posts: 6397
Joined: Sun May 09, 2010 6:15 pm
Location: On the roof.

Re: (Linux) server hangs at unloading a map

Post by sektor2111 »

Duh, some Map-Vote can be featured with this "collector-type". I did a separate Actor for this job but I guess I can move it.
__
I recall someone speaking about an endless factory spawning over and over craps (sbCreatureFactory like a recall through a mind-fog), boomerang returned at owner :mrgreen: .
User avatar
Barbie
Godlike
Posts: 2770
Joined: Fri Sep 25, 2015 9:01 pm
Location: moved without proper hashing

Re: (Linux) server hangs at unloading a map

Post by Barbie »

sektor2111 wrote:sbCreatureFactory
Yep, the first version didn't keep track of the total amount of living created prototypes and so it has produced more and more and more... But now it has a upper limit.
"Multiple exclamation marks," he went on, shaking his head, "are a sure sign of a diseased mind." --Terry Pratchett
User avatar
sektor2111
Godlike
Posts: 6397
Joined: Sun May 09, 2010 6:15 pm
Location: On the roof.

Re: (Linux) server hangs at unloading a map

Post by sektor2111 »

Then is good...
Other thing which I could figure. Some times an OS seems to develop a lot of fragmented data in memory. Also I'm guessing at a moment is unable to manage those blocks any more because they might be badly spread. In the past, when I was hosting servers at home I was scheduling a "power-down" at 4:48 AM and a restart VIA Bios at 5:00 AM. In this way things went a bit more stable due to such "refresh". All these are available when things are in your yard, for other rented services I cannot bet that you'll have the best ever memory management. :|
User avatar
Barbie
Godlike
Posts: 2770
Joined: Fri Sep 25, 2015 9:01 pm
Location: moved without proper hashing

Re: (Linux) server hangs at unloading a map

Post by Barbie »

Barbie wrote:didn't keep track of the total amount of living created prototypes
I just played "MH-Tarmation2+fix2" and wondered why monster count was increasing without end... Asgard2.u has the issue of creating ScriptedPawns without an upper limit:

Code: Select all

class swarm extends hatchling;

function TweenToRunning(float tweentime) {
local hatchling G;
local vector X,Y,Z, projStart;
	GetAxes(Rotation,X,Y,Z);
	projStart = Location + 0.5 * CollisionRadius * X - 0.3 * CollisionHeight * Z;
	if ( (numChildren > 1) || (FRand() > 0.5) )
	{
		G = spawn(class 'hatchling' ,,'',projStart + (4.2 * CollisionRadius + class'hatchling'.Default.CollisionRadius) * X);
		if ( G != None )
		{
			G.Parentcarrier = self;
			numChildren++;
			if (AnimSequence != 'Crawl' || !bAnimLoop)
				TweenAnim('Crawl', tweentime);
		}
	}
}
"Multiple exclamation marks," he went on, shaking his head, "are a sure sign of a diseased mind." --Terry Pratchett
MrLoathsome
Inhuman
Posts: 958
Joined: Wed Mar 31, 2010 9:02 pm
Personal rank: I am quite rank.
Location: MrLoathsome fell out of the world!

Re: (Linux) server hangs at unloading a map

Post by MrLoathsome »

This is a very interesting thread for me at this point, as I have been migrating all my servers from a Linux box
to a Windows server this month. Had some hardware issues with the antique machine that has been running
4 UT UTPG 440 servers for well over 10 years.

For some reasons I cannot recall, I was under the impression that 451 was best for windows servers, and 440 was
the best to use for linux servers. I am starting to think this is not right. I am starting to think the UTPG stuff sucks.

Even before I was forced to move the servers due to hardware issues, I had been considering downgrading
my servers from 440 to 436 ever since Leo pointed out to me that spider physics was BROKEN in 440.... :mad2:
What else is broken in 440 that I don't know about yet?

Over the years I have always had to keep an eye on the Linux box, as it would seemingly lock up on a random
basis with the UT process pinning one of the 2 cpu's at 100%.
I would just have to restart that UT server again, and it would run again.
Sometimes for weeks or months with no issue, other times it would lock up the same way the next day.
Guess I just got used to it, assumed it was because of something extra I had running on the server.
It made no difference if the linux server itself had been restarted that same day, or had uptime > 200 days.

Have 2 coop servers and a DOM server all running custom gametypes on the new windows 436 server box now.
And a test server running default UT DeathMatchPlus, with a few custom weapons and monsters on it.
Been up 2 weeks on that, and none of the dedicated servers have locked up. No issues at all.

I was considering upgrading them to 451 at some point, but I am not sure if I will bother. Why?
Do spider physics work online with 451? Did it break anything else?
:?: :what: :loool: :???:
blarg
User avatar
sektor2111
Godlike
Posts: 6397
Joined: Sun May 09, 2010 6:15 pm
Location: On the roof.

Re: (Linux) server hangs at unloading a map

Post by sektor2111 »

I believe I did not rammed anything in connection (not a single OS install with the same results), I could see a lot of PlayerCanSeeMe() crashes in Win 451 any a, b,... x version, in 436 they were a bit rare but still bullshitting. This function is not sanitized and neither works properly - see Higor's description + I logged something in a map. Is utterly trash even in that awesome 451. When player somehow develop another false connection slot, if that slot is closing by Engine and this query happens by some actor server will crash. Owner of Connection being None and/or Connection being also None later, make that C++ crap to ruin everything - it's my deduction and observations - Some day I'll look at it... for memories... Higors's one is 100 times better, even more.

The rest of fixes did not fascinated me. I'm currently happy with 436 and XC_v16.
Visible things in 451:
- 255 ban locations = WHAT FOR ? In the past I had a dynamic IP even 4 years on a DSL line;
- Screwed ShowPath on-line = Not fixed the mostly it was ruined correctly;
- Collision crash is not fixed - still to speak about it;
- String issues ? Not surprised - first lines of log shows whatever Processor (maybe an alien one) in how looks written.
I'm not interested about more issues since the first is really nasty.
What I wanna point out is that 451 supposed better is not better.

For the rest of Linux experiments I'm not curious, I could see enough crashes posted in forums and some of them are coming from Engine.u + dll/so.
User avatar
Barbie
Godlike
Posts: 2770
Joined: Fri Sep 25, 2015 9:01 pm
Location: moved without proper hashing

Re: (Linux) server hangs at unloading a map

Post by Barbie »

Barbie wrote:For now I've implemented this in my base mutator: 60 min after the last player left or the last garbage collection was done, the garbage collector is called.
Interim report: some maps produce no garbage while inactive (always same count of objects and refs), some are dynamics.
Example for Garbage collector on map MH-CrystalMine3
2016-04-01 08:36:56 Garbage: objects: 38623->38597; refs: 838441
2016-04-01 09:31:30 Garbage: objects: 38680->38680; refs: 841761
2016-04-01 10:26:04 Garbage: objects: 38571->38571; refs: 837402
2016-04-01 11:20:38 Garbage: objects: 38668->38668; refs: 841282
2016-04-01 12:15:12 Garbage: objects: 38588->38588; refs: 838082
2016-04-01 13:09:47 Garbage: objects: 38636->38636; refs: 840001
2016-04-01 14:46:05 Garbage: objects: 38766->38647; refs: 839624
2016-04-01 15:40:39 Garbage: objects: 38689->38689; refs: 841304
2016-04-01 16:35:13 Garbage: objects: 38694->38694; refs: 841504
2016-04-01 17:29:47 Garbage: objects: 38696->38696; refs: 841586
2016-04-01 18:24:21 Garbage: objects: 38719->38719; refs: 842506
2016-04-01 19:18:55 Garbage: objects: 38687->38687; refs: 841224
2016-04-01 20:13:29 Garbage: objects: 38624->38624; refs: 838704
2016-04-01 21:08:03 Garbage: objects: 38598->38598; refs: 837664
2016-04-01 22:02:37 Garbage: objects: 38627->38627; refs: 838825
2016-04-01 22:57:12 Garbage: objects: 38614->38614; refs: 838304
No hungs detected so far.
"Multiple exclamation marks," he went on, shaking his head, "are a sure sign of a diseased mind." --Terry Pratchett
User avatar
sektor2111
Godlike
Posts: 6397
Joined: Sun May 09, 2010 6:15 pm
Location: On the roof.

Re: (Linux) server hangs at unloading a map

Post by sektor2111 »

Empty server won't develop garbage for collecting it unless, like Higor said we have smokes, precipitation particles, drips generated, etc. These are stacking in a big quantity and later they become a hard management.
User avatar
Chamberly
Godlike
Posts: 1963
Joined: Sat Sep 17, 2011 4:32 pm
Personal rank: Dame. Vandora
Location: TN, USA

Re: (Linux) server hangs at unloading a map

Post by Chamberly »

Thinking about these generators... I guess I would've made an actor that automatically kill those generator just to prevent any much of an issue. (Log spam is one of them lol)
Image
Image
Image Edit: Why does my sig not work anymore?
User avatar
sektor2111
Godlike
Posts: 6397
Joined: Sun May 09, 2010 6:15 pm
Location: On the roof.

Re: (Linux) server hangs at unloading a map

Post by sektor2111 »

Chamberly wrote:(Log spam is one of them lol)
Yes, it's both fault modder creating stuff and mapper who configured that. Best example old "PrecipitationGenerator".
It spawns particles which aren't checked if they have been launched, but mapper never understood how to setup range for this lousy thing and then... generator try to spawn them outside of playable zones.

Here we speak about lacks of explanations for new stuff - the same as old messed up MonsterArena where mainly all "mappers" did not get how to deal with MonsterArenaEnd trigger, and generally actors with a custom size (NOT PAWNS).
User avatar
Barbie
Godlike
Posts: 2770
Joined: Fri Sep 25, 2015 9:01 pm
Location: moved without proper hashing

Re: (Linux) server hangs at unloading a map

Post by Barbie »

sektor2111 wrote:smokes, precipitation particles, drips generated, etc.
All these visual effects (=not interacting with anything else) should run on clients only, they even don't have to be synchronous on different clients. Isn't there a setting to achieve this?
"Multiple exclamation marks," he went on, shaking his head, "are a sure sign of a diseased mind." --Terry Pratchett