XC_Engine [20] - XC_Core [7b] - XC_IpDrv

User avatar
sektor2111
Godlike
Posts: 5388
Joined: Sun May 09, 2010 6:15 pm
Location: On the roof.

Re: XC_GameEngine [build 12]

Post by sektor2111 » Wed May 20, 2015 3:32 pm

Did a few speed tests as long as I could made things to work with XC_Engine without to query presence just to see what I did with custom game-types and the rest of stuff. I'll copy-paste lines saved by me and completed after they have been written by XC_Engine.

Code: Select all

ScriptLog: Helper reports the Number of Actors = 2361 - Foreach AllActors (class'Actor',A)
ScriptLog: Time needed to execute: 0.000366 seconds.

PawnList 106 Monsters + 1 Player + 4 Bots
ScriptLog: Finished Pawnlist in 0.000488 seconds.
Now for me I can drew conclusions related to coding stuff for UT.
Almighty Botpack is done for let's say 4 people + 4 Bots - decent load which make PawnList to work faster enough.
BUT using PawnList in games with 700 Monsters, performance goes into trash-box as long as iterator Foreach (including pawns inside) is faster - at least I got rid of it from A.I. mission call MH related but I was using PawnList which is not fast comparing with Foreach. Gotta rewrite my Controllers as long as I don't like these results. What was written about Foreach ? Slow ? I'm not sure.
If results granted by XC_Engine are based on RealTimeClock then... I think I'll use Pawnlist depending on number of Pawns switching at Foreach if PawnList grows, LOL.

I'm going to test VisibleCollidingActors which was claimed FAST. I'll be back.

Edit Bot.ForeachVisibleCollidingActors class Actor A if A !=None Act++ log bla bla

Code: Select all

ScriptLog: MH-BR-Village.WatchIt0 spawned for Drimacus
ScriptLog: Drimacus claim to see 30 actors.
ScriptLog: Detected Actors in 0.000488 seconds.
ScriptLog: MH-BR-Village.WatchIt1 spawned for Kira
ScriptLog: Kira claim to see 31 actors.
ScriptLog: Detected Actors in 0.000000 seconds.
ScriptLog: MH-BR-Village.WatchIt2 spawned for Necroth
ScriptLog: Necroth claim to see 30 actors.
ScriptLog: Detected Actors in 0.000000 seconds.
ScriptLog: MH-BR-Village.WatchIt3 spawned for Cilia
ScriptLog: Cilia claim to see 30 actors.
ScriptLog: Detected Actors in 0.000488 seconds.
I don't understand yet why is unstable. Other checks ? Bugs ? Anyway at a moment seems to react instantly ?

Higor
Godlike
Posts: 1832
Joined: Sun Mar 04, 2012 6:47 pm

Re: XC_GameEngine [build 12]

Post by Higor » Wed May 20, 2015 4:47 pm

Octree systems are way faster than hashes for anything other than line queries.
If your watcher happens to be querying a single octree box, you may get insanely fast queries.

Things about querying actors:
- You need to query the entire actor list, use AllActors
- You need to query the entire pawn subset, use Botz.PawnActors (i should copy this code to XC_GE)
- You need to query actors in a certain radius, use RadiusActors.
- You need to query colliding actors in a certain radius, use XC_GameEngine.CollidingActors.
- You need to query visible colliding actors, use VisibleCollidingActors.
- You need to query visible colliding actors with very specific things in mind, use XC_GameEngine.CollidingActors, filter, then perform FastTrace.
- You need to find visible enemy candidates you can actually hit, use XC_GameEngine.CollidingActors, filter, then perform FastTrace.
- You need to build a matrix of pawns in the whole map or around the query actor for whatever reason, use Botz.PawnActors.

Using the raw PawnList in a map with over 200 pawns is a guaranteed slowdown.
Botz takes care of this by implementing a basic loop at c++ level, and I'm considering moving this to XC_GameEngine as well:

Code: Select all

//native final function iterator PawnActors( class<Pawn> PawnClass, out pawn P, optional float Distance, optional vector VOrigin, optional bool bHasPRI);
void ABotz::execPawnActors(FFrame &Stack, RESULT_DECL)
{
	P_GET_CLASS(BaseClass);
	P_GET_PAWN_REF(P);
	P_GET_FLOAT_OPTX(dist,0.0);
	P_GET_VECTOR_OPTX(aVec,this->Location);
	P_GET_UBOOL_OPTX(bPRI,false);
	P_FINISH;

	BaseClass = BaseClass ? BaseClass : APawn::StaticClass();
	APawn *PP = Level->PawnList;
	dist = dist * dist;

	PRE_ITERATOR;
		*P = NULL;
		while( PP && *P == NULL )
		{
			if ( PP->IsA(BaseClass) )
				if ( !bPRI || PP->PlayerReplicationInfo != NULL )
				{
					if ( dist <= 0.0 || FVector(PP->Location - aVec).SizeSquared() < dist )
						*P = (APawn*)PP;
				}
			PP = PP->nextPawn;
		}
		if ( *P == NULL )
		{
			Stack.Code = &Stack.Node->Script(wEndOffset + 1);
			break;
		}
	POST_ITERATOR;
}
=================================================================================
On a different note, I've made a breakthrough and managed to replicate actors using XC_GameEngine code.

Differences:
- Actor consider list is premade before each individual connection is polled (as seen in newer Unreal Engine, faster)
- Player, player owned, player instigated actors receive highest priority and are immediately marked as 'relevant' before the relevancy check function is marked.
-- No need to do relevancy check on these.
-- Maps with 1023 replicable elements no longer prevent player from joining the game, also Nexgen client, Mapvote stuff, etc are replicated to clients preventing kicks on badly made maps.
- Visibility checks are done to a random point on the candidate's box, this eliminates the need of Hide'n'Peek mutator (slower).
- Visibility checks now also use the NF_NotCsg flag, these can go through translucent and invisible solids.

Will probably add more stuff there later.
ImageImage
Image unreal://23.111.157.138:7777
Image unreal://46.228.199.205:7788

User avatar
sektor2111
Godlike
Posts: 5388
Joined: Sun May 09, 2010 6:15 pm
Location: On the roof.

Re: XC_GameEngine [build 12]

Post by sektor2111 » Wed May 20, 2015 5:57 pm

Higor wrote:this eliminates the need of Hide'n'Peek mutator (slower).
That one is a bad idea for maps with 1200+ monsters... Already in my MH2 I was testing a lot of dumb and smart things at once.

Crawling into MBot code I was changing tiny things. Example one: Bot Bump a pawn. Original Code scan "PawnList" (see huge MH levels) in order to gain info about a closer pawn to turn around. I'm interested why not a radius-check around Bot in 10*CollisionRadius size. Pawnlist will check even pawns far away. Need this entire cycle since it could check only certain radius ?
I have changed that and all such related checks: pawnlist + visibility + range 1600 replaced with VisibleCollidingActors in 1600 - from function "Killed". Certain things after testing speed doesn't make sense for me now, LOL.

ShaiHulud
Adept
Posts: 427
Joined: Sat Dec 22, 2012 6:37 am

Re: XC_GameEngine [build 12]

Post by ShaiHulud » Wed May 20, 2015 10:20 pm

This is a little embarrassing, but I'm not sure how to interpret the instruction "If you don't have a XC_GameEngine entry, copy it from GameEngine instead of making a new one or ugly things happen."

Does this mean that I should have two engine sections, e.g.

Code: Select all

[Engine.Engine]
GameRenderDevice=SoftDrv.SoftwareRenderDevice
AudioDevice=Galaxy.GalaxyAudioSubsystem
NetworkDevice=IpDrv.TcpNetDriver
DemoRecordingDevice=Engine.DemoRecDriver
Console=UTMenu.UTConsole
Language=int
;GameEngine=Engine.GameEngine
GameEngine=XC_Engine.XC_GameEngine
EditorEngine=Editor.EditorEngine
WindowedRenderDevice=SoftDrv.SoftwareRenderDevice
RenderDevice=GlideDrv.GlideRenderDevice
DefaultGame=Botpack.DeathMatchPlus
DefaultServerGame=Botpack.DeathMatchPlus
ViewportManager=WinDrv.WindowsClient
Render=Render.Render
Input=Engine.Input
Canvas=Engine.Canvas

[XC_Engine.XC_GameEngine]
bDisableTimingFix=True
bFasterUpload=False
GameRenderDevice=SoftDrv.SoftwareRenderDevice
AudioDevice=Galaxy.GalaxyAudioSubsystem
NetworkDevice=IpDrv.TcpNetDriver
DemoRecordingDevice=Engine.DemoRecDriver
Console=UTMenu.UTConsole
Language=int
GameEngine=XC_Engine.XC_GameEngine
EditorEngine=Editor.EditorEngine
WindowedRenderDevice=SoftDrv.SoftwareRenderDevice
RenderDevice=GlideDrv.GlideRenderDevice
DefaultGame=Botpack.DeathMatchPlus
DefaultServerGame=Botpack.DeathMatchPlus
ViewportManager=WinDrv.WindowsClient
Render=Render.Render
Input=Engine.Input
Canvas=Engine.Canvas
...or that I should remove the original [Engine.Engine] section entirely, and just have a [XC_Engine.XC_GameEngine] section containing everything that was in the [Engine.Engine] section? At the moment I have both, and my test server refuses to start (it's an i3d hosted 451 server running on Windows Server 2008 R2 Web). It doesn't seem to be possible to access the log file, it's always "0" bytes when I use their web based server stop command. Thanks for your help

SC]-[WARTZ_{HoF}
Adept
Posts: 425
Joined: Tue Feb 21, 2012 7:29 pm

Re: XC_GameEngine [build 12]

Post by SC]-[WARTZ_{HoF} » Thu May 21, 2015 12:21 am

Just one Game Engine section.

All you need to do is replace this line "[Engine.Engine]" with this one "[XC_Engine.XC_GameEngine]" and this line "GameEngine=Engine.GameEngine" with "GameEngine=XC_Engine.XC_GameEngine" when installing XCGE. Of course don't forget too add those extra setting bDisableTimingFix=True, bFasterUpload=False and ect.
Image
Image
Image

ShaiHulud
Adept
Posts: 427
Joined: Sat Dec 22, 2012 6:37 am

Re: XC_GameEngine [build 12]

Post by ShaiHulud » Thu May 21, 2015 1:54 am

Thanks for the confirmation SC]-[LONG, appreciate it.

SC]-[WARTZ_{HoF}
Adept
Posts: 425
Joined: Tue Feb 21, 2012 7:29 pm

Re: XC_GameEngine [build 12]

Post by SC]-[WARTZ_{HoF} » Thu May 21, 2015 2:02 am

Was my pleasure. Hope it works well for you as it does for {HoF}.
Image
Image
Image

ShaiHulud
Adept
Posts: 427
Joined: Sat Dec 22, 2012 6:37 am

Re: XC_GameEngine [build 12]

Post by ShaiHulud » Thu May 21, 2015 3:06 am

EDIT - removed log file entry and related question. OK, I think I've realised where I was going wrong. I'll just include it here for future reference.

In the UnrealTournament.ini file there are two sections

Code: Select all

[Engine.Engine]
GameRenderDevice=SoftDrv.SoftwareRenderDevice
...
AND

Code: Select all

[Engine.GameEngine]
CacheSizeMegs=128
...
I was renaming the wrong section - I think :) I believe that I needed to rename the second of these and not the first. And to put the "GameEngine=XC_Engine.XC_GameEngine" line under the first section, replacing the existing line there. In other words:

Code: Select all

[Engine.Engine]
GameRenderDevice=SoftDrv.SoftwareRenderDevice
...
;GameEngine=Engine.GameEngine
GameEngine=XC_Engine.XC_GameEngine

...

;[Engine.GameEngine]
[XC_Engine.XC_GameEngine]
bFasterUpload=False
bDisableTimingFix=True
CacheSizeMegs=128
Having done this, the server starts. I'm seeing this in the log file (which is thankfully allowing me to access it tonight) so seems to be working fine:

Code: Select all

XC_Engine: ========= XC_ENGINE - Test build 12 =========
XC_Engine: ================= By Higor ==================
XC_Engine: Detected version 451 GameEngine

User avatar
sektor2111
Godlike
Posts: 5388
Joined: Sun May 09, 2010 6:15 pm
Location: On the roof.

Re: XC_GameEngine [build 12]

Post by sektor2111 » Thu May 21, 2015 6:04 am

Higor wrote:Using the raw PawnList in a map with over 200 pawns is a guaranteed slowdown.
Now I'm bite the ground... Monsters are using this, too. Running smoother might come with following options:
- Getting rid of "Levels" with 1000-2000+ Creatures;
- Getting rid of "Levels" with awesome "Pawns" which uses default 0 tuned SP code with 0 fixes and new bugs successfully added;
- Coding a MonsterReplacer in order to change all stock stuff as broken weaponry - I was thinking at something posted at Ecoop due to some troubles which are about to happen at replacing monsters if replacement occurs too fast;
- RE-Coding my NsMonster since I don't want other things - NsMonster is MH ready as I could see, and it was in purpose to be Team-Ready, they don't fight each-other ever if have the same team-byte and are injected in a bTeamGame=True - more cute code to avoid a nasty load. And also I did them as replacement for default stock except Nali and Cow which for me are too stupid for breathing air on Earth.
Good... then I have to see the magic "PawnList" which gets called in "StartRoaming" "Killed" default SkaarjBerserker's "WhaToDoNext" crap, ect.
Now I have a homework to do again :roll:

Edit:
Else if you can hook PawnList problems directly with XC_Engine then I can avoid a lot of hours of working and testing.

Higor
Godlike
Posts: 1832
Joined: Sun Mar 04, 2012 6:47 pm

Re: XC_GameEngine [build 12]

Post by Higor » Thu May 21, 2015 6:53 am

Cannot really hook unrealscript pawnlist iterators.
I'd have to replace the functions instead and have them call PawnActors() lol.

I know, show me what stock code makes use of these heavy iterators, not one-shot events but instead stuff that can be triggered more than once per second.
- DistanceTrigger is one I know of, was slowing down Gopo's defense map project.

=======================
In the meantime, modified the stats display on the new replication code.
And MAYBE I'll be able to multithread the replication checks (like 10% sure it'll work lol).
XC_GE_userflag.PNG
=======================================
EDIT:

Hahahaha, suck on that UTPG.
Still messing up the structs and making binaries not compatible? I got around your shit boys.
BTW, Just tested and I can join servers where there's over 1024 bAlwaysRelevant actors with the new replication code, and all of my mapvote, nexgen, ace actors come with higher priority as well so I don't get kicked.
v451 rep only prioritizes the player, my stuff prioritizes your nexgen login actor and nploader, you no longer get your ass kicked.

Code: Select all

	static INT b451NetSetup = 0;
	static DWORD SentTemporariesOffset = 0;
	static DWORD ActorChannelsOffset = 0;
	if ( !b451NetSetup && b451Setup ) //Fix UTPG's mess
	{
		b451NetSetup = 1;
		SentTemporariesOffset = 20 * b451Setup;
		ActorChannelsOffset = 20 * b451Setup;
	}
You do not have the required permissions to view the files attached to this post.
ImageImage
Image unreal://23.111.157.138:7777
Image unreal://46.228.199.205:7788

User avatar
sektor2111
Godlike
Posts: 5388
Joined: Sun May 09, 2010 6:15 pm
Location: On the roof.

Re: XC_GameEngine [build 12]

Post by sektor2111 » Thu May 21, 2015 6:04 pm

Higor wrote:I know, show me what stock code makes use of these heavy iterators, not one-shot events but instead stuff that can be triggered more than once per second.
Normally I exported all scripts which can be exported. Using a good file search tool I was searching files UC containing word PawnList....
Now ScriptedPawn
Monster killed something (or a group of monsters) - will check in 500 radius a visible pawn doing setenemy() query in order to attack next threat, too. Let's say is not really harmful as long as people removed monster in monster damage so it happens at killing players - not a very big deal. But... if monster killed something and it's nothing else to do will attempt to roam. Will check again from pawnlist how many roamers exist in level, if are many is about to quit roaming. In "WhatToDoNext" used by SkaarjBerserker server can even crash depending on certain occurrence especially if moron has nothing else to do.
These might be my Brain-farts in how to gain speed but now I'm sure why extra-loaded levels tend to slow down frames. Default MH has a bad structure. Example:
I set a Level with triggered IceSkaarj to run into an alarm. Reached at alarm, destroy some tagged WoodenBoxes, Birds, NaliRabbits, ect. It works in more game-types but in default MH has chances to not very work - this is just a note about a heavy controller which is more stressful than efficient.
Higor wrote:DistanceTrigger is one I know of, was slowing down Gopo's defense map project.
I was working at such chapter. Is not a big deal if we have 4 monsters (see roaming chapter), else 20 monsters using the same Navigation are going to get stuck with 0 attack or 16 possible to quit roaming (in hope to not call roaming and to use "hunting" but I have doubts) - so this chapter is addressing to 4 cool guys which won't harm spirits. I gave him an example and I was using such things in other levels which were too relaxed (really retarded).

So, I guess because of those Accessed Nones involving MeleeDamageTarget() and so on, I'm gonna head to replacements. More insane, more stable, more roamers, more friends against people, don't attack doors, don't attack themselves, they are better after all. These could be thousand times easier if I would lead ScriptedPawn call to other package with a reworked one not UnrealShare.ScriptedPawn. I really don't know where and what I need to address in here to hijack this class.
I have even idea to configure class depending on server used:
- wick server = 10 - 20 roamers;
- more powerful = 100 roamers;
- NASA released a machine for an UT server ? Okay, 1000 roamers. It's enough ? We can continue to dream numbers as needed.
But at this moment is only a project because I don't have the key for hijacking ScriptedPawn.

Higor
Godlike
Posts: 1832
Joined: Sun Mar 04, 2012 6:47 pm

Re: XC_GameEngine [build 13 - Server update]

Post by Higor » Mon Jun 01, 2015 9:20 pm

Added [1719]IsInPackageMap() 227 based native, with an extra parameter for ServerPackages list only checks.
Added alternative XCGE version detector: ConsoleCommand("XC_Engine") returns the XCGE version.
Added ToggleRelevancy command that enables bUseNewRelevancy var.
Moved XC_ServerActor and XC_ConnectionHandler config to XC_Engine.ini

Created XC_ConnectionHandler
- Adds basic protection against connection attempt overflows.
- Can add extra timeouts to dataless connections.
- Can force extra TCPNetDriver polls to address a design flaw that causes lag to increase during an attack.
- Can operate on non XC_Engine servers by spawning it (ServerActors).
(The only reason I deployed a Linux build, as base XC_GE has a critical bug)

UXC_Level now runs it's own replication loop
- Actor consider list is created once per frame instead of every player check.
- Visibility checks happen less often on relevant actors.
- Visibility checks go through most transparent/invisible surfaces.
- Visibility checks are done against a random point of the actor's box.
- Player, Viewtarget and Player owned actors are given higher priority.
- Actors that pass relevancy aren't checked again for a short while.

Removed bUseReplicationHack variable.
Added bUseLevelHook variable.
- Hooks the MyLevel object during map load.
Added bUseNewRelevancy variable, requires bUseLevelHook=True.
- Causes the hooked level to run more UT3 like relevancy checks (see above).

Extra properties available from Level via GetPropertyText:
- ActorListSize
- iFirstDynamicActor
- iFirstNetRelevantActor

Updated most documentation files and added XC_ConnectionHandler.txt:
=================================
XC_ConnectionHandler actor readme
=================================

This actor is automatically spawned on net servers running XC_GameEngine.
It was made to operate independantly of XC_GameEngine so you can spawn it
on a non XC_GameEngine server by simply adding it to the ServerActors list
as follows:

[Engine.GameEngine]
...
ServerActors=XC_Engine.XC_ConnectionHandler


=== Features:
Can timeout connections without channel data at much lower values.
Timeout value can switch to a lower 'critical' timeout when connection count
exceeds a user defined value as a way to aggresively enforce less resource
usage on the server.

Can query the TCPNetDriver object an extra amount of times per frame, this
is done to mitigate a design flaw in the driver where a bad set of packets
can delay the client's incoming data by a 'n' amount of frames causing
said data to be discarded by the server due to artificially increased
delay.
By querying the TCPNetDriver an extra (user defined) times per frame, we
ensure than the malicious packets causing artificial delay are gone from
the socket at a much faster rate.


=== Settings (On XC_Engine.ini):
The default settings are as follow:

[XC_Engine.XC_ConnectionHandler]
DatalessTimeout=5.0
CriticalTimeout=2.0
CriticalConnCount=10
ExtraTCPQueries=2

DatalessTimeout: Timeout for dataless connections in normal conditions
CriticalTimeout: Timeout for dataless connections in critical conditions
CriticalConnCount: Amount of dataless connections needed to trigger critical mode
ExtraTCPQueries: Extra TCPNetDriver queries per frame

There are some restrictions:
- DatalessTimeout may not be lower than 2.0
- CriticalConnCount may not be lower than 2
Lowering said values to anything below will reset those to their defaults.


=== Notes on attacks:
This is best run in conjunction with ServerCrashFix.
As SCF handles exploits, this handles data overflow and a TCPNetDriver flaw.

The values are completely arbitrary and depend on the volume of an attack.
At extreme DDoS conditions these values had to be used:
DatalessTimeout=3.0
CriticalTimeout=0.0
CriticalConnCount=6
ExtraTCPQueries=50
With some remote ports firewalled.

Firewalling remote ports is important to prevent bandwidth amplification
attacks from reaching UT.
You can easily spot those amplification attacks as you notice thousands
of different IP addresses, but very common remote ports where said data
originates from.


Example:
SSDP reflected DoS attack.

Consists of vulnerable routers returning 30x the attacker's bandwidth to
the game server via spoofing, these connection attempts can be easily spotted:
random.ip.address:1900 -> multiple times

All you have to do is firewall all incoming connections with remote port 1900.
ImageImage
Image unreal://23.111.157.138:7777
Image unreal://46.228.199.205:7788

Palestine
Posts: 1
Joined: Mon Jun 01, 2015 11:16 pm

Re: XC_GameEngine [build 13 - Server update]

Post by Palestine » Mon Jun 01, 2015 11:25 pm

At the firewall rule, I have successfuly added a rule that drops the ssdp requests.

LOG all -- anywhere anywhere limit: avg 5/min burst 5 LOG level debug prefix `SRCPORT 1900 DROP: '
DROP tcp -- anywhere anywhere tcp spt:ssdp


This might help deflect the attack, do not reject the packets, deny them! Rejecting answers to the host which will still cause high bandwidth utilization!

ShaiHulud
Adept
Posts: 427
Joined: Sat Dec 22, 2012 6:37 am

Re: XC_GameEngine [build 13 - Server update]

Post by ShaiHulud » Tue Jun 02, 2015 7:53 am

I don't want to clog up the thread with "thanks" messages, so I'll just post this once and you'll have to assume it applies for all subsequent updates ;) But thanks Higor, much appreciated.

RocketJedi
Inhuman
Posts: 824
Joined: Wed Mar 12, 2008 7:14 pm
Personal rank: I.T Master
Location: New York

Re: XC_GameEngine [build 13 - Server update]

Post by RocketJedi » Tue Jun 02, 2015 2:43 pm

epic and a good read although I'm clueless about much of it LOL been running on MH2 server with nw3 trying new version today! :)
https://www.vulpinemission.com
Image ROCKET-X8 Server
Image MONSTERHUNT w/ NALI WEAPONS 3 + RX8
Image BUNNYTRACK NY
Image SNIPER DEATHMATCH
Image InstaGib + ComboGib + Jailbreak
Image ROSEBUM ROCKET-X RB