XC_Engine [20] - XC_Core [7b] - XC_IpDrv

[Higor]

I started doing some small reverse engineering on how map switch works, as well as how subsystems interacted with it and without realizing, stumbled upon something I wanted to fix years ago: traces going through movers on dedicated servers.
Continued to that, I kept making experiments with the GameEngine and got to achieve a few interesting things, things you'd see in a game patch now deployed into an extension.

=========================================================
XC_Core - Base extension for native UT99 addons by Higor

Releases
7b: GIT UT99.org mirror

XC_Core_7.7z

Spoiler

Version 7b

=============
Installation:
=============
XC_Core.u
XC_Core.int
XC_Core.dll (win32)
XC_Core.so (linux)
LZMA.dll (win32)
LZMA.so (linux)
>>> ~UnrealTournament\System\


=================================
Setting up LZMA channel upload:
(optional, unredirected servers)
=================================
~UnrealTournament.ini
>>>
[IpDrv.TcpNetDriver] or [XC_IpDrv.XC_TcpNetDriver]
AllowDownloads=True
...
DownloadManagers=IpDrv.HTTPDownload
DownloadManagers=XC_Core.XC_ChannelDownload
DownloadManagers=Engine.ChannelDownload

Then keep the .LZMA (or .UZ) files on the same directory as the uncompressed versions.


=============================
LZMA Compression commandlets:
=============================

You can LZMA compress using a XC_Core commandlet:
UCC LZMACompress ..\Maps\CTF-Coret.unr

You can LZMA decompress using 7zip, WinRar or:
UCC LZMADecompress ..\Maps\CTF-Coret.unr.lzma

Both commandlets support wildcards.


===================
Additional natives:
===================
See XC_CoreStatics class (Object subclass).


===================
UBinary serializer:
===================
Now merged into XC_Core, proves minimal binary file handling to UnrealScript.
Check classes BinarySerializer (Object) and BinaryTester (Actor) for usage guidelines.

NOTES:
For security measures file writer doesn't allow creating files outside of the game directory.


======
Notes:
======
Having XC_Core files installed allows you to use LZMA channel download from servers that have this feature.
This package is required to run XC_Engine and XC_IpDrv.


========================
c++ headers and linking:
========================
This package contains headers that allow the user to utilize XC_Core features in own native packages.
Just add ..\XC_Core\Inc to include settings and link to XC_Core.lib (or XC_Core.so in Linux)

Don't forget to define this macro somewhere in your code (or preprocessor):
#define XC_CORE_API DLL_IMPORT

Download.


NEW

XC_Engine_20.7z

XC_Engine_19.7z

XC_Engine_18.7z

When reporting crashes, please attach the critical error logs if possible and tell what package/class is causing it.
=======================
XC_Engine - XC_GameEngine extension for UT99 by Higor.


===========
Setting up:
===========
** REQUIRES AT LEAST XC_CORE VERSION 7
Place XC_Engine binaries in your ~UT/System/ directory.
The new GameEngine we want to load has to be specified in UnrealTournament.ini (or it's server equivalent) as follows.

[Engine.Engine]
;GameEngine=Engine.GameEngine
GameEngine=XC_Engine.XC_GameEngine
;NetworkDevice=IpDrv.TcpNetDriver
NetworkDevice=XC_IpDrv.XC_TcpNetDriver

Be adviced, when editing ServerPackages and ServerActors in a XC_Engine server, find the [XC_Engine.XC_GameEngine] entry!!!
Either remove it (and apply on GameEngine), or apply the changes on said (XC_GameEngine) entry instead.

Safe to use in v436-v451, and on ACE servers since most hacks are reverted during online sessions.
Just avoid AnthChecker servers until they have whitelisted this binary.


=================
Global features:
=================
- (Server, Standalone) Runtime UnrealScript/Native function replacer plus existing premade replacements (bugfixes, optimizations).
- (Conditional) Big collection of new native functions to use, check the UnrealScript source for documentation.
- (Linux) Server/Client communication no longer borks strings with non-standard characters.
- (All) Version 451 GET and SET command functionality + crashfix.
- (Server) Fake player and malformed string crash exploits fix, players cannot bypass password protection, massive log spam prevention.
- (Server) New Relevancy loop code that allows better/extended visibility conditions on actor replication, also fixes a few flaws on net priority.
- (Editor) New navigation network builder.
- (Client) Renderer-independant framerate limiter, combined with a fixed launcher can solve speed/timing issues: type "FPS [limit]".

- (Conditional) XC_Core natives have their numbered opcodes enabled for use without package dependancy.
- (Dedicated Server) Moving Brush Tracker in Dedicated servers, specific maps can be ignored.
- (Server) Ability to send 'undownloadable' maps.
- (All) Makes several properties from native only classes visible to UnrealScript, player commands and edit windows (win32). Below table for more info.
- (All) Timing fix for computers that change CPU frequencies (DOESN'T ADDRESS AMD TIMERS)
- (Server) TravelManager subsystem providing extended inventory handling for Coop games (don't lose items on reconnect).
- (Client) Automatic cache conversion to desired directories.
- (All) Collision Grid replacing the old hash, per-actor custom primitives support.
- (All) Cleaner map switch by nulling out transient actor references to the main level.
- (Server) Worked around RELIABLE_BUFFER crash on Windows servers.
- (Server, Win32) LZMA autocompressor can be run on a separate thread as soon as a map is loaded.
- (All) Lower memory usage and faster map cleanup on long games.
- (Server, experimental) Sliding player bug workaround.

Note:
(Conditional) Means
- (All) in Linux builds
- (Server, Standalone player) In Win32 builds, auto-disables itself when joining a server.

=================
XC_IpDrv
Enhanced Net Driver and file downloaders.
=================
Net Driver:
- ICMP unreachable exploit patched.
- Connection limit, kills dataless connections.
- (Experimental) Can connect to redirects via proxy.

HTTP LZMA file downloader.
- (Experimental) Can connect to redirects via proxy.


================
Extra commands.
Check other documentation files for more commands.
================
- EditObject Name=Objectname Skip=skipcount
Client, Win32 only.
Brings up a property editor dialog of an object with a specified name.
Skip= is optional and can be used to bring up a newer objects with said name.

Example: "EditObject Name=MyLevel Skip=1" Brings up play level's XLevel properties.
Example: "EditObject Name=MyLevel" Brings up Entry level's XLevel properties.

- FPS fpslimit
Sets internal framerate limiter between 4 and 200, lower than 4 means disable.

- DumpObject Name=Objectname
Dumps object in question's memory block into a file (with the object's name), only dumps the first object with matching name.
If the object is a UFunction, then it will also save a file name FUNCTIONDATA.bin with the script code (serialized TArray<BYTE>).

- LogFields Name=classname
Logs all of the UnrealScript visible properties of the specified class, with property flags, offset, size and array count.
Boolean properties have their bitmask info logged instead of array size.

- LogClassSizes Outer=packagename(optional)
Prints in log a huge list of classes and their size in memory.
If the Outer=packagename parameter isn't used (or fails), it will print all classes's sizes.

- ToggleTimingFix - TimingFix
Toggles the timing fix on/off.
Timing fix is enabled by default and is saved in [XC_Engine.XC_GameEngine] config entry.

- ToggleDebugLogs - DebugLogs
Toggles additional logging, for developers.
Disabled by default, saved in [XC_Engine.XC_GameEngine] config entry.

- ToggleRelevancy - ToggleRelevant
Requires bUseLevelHook.
Toggles XC_Level relevancy loop on net servers, see "Relevancy loop.txt" for details.

- AdminLoginHook actor_name
For code usage, lets an actor register itself as the AdminLoginHook actor (overrides previous one).

- OctreeDebug Name=actor_name
Prints FCollisionCacus information for an actor (if contained in the Octree system).

- TimeFactor
Displays the Time Manager's current time scaler, if active.
Values other than 1 (or approximate) indicate that XC_Engine is the one responsible
for keeping your game running at normal speed.


===================
Exposed properties:
===================
Additional properties are now visible on certain native classes and their subclasses, these increase the potential functionality of servers and clients running mods coded to access them via GetPropertyText() or GET commands.
See "Relevancy loop.txt" for extra properties in Actor.
= CLASS -> CPP_PropertyName -> UScript_PropertyName (type) (flags)

- GameEngine -> GLevel -> Level (Level) (const, editconst)
- GameEngine -> GEntry -> Entry (Level) (const, editconst)
- DemoRecDriver -> DemoFileName -> DemoFileName (string) (const, editconst)
- LevelBase -> NetDriver -> NetDriver (obj NetDriver) (const, editconst)
- LevelBase -> DemoRecDriver -> DemoRecDriver (obj NetDriver) (const, editconst)
- LevelBase -> Engine -> Engine (obj Engine) (const, editconst)
- LevelBase -> URL.Protocol -> URL_Protocol (string) (const, editconst)
- LevelBase -> URL.Host -> URL_Host (string) (const, editconst)
- LevelBase -> URL.Port -> URL_Port (int) (const, editconst)
- LevelBase -> URL.Map -> URL_Map (string) (const, editconst)
- LevelBase -> URL.Op -> URL_Options (array<string>) (const, editconst)
- LevelBase -> URL.Portal -> URL_Portal (string) (const, editconst)
- LevelBase -> Actors.Num() -> ActorListSize (int) (const, editconst)
- Level -> iFirstDynamicActor -> iFirstDynamicActor (int) (const, editconst)
- Level -> iFirstNetRelevantActor -> iFirstNetRelevantActor (int) (const, editconst)
- NetDriver -> ClientConnections -> ClientConnections (array<obj NetConnection>) (const, editconst)
- NetDriver -> ServerConnection -> ServerConnection (obj NetConnection) (const, editconst)

====================================
Functions patched/hooked in runtime:
====================================
See XC_Engine_Actor and XC_Engine_UT99_Actor for a full list of script patches.

Additionally this hooks still remains forced by internal code:
UWindowList.Sort -> Super fast, doesn't crash practice session when map count exceeds ~4000

=============================
Own XC_GameEngine properties:
=============================
- XC_Version (int) (const, editconst)
Stored in defaults, accesible via "GET INI:ENGINE.ENGINE.GAMEENGINE XC_VERSION"
Also accesible via "XC_Engine" console command.

- bHackingTracker (bool) (const, editconst)
Indicates that XC_GameEngine is handling the moving brush tracker.
Only true in Dedicated servers where the loaded map has movers.

- bDisableTimingFix (bool) (config) (default=false)
Disables timing fix.

- bDisableBrushTracker (bool) (config) (default=false)
Disables XC_Engine's addition of Brush Tracker.

- bSortMaplistByFolder (bool) (config) (default=false)
Sort the map cache alphabetically by folder.

- bSortMaplistGlobal (bool) (config) (default=false)
Sort the entire map cache alphabetically. This overrides the previous setting.

- bAutoTravelManager (bool) (config) (default=true)
TravelManager rebuilds the travel lists every two seconds.

- bCacheConvertAtJoin (bool) (config) (default=false)
Copy and rename cache files from current server into it's corresponding system paths.

- bFasterUpload (bool) (config) (default=true)
Forces a temporary NetSpeed of 1000001 on incoming clients for non-redirected downloads, reverts when finished.
The initial NetSpeed value is stored in ConfiguredNetSpeed during file download.

- bCollisionHashHook (bool) (config) (default=true)
Initializes levels using the FCollisionCacus octree system instead of the FCollisionHash.

- bEnableDebugLogs (bool) (config) (default=false)
Displays additional logs aimed at developers.

- NoBrushTrackerFix (array<string>) (config)
List of maps we want to exclude from using the brush tracker fixer (only works in dedicated server mode), ex (~UT.ini):
NoBrushTrackerFix=CTF-Niven
NoBrushTrackerFix=etc...

- bUseLevelHook (bool) (config) (default=true)
Hooks the level object after map load, allows it to execute XC_Engine code.

- bForceLevelHook (bool) (config) (default=false)
Forces level hook if net client.

- bUseNewRelevancy (bool) (config) (default=true)
Requires bUseLevelHook=True. See: - ToggleRelevancy - ToggleRelevant commands

- AdminLoginHook (actor) (const, editconst)
Actor handling AdminLogin requests, already passed the checks needed to function.

- bAutoCompressLZMA (bool) (config) (default=false)
Spawns a thread to LZMA compress every package loaded by the server.
To be used in conjunction with XC_Core's XC_ChannelDownload.

- bScriptDebug (bool) (config) (default=false)
Displays script callstack on crashlogs, useful for developers and server admins under attack.


=================
Credits:
=================
I would like to thank my fellow betatesters
- Chamberly
- ~V~
- Nelsona
- SC]-[LONG_{HoF}
- $carface (and the legions of Siege apes)
- AnthRAX
- SicilianKill

And all of Cham's development server visitors for the help in bugfixing this.
And to the website owners where I downloaded very educational sample codes of Unreal Engine 2 and 3 (lol!)

[Feralidragon]

Well, by making your own engine subclass (as well as most of the other things), it's a given that you can modify pretty anything at that level
I will probably also extend the game engine within some time (mostly because I really need it for a few things), but the sheer amount of things that can be done by doing so are quite something.

- Hotswapping maps pretty quickly without any loading, let's for example say you're playing a Bot match, and decide to join a game.
Instead of closing the bot game, it will be frozen and the online game comes up as primary level, when you disconnect you'll be sent back to the Bot match.

Sounds nice, although to be completely honest I don't see the real practical use of that use-case you presented. If a player wants to play in a server, he would care less about going back later. It's still an interesting idea though.

- Loading maps in the background (if there's a way to open a second thread in UT).

It's possible to open any threads you want there, and I already did it myself for testing purposes (when I made a couple of tests with cURL integration into UT, since multi-threading is vital for asynchronous HTTP requests, I made a test by calling it from UScript, to then create a thread for cURL to make its request independently of the main thread of UT, and then make the callback to call a UScript event, so technically it's even possible to make multi-threaded UScript in a certain sense), and it works very very well so far. All there is to it is to simply create a thread like you would do in any normal C++ program, but you have to be very very careful with thread-safety, however when it comes to loading packages themselves I didn't test it out yet, but I think it may get a little bit tricky with the memory allocations and some internal engine flags switching back and forth during the package load.

One thing I want to do for my own purposes in the near future, is not exactly loading maps in the background, but something very similar: loading resources (some to UT, like texture packages, sounds, etc, some others directly to other subsystems and the graphic card itself using OpenGL). But I won't try to "compete" with the UT main thread for this, instead when it comes to UT resources, I will probably just watch the main thread and only load stuff whenever the main one isn't loading anything. How to do this exactly I still don't know (didn't look into it yet), but there's probably some flag of some sort when the engine is loading something by itself.

[Higor]

BTW, 451 version works now.

UTPG made the VEEEEERY stupid mistake of adding the MinClientVersion to GameEngine at the beginning of the class declaration, before the GLevel pointer.
That forced me to do a version check and hack the pointer location for 451 games, FFS my hate for v451 grows with each passing day...

If the game's source ever goes open, this should be the very first thing to correct!!!!

BTW, XC_GameEngine running in the Development Server (v451 windows) just fine.

[noccer]

- Hotswapping maps pretty quickly without any loading, let's for example say you're playing a Bot match, and decide to join a game.

I would LOOOOVE this one!!!
Would be VERY usefull if you just want to play some 1v1 games and dont want to wait all the time again but quickly jump in.
For the same map a score reset would be just great, for changing maps you could make a set of maps to be played X times in this order etc...

A functionality i absolutely like !!!

[Higor]

Added ability to edit objects, mostly useful for debugging as it requires the exact object name to be typed into the console.


EDIT:>
Allows clients to request maps that haven't been marked as 'AllowDownload', server still rejects sending those, but redirects dont!!
Hosting Unreal SP maps is now possible.

[papercoffee]

Your image is not visible.

[Dr.Flay]

I am wondering if the members of the UTPG project have permission to release a hotfix.
Smirftsch is no longer interested in it, but if you gave him all the required code and info, and it is something he can use in Unreal, then it would be easier for him to make something "official" as he has the required source.

[Higor]

The moving brush tracker thing was fixed in U227
Dunno how the file transfer and redirect thing is being handled anyways.

Although, since levels can be loaded manually or using a teleporter, some case discrepancies with the actual filename will cause Linux based redirects to fail to send the file (found out during the test run yesterday).
So I forced filename correction to all cases, not just to maps without AllowDownload enabled.

Since I don't have the game's code, all i'm doing is changing variables on existing objects and fixing the movers in my XC_GameEngine's Tick event, after the default GameEngine's Tick finished updating the world.
Did find a few things that can fix utils like PackageHelper (and therefore ACE)...


BTW:
The moving brush tracker is initialized AFTER the GameInfo's InitGame in both XC_Engine and normal Viewports, meaning you can run TraceActors and FastTrace through Movers at that stage!!
(Which is why Botz's Door paths worked, regardless of the ingame tests I made)

[JackGriffin]

I am wondering if the members of the UTPG project have permission to release a hotfix.
Smirftsch is no longer interested in it, but if you gave him all the required code and info, and it is something he can use in Unreal, then it would be easier for him to make something "official" as he has the required source.

My (cough) guess (cough) is that Smirf is very tightly bound by an NDA that limits what he can and cannot do. It's much more involved to 'fix' 451 than that sounds and it's not a project to be entered lightly. Yeah, he's made big improvements to 227 but it's taken many hundreds of hours of work and testing and almost none of that directly translates to UT's current issues.

[Dr.Flay]

Yeah, I couldn't see him doing it unless it was as easy as rebuilding the UT and Deus Ex renderers, which I am aware we only have because of the similarity to what he already did for 227.
Too far down the road now

[Higor]

On a different note, I'll probably make the DLL version dependant in order to avoid lots of future issues.
Using a patching style I saw in Seven Kingdoms II code, where you'd simply use IFDEFS in order to differ version specific code.

UnGame.h would look like this:

Code: Select all

/*=============================================================================
	UnGame.h: Unreal game class.
	Copyright 1997-1999 Epic Games, Inc. All Rights Reserved.

	Revision history:
		* Created by Tim Sweeney
		* Added v451 variable
=============================================================================*/

/*-----------------------------------------------------------------------------
	Unreal game engine.
-----------------------------------------------------------------------------*/

//
// The Unreal game engine.
//
class ENGINE_API UGameEngine : public UEngine
{
	DECLARE_CLASS(UGameEngine,UEngine,CLASS_Config|CLASS_Transient,Engine)

	// Variables.
#ifdef VERSION_451
	INT				MinClientVersion;
#endif
	ULevel*			GLevel;
	ULevel*			GEntry;
	UPendingLevel*	GPendingLevel;
	FURL			LastURL;
	TArray<FString> ServerActors;
	TArray<FString> ServerPackages;

	// Constructors.
	UGameEngine();
	void StaticConstructor();

	// UObject interface.
	void Serialize( FArchive& Ar );
	void Destroy();

	// UEngine interface.
	void Init();
	void Exit();
	void Tick( FLOAT DeltaSeconds );
	void Draw( UViewport* Viewport, UBOOL Blit=1, BYTE* HitData=NULL, INT* HitSize=NULL );
	UBOOL Exec( const TCHAR* Cmd, FOutputDevice& Ar=*GLog );
	void MouseDelta( UViewport*, DWORD, FLOAT, FLOAT );
	void MousePosition( class UViewport*, DWORD, FLOAT, FLOAT );
	void Click( UViewport*, DWORD, FLOAT, FLOAT );
	void SetClientTravel( UPlayer* Viewport, const TCHAR* NextURL, UBOOL bItems, ETravelType TravelType );
	FLOAT GetMaxTickRate();
	INT ChallengeResponse( INT Challenge );
	void SetProgress( const TCHAR* Str1, const TCHAR* Str2, FLOAT Seconds );

	// UGameEngine interface.
	virtual UBOOL Browse( FURL URL, const TMap<FString,FString>* TravelInfo, FString& Error );
	virtual ULevel* LoadMap( const FURL& URL, UPendingLevel* Pending, const TMap<FString,FString>* TravelInfo, FString& Error );
	virtual void SaveGame( INT Position );
	virtual void CancelPending();
	virtual void PaintProgress();
	virtual void UpdateConnectingMessage();
	virtual void BuildServerMasterMap( UNetDriver* NetDriver, ULevel* InLevel );
	virtual void NotifyLevelChange();
	void FixUpLevel();
};

/*-----------------------------------------------------------------------------
	The End.
-----------------------------------------------------------------------------*/

I wonder if we should make a collection of headers we know that were changed in the versions...
Since these are based on public headers, we shouldn't have problems right?
UDemo's source code appears to contain references to other changed classes.

[*Kr!D_o)]

That's brilliant Higor!

[Higor]

Added faster map searching algorithm that's hacked into the function Actor.GetMapName( string Prefix, string CurMap, int Dir);
It will cache the map list and keep it in memory for one frame before doing a full cache on next request.

Most mapvotes and menues do call GetMapName between 20-500 times depending on map count and the process can be insanely slow.
This loops the whole map directories once and creates a combined list of map names (for when there's multiple directories), then stores the last returned map's position to avoid loops.
If the UnrealScript code using this isn't doing weird things, the whole map list creation can take as little as 1/20 of what it used to.

Of course, the 'hacking the Native function table' part is only done partially.
If you're connected to a server, the table gets reverted to original to prevent you from getting kicked by ACE, still good if you're playing single player or hosting a server.

[Saya-chan]

Hearing about something useful being proprietary out of fear saddens me.

You're doing a good job but... it's just that.

Just out of curiosity... how MUCH do you know about the internals of the engine? I'm curious about certain aspects of it.

[Higor]

I'm just keeping it in check because I'm constantly adding features to it, and because small mistakes break game stability in amazing ways, also because I don't want to be constantly updating the posts.
Some ppl does have my green light to PM away and test it, my main target is to get this up for testing in a couple of Coop or MH servers since it's where it brings the greatest benefit.

About knowledge on the engine:
- Got UT's UnRender.cpp.
- Got Echelon's Engine cpp sources (modified UE2)
- Got some modified UE3 Core cpp sources (UnrealEngine's entire scripting engine isn't that mysterious, not too different from UE1)
- FerBotz (product of my experiments) and UT Community SDK.

I was thinking with remaking the PackageHelper ACE currently uses, right now it has a huge amount of limitations and can't read for files beyond default paths.