XC_Engine megathread

XC_Engine megathread

Postby Higor » Sat Feb 17, 2018 3:45 am

Discuss all stuff related to XC_Engine and others.

Current builds:

XC_Core - 8:
Spoiler: show
XC_Core - Base extension for native UT99 addons by Higor

Version 8

XC_Core.dll (win32)
XC_Core.so (linux)
LZMA.dll (win32)
LZMA.so (linux)
>>> ~UnrealTournament\System\

Setting up LZMA channel upload:
(optional, unredirected servers)
[IpDrv.TcpNetDriver] or [XC_IpDrv.XC_TcpNetDriver]

Then keep the .LZMA (or .UZ) files on the same directory as the uncompressed versions.

LZMA Compression commandlets:

You can LZMA compress using a XC_Core commandlet:
UCC LZMACompress ..\Maps\CTF-Coret.unr

You can LZMA decompress using 7zip, WinRar or:
UCC LZMADecompress ..\Maps\CTF-Coret.unr.lzma

Both commandlets support wildcards.

Additional natives:
See XC_CoreStatics class (Object subclass).

UBinary serializer:
Now merged into XC_Core, proves minimal binary file handling to UnrealScript.
Check classes BinarySerializer (Object) and BinaryTester (Actor) for usage guidelines.
For security measures file writer doesn't allow creating files outside of the game directory.

This package is required to run all other XC Tools.

c++ headers and linking:
This package contains headers that allow the user to utilize XC_Core features in own native packages.
Just add ..\XC_Core\Inc to include settings and link to XC_Core.lib (or XC_Core.so in Linux)

Don't forget to define this macro somewhere in your code (or preprocessor):

XC_Engine - 21:
Spoiler: show
XC_Engine - XC_GameEngine extension for UT99 by Higor.

Setting up:
Place XC_Engine files in your ~UT/System/ directory.

/** Auto-installer scripts

Run XC_Enable.bat/XC_Enable_nohome.sh scripts in order to auto-config XC_Engine stuff
The scripts will enable the engine, net driver and editor addons.

See "XC_Setup.txt" for more info.

In case the above fails, or a different setup is needed, follow these steps.
The new GameEngine we want to load has to be specified in UnrealTournament.ini (or it's server equivalent) as follows.


Be adviced, when editing ServerPackages and ServerActors in a XC_Engine server, find the [XC_Engine.XC_GameEngine] entry!!!
Either remove it (and apply on GameEngine), or apply the changes on said (XC_GameEngine) entry instead.

Safe to use in v436-v451, and on ACE servers since most hacks are reverted during online sessions.
In v436 Linux, make sure you're using the Core.so build that exports "_9__Context.Env" symbol.
Just avoid AnthChecker servers until they have whitelisted this binary.


- Global
Version 451 GET and SET command functionality + overflow crashfix.
Makes several properties from native only classes visible to UnrealScript, player commands and edit windows (win32). Below table for more info.
Collision Grid replacing the old hash, loaded from CollisionGrid (.dll/.so)
Cleaner map switch by nulling out potentially dangerous actor references to the main level.
Lower memory usage and faster map cleanup on long games by recycling actor names.
Log file size reduction by grouping log spam and displaying how much log messages repeat.

- Server
Moving Brush Tracker in Dedicated servers (movers block visibility checks), specific maps can be ignored.
Various exploits patched, see "Server exploits.txt" for more info.
New Relevancy loop code, see "Relevancy loop.txt" for more info.
Enhanced coop/SP games in online play, see "TravelManager.txt" for more info.
Ability to send maps marked as 'no download'.
LZMA autocompressor can be run on a separate thread as soon as a map is loaded.
(Experimental) Sliding player bug workaround by reordering the package order, putting textures last.

- Server / Player ** these are selectively disabled upon joining a server **
Runtime UnrealScript/Native function replacer plus existing premade replacements (bugfixes, optimizations).
Big collection of new native functions to use, check the UnrealScript source for documentation.
XC_Core natives have their numbered opcodes enabled for use without package dependancy.
Thread safe memory allocator (if not running XC_Launcher)

- Linux
Server/Client communication no longer borks strings with non-standard characters.
Added SIGSEGV and SIGIOT handlers, crash logs display the call history almost like windows UT.

- Editor:
Enhanced navigation network builder, see "XC_PathBuilder.txt" for more info.
New Unreal Editor addons added to the brush builder pane.

- Client / Player:
Built-in framerate limiter, see "Framerate limiter.txt" for more info.
Ingame cache converter, see "AutoCacheConverter.txt" for more info.
Prevents servers from using 'Open' and 'ClientTravel' commands to open local files on the client.
Clients no longer send options 'Game' and 'Mutator' in their login string.
In most cases of package mismatch when joining, clients will load/download from other sources instead of failing to connect.
More info displayed during file download: amount of files, data pending installation.

Enhanced Net Driver and file downloaders.
Net Driver:
- ICMP unreachable exploit patched.
- Connection limit, kills dataless connections.

HTTP LZMA file downloader.
- (Experimental) Can connect to redirects via proxy.

Extra commands.
Check other documentation files for more commands.
- EditObject Name=Objectname Skip=skipcount
Client, Win32 only.
Brings up a property editor dialog of an object with a specified name.
Skip= is optional and can be used to bring up a newer objects with said name.

Example: "EditObject Name=MyLevel Skip=1" Brings up play level's XLevel properties.
Example: "EditObject Name=MyLevel" Brings up Entry level's XLevel properties.

- DumpObject Name=Objectname
Dumps object in question's memory block into a file (with the object's name), only dumps the first object with matching name.
If the object is a UFunction, then it will also save a file name FUNCTIONDATA.bin with the script code (serialized TArray<BYTE>).

- LogFields Name=classname
Logs all of the UnrealScript visible properties of the specified class, with property flags, offset, size and array count.
Boolean properties have their bitmask info logged instead of array size.

- LogClassSizes Outer=packagename(optional)
Prints in log a huge list of classes and their size in memory.
If the Outer=packagename parameter isn't used (or fails), it will print all classes's sizes.

- ToggleTimingFix - TimingFix
Toggles the timing fix on/off.
Timing fix is enabled by default and is saved in [XC_Engine.XC_GameEngine] config entry.

- ToggleDebugLogs - DebugLogs
Toggles additional logging, for developers.
Disabled by default, saved in [XC_Engine.XC_GameEngine] config entry.

- ToggleRelevancy - ToggleRelevant
Requires bUseLevelHook.
Toggles XC_Level relevancy loop on net servers, see "Relevancy loop.txt" for details.

- TimeFactor
Displays the Time Manager's current time scaler, if active.
Values other than 1 (or approximate) indicate that XC_Engine is the one responsible
for keeping your game running at normal speed.

Exposed properties:
Additional properties are now visible on certain native classes and their subclasses, these increase the potential functionality of servers and clients running mods coded to access them via GetPropertyText() or GET commands.
See "Relevancy loop.txt" for extra properties in Actor.
= CLASS -> CPP_PropertyName -> UScript_PropertyName (type) (flags)

- GameEngine -> GLevel -> Level (Level) (const, editconst)
- GameEngine -> GEntry -> Entry (Level) (const, editconst)
- DemoRecDriver -> DemoFileName -> DemoFileName (string) (const, editconst)
- LevelBase -> NetDriver -> NetDriver (obj NetDriver) (const, editconst)
- LevelBase -> DemoRecDriver -> DemoRecDriver (obj NetDriver) (const, editconst)
- LevelBase -> Engine -> Engine (obj Engine) (const, editconst)
- LevelBase -> URL.Protocol -> URL_Protocol (string) (const, editconst)
- LevelBase -> URL.Host -> URL_Host (string) (const, editconst)
- LevelBase -> URL.Port -> URL_Port (int) (const, editconst)
- LevelBase -> URL.Map -> URL_Map (string) (const, editconst)
- LevelBase -> URL.Op -> URL_Options (array<string>) (const, editconst)
- LevelBase -> URL.Portal -> URL_Portal (string) (const, editconst)
- LevelBase -> Actors.Num() -> ActorListSize (int) (const, editconst)
- Level -> iFirstDynamicActor -> iFirstDynamicActor (int) (const, editconst)
- Level -> iFirstNetRelevantActor -> iFirstNetRelevantActor (int) (const, editconst)
- NetDriver -> ClientConnections -> ClientConnections (array<obj NetConnection>) (const, editconst)
- NetDriver -> ServerConnection -> ServerConnection (obj NetConnection) (const, editconst)

Functions patched/hooked in runtime:
See XC_Engine_Actor and XC_Engine_UT99_Actor for a full list of script patches.

Additionally this hook still remains forced by internal code:
UWindowList.Sort -> Super fast, doesn't crash practice session when map count exceeds ~4000

I would like to thank my fellow betatesters
- Chamberly
- ~V~
- Nelsona
- SC]-[LONG_{HoF}
- $carface (and the legions of Siege apes)
- AnthRAX
- SicilianKill

And all of Cham's development server visitors for the help in bugfixing this.
And to the website owners where I downloaded very educational sample codes of Unreal Engine 2 and 3 (lol!)

UCC2 - 3:

XC_Launch - 1:
Posts: 1644
Joined: Sun Mar 04, 2012 6:47 pm

Re: XC_Engine megathread

Postby Higor » Sat Apr 14, 2018 6:24 am

Time to kick off the new discussion thread.

Let's see who figures out what I'm doing here...
Hey @anth !
Posts: 1644
Joined: Sun Mar 04, 2012 6:47 pm

Re: XC_Engine megathread

Postby Feralidragon » Sat Apr 14, 2018 11:36 am

I am just going to make a really wild guess, which is probably wrong, and say that you're on your way to remap UnrealEd/UCC functions so you can do something close to creating your own UnrealEd/UCC version, or at least so you can add more powerful native plugins (and more easily).
User avatar
Posts: 5001
Joined: Wed Feb 27, 2008 6:24 pm
Location: Liandri
Personal rank: Work In Progress

Re: XC_Engine megathread

Postby sektor2111 » Mon Apr 16, 2018 5:51 am

Are you doing something for Android ? Excuse my question if I'm wrong...
User avatar
Posts: 3623
Joined: Sun May 09, 2010 6:15 pm
Location: vect(1,1,1)

Re: XC_Engine megathread

Postby Higor » Mon Apr 16, 2018 7:46 am

That would be a new 'bridge' library for Linux, which I'll name Core_GCC for now.

When GCC was updated to version 3 the name mangling scheme was modified, making c++ shared libraries built on GCC 3.x incompatible with GCC 2.x ones (Along a few other things).
What this new library does is load all the usable exports from Core.so, and re-export them in Core_GCC.so on the new name mangling scheme.

What this does (with the help of modified headers), is allow me to build UT99 packages like XC_Core, etc using newer GCC compilers.
These packages should load the bridge binary automatically during initialization.

The expected behaviour us:
- Load XC_Core.so (new compiler)
- During XC_Core.so binary initialization, Core_GCC.so is loaded. (interrupts operation until Core_GCC is fully intialized)
- During Core_GCC.so binary initialization, the old Core.so symbols are imported and copied into the exports in the new format.
- After Core_GCC.so finishes initializing, XC_Core.so continues initialization and succesfully locates all the newly translated imports.

If this experiment is a success, I'll do it's Engine equivalent and ditch both old compilers (VC++6 and GCC 2.95), improve vector math, use more C++11 features and get rid of some ugly hacks I needed for v440/v451 compatibility.
May as well release the modified headers so that native coding on Linux without weird compiler mumbo-jumbos becomes a simple reality.

Another advantage of a bridge binary is exporting the __Context::Env symbols in the new format.
Even if Core.so happens to be the old v436 build that has no exception handlers, the bridge binary can manually take care of that (if it can't load Core.so __Context::Env, then create one in the bridge).
Posts: 1644
Joined: Sun Mar 04, 2012 6:47 pm

Re: XC_Engine megathread

Postby Chris » Fri Apr 20, 2018 3:27 pm

Aww I was too late! Oh well, guess that's what you get for being inactive.

While being on the subject of GCC, you don't happen to know if GCC (7.x.x) and MSVC shares the same mangling scheme? (Please say yes!) In case it does, that would really make my job easier. :loool:

While writing my own COFF linker and loader, I noticed some interesting desicions by GCC.

When targeting ELF (Linux GCC) the compiler creates an extra indirection to access static variables (globals or local statics).
For example say that you have a piece of code that looks like this
Code: Select all
int SomeVar;

void main()
       SomeVar = 100;

This would translate into something like this:

Code: Select all
mov rax, QWORD PTR[rip + offset]
mov DWORD PTR[rax], 64h

Any idea why that is?
It's not doing that when targeting COFF (GCC MinGW)

However GCC targeting COFF does something like that for extern declared variables (I don't remember if that applies to functions too but I would think it does).
Yes ELF is a lot more convenient when it comes to resolving relocatables. I've got a functional linker for COFF as well now!)

Oh and another thing, while GCC adds the real offset value into the relocation table for ELF, it doesn't do that for COFF. Instead, in COFF, the relocation entry only contains the name of the segment to which the symbol belongs. So where do we find the actual segment offset? Well it turns out that they baked that into the machine code already.. It took me a while to figure that out while it was right in front of me all along. :ironic:
Posts: 113
Joined: Mon Nov 24, 2014 9:27 am

Re: XC_Engine megathread

Postby Higor » Fri Apr 20, 2018 6:40 pm

Sorry, haven't made any extensive research on unix linkers.

The indirection (MAYBE) could be there to address the dynamic starting address of the shared library.

In windows DLL's you won't see the indirection because all absolute addresses are modified at load time (there's a compressed offset table at the end of the DLL), so during dynamic linking time all of the locations of absolute addresses are offset by [DYNAMICBASE-DLLBASE].
The problem with this is that you can't edit a DLL file and remove an absolute addrees (replacing it with code) because the linker will still offset that pointer, breaking your edited DLL (so you have to edit the table as well).

Example in Win32 (adding 16 to a global var):
Code: Select all
mov eax,dword ptr 0x00001000
mov edx,[eax]
add edx,0x10
mov [eax],edx

I can NOP the entire code and prevent the operation (guessing the amount of nops lol)...
Code: Select all

which looks like 909090909090909090909090 in a HEX editor

But thanks to the DLL's global address table saying there's a pointer (supposedly 0x00001000) on that code, the linking process will offset that pointer.
Code: Select all
DLLBASE: 0x00001000 (pointer is base address for simplicity)
DYNAMICBASE: 0x00002000 (dll was loaded, but code was placed at diff memory address)

So the 4 bytes between the brackets which are the pointer: 909090[90909090]90...etc
Are offset by 0x00002000 - 0x00001000 (0x1000)
The resulting code in memory looks like: 909090[90A09090]909090... DON'T FORGET ABOUT ENDIANNESS!!!

I don't know exactly how this goes in linux but a simple indirection with [startaddr+offset] can do the job done just as good, while the instruction size is larger and maybe it can take half to one more cycle to process, it can be hacked/edited without the Pointer Table nightmare scenario dll's have.
Posts: 1644
Joined: Sun Mar 04, 2012 6:47 pm

Re: XC_Engine megathread

Postby Chris » Sat Apr 21, 2018 2:14 pm

I didn't make much research on existing unix linkers either, however I did quite some research on the ELF relocatable (the compiled but not yet linked .o files) and it is much easier and more convenient in many ways compared to the COFF files outputted from a Win32 compiler (not linked). I managed to create a fully functional linker for both ELF relocatables and for COFF relocatables. the ELF version was very straightforward.
Just iterate over the relocation segment and resolve all the symbols if it exists in any of the current compilation units, otherwise check libraries and relocate at loadtime.
Since both the .so and the .dll has a symbol table, the runtime linker would apply the offset addend for both filetypes, however I think you're onto something there.
The linker wouldn't have to mess with the code segment, only the segment that cotains the indirections.
I can see a major disadvantage here however;
Due to the way the compiler outputs the assembly for the indirection, it loses the advantage of relative offsets:

Code: Select all
mov rax, QWORD PTR[rip + offset] ;<-Relative offsets, image local variables can be resolved with static linking.
mov DWORD PTR[rax], 64h ;<-!!! this is no longer a relative offset, which means this needs to be resolved at load time.

How common is it to hex edit the code segment of dll and so files? I guess it would be pretty convenient in some situations.

The dynamic linker should have a simple sanity condition so you can easily just NULL out a symbol entry without having to resize the symbol table (Saves you from having to relocate every segment that follows and modify the headers which suddenly requires a lot more work).

The compressed segments was also a bit of a pain to get right in COFF, because not only are there two options (short name vs long name) where the short name is directly added to the 8 byte descriptor, vs a long name where the descriptor contains an offset into the string table (+4 bytes for the size of the string table).
There is also a thrid alternative where the offset is for some reason encoded as an UTF-8 string.
Code: Select all

   union {
      BYTE    ShortName[8];
      struct {
         DWORD   Short;     // if 0, use LongName
         DWORD   Long;      // offset into string table
      } Name;
      DWORD   LongName[2];    // PBYTE [2]
   } N;
   DWORD VirtualSize;
   DWORD Offset;      //Segment Offset
   DWORD Size;         //Size of data
   DWORD Position;      //Position in file
   DWORD RelPosition;   //Position in relocation sector
   DWORD LnPosition;   //Position in Line number sector
   WORD  NumReloc;      //Number of relocations
   WORD  NumLn;      //Number of line numbers
   DWORD Flags;

   char* GetName(OBJECT_FILE* O)
      if (!N.Name.Short)
         if (N.Name.Long < 4) //Invalid offset
            return "";
         return (O->StringTable + N.Name.Long - 4);
      else if(N.ShortName[0] == 0x2F) // '/' indicates encoded offset
         return O->StringTable + ToNum((char*)N.ShortName + 1) - 4;
      return (char*)N.ShortName;

I guess the big mashup of unions and structs are leftovers from the early NT (or pre-NT) era..
Posts: 113
Joined: Mon Nov 24, 2014 9:27 am

Re: XC_Engine megathread

Postby Higor » Wed Dec 12, 2018 12:12 am

So I'm doing a full codebase update on the entire XC stuff.
Going to be finally ditching the old VC++6 compiler for Visual Studio 2015, as well as adding a dependancy to something called CacusLib to where I moved some things into (threading API, UTF-8 logging, atomics, timers).
The XC_ binaries won't require the new Visual C++ runtimes at all, and CacusLib should have it statically linked (so you won't have to install it).

Those v432 headers sure had problems with C++ standards as well as other things, so I'm putting improved versions of them in this folder https://github.com/CacoFFF/XC-UT99/tree ... formUpdate which is prioritized over the original ones.
What's certain is that the new builds will have much smaller and better binaries.

In addition to the S3TC stuff I'll be adding a nice button to unreference unused textures from brushes (not visible after BSP construction) and to shrink the actor list, because we can't have enough editor improvements.
Posts: 1644
Joined: Sun Mar 04, 2012 6:47 pm

Re: XC_Engine megathread

Postby papercoffee » Wed Dec 12, 2018 1:05 am

Higor wrote:because we can't have enough editor improvements.

Absolutely agree on this.

More stability would be also awesome.
User avatar
Site Staff
Posts: 8923
Joined: Wed Jul 15, 2009 11:36 am
Location: Cologne, the city with the big cathedral.
Personal rank: coffee addicted !!!

Re: XC_Engine megathread

Postby sektor2111 » Wed Dec 12, 2018 6:59 am

Not big improvements but "Helpers" - perhaps Editor is stable enough for me but poor in options. Definitely Higor can write more stuff... maybe I have to explain what I'm thinking:
- Select - push a button - said brush has 2 NON-COPLANAR surfaces (this is a BAD point in a map causing PAIN);
- Select brush - push button - answer Brush has a NON-SOLID surface caused by a funky build;
- Two selected NavigationPoints - Are they well connected ? Which are ReachSpecs ? Adding (human readable) ReachSpecs at pushing button like that EZConnect.... from XC_EngineActor... and reporting them in readable string format (I think we can use Original DescribeSpecs if not a better one):
1. using ("manually") fly directives for those maps requiring Jets and all that stuff - for MH = FLY MANTA WARLORD and so on;
2. creating some special nodes on button push ? (optional assets/plugins compiled and MyLevel-ed instantly on a button press ?);
- showing that PreviousPath Bug from navigation because THAT ONE is causing stupid Pawn loops and YES, this is an ugly bug very obscure and well hidden from mappers by Polge himself - even XC_PathBuilder has this bug - I saw this a day before yesterday. I was thinking to do a builder scanner but iterations limit specific crash it's part of Editor too (yeah, read again), where I think it should not be like in game - lags in Editor ? WTF, who cares ? Editing it's taking time anyway. And... at this moment I could not do a state running in Editor - because I have 2 minutes patience - for debugging rather than sharing some lousy bugged Level.

Does DevPath is ready for fliers or has only "//fix me" dumb strings ? I suppose EPIC were not loving monsters - considering "lolzing" stuff from UnrealShare.ScriptedPawn and deals with their own bool values bIsPlayer stupidity here - so I'm expecting more bubbles existent in natives which I did not see yet but are sitting there... showing up some day (or some night) and causing a sudden heart attack...
We can forget a bit stability and we might want focus over some DEBUG type buttons, right ?

Revision at DevPath ? I see that nodes over movers are now navigable making Bot to stupidly fall when mover is not in the right position (should trace ground if spot can be a BASE for Walkers/Runners rather than blindly pointing Pawn to fall) - if that it's new stuff perhaps it's not what I'm expecting. Using real CPU power rather than hard-coded cycles ? Eh... I know that more mappers cannot map paths over a mover bridge but... I can, because it's a piece of cake. Currently it's needed a special node, else a default PathNode in plain vanilla UT would be enough...
User avatar
Posts: 3623
Joined: Sun May 09, 2010 6:15 pm
Location: vect(1,1,1)

Re: XC_Engine megathread

Postby Chamberly » Wed Dec 12, 2018 11:48 am

I gotta say save the past projects, you'll never know how you can use it later to compare. :idea: :mrgreen:
irc.globalgamers.net #uscript
User avatar
Posts: 1634
Joined: Sat Sep 17, 2011 4:32 pm
Location: TN, USA
Personal rank: Dame. Vandora

Re: XC_Engine megathread

Postby Higor » Mon Dec 17, 2018 4:33 am

Noticed the autocompressor shouldn't be compressing some files that will never be sent, I'm a bit lazy to expand the list so if there's another set of files UT absolutely requires in order to launch, let me know.
UWindow stuff is never sent anyways so no need to include here.

Code: Select all
static UBOOL IsDefaultPackage( const TCHAR* Pkg)
   //Get rid of paths...
   const TCHAR* Filename;
   for ( Filename=Pkg ; *Pkg ; Pkg++ )
      if ( *Pkg == '\\' || *Pkg == '/' )
         Filename = Pkg + 1;
   //Save as ANSI text
   static const TCHAR* DefaultList[] =
      {   TEXT("Botpack.u")
      ,   TEXT("Engine.u")
      ,   TEXT("Core.u")
      ,   TEXT("Unreali.u")
      ,   TEXT("UnrealShare.u")
      ,   TEXT("Editor.u")
      ,   TEXT("Fire.u")
      ,   TEXT("Credits.utx")
      ,   TEXT("LadderFonts.utx")
      ,   TEXT("LadrStatic.utx")
      ,   TEXT("LadrArrow.utx")   };

   // Compare
   const int max = ARRAY_COUNT( DefaultList);
   for ( int i=0 ; i<max ; i++ )
      if ( !appStricmp( Filename, DefaultList[i]) )
         return 1;
   return 0;

In the meantime...
Raw mouse input for UT in next build
(Experimental feature)
Posts: 1644
Joined: Sun Mar 04, 2012 6:47 pm

Return to Discussions

Who is online

Users browsing this forum: No registered users and 2 guests