XC_Engine megathread

XC_Engine megathread

Postby Higor » Sat Feb 17, 2018 3:45 am

Discuss all stuff related to XC_Engine and others.

Current builds:


XC_Core - 8:
viewtopic.php?f=63&t=12724
Spoiler: show
=========================================================
XC_Core - Base extension for native UT99 addons by Higor

Version 8

=============
Installation:
=============
XC_Core.u
XC_Core.int
XC_Core.dll (win32)
XC_Core.so (linux)
LZMA.dll (win32)
LZMA.so (linux)
>>> ~UnrealTournament\System\


=================================
Setting up LZMA channel upload:
(optional, unredirected servers)
=================================
~UnrealTournament.ini
>>>
[IpDrv.TcpNetDriver] or [XC_IpDrv.XC_TcpNetDriver]
AllowDownloads=True
...
DownloadManagers=IpDrv.HTTPDownload
DownloadManagers=XC_Core.XC_ChannelDownload
DownloadManagers=Engine.ChannelDownload

Then keep the .LZMA (or .UZ) files on the same directory as the uncompressed versions.


=============================
LZMA Compression commandlets:
=============================

You can LZMA compress using a XC_Core commandlet:
UCC LZMACompress ..\Maps\CTF-Coret.unr

You can LZMA decompress using 7zip, WinRar or:
UCC LZMADecompress ..\Maps\CTF-Coret.unr.lzma

Both commandlets support wildcards.


===================
Additional natives:
===================
See XC_CoreStatics class (Object subclass).


===================
UBinary serializer:
===================
Now merged into XC_Core, proves minimal binary file handling to UnrealScript.
Check classes BinarySerializer (Object) and BinaryTester (Actor) for usage guidelines.
For security measures file writer doesn't allow creating files outside of the game directory.


======
Notes:
======
This package is required to run all other XC Tools.


========================
c++ headers and linking:
========================
This package contains headers that allow the user to utilize XC_Core features in own native packages.
Just add ..\XC_Core\Inc to include settings and link to XC_Core.lib (or XC_Core.so in Linux)

Don't forget to define this macro somewhere in your code (or preprocessor):
#define XC_CORE_API DLL_IMPORT


XC_Engine - 21:
viewtopic.php?f=63&t=12725
Spoiler: show
XC_Engine - XC_GameEngine extension for UT99 by Higor.


===========
Setting up:
===========
** REQUIRES AT LEAST XC_CORE VERSION 8
Place XC_Engine files in your ~UT/System/ directory.

/** Auto-installer scripts

Run XC_Enable.bat/XC_Enable_nohome.sh scripts in order to auto-config XC_Engine stuff
The scripts will enable the engine, net driver and editor addons.

See "XC_Setup.txt" for more info.
**/

In case the above fails, or a different setup is needed, follow these steps.
The new GameEngine we want to load has to be specified in UnrealTournament.ini (or it's server equivalent) as follows.

[Engine.Engine]
;GameEngine=Engine.GameEngine
GameEngine=XC_Engine.XC_GameEngine
;NetworkDevice=IpDrv.TcpNetDriver
NetworkDevice=XC_IpDrv.XC_TcpNetDriver

Be adviced, when editing ServerPackages and ServerActors in a XC_Engine server, find the [XC_Engine.XC_GameEngine] entry!!!
Either remove it (and apply on GameEngine), or apply the changes on said (XC_GameEngine) entry instead.

Safe to use in v436-v451, and on ACE servers since most hacks are reverted during online sessions.
In v436 Linux, make sure you're using the Core.so build that exports "_9__Context.Env" symbol.
Just avoid AnthChecker servers until they have whitelisted this binary.


=================
Features:
=================

- Global
Version 451 GET and SET command functionality + overflow crashfix.
Makes several properties from native only classes visible to UnrealScript, player commands and edit windows (win32). Below table for more info.
Collision Grid replacing the old hash, loaded from CollisionGrid (.dll/.so)
Cleaner map switch by nulling out potentially dangerous actor references to the main level.
Lower memory usage and faster map cleanup on long games by recycling actor names.
Log file size reduction by grouping log spam and displaying how much log messages repeat.

- Server
Moving Brush Tracker in Dedicated servers (movers block visibility checks), specific maps can be ignored.
Various exploits patched, see "Server exploits.txt" for more info.
New Relevancy loop code, see "Relevancy loop.txt" for more info.
Enhanced coop/SP games in online play, see "TravelManager.txt" for more info.
Ability to send maps marked as 'no download'.
LZMA autocompressor can be run on a separate thread as soon as a map is loaded.
(Experimental) Sliding player bug workaround by reordering the package order, putting textures last.

- Server / Player ** these are selectively disabled upon joining a server **
Runtime UnrealScript/Native function replacer plus existing premade replacements (bugfixes, optimizations).
Big collection of new native functions to use, check the UnrealScript source for documentation.
XC_Core natives have their numbered opcodes enabled for use without package dependancy.
Thread safe memory allocator (if not running XC_Launcher)

- Linux
Server/Client communication no longer borks strings with non-standard characters.
Added SIGSEGV and SIGIOT handlers, crash logs display the call history almost like windows UT.

- Editor:
Enhanced navigation network builder, see "XC_PathBuilder.txt" for more info.
New Unreal Editor addons added to the brush builder pane.

- Client / Player:
Built-in framerate limiter, see "Framerate limiter.txt" for more info.
Ingame cache converter, see "AutoCacheConverter.txt" for more info.
Prevents servers from using 'Open' and 'ClientTravel' commands to open local files on the client.
Clients no longer send options 'Game' and 'Mutator' in their login string.
In most cases of package mismatch when joining, clients will load/download from other sources instead of failing to connect.
More info displayed during file download: amount of files, data pending installation.


=================
XC_IpDrv
Enhanced Net Driver and file downloaders.
=================
Net Driver:
- ICMP unreachable exploit patched.
- Connection limit, kills dataless connections.

HTTP LZMA file downloader.
- (Experimental) Can connect to redirects via proxy.


================
Extra commands.
Check other documentation files for more commands.
================
- EditObject Name=Objectname Skip=skipcount
Client, Win32 only.
Brings up a property editor dialog of an object with a specified name.
Skip= is optional and can be used to bring up a newer objects with said name.

Example: "EditObject Name=MyLevel Skip=1" Brings up play level's XLevel properties.
Example: "EditObject Name=MyLevel" Brings up Entry level's XLevel properties.

- DumpObject Name=Objectname
Dumps object in question's memory block into a file (with the object's name), only dumps the first object with matching name.
If the object is a UFunction, then it will also save a file name FUNCTIONDATA.bin with the script code (serialized TArray<BYTE>).

- LogFields Name=classname
Logs all of the UnrealScript visible properties of the specified class, with property flags, offset, size and array count.
Boolean properties have their bitmask info logged instead of array size.

- LogClassSizes Outer=packagename(optional)
Prints in log a huge list of classes and their size in memory.
If the Outer=packagename parameter isn't used (or fails), it will print all classes's sizes.

- ToggleTimingFix - TimingFix
Toggles the timing fix on/off.
Timing fix is enabled by default and is saved in [XC_Engine.XC_GameEngine] config entry.

- ToggleDebugLogs - DebugLogs
Toggles additional logging, for developers.
Disabled by default, saved in [XC_Engine.XC_GameEngine] config entry.

- ToggleRelevancy - ToggleRelevant
Requires bUseLevelHook.
Toggles XC_Level relevancy loop on net servers, see "Relevancy loop.txt" for details.

- TimeFactor
Displays the Time Manager's current time scaler, if active.
Values other than 1 (or approximate) indicate that XC_Engine is the one responsible
for keeping your game running at normal speed.


===================
Exposed properties:
===================
Additional properties are now visible on certain native classes and their subclasses, these increase the potential functionality of servers and clients running mods coded to access them via GetPropertyText() or GET commands.
See "Relevancy loop.txt" for extra properties in Actor.
= CLASS -> CPP_PropertyName -> UScript_PropertyName (type) (flags)

- GameEngine -> GLevel -> Level (Level) (const, editconst)
- GameEngine -> GEntry -> Entry (Level) (const, editconst)
- DemoRecDriver -> DemoFileName -> DemoFileName (string) (const, editconst)
- LevelBase -> NetDriver -> NetDriver (obj NetDriver) (const, editconst)
- LevelBase -> DemoRecDriver -> DemoRecDriver (obj NetDriver) (const, editconst)
- LevelBase -> Engine -> Engine (obj Engine) (const, editconst)
- LevelBase -> URL.Protocol -> URL_Protocol (string) (const, editconst)
- LevelBase -> URL.Host -> URL_Host (string) (const, editconst)
- LevelBase -> URL.Port -> URL_Port (int) (const, editconst)
- LevelBase -> URL.Map -> URL_Map (string) (const, editconst)
- LevelBase -> URL.Op -> URL_Options (array<string>) (const, editconst)
- LevelBase -> URL.Portal -> URL_Portal (string) (const, editconst)
- LevelBase -> Actors.Num() -> ActorListSize (int) (const, editconst)
- Level -> iFirstDynamicActor -> iFirstDynamicActor (int) (const, editconst)
- Level -> iFirstNetRelevantActor -> iFirstNetRelevantActor (int) (const, editconst)
- NetDriver -> ClientConnections -> ClientConnections (array<obj NetConnection>) (const, editconst)
- NetDriver -> ServerConnection -> ServerConnection (obj NetConnection) (const, editconst)

====================================
Functions patched/hooked in runtime:
====================================
See XC_Engine_Actor and XC_Engine_UT99_Actor for a full list of script patches.

Additionally this hook still remains forced by internal code:
UWindowList.Sort -> Super fast, doesn't crash practice session when map count exceeds ~4000

=================
Credits:
=================
I would like to thank my fellow betatesters
- Chamberly
- ~V~
- Nelsona
- SC]-[LONG_{HoF}
- $carface (and the legions of Siege apes)
- AnthRAX
- SicilianKill

And all of Cham's development server visitors for the help in bugfixing this.
And to the website owners where I downloaded very educational sample codes of Unreal Engine 2 and 3 (lol!)



UCC2 - 3:
viewtopic.php?f=63&t=12727

XC_Launch - 1:
viewtopic.php?f=63&t=12726
Higor
Godlike
 
Posts: 1611
Joined: Sun Mar 04, 2012 6:47 pm

Re: XC_Engine megathread

Postby Higor » Sat Apr 14, 2018 6:24 am

Time to kick off the new discussion thread.

Let's see who figures out what I'm doing here...
Hey @anth !
Attachments
Translation.PNG
Higor
Godlike
 
Posts: 1611
Joined: Sun Mar 04, 2012 6:47 pm

Re: XC_Engine megathread

Postby Feralidragon » Sat Apr 14, 2018 11:36 am

I am just going to make a really wild guess, which is probably wrong, and say that you're on your way to remap UnrealEd/UCC functions so you can do something close to creating your own UnrealEd/UCC version, or at least so you can add more powerful native plugins (and more easily).
User avatar
Feralidragon
Godlike
 
Posts: 4836
Joined: Wed Feb 27, 2008 6:24 pm
Location: Liandri
Personal rank: Work In Progress

Re: XC_Engine megathread

Postby sektor2111 » Mon Apr 16, 2018 5:51 am

Are you doing something for Android ? Excuse my question if I'm wrong...
User avatar
sektor2111
Godlike
 
Posts: 3414
Joined: Sun May 09, 2010 6:15 pm
Location: vect(1,1,1)

Re: XC_Engine megathread

Postby Higor » Mon Apr 16, 2018 7:46 am

That would be a new 'bridge' library for Linux, which I'll name Core_GCC for now.

When GCC was updated to version 3 the name mangling scheme was modified, making c++ shared libraries built on GCC 3.x incompatible with GCC 2.x ones (Along a few other things).
What this new library does is load all the usable exports from Core.so, and re-export them in Core_GCC.so on the new name mangling scheme.

What this does (with the help of modified headers), is allow me to build UT99 packages like XC_Core, etc using newer GCC compilers.
These packages should load the bridge binary automatically during initialization.

The expected behaviour us:
- Load XC_Core.so (new compiler)
- During XC_Core.so binary initialization, Core_GCC.so is loaded. (interrupts operation until Core_GCC is fully intialized)
- During Core_GCC.so binary initialization, the old Core.so symbols are imported and copied into the exports in the new format.
- After Core_GCC.so finishes initializing, XC_Core.so continues initialization and succesfully locates all the newly translated imports.

If this experiment is a success, I'll do it's Engine equivalent and ditch both old compilers (VC++6 and GCC 2.95), improve vector math, use more C++11 features and get rid of some ugly hacks I needed for v440/v451 compatibility.
May as well release the modified headers so that native coding on Linux without weird compiler mumbo-jumbos becomes a simple reality.


==========
Edit:
Another advantage of a bridge binary is exporting the __Context::Env symbols in the new format.
Even if Core.so happens to be the old v436 build that has no exception handlers, the bridge binary can manually take care of that (if it can't load Core.so __Context::Env, then create one in the bridge).
Higor
Godlike
 
Posts: 1611
Joined: Sun Mar 04, 2012 6:47 pm

Re: XC_Engine megathread

Postby Chris » Fri Apr 20, 2018 3:27 pm

Aww I was too late! Oh well, guess that's what you get for being inactive.

While being on the subject of GCC, you don't happen to know if GCC (7.x.x) and MSVC shares the same mangling scheme? (Please say yes!) In case it does, that would really make my job easier. :loool:

While writing my own COFF linker and loader, I noticed some interesting desicions by GCC.

When targeting ELF (Linux GCC) the compiler creates an extra indirection to access static variables (globals or local statics).
For example say that you have a piece of code that looks like this
Code: Select all
int SomeVar;

void main()
{
       SomeVar = 100;
}


This would translate into something like this:

Code: Select all
mov rax, QWORD PTR[rip + offset]
mov DWORD PTR[rax], 64h


Any idea why that is?
It's not doing that when targeting COFF (GCC MinGW)

However GCC targeting COFF does something like that for extern declared variables (I don't remember if that applies to functions too but I would think it does).
Yes ELF is a lot more convenient when it comes to resolving relocatables. I've got a functional linker for COFF as well now!)

Oh and another thing, while GCC adds the real offset value into the relocation table for ELF, it doesn't do that for COFF. Instead, in COFF, the relocation entry only contains the name of the segment to which the symbol belongs. So where do we find the actual segment offset? Well it turns out that they baked that into the machine code already.. It took me a while to figure that out while it was right in front of me all along. :ironic:
Chris
Experienced
 
Posts: 103
Joined: Mon Nov 24, 2014 9:27 am

Re: XC_Engine megathread

Postby Higor » Fri Apr 20, 2018 6:40 pm

Sorry, haven't made any extensive research on unix linkers.

The indirection (MAYBE) could be there to address the dynamic starting address of the shared library.

In windows DLL's you won't see the indirection because all absolute addresses are modified at load time (there's a compressed offset table at the end of the DLL), so during dynamic linking time all of the locations of absolute addresses are offset by [DYNAMICBASE-DLLBASE].
The problem with this is that you can't edit a DLL file and remove an absolute addrees (replacing it with code) because the linker will still offset that pointer, breaking your edited DLL (so you have to edit the table as well).

Example in Win32 (adding 16 to a global var):
Code: Select all
mov eax,dword ptr 0x00001000
mov edx,[eax]
add edx,0x10
mov [eax],edx


I can NOP the entire code and prevent the operation (guessing the amount of nops lol)...
Code: Select all
nop,nop,nop,nop,nop
nop,nop,nop
nop,nop,nop,nop
nop,nop,nop,nop

which looks like 909090909090909090909090 in a HEX editor


But thanks to the DLL's global address table saying there's a pointer (supposedly 0x00001000) on that code, the linking process will offset that pointer.
Code: Select all
DLLBASE: 0x00001000 (pointer is base address for simplicity)
DYNAMICBASE: 0x00002000 (dll was loaded, but code was placed at diff memory address)

So the 4 bytes between the brackets which are the pointer: 909090[90909090]90...etc
Are offset by 0x00002000 - 0x00001000 (0x1000)
The resulting code in memory looks like: 909090[90A09090]909090... DON'T FORGET ABOUT ENDIANNESS!!!


I don't know exactly how this goes in linux but a simple indirection with [startaddr+offset] can do the job done just as good, while the instruction size is larger and maybe it can take half to one more cycle to process, it can be hacked/edited without the Pointer Table nightmare scenario dll's have.
Higor
Godlike
 
Posts: 1611
Joined: Sun Mar 04, 2012 6:47 pm

Re: XC_Engine megathread

Postby Chris » Sat Apr 21, 2018 2:14 pm

I didn't make much research on existing unix linkers either, however I did quite some research on the ELF relocatable (the compiled but not yet linked .o files) and it is much easier and more convenient in many ways compared to the COFF files outputted from a Win32 compiler (not linked). I managed to create a fully functional linker for both ELF relocatables and for COFF relocatables. the ELF version was very straightforward.
Just iterate over the relocation segment and resolve all the symbols if it exists in any of the current compilation units, otherwise check libraries and relocate at loadtime.
Since both the .so and the .dll has a symbol table, the runtime linker would apply the offset addend for both filetypes, however I think you're onto something there.
The linker wouldn't have to mess with the code segment, only the segment that cotains the indirections.
I can see a major disadvantage here however;
Due to the way the compiler outputs the assembly for the indirection, it loses the advantage of relative offsets:

Code: Select all
mov rax, QWORD PTR[rip + offset] ;<-Relative offsets, image local variables can be resolved with static linking.
mov DWORD PTR[rax], 64h ;<-!!! this is no longer a relative offset, which means this needs to be resolved at load time.

How common is it to hex edit the code segment of dll and so files? I guess it would be pretty convenient in some situations.

The dynamic linker should have a simple sanity condition so you can easily just NULL out a symbol entry without having to resize the symbol table (Saves you from having to relocate every segment that follows and modify the headers which suddenly requires a lot more work).

The compressed segments was also a bit of a pain to get right in COFF, because not only are there two options (short name vs long name) where the short name is directly added to the 8 byte descriptor, vs a long name where the descriptor contains an offset into the string table (+4 bytes for the size of the string table).
There is also a thrid alternative where the offset is for some reason encoded as an UTF-8 string.
Code: Select all

struct HYP_IMAGE_SECTION_HEADER
{
   union {
      BYTE    ShortName[8];
      struct {
         DWORD   Short;     // if 0, use LongName
         DWORD   Long;      // offset into string table
      } Name;
      DWORD   LongName[2];    // PBYTE [2]
   } N;
   DWORD VirtualSize;
   DWORD Offset;      //Segment Offset
   DWORD Size;         //Size of data
   DWORD Position;      //Position in file
   DWORD RelPosition;   //Position in relocation sector
   DWORD LnPosition;   //Position in Line number sector
   WORD  NumReloc;      //Number of relocations
   WORD  NumLn;      //Number of line numbers
   DWORD Flags;

   char* GetName(OBJECT_FILE* O)
   {
      if (!N.Name.Short)
      {
         if (N.Name.Long < 4) //Invalid offset
            return "";
         return (O->StringTable + N.Name.Long - 4);
      }
      else if(N.ShortName[0] == 0x2F) // '/' indicates encoded offset
      {
         return O->StringTable + ToNum((char*)N.ShortName + 1) - 4;
      }
      return (char*)N.ShortName;
   }
};



I guess the big mashup of unions and structs are leftovers from the early NT (or pre-NT) era..
Chris
Experienced
 
Posts: 103
Joined: Mon Nov 24, 2014 9:27 am


Return to Discussions

Who is online

Users browsing this forum: No registered users and 1 guest