XC_Engine megathread
XC_Engine megathread
Discuss all stuff related to XC_Engine and others.
Current builds:
XC_Engine - 25 (for UT v469a):
viewtopic.php?f=63&t=14121
Current builds:
XC_Engine - 25 (for UT v469a):
viewtopic.php?f=63&t=14121
Last edited by Higor on Wed Sep 23, 2020 5:50 pm, edited 3 times in total.
Re: XC_Engine megathread
Time to kick off the new discussion thread.
Let's see who figures out what I'm doing here...
Hey @anth !
Let's see who figures out what I'm doing here...
Hey @anth !
- Feralidragon
- Godlike
- Posts: 5493
- Joined: Wed Feb 27, 2008 6:24 pm
- Personal rank: Work In Progress
- Location: Liandri
Re: XC_Engine megathread
I am just going to make a really wild guess, which is probably wrong, and say that you're on your way to remap UnrealEd/UCC functions so you can do something close to creating your own UnrealEd/UCC version, or at least so you can add more powerful native plugins (and more easily).
- sektor2111
- Godlike
- Posts: 6410
- Joined: Sun May 09, 2010 6:15 pm
- Location: On the roof.
Re: XC_Engine megathread
That would be a new 'bridge' library for Linux, which I'll name Core_GCC for now.
When GCC was updated to version 3 the name mangling scheme was modified, making c++ shared libraries built on GCC 3.x incompatible with GCC 2.x ones (Along a few other things).
What this new library does is load all the usable exports from Core.so, and re-export them in Core_GCC.so on the new name mangling scheme.
What this does (with the help of modified headers), is allow me to build UT99 packages like XC_Core, etc using newer GCC compilers.
These packages should load the bridge binary automatically during initialization.
The expected behaviour us:
- Load XC_Core.so (new compiler)
- During XC_Core.so binary initialization, Core_GCC.so is loaded. (interrupts operation until Core_GCC is fully intialized)
- During Core_GCC.so binary initialization, the old Core.so symbols are imported and copied into the exports in the new format.
- After Core_GCC.so finishes initializing, XC_Core.so continues initialization and succesfully locates all the newly translated imports.
If this experiment is a success, I'll do it's Engine equivalent and ditch both old compilers (VC++6 and GCC 2.95), improve vector math, use more C++11 features and get rid of some ugly hacks I needed for v440/v451 compatibility.
May as well release the modified headers so that native coding on Linux without weird compiler mumbo-jumbos becomes a simple reality.
==========
Edit:
Another advantage of a bridge binary is exporting the __Context::Env symbols in the new format.
Even if Core.so happens to be the old v436 build that has no exception handlers, the bridge binary can manually take care of that (if it can't load Core.so __Context::Env, then create one in the bridge).
When GCC was updated to version 3 the name mangling scheme was modified, making c++ shared libraries built on GCC 3.x incompatible with GCC 2.x ones (Along a few other things).
What this new library does is load all the usable exports from Core.so, and re-export them in Core_GCC.so on the new name mangling scheme.
What this does (with the help of modified headers), is allow me to build UT99 packages like XC_Core, etc using newer GCC compilers.
These packages should load the bridge binary automatically during initialization.
The expected behaviour us:
- Load XC_Core.so (new compiler)
- During XC_Core.so binary initialization, Core_GCC.so is loaded. (interrupts operation until Core_GCC is fully intialized)
- During Core_GCC.so binary initialization, the old Core.so symbols are imported and copied into the exports in the new format.
- After Core_GCC.so finishes initializing, XC_Core.so continues initialization and succesfully locates all the newly translated imports.
If this experiment is a success, I'll do it's Engine equivalent and ditch both old compilers (VC++6 and GCC 2.95), improve vector math, use more C++11 features and get rid of some ugly hacks I needed for v440/v451 compatibility.
May as well release the modified headers so that native coding on Linux without weird compiler mumbo-jumbos becomes a simple reality.
==========
Edit:
Another advantage of a bridge binary is exporting the __Context::Env symbols in the new format.
Even if Core.so happens to be the old v436 build that has no exception handlers, the bridge binary can manually take care of that (if it can't load Core.so __Context::Env, then create one in the bridge).
Re: XC_Engine megathread
Aww I was too late! Oh well, guess that's what you get for being inactive.
While being on the subject of GCC, you don't happen to know if GCC (7.x.x) and MSVC shares the same mangling scheme? (Please say yes!) In case it does, that would really make my job easier.
While writing my own COFF linker and loader, I noticed some interesting desicions by GCC.
When targeting ELF (Linux GCC) the compiler creates an extra indirection to access static variables (globals or local statics).
For example say that you have a piece of code that looks like this
This would translate into something like this:
Any idea why that is?
It's not doing that when targeting COFF (GCC MinGW)
However GCC targeting COFF does something like that for extern declared variables (I don't remember if that applies to functions too but I would think it does).
Yes ELF is a lot more convenient when it comes to resolving relocatables. I've got a functional linker for COFF as well now!)
Oh and another thing, while GCC adds the real offset value into the relocation table for ELF, it doesn't do that for COFF. Instead, in COFF, the relocation entry only contains the name of the segment to which the symbol belongs. So where do we find the actual segment offset? Well it turns out that they baked that into the machine code already.. It took me a while to figure that out while it was right in front of me all along.![Ironic :ironic:](./images/smilies/HeftigeIronie.gif)
While being on the subject of GCC, you don't happen to know if GCC (7.x.x) and MSVC shares the same mangling scheme? (Please say yes!) In case it does, that would really make my job easier.
![loool :loool:](./images/smilies/HaudenLukasoderso.gif)
While writing my own COFF linker and loader, I noticed some interesting desicions by GCC.
When targeting ELF (Linux GCC) the compiler creates an extra indirection to access static variables (globals or local statics).
For example say that you have a piece of code that looks like this
Code: Select all
int SomeVar;
void main()
{
SomeVar = 100;
}
Code: Select all
mov rax, QWORD PTR[rip + offset]
mov DWORD PTR[rax], 64h
It's not doing that when targeting COFF (GCC MinGW)
However GCC targeting COFF does something like that for extern declared variables (I don't remember if that applies to functions too but I would think it does).
Yes ELF is a lot more convenient when it comes to resolving relocatables. I've got a functional linker for COFF as well now!)
Oh and another thing, while GCC adds the real offset value into the relocation table for ELF, it doesn't do that for COFF. Instead, in COFF, the relocation entry only contains the name of the segment to which the symbol belongs. So where do we find the actual segment offset? Well it turns out that they baked that into the machine code already.. It took me a while to figure that out while it was right in front of me all along.
![Ironic :ironic:](./images/smilies/HeftigeIronie.gif)
Re: XC_Engine megathread
Sorry, haven't made any extensive research on unix linkers.
The indirection (MAYBE) could be there to address the dynamic starting address of the shared library.
In windows DLL's you won't see the indirection because all absolute addresses are modified at load time (there's a compressed offset table at the end of the DLL), so during dynamic linking time all of the locations of absolute addresses are offset by [DYNAMICBASE-DLLBASE].
The problem with this is that you can't edit a DLL file and remove an absolute addrees (replacing it with code) because the linker will still offset that pointer, breaking your edited DLL (so you have to edit the table as well).
Example in Win32 (adding 16 to a global var):
I can NOP the entire code and prevent the operation (guessing the amount of nops lol)...
But thanks to the DLL's global address table saying there's a pointer (supposedly 0x00001000) on that code, the linking process will offset that pointer.
I don't know exactly how this goes in linux but a simple indirection with [startaddr+offset] can do the job done just as good, while the instruction size is larger and maybe it can take half to one more cycle to process, it can be hacked/edited without the Pointer Table nightmare scenario dll's have.
The indirection (MAYBE) could be there to address the dynamic starting address of the shared library.
In windows DLL's you won't see the indirection because all absolute addresses are modified at load time (there's a compressed offset table at the end of the DLL), so during dynamic linking time all of the locations of absolute addresses are offset by [DYNAMICBASE-DLLBASE].
The problem with this is that you can't edit a DLL file and remove an absolute addrees (replacing it with code) because the linker will still offset that pointer, breaking your edited DLL (so you have to edit the table as well).
Example in Win32 (adding 16 to a global var):
Code: Select all
mov eax,dword ptr 0x00001000
mov edx,[eax]
add edx,0x10
mov [eax],edx
Code: Select all
nop,nop,nop,nop,nop
nop,nop,nop
nop,nop,nop,nop
nop,nop,nop,nop
which looks like 909090909090909090909090 in a HEX editor
Code: Select all
DLLBASE: 0x00001000 (pointer is base address for simplicity)
DYNAMICBASE: 0x00002000 (dll was loaded, but code was placed at diff memory address)
So the 4 bytes between the brackets which are the pointer: 909090[90909090]90...etc
Are offset by 0x00002000 - 0x00001000 (0x1000)
The resulting code in memory looks like: 909090[90A09090]909090... DON'T FORGET ABOUT ENDIANNESS!!!
Re: XC_Engine megathread
I didn't make much research on existing unix linkers either, however I did quite some research on the ELF relocatable (the compiled but not yet linked .o files) and it is much easier and more convenient in many ways compared to the COFF files outputted from a Win32 compiler (not linked). I managed to create a fully functional linker for both ELF relocatables and for COFF relocatables. the ELF version was very straightforward.
Just iterate over the relocation segment and resolve all the symbols if it exists in any of the current compilation units, otherwise check libraries and relocate at loadtime.
Since both the .so and the .dll has a symbol table, the runtime linker would apply the offset addend for both filetypes, however I think you're onto something there.
The linker wouldn't have to mess with the code segment, only the segment that cotains the indirections.
I can see a major disadvantage here however;
Due to the way the compiler outputs the assembly for the indirection, it loses the advantage of relative offsets:
How common is it to hex edit the code segment of dll and so files? I guess it would be pretty convenient in some situations.
The dynamic linker should have a simple sanity condition so you can easily just NULL out a symbol entry without having to resize the symbol table (Saves you from having to relocate every segment that follows and modify the headers which suddenly requires a lot more work).
The compressed segments was also a bit of a pain to get right in COFF, because not only are there two options (short name vs long name) where the short name is directly added to the 8 byte descriptor, vs a long name where the descriptor contains an offset into the string table (+4 bytes for the size of the string table).
There is also a thrid alternative where the offset is for some reason encoded as an UTF-8 string.
I guess the big mashup of unions and structs are leftovers from the early NT (or pre-NT) era..
Just iterate over the relocation segment and resolve all the symbols if it exists in any of the current compilation units, otherwise check libraries and relocate at loadtime.
Since both the .so and the .dll has a symbol table, the runtime linker would apply the offset addend for both filetypes, however I think you're onto something there.
The linker wouldn't have to mess with the code segment, only the segment that cotains the indirections.
I can see a major disadvantage here however;
Due to the way the compiler outputs the assembly for the indirection, it loses the advantage of relative offsets:
Code: Select all
mov rax, QWORD PTR[rip + offset] ;<-Relative offsets, image local variables can be resolved with static linking.
mov DWORD PTR[rax], 64h ;<-!!! this is no longer a relative offset, which means this needs to be resolved at load time.
The dynamic linker should have a simple sanity condition so you can easily just NULL out a symbol entry without having to resize the symbol table (Saves you from having to relocate every segment that follows and modify the headers which suddenly requires a lot more work).
The compressed segments was also a bit of a pain to get right in COFF, because not only are there two options (short name vs long name) where the short name is directly added to the 8 byte descriptor, vs a long name where the descriptor contains an offset into the string table (+4 bytes for the size of the string table).
There is also a thrid alternative where the offset is for some reason encoded as an UTF-8 string.
Code: Select all
struct HYP_IMAGE_SECTION_HEADER
{
union {
BYTE ShortName[8];
struct {
DWORD Short; // if 0, use LongName
DWORD Long; // offset into string table
} Name;
DWORD LongName[2]; // PBYTE [2]
} N;
DWORD VirtualSize;
DWORD Offset; //Segment Offset
DWORD Size; //Size of data
DWORD Position; //Position in file
DWORD RelPosition; //Position in relocation sector
DWORD LnPosition; //Position in Line number sector
WORD NumReloc; //Number of relocations
WORD NumLn; //Number of line numbers
DWORD Flags;
char* GetName(OBJECT_FILE* O)
{
if (!N.Name.Short)
{
if (N.Name.Long < 4) //Invalid offset
return "";
return (O->StringTable + N.Name.Long - 4);
}
else if(N.ShortName[0] == 0x2F) // '/' indicates encoded offset
{
return O->StringTable + ToNum((char*)N.ShortName + 1) - 4;
}
return (char*)N.ShortName;
}
};
Re: XC_Engine megathread
So I'm doing a full codebase update on the entire XC stuff.
Going to be finally ditching the old VC++6 compiler for Visual Studio 2015, as well as adding a dependancy to something called CacusLib to where I moved some things into (threading API, UTF-8 logging, atomics, timers).
The XC_ binaries won't require the new Visual C++ runtimes at all, and CacusLib should have it statically linked (so you won't have to install it).
Those v432 headers sure had problems with C++ standards as well as other things, so I'm putting improved versions of them in this folder https://github.com/CacoFFF/XC-UT99/tree ... formUpdate which is prioritized over the original ones.
What's certain is that the new builds will have much smaller and better binaries.
In addition to the S3TC stuff I'll be adding a nice button to unreference unused textures from brushes (not visible after BSP construction) and to shrink the actor list, because we can't have enough editor improvements.
Going to be finally ditching the old VC++6 compiler for Visual Studio 2015, as well as adding a dependancy to something called CacusLib to where I moved some things into (threading API, UTF-8 logging, atomics, timers).
The XC_ binaries won't require the new Visual C++ runtimes at all, and CacusLib should have it statically linked (so you won't have to install it).
Those v432 headers sure had problems with C++ standards as well as other things, so I'm putting improved versions of them in this folder https://github.com/CacoFFF/XC-UT99/tree ... formUpdate which is prioritized over the original ones.
What's certain is that the new builds will have much smaller and better binaries.
In addition to the S3TC stuff I'll be adding a nice button to unreference unused textures from brushes (not visible after BSP construction) and to shrink the actor list, because we can't have enough editor improvements.
- papercoffee
- Godlike
- Posts: 10448
- Joined: Wed Jul 15, 2009 11:36 am
- Personal rank: coffee addicted !!!
- Location: Cologne, the city with the big cathedral.
- Contact:
Re: XC_Engine megathread
Absolutely agree on this.Higor wrote:because we can't have enough editor improvements.
More stability would be also awesome.
- sektor2111
- Godlike
- Posts: 6410
- Joined: Sun May 09, 2010 6:15 pm
- Location: On the roof.
Re: XC_Engine megathread
Not big improvements but "Helpers" - perhaps Editor is stable enough for me but poor in options. Definitely Higor can write more stuff... maybe I have to explain what I'm thinking:
- Select - push a button - said brush has 2 NON-COPLANAR surfaces (this is a BAD point in a map causing PAIN);
- Select brush - push button - answer Brush has a NON-SOLID surface caused by a funky build;
- Two selected NavigationPoints - Are they well connected ? Which are ReachSpecs ? Adding (human readable) ReachSpecs at pushing button like that EZConnect.... from XC_EngineActor... and reporting them in readable string format (I think we can use Original DescribeSpecs if not a better one):
1. using ("manually") fly directives for those maps requiring Jets and all that stuff - for MH = FLY MANTA WARLORD and so on;
2. creating some special nodes on button push ? (optional assets/plugins compiled and MyLevel-ed instantly on a button press ?);
- showing that PreviousPath Bug from navigation because THAT ONE is causing stupid Pawn loops and YES, this is an ugly bug very obscure and well hidden from mappers by Polge himself - even XC_PathBuilder has this bug - I saw this a day before yesterday. I was thinking to do a builder scanner but iterations limit specific crash it's part of Editor too (yeah, read again), where I think it should not be like in game - lags in Editor ? WTF, who cares ? Editing it's taking time anyway. And... at this moment I could not do a state running in Editor - because I have 2 minutes patience - for debugging rather than sharing some lousy bugged Level.
Does DevPath is ready for fliers or has only "//fix me" dumb strings ? I suppose EPIC were not loving monsters - considering "lolzing" stuff from UnrealShare.ScriptedPawn and deals with their own bool values bIsPlayer stupidity here - so I'm expecting more bubbles existent in natives which I did not see yet but are sitting there... showing up some day (or some night) and causing a sudden heart attack...
We can forget a bit stability and we might want focus over some DEBUG type buttons, right ?
GameEngine
Revision at DevPath ? I see that nodes over movers are now navigable making Bot to stupidly fall when mover is not in the right position (should trace ground if spot can be a BASE for Walkers/Runners rather than blindly pointing Pawn to fall) - if that it's new stuff perhaps it's not what I'm expecting. Using real CPU power rather than hard-coded cycles ? Eh... I know that more mappers cannot map paths over a mover bridge but... I can, because it's a piece of cake. Currently it's needed a special node, else a default PathNode in plain vanilla UT would be enough...
- Select - push a button - said brush has 2 NON-COPLANAR surfaces (this is a BAD point in a map causing PAIN);
- Select brush - push button - answer Brush has a NON-SOLID surface caused by a funky build;
- Two selected NavigationPoints - Are they well connected ? Which are ReachSpecs ? Adding (human readable) ReachSpecs at pushing button like that EZConnect.... from XC_EngineActor... and reporting them in readable string format (I think we can use Original DescribeSpecs if not a better one):
1. using ("manually") fly directives for those maps requiring Jets and all that stuff - for MH = FLY MANTA WARLORD and so on;
2. creating some special nodes on button push ? (optional assets/plugins compiled and MyLevel-ed instantly on a button press ?);
- showing that PreviousPath Bug from navigation because THAT ONE is causing stupid Pawn loops and YES, this is an ugly bug very obscure and well hidden from mappers by Polge himself - even XC_PathBuilder has this bug - I saw this a day before yesterday. I was thinking to do a builder scanner but iterations limit specific crash it's part of Editor too (yeah, read again), where I think it should not be like in game - lags in Editor ? WTF, who cares ? Editing it's taking time anyway. And... at this moment I could not do a state running in Editor - because I have 2 minutes patience - for debugging rather than sharing some lousy bugged Level.
Does DevPath is ready for fliers or has only "//fix me" dumb strings ? I suppose EPIC were not loving monsters - considering "lolzing" stuff from UnrealShare.ScriptedPawn and deals with their own bool values bIsPlayer stupidity here - so I'm expecting more bubbles existent in natives which I did not see yet but are sitting there... showing up some day (or some night) and causing a sudden heart attack...
We can forget a bit stability and we might want focus over some DEBUG type buttons, right ?
GameEngine
Revision at DevPath ? I see that nodes over movers are now navigable making Bot to stupidly fall when mover is not in the right position (should trace ground if spot can be a BASE for Walkers/Runners rather than blindly pointing Pawn to fall) - if that it's new stuff perhaps it's not what I'm expecting. Using real CPU power rather than hard-coded cycles ? Eh... I know that more mappers cannot map paths over a mover bridge but... I can, because it's a piece of cake. Currently it's needed a special node, else a default PathNode in plain vanilla UT would be enough...
- Chamberly
- Godlike
- Posts: 1963
- Joined: Sat Sep 17, 2011 4:32 pm
- Personal rank: Dame. Vandora
- Location: TN, USA
- Contact:
Re: XC_Engine megathread
I gotta say save the past projects, you'll never know how you can use it later to compare.
![Mr. Green :mrgreen:](./images/smilies/icon_mrgreen.gif)
![Idea :idea:](./images/smilies/gecheckt.gif)
![Mr. Green :mrgreen:](./images/smilies/icon_mrgreen.gif)
Re: XC_Engine megathread
Noticed the autocompressor shouldn't be compressing some files that will never be sent, I'm a bit lazy to expand the list so if there's another set of files UT absolutely requires in order to launch, let me know.
UWindow stuff is never sent anyways so no need to include here.
===
In the meantime...
Raw mouse input for UT in next build
(Experimental feature)
UWindow stuff is never sent anyways so no need to include here.
Code: Select all
static UBOOL IsDefaultPackage( const TCHAR* Pkg)
{
//Get rid of paths...
const TCHAR* Filename;
for ( Filename=Pkg ; *Pkg ; Pkg++ )
if ( *Pkg == '\\' || *Pkg == '/' )
Filename = Pkg + 1;
//Save as ANSI text
static const TCHAR* DefaultList[] =
{ TEXT("Botpack.u")
, TEXT("Engine.u")
, TEXT("Core.u")
, TEXT("Unreali.u")
, TEXT("UnrealShare.u")
, TEXT("Editor.u")
, TEXT("Fire.u")
, TEXT("Credits.utx")
, TEXT("LadderFonts.utx")
, TEXT("LadrStatic.utx")
, TEXT("LadrArrow.utx") };
// Compare
const int max = ARRAY_COUNT( DefaultList);
for ( int i=0 ; i<max ; i++ )
if ( !appStricmp( Filename, DefaultList[i]) )
return 1;
return 0;
}
In the meantime...
Raw mouse input for UT in next build
(Experimental feature)
- sektor2111
- Godlike
- Posts: 6410
- Joined: Sun May 09, 2010 6:15 pm
- Location: On the roof.