Page 1 of 10

XC_Engine megathread

Posted: Sat Feb 17, 2018 3:45 am
by Higor
Discuss all stuff related to XC_Engine and others.

Current builds:

XC_Engine - 23 (full):

Re: XC_Engine megathread

Posted: Sat Apr 14, 2018 6:24 am
by Higor
Time to kick off the new discussion thread.

Let's see who figures out what I'm doing here...
Hey @anth !

Re: XC_Engine megathread

Posted: Sat Apr 14, 2018 11:36 am
by Feralidragon
I am just going to make a really wild guess, which is probably wrong, and say that you're on your way to remap UnrealEd/UCC functions so you can do something close to creating your own UnrealEd/UCC version, or at least so you can add more powerful native plugins (and more easily).

Re: XC_Engine megathread

Posted: Mon Apr 16, 2018 5:51 am
by sektor2111
Are you doing something for Android ? Excuse my question if I'm wrong...

Re: XC_Engine megathread

Posted: Mon Apr 16, 2018 7:46 am
by Higor
That would be a new 'bridge' library for Linux, which I'll name Core_GCC for now.

When GCC was updated to version 3 the name mangling scheme was modified, making c++ shared libraries built on GCC 3.x incompatible with GCC 2.x ones (Along a few other things).
What this new library does is load all the usable exports from, and re-export them in on the new name mangling scheme.

What this does (with the help of modified headers), is allow me to build UT99 packages like XC_Core, etc using newer GCC compilers.
These packages should load the bridge binary automatically during initialization.

The expected behaviour us:
- Load (new compiler)
- During binary initialization, is loaded. (interrupts operation until Core_GCC is fully intialized)
- During binary initialization, the old symbols are imported and copied into the exports in the new format.
- After finishes initializing, continues initialization and succesfully locates all the newly translated imports.

If this experiment is a success, I'll do it's Engine equivalent and ditch both old compilers (VC++6 and GCC 2.95), improve vector math, use more C++11 features and get rid of some ugly hacks I needed for v440/v451 compatibility.
May as well release the modified headers so that native coding on Linux without weird compiler mumbo-jumbos becomes a simple reality.

Another advantage of a bridge binary is exporting the __Context::Env symbols in the new format.
Even if happens to be the old v436 build that has no exception handlers, the bridge binary can manually take care of that (if it can't load __Context::Env, then create one in the bridge).

Re: XC_Engine megathread

Posted: Fri Apr 20, 2018 3:27 pm
by Chris
Aww I was too late! Oh well, guess that's what you get for being inactive.

While being on the subject of GCC, you don't happen to know if GCC (7.x.x) and MSVC shares the same mangling scheme? (Please say yes!) In case it does, that would really make my job easier. :loool:

While writing my own COFF linker and loader, I noticed some interesting desicions by GCC.

When targeting ELF (Linux GCC) the compiler creates an extra indirection to access static variables (globals or local statics).
For example say that you have a piece of code that looks like this

Code: Select all

int SomeVar;

void main()
       SomeVar = 100;
This would translate into something like this:

Code: Select all

mov rax, QWORD PTR[rip + offset]
mov DWORD PTR[rax], 64h
Any idea why that is?
It's not doing that when targeting COFF (GCC MinGW)

However GCC targeting COFF does something like that for extern declared variables (I don't remember if that applies to functions too but I would think it does).
Yes ELF is a lot more convenient when it comes to resolving relocatables. I've got a functional linker for COFF as well now!)

Oh and another thing, while GCC adds the real offset value into the relocation table for ELF, it doesn't do that for COFF. Instead, in COFF, the relocation entry only contains the name of the segment to which the symbol belongs. So where do we find the actual segment offset? Well it turns out that they baked that into the machine code already.. It took me a while to figure that out while it was right in front of me all along. :ironic:

Re: XC_Engine megathread

Posted: Fri Apr 20, 2018 6:40 pm
by Higor
Sorry, haven't made any extensive research on unix linkers.

The indirection (MAYBE) could be there to address the dynamic starting address of the shared library.

In windows DLL's you won't see the indirection because all absolute addresses are modified at load time (there's a compressed offset table at the end of the DLL), so during dynamic linking time all of the locations of absolute addresses are offset by [DYNAMICBASE-DLLBASE].
The problem with this is that you can't edit a DLL file and remove an absolute addrees (replacing it with code) because the linker will still offset that pointer, breaking your edited DLL (so you have to edit the table as well).

Example in Win32 (adding 16 to a global var):

Code: Select all

mov eax,dword ptr 0x00001000
mov edx,[eax]
add edx,0x10
mov [eax],edx
I can NOP the entire code and prevent the operation (guessing the amount of nops lol)...

Code: Select all


which looks like 909090909090909090909090 in a HEX editor
But thanks to the DLL's global address table saying there's a pointer (supposedly 0x00001000) on that code, the linking process will offset that pointer.

Code: Select all

DLLBASE: 0x00001000 (pointer is base address for simplicity)
DYNAMICBASE: 0x00002000 (dll was loaded, but code was placed at diff memory address)

So the 4 bytes between the brackets which are the pointer: 909090[90909090]90...etc
Are offset by 0x00002000 - 0x00001000 (0x1000)
The resulting code in memory looks like: 909090[90A09090]909090... DON'T FORGET ABOUT ENDIANNESS!!! 
I don't know exactly how this goes in linux but a simple indirection with [startaddr+offset] can do the job done just as good, while the instruction size is larger and maybe it can take half to one more cycle to process, it can be hacked/edited without the Pointer Table nightmare scenario dll's have.

Re: XC_Engine megathread

Posted: Sat Apr 21, 2018 2:14 pm
by Chris
I didn't make much research on existing unix linkers either, however I did quite some research on the ELF relocatable (the compiled but not yet linked .o files) and it is much easier and more convenient in many ways compared to the COFF files outputted from a Win32 compiler (not linked). I managed to create a fully functional linker for both ELF relocatables and for COFF relocatables. the ELF version was very straightforward.
Just iterate over the relocation segment and resolve all the symbols if it exists in any of the current compilation units, otherwise check libraries and relocate at loadtime.
Since both the .so and the .dll has a symbol table, the runtime linker would apply the offset addend for both filetypes, however I think you're onto something there.
The linker wouldn't have to mess with the code segment, only the segment that cotains the indirections.
I can see a major disadvantage here however;
Due to the way the compiler outputs the assembly for the indirection, it loses the advantage of relative offsets:

Code: Select all

mov rax, QWORD PTR[rip + offset] ;<-Relative offsets, image local variables can be resolved with static linking.
mov DWORD PTR[rax], 64h ;<-!!! this is no longer a relative offset, which means this needs to be resolved at load time.
How common is it to hex edit the code segment of dll and so files? I guess it would be pretty convenient in some situations.

The dynamic linker should have a simple sanity condition so you can easily just NULL out a symbol entry without having to resize the symbol table (Saves you from having to relocate every segment that follows and modify the headers which suddenly requires a lot more work).

The compressed segments was also a bit of a pain to get right in COFF, because not only are there two options (short name vs long name) where the short name is directly added to the 8 byte descriptor, vs a long name where the descriptor contains an offset into the string table (+4 bytes for the size of the string table).
There is also a thrid alternative where the offset is for some reason encoded as an UTF-8 string.

Code: Select all

	union {
		BYTE    ShortName[8];
		struct {
			DWORD   Short;     // if 0, use LongName
			DWORD   Long;      // offset into string table
		} Name;
		DWORD   LongName[2];    // PBYTE [2]
	} N;
	DWORD VirtualSize;
	DWORD Offset;		//Segment Offset
	DWORD Size;			//Size of data
	DWORD Position;		//Position in file
	DWORD RelPosition;	//Position in relocation sector
	DWORD LnPosition;	//Position in Line number sector
	WORD  NumReloc;		//Number of relocations
	WORD  NumLn;		//Number of line numbers
	DWORD Flags;

	char* GetName(OBJECT_FILE* O)
		if (!N.Name.Short)
			if (N.Name.Long < 4) //Invalid offset
				return "";
			return (O->StringTable + N.Name.Long - 4);
		else if(N.ShortName[0] == 0x2F) // '/' indicates encoded offset
			return O->StringTable + ToNum((char*)N.ShortName + 1) - 4;
		return (char*)N.ShortName;

I guess the big mashup of unions and structs are leftovers from the early NT (or pre-NT) era..

Re: XC_Engine megathread

Posted: Wed Dec 12, 2018 12:12 am
by Higor
So I'm doing a full codebase update on the entire XC stuff.
Going to be finally ditching the old VC++6 compiler for Visual Studio 2015, as well as adding a dependancy to something called CacusLib to where I moved some things into (threading API, UTF-8 logging, atomics, timers).
The XC_ binaries won't require the new Visual C++ runtimes at all, and CacusLib should have it statically linked (so you won't have to install it).

Those v432 headers sure had problems with C++ standards as well as other things, so I'm putting improved versions of them in this folder ... formUpdate which is prioritized over the original ones.
What's certain is that the new builds will have much smaller and better binaries.

In addition to the S3TC stuff I'll be adding a nice button to unreference unused textures from brushes (not visible after BSP construction) and to shrink the actor list, because we can't have enough editor improvements.

Re: XC_Engine megathread

Posted: Wed Dec 12, 2018 1:05 am
by papercoffee
Higor wrote:because we can't have enough editor improvements.
Absolutely agree on this.

More stability would be also awesome.

Re: XC_Engine megathread

Posted: Wed Dec 12, 2018 6:59 am
by sektor2111
Not big improvements but "Helpers" - perhaps Editor is stable enough for me but poor in options. Definitely Higor can write more stuff... maybe I have to explain what I'm thinking:
- Select - push a button - said brush has 2 NON-COPLANAR surfaces (this is a BAD point in a map causing PAIN);
- Select brush - push button - answer Brush has a NON-SOLID surface caused by a funky build;
- Two selected NavigationPoints - Are they well connected ? Which are ReachSpecs ? Adding (human readable) ReachSpecs at pushing button like that EZConnect.... from XC_EngineActor... and reporting them in readable string format (I think we can use Original DescribeSpecs if not a better one):
1. using ("manually") fly directives for those maps requiring Jets and all that stuff - for MH = FLY MANTA WARLORD and so on;
2. creating some special nodes on button push ? (optional assets/plugins compiled and MyLevel-ed instantly on a button press ?);
- showing that PreviousPath Bug from navigation because THAT ONE is causing stupid Pawn loops and YES, this is an ugly bug very obscure and well hidden from mappers by Polge himself - even XC_PathBuilder has this bug - I saw this a day before yesterday. I was thinking to do a builder scanner but iterations limit specific crash it's part of Editor too (yeah, read again), where I think it should not be like in game - lags in Editor ? WTF, who cares ? Editing it's taking time anyway. And... at this moment I could not do a state running in Editor - because I have 2 minutes patience - for debugging rather than sharing some lousy bugged Level.

Does DevPath is ready for fliers or has only "//fix me" dumb strings ? I suppose EPIC were not loving monsters - considering "lolzing" stuff from UnrealShare.ScriptedPawn and deals with their own bool values bIsPlayer stupidity here - so I'm expecting more bubbles existent in natives which I did not see yet but are sitting there... showing up some day (or some night) and causing a sudden heart attack...
We can forget a bit stability and we might want focus over some DEBUG type buttons, right ?

Revision at DevPath ? I see that nodes over movers are now navigable making Bot to stupidly fall when mover is not in the right position (should trace ground if spot can be a BASE for Walkers/Runners rather than blindly pointing Pawn to fall) - if that it's new stuff perhaps it's not what I'm expecting. Using real CPU power rather than hard-coded cycles ? Eh... I know that more mappers cannot map paths over a mover bridge but... I can, because it's a piece of cake. Currently it's needed a special node, else a default PathNode in plain vanilla UT would be enough...

Re: XC_Engine megathread

Posted: Wed Dec 12, 2018 11:48 am
by Chamberly
I gotta say save the past projects, you'll never know how you can use it later to compare. :idea: :mrgreen:

Re: XC_Engine megathread

Posted: Mon Dec 17, 2018 4:33 am
by Higor
Noticed the autocompressor shouldn't be compressing some files that will never be sent, I'm a bit lazy to expand the list so if there's another set of files UT absolutely requires in order to launch, let me know.
UWindow stuff is never sent anyways so no need to include here.

Code: Select all

static UBOOL IsDefaultPackage( const TCHAR* Pkg)
	//Get rid of paths...
	const TCHAR* Filename;
	for ( Filename=Pkg ; *Pkg ; Pkg++ )
		if ( *Pkg == '\\' || *Pkg == '/' )
			Filename = Pkg + 1;
	//Save as ANSI text
	static const TCHAR* DefaultList[] = 
		{	TEXT("Botpack.u")
		,	TEXT("Engine.u")
		,	TEXT("Core.u")
		,	TEXT("Unreali.u")
		,	TEXT("UnrealShare.u")
		,	TEXT("Editor.u")
		,	TEXT("Fire.u")
		,	TEXT("Credits.utx")
		,	TEXT("LadderFonts.utx")
		,	TEXT("LadrStatic.utx")
		,	TEXT("LadrArrow.utx")	};

	// Compare
	const int max = ARRAY_COUNT( DefaultList);
	for ( int i=0 ; i<max ; i++ )
		if ( !appStricmp( Filename, DefaultList[i]) )
			return 1;
	return 0;
In the meantime...
Raw mouse input for UT in next build
(Experimental feature)

Re: XC_Engine megathread

Posted: Mon Dec 17, 2018 5:21 pm
by sektor2111
And all those from a fresh UT install (not goty), all stock so to speak...

Re: XC_Engine megathread

Posted: Tue Dec 18, 2018 7:14 am
by Higor
Well you weren't expecting this one didn't you?