Page 1 of 1

UScript debug helper library

Posted: Thu Oct 10, 2019 9:31 pm
by Chris
I created this debug helper mainly to make it easier for myself to find script errors (accessed nones and such).
When the binary is hooked, it'll display the name of the variable/function that was accessed through a none, as well as the entire stack leading to the null pointer (casts, variables...).
I thought it might be of use to others, so I'm posting it here, as well as the full C++ source code for it.

Here is an example of how it might look:

Code: Select all

ScriptWarning: TeamSpawnpoint MH-LostSouls.TeamSpawnpoint9 (Function TeamMonster.TeamSpawnpoint.Create:0032) Accessed None 'prototype' Next Op '1' Stack: Try Fetching context >>  , Instance variable: Factory Null 
ScriptWarning: SkaarjWarrior MH-LostSouls.SkaarjWarrior2 (Function UnrealShare.Skaarj.SpinDamageTarget:002D) Accessed None 'Location' Next Op '1' Stack: Try Fetching context >>  , Instance variable: Target Null 
Where the name in the first quoute is the name of the variable or function (including native functions).
The quote after Next Op is the next opcode index to be executed.
The context fetch is the member access operator "." or dot. After that, you'll get a list of all the variable reference / casting opcodes, property name, and whether any of them are null so you know where the issue lies.
An instance variable is an object defined variable, i.e defined in a class scope.
A Local variable is a local function variable, or a parameter.
A default variable is a class default variable (accessed using .default.)

The library is fairly simple. It replaces some of the variable reference opcodes, cast opcodes, context opcodes, assign opcode and so on in the dll.
The UScript class is empty, it's only there as a dummy reference to make the engine load it and bind the dll.
Due to the extra bookkeeping and lookups, they will take slightly longer to execute.

In order to use it, just load it as a mutator or spawn the DebugDummy.

Re: UScript debug helper library

Posted: Thu Oct 10, 2019 11:40 pm
by PrinceOfFunky
Awesome! Any way to get the line number and a opcode table to know what those numbers reference to?

Re: UScript debug helper library

Posted: Fri Oct 11, 2019 3:32 pm
by Barbie
Because the source code is compiled into byte code and the byte code can be executed although the source code could have been deleted it indicates that there is no connection between source and byte code. But let's see what Chris replies.

Re: UScript debug helper library

Posted: Fri Oct 11, 2019 5:10 pm
by sektor2111
I downloaded this and I did not asked to myself such things. Usually if I check my mod(s) and I want to find where the trouble is, definitely I won't remove source-code if mod is not ready... Right ?
I know those "Target" things and "Prototype" problem but... other newer toys which I'm doing might need some debugging...
Greetings Chris !

Re: UScript debug helper library

Posted: Fri Oct 11, 2019 10:16 pm
by Chris
The execution of the compiled code do not rely on the source code at all. They simply made it so that the serializer dumps the source code into the package along with the compiled bytecode. UScript uses UProperty meta objects to briefly describe member types, names and their offsets. This is what allows functions such as GetPropertyText and SetPropertyText to function using the name of a member. The compiler reserves 4 bytes in the byte code right after the 1 byte opcode number for one of the following member types ;
- UBoolProperty
- UIntProperty
- UStrProperty
- UNameProperty
- UFloatProperty
- UByteProperty
- UFixedArrayProperty
- UClassProperty
- UArrayProperty (Dynamic array)
- UMapProperty (Mapped array)
- UStructProperty
- UObjectProperty

The dynamic UScript linker then iterates over these reserved slots and fills them in with the address to the meta objects in runtime, similar to how WIndows and Linux loads dlls and so files. This gives the uscript instruction direct access to the meta object without having to look for it, in order to get the offset for the desired member variable in the given context (local, instance, default). It would look something like this:

Code: Select all

	GProperty = (UProperty*)Stack.ReadObject();
	GPropAddr = Stack.Locals + GProperty->Offset;
GPropAddr would now store the address to the variable. It's messy and slow, but this is how they made it.
Functions are even slower. Any virtual function is being searched for (requires iterations) when called. Only final functions are treated the same way as the above (address stored in the byte code itself).
Virtual functions are poorly hashed using the first 8 bits of the name hash and placed in hash bins that are searched everytime you call a virtual UScript function.

Therefore, it's possible to determine the name of the variables, functions and classes, but not what line the code corresponds to. That would take up a lot of extra, unnecessary space in the byte code, and would be impossible to do without rewriting the compiler.
It would be interesting to rewrite the compiler to generate an external debug database, similar to what native debuggers use to break at certain lines. That would then make it possible to determine the line the code corresponds to.

The opcode printed in the log isn't particularly useful unless you look at, and understand the native part of the Core. The opcode will correspond to an index into the GNatives table.
Epic initially made opcodes occupy 1 byte, this means a maximum of 256 opcodes could be used. This created severe limitations since it would limit the number of native functions + intrinsics to 256.
Therefore they extended it to 2 bytes, a so called High Native. The engine first reads the 1 byte (Stack.Step) like it's in an 8-bit mode, jumps to that index in the GNatives table, and if the index is >= 0x60, that index will point to a native function in the table that reads a second byte, adds them together (sort of like a dispatch that turns the engine into 16-bit mode).
It would then jump to the target native function using the new 16 bit opcode. Yeah the engine is pretty messy and hackish. So for native functions (declared in UScript as native), the printed opcode would point to the opcode number of that function. For opcodes < 0x60, it would correspond to either an internal script instruction, or the few low native functions declared in Object.uc. I put it in there to help me debug the debug library :loool:

The reason it isn't possible to simply open a package in a text editor, remove the source code and then save it again is because the package format contain headers that store offsets and sizes. If you simply erase the source code, without having all of those offsets and sizes changed accordingly, the package becomes corrupt and useless. In order to remove source code, the engine has to load the serialized objects, empty the text buffers, then save the objects again using recalculated offsets and sizes.

Re: UScript debug helper library

Posted: Sat Oct 19, 2019 3:54 pm
by PrinceOfFunky
Can you make it work with replication?
When a player joins a server with UDebug, it won't print in the client logs.

Re: UScript debug helper library

Posted: Sun Oct 20, 2019 2:52 am
by Gustavo6046
Is it maybe possible to make a parser that maps predicted bytecode positions to line numbers when reading a UScript file, and saves those, so they can be used by UDebug to look the line numbers up later?

Re: UScript debug helper library

Posted: Mon Jan 06, 2020 5:30 pm
by PrinceOfFunky
removed_cause_Debug_dll.png
removed_cause_Debug_dll.png (20.96 KiB) Viewed 1454 times
I was playing a single player match with Debug.dll hooked before joining this server, was that the cause?

Re: UScript debug helper library

Posted: Mon May 25, 2020 7:18 pm
by Chris
Gustavo6046 wrote: Sun Oct 20, 2019 2:52 am Is it maybe possible to make a parser that maps predicted bytecode positions to line numbers when reading a UScript file, and saves those, so they can be used by UDebug to look the line numbers up later?
Pardon my late reply.
This would be possible if the compiler would be rewritten to output an additional debug database of some sort, just like MSVC does it. That database file could then be used during runtime debugging.
PrinceOfFunky wrote: Mon Jan 06, 2020 5:30 pm
removed_cause_Debug_dll.png
I was playing a single player match with Debug.dll hooked before joining this server, was that the cause?
You need to tell ACE to whitelist the mod. Is this your own server?

Re: UScript debug helper library

Posted: Tue May 26, 2020 5:42 pm
by Gustavo6046
Database? Just assign each symbol (and instruction) its associated line number (and perhaps column number) in compile time. It is a matter of adding two members to each struct, and knowing what line position you are in during the tokenization and parsing steps. Microsoft stuff tends to boil down to Rube Goldberg, but either way, you get the point.

Re: UScript debug helper library

Posted: Sun Sep 05, 2021 5:12 pm
by a9902957@nepwk.com
Thanks! This might come in handy. Already did. Thanks for this indeed! :gj: