Fast UZ compressor, drop-in replacement UCC.exe for compress/decompress unreal .uz files

External tools, software and sites for creating or editing UT99 resources.
Deaod
Novice
Posts: 5
Joined: Fri Jul 10, 2020 9:22 am

Re: Fast UZ compressor, drop-in replacement UCC.exe for compress/decompress unreal .uz files

Post by Deaod » Sat Sep 05, 2020 9:47 pm

Okay, ill do this when i have a few spare days.

Btw, the fact that you think im proposing a new sorting algorithm says a lot about how deep you dug into what i was proposing, and how well you understand what youre working on. At least have the courtesy to not assume youre talking to idiots.

User avatar
sektor2111
Godlike
Posts: 4709
Joined: Sun May 09, 2010 6:15 pm
Location: On the roof.

Re: Fast UZ compressor, drop-in replacement UCC.exe for compress/decompress unreal .uz files

Post by sektor2111 » Sat Sep 05, 2020 11:01 pm

Meh... You are a genius.
I don't know how did you compile these, I did not do any tweaks or DLL installs, no need super duper settings, it was compressed under 100 seconds (that CTF-Blice... map).

My two cents about compression vs time
Perhaps compression matters, that's why we do speak about redirects, they have such a purpose: gain speed. I can wait 100 seconds not 80 for compression process if file results smaller in size, it's why we are compressing files. In XC is used LZMA, faster and better compressed - of course, that's another story.
To not forget that UZ file generated for a very small package is crashing UT player at decompression process from UnrealTournament.exe - it's a plain default bug, ask Barbie for more such instances. Definitely a compressed file has to be smaller than source file unless this is not what we call package compression.
You can do an experiment with a very small package. See what happens with client at decompression task - not with UCC.EXE. If you are solving this problem it would be priceless.

Buggie
Skilled
Posts: 232
Joined: Sat Mar 21, 2020 5:32 am

Re: Fast UZ compressor, drop-in replacement UCC.exe for compress/decompress unreal .uz files

Post by Buggie » Sat Sep 05, 2020 11:15 pm

New version 1.4

Better handling for compress params.
Add 'buffer=N' param for compress. Now you can play with buffer size and get better but slower compression if you want.

https://github.com/SeriousBuggie/UZ/releases/latest
What happens if you use different buffer sizes
Show

Code: Select all

uz compress CTF-Blice-Alpha.unr buffer=0x1000
BWT buffer size limited to 4096 (0x1000)
Compressing CTF-Blice-Alpha.unr (87328898 bytes)
stage 0: 4.290000 secs
stage 1: 38.111000 secs
stage 2: 3.042000 secs
stage 3: 4.009000 secs
Total: 49.452000 secs
Compressed CTF-Blice-Alpha.unr -> CTF-Blice-Alpha.unr.uz (23%)

Code: Select all

uz compress CTF-Blice-Alpha.unr buffer=0x2000
BWT buffer size limited to 8192 (0x2000)
Compressing CTF-Blice-Alpha.unr (87328898 bytes)
stage 0: 4.290000 secs
stage 1: 52.542000 secs
stage 2: 2.870000 secs
stage 3: 3.885000 secs
Total: 63.586998 secs
Compressed CTF-Blice-Alpha.unr -> CTF-Blice-Alpha.unr.uz (22%)

Code: Select all

uz compress CTF-Blice-Alpha.unr buffer=0x4000
BWT buffer size limited to 16384 (0x4000)
Compressing CTF-Blice-Alpha.unr (87328898 bytes)
stage 0: 4.586000 secs
stage 1: 123.787003 secs
stage 2: 2.855000 secs
stage 3: 3.853000 secs
Total: 135.080994 secs
Compressed CTF-Blice-Alpha.unr -> CTF-Blice-Alpha.unr.uz (21%)

Code: Select all

uz compress CTF-Blice-Alpha.unr buffer=0x8000
BWT buffer size limited to 32768 (0x8000)
Compressing CTF-Blice-Alpha.unr (87328898 bytes)
stage 0: 4.352000 secs
stage 1: 304.295990 secs
stage 2: 2.746000 secs
stage 3: 3.869000 secs
Total: 315.262970 secs
Compressed CTF-Blice-Alpha.unr -> CTF-Blice-Alpha.unr.uz (21%)
-----

Now you can choose buffer size and if you want wait but get better compression - set appropriate buffer size.

Attach sample of small package which crash UT on unpack. I see what can be done.

User avatar
sektor2111
Godlike
Posts: 4709
Joined: Sun May 09, 2010 6:15 pm
Location: On the roof.

Re: Fast UZ compressor, drop-in replacement UCC.exe for compress/decompress unreal .uz files

Post by sektor2111 » Sun Sep 06, 2020 12:11 am

Barbie did small files for server, but... I know a file called relicsbindings.u or such which I used as a test. You can do such a package having a new monster, SkaarjTrooper with weapon sniperrifle and that's all, no code just other properties. Compress package in plain UZ compression then... add it in ServerPackages, and go figure what client does when package is smaller than UZ archive - UT436.

Of course, to me that's not a "HowTo" make monsters, perhaps changing properties on fly or adding more monsters in a bigger package would be ideal case and not 10 small pieces because of 10 variables changed, lol.

Thanks for offering compression flexibility :D :highfive:

User avatar
Barbie
Godlike
Posts: 1929
Joined: Fri Sep 25, 2015 9:01 pm
Location: moved without proper hashing

Re: Fast UZ compressor, drop-in replacement UCC.exe for compress/decompress unreal .uz files

Post by Barbie » Sun Sep 06, 2020 12:35 am

Buggie wrote:
Sat Sep 05, 2020 11:15 pm
Attach sample of small package which crash UT on unpack.
See thread Critical Error on client while redirected download.
"Multiple exclamation marks," he went on, shaking his head, "are a sure sign of a diseased mind." --Terry Pratchett

Buggie
Skilled
Posts: 232
Joined: Sat Mar 21, 2020 5:32 am

Re: Fast UZ compressor, drop-in replacement UCC.exe for compress/decompress unreal .uz files

Post by Buggie » Sun Sep 06, 2020 1:03 am

I try reproduce crash with relicsbindings.u.
v436 - crashed.
v469 - not.
So look like in v469 this crash already fixed.
---
Also if this happens only if compression rate >= 100% then do not store such files at all.
Parse output of UCC or UZ and if compression rate bigger 99% - delete .uz file.

Also look like this happens only for small files. So what point of .uz file at all if file small? It can served by server directly.
----
I am not look deep on source of issue, but look like something going wrong when compressed size is bigger then source.
Related code:

Code: Select all

	UBOOL Decode( FArchive& In, FArchive& Out )
	{
		guard(FCodecHuffman::Decode);
		INT Total;
		In << Total;
		TArray<BYTE> InArray( In.TotalSize()-In.Tell() );
		In.Serialize( &InArray(0), InArray.Num() );
		FBitReader Reader( &InArray(0), InArray.Num()*8 );
		FHuffman Root(-1);
		Root.ReadTable( Reader );
		while( Total-- > 0 )
		{
			check(!Reader.AtEnd());
			FHuffman* Node = &Root;
			while( Node->Ch==-1 )
				Node = Node->Child( Reader.ReadBit() );
			BYTE B = Node->Ch;
			Out << B;
		}
		return 1;
		unguard;
	}
Fail happens on line
check(!Reader.AtEnd());
So Reader currently at end. So read from it impossible.

Also this bug happens only on UT.exe, not in UCC.exe which lead us to (possible) different implementation or different conditions.

Because this is hardly reproduce in controlled environment, like debugged UZ, I can not say much about what happens actually here.

IDK what going wrong when compression above 100% and why UCC work fine, but UT crash. I do not see any effort for deep dig this issue, because above solution is simple and effective.
------
If you want automatize process: store all log of compress and use grep and xargs for delete files which compressed badly. Like this:

Code: Select all

uz compress relicsbindings.u | grep "([1-9][0-9][0-9][0-9]*%)" | grep "[^ ]*.uz" -o
relicsbindings.u.uz
Just not print, but redirect to xargs or similar tool for delete file.

User avatar
Barbie
Godlike
Posts: 1929
Joined: Fri Sep 25, 2015 9:01 pm
Location: moved without proper hashing

Re: Fast UZ compressor, drop-in replacement UCC.exe for compress/decompress unreal .uz files

Post by Barbie » Sun Sep 06, 2020 2:39 am

Buggie wrote:
Sun Sep 06, 2020 1:03 am
Also if this happens only if compression rate >= 100% then do not store such files at all.
I configured my server so that it never will send any uncompressed files - all files are send via the redirect server. I cannot remember why I did so but there must have been a reason. ;)

Buggie wrote:
Sun Sep 06, 2020 1:03 am
Parse output of UCC or UZ and if compression rate bigger 99%
I have a shell script that creates compressed files if they don't exist. Here I issue a warning if filesize(uncompressed) <= filesize(compressed):

Code: Select all

for f in ../$DIR; do
	...
	FZCOMPRESSED=$(wc -c < "$f.uz")
	FZUNCOMPRESSED=$(wc -c < "$f")
	test $FZCOMPRESSED -ge $FZUNCOMPRESSED && echo "WARNING: compression ratio is above 100% for file \"$f\". Clients cannot process this!"
"Multiple exclamation marks," he went on, shaking his head, "are a sure sign of a diseased mind." --Terry Pratchett

User avatar
Que
Skilled
Posts: 173
Joined: Mon Dec 09, 2019 5:49 am
Personal rank: ...
Contact:

Re: Fast UZ compressor, drop-in replacement UCC.exe for compress/decompress unreal .uz files

Post by Que » Sun Sep 06, 2020 5:27 am

so if it manages to Compress CTF-Blice, is UCC capable of decompressing it Clientside?
>>> PSYCHOSERVERS <<< Address: 103.125.218.185 Port: 7760

Buggie
Skilled
Posts: 232
Joined: Sat Mar 21, 2020 5:32 am

Re: Fast UZ compressor, drop-in replacement UCC.exe for compress/decompress unreal .uz files

Post by Buggie » Sun Sep 06, 2020 5:39 am

Yep. It is UCC capable. I show this above.
viewtopic.php?p=122222#p122222
----
As I say:
Buggie wrote:
Sun Sep 06, 2020 1:03 am
Also this bug happens only on UT.exe, not in UCC.exe which lead us to (possible) different implementation or different conditions.
UCC decode with IpDrv.dll.
UT decode with Engine.dll.
Code stored in header, so every module get own copy of it.
Setup.exe for example use own copy too.

Return to previously posted code.
In.TotalSize() must return size of .uz (0x2a2) but return size of original file (0x246).
So it is cause of crash.
----
Image
I look at Cache*.tmp files and found here part of .uz file truncated to size of .u.
Now I get full picture: UT contains error - on download .uz file use size of .u file. :facepalm:
So until .u file bigger it is not a problem.
But when not, downloaded .uz file truncated and can not be decompressed.

There nothing can be "fixed" at compress stage.
Some things limited by format.
Some files can not be compressed at all. Fortunately, usually, all unreal files can be compressed less or more.
Unfortunately, if file really small, produced win of bytes not enough for cover system needs for store header and other stuff of .uz file. So result is bigger from original.
You can not rewrite exists UT clients, so do not use such .uz files.
----
Barbie wrote:
Sun Sep 06, 2020 2:39 am
all files are send via the redirect server. I cannot remember why I did so but there must have been a reason.
You can add some rules for your redirect server. For example if requested file size less from 8 KB return 404 error. And client go download package directly from server.

User avatar
sektor2111
Godlike
Posts: 4709
Joined: Sun May 09, 2010 6:15 pm
Location: On the roof.

Re: Fast UZ compressor, drop-in replacement UCC.exe for compress/decompress unreal .uz files

Post by sektor2111 » Sun Sep 06, 2020 9:09 am

Let me to setup some memory refresh.

If a small file does damage you won't have it in redirect. Here is the Plain Problem OVER KNOWN by all old players and ignored by "admins". If client starts downloading a single file directly from server, next files are downloaded from SERVER TOO unless client knows what to do, HIT F10 and then reconnect or else every single next downloaded file will happen at a doggy slow speed directly from server (except XC or 469 well configured servers of course :pfff: ).

And then ? And then if redirect is not using any compression it takes more space for hosting files but... you won't have any damage for clients... except screwed packages like MapVoteLA13 and others from a random MH server... (geniuses fu..ed them properly).

Next ? Yes, I think those very small files... are just stupid - wrecks with no purpose to be like that. Seriously they can be thrown in bigger packages and called from there. I already made new troopers without any new package, right from map's MyLevel using custom actors, even without using files like "Pupae.u" which can be cloned by a custom Factory as long as monster has NOTHING new in combat intelligence. So we don't need any new package for a new variable...
Que wrote:
Sun Sep 06, 2020 5:27 am
so if it manages to Compress CTF-Blice, is UCC capable of decompressing it Clientside?
Why would have a CTF map destroying a FLAG - game objective ? Here is needed a FIX not a compression.
And those over 6000+ reachSpecs can go to around 1900 - inside engine boundaries... As btw map can be rebuild... UGold227h available for everyone can do this. In UT map is OVER ENGINE crashing at Max_Points boundary during build. More than 10000 actors can be cleaned by MapGarbage losing another 0.4 MB junk bytes from it, red builder left can be also reduced to a point, etc.

User avatar
Feralidragon
Godlike
Posts: 5218
Joined: Wed Feb 27, 2008 6:24 pm
Personal rank: Work In Progress
Location: Liandri

Re: Fast UZ compressor, drop-in replacement UCC.exe for compress/decompress unreal .uz files

Post by Feralidragon » Sun Sep 06, 2020 2:31 pm

sektor2111 wrote:
Sun Sep 06, 2020 9:09 am
Que wrote:
Sun Sep 06, 2020 5:27 am
so if it manages to Compress CTF-Blice, is UCC capable of decompressing it Clientside?
Why would have a CTF map destroying a FLAG - game objective ? Here is needed a FIX not a compression.
And those over 6000+ reachSpecs can go to around 1900 - inside engine boundaries... As btw map can be rebuild... UGold227h available for everyone can do this. In UT map is OVER ENGINE crashing at Max_Points boundary during build. More than 10000 actors can be cleaned by MapGarbage losing another 0.4 MB junk bytes from it, red builder left can be also reduced to a point, etc.
Trying to not steer this topic off rails, I will just reply with this: again, Blice is mostly meant as a benchmark map, so keep this in mind when considering and highlighting any "issues" concerning this specific map.
Every single issue the map has is actually an issue that must be fixed (and most of them already were) in the engine itself.

And yes, this in fact includes the CloudZone issue, which is an issue with the game, not the map, because a CloudZone should never destroy things like flags.
As a matter of fact, a while ago I was about to actually submit a CloudZone fix for the 469 patch, but I didn't have time nor much motivation to do it yet, and I would like it to be more flexible and not specific to the flag only, but for that I need to consider things like network compatibility and such, which is why I never released a fixed version, yet.

Solving the engine issues that plague this map means that the map gets fixed, and that any map actually meant to be played will be far more stable and faster as a result and will be a lot less painful to build with less problems.

So, the fact that UCC was unable to compress this map is in fact an engine problem, which Buggie just fixed with his own UZ version.
UCC should never fail to compress any package whatsoever, regardless of its contents, period.

The same is true for things like BSP building, lighting building, path building, etc, no matter how crazy it gets, it should simply work, and if it doesn't, then that's a problem or limitation of the engine, and that's the whole point of this map and keeping as it is.

User avatar
sektor2111
Godlike
Posts: 4709
Joined: Sun May 09, 2010 6:15 pm
Location: On the roof.

Re: Fast UZ compressor, drop-in replacement UCC.exe for compress/decompress unreal .uz files

Post by sektor2111 » Sun Sep 06, 2020 5:54 pm

Understandable, in big parts except hard-coded limitations... As for UZ.EXE by Buggie I think I'll stand with buffer=32768. Even if compression takes time, we can have smaller files. My MH-Sk_Godz map went with 4 MB smaller. When player has whatever data limitations (mobile connections) this is helping with "data plans" and saving servers from stress, longer packages files being downloaded this way faster due to newer compression Level. Next time when I'll setup a server from point zero I'll use this compression doesn't matter how long does it takes for making redirect. I had years ago some deductions about this UZ compression which I was discussing with Kelly, in that time nobody was dropping any eye at these things... I would like to have some C++ coding skills.

Buggie
Skilled
Posts: 232
Joined: Sat Mar 21, 2020 5:32 am

Re: Fast UZ compressor, drop-in replacement UCC.exe for compress/decompress unreal .uz files

Post by Buggie » Sun Sep 06, 2020 6:21 pm

Default one (and biggest possible), used by UCC is 0x40000 which 262144. So your compression is little worst against default UCC.

User avatar
Feralidragon
Godlike
Posts: 5218
Joined: Wed Feb 27, 2008 6:24 pm
Personal rank: Work In Progress
Location: Liandri

Re: Fast UZ compressor, drop-in replacement UCC.exe for compress/decompress unreal .uz files

Post by Feralidragon » Sun Sep 06, 2020 7:14 pm

sektor2111 wrote:
Sun Sep 06, 2020 5:54 pm
When player has whatever data limitations (mobile connections) this is helping with "data plans" and saving servers from stress, longer packages files being downloaded this way faster due to newer compression Level.
Be careful with this line of reasoning, we're no longer in 1999, this is the year 2020.
That is to say that while data plans and such still do exist (unfortunately), you're not helping a whole lot by shaving off just a 1 or 2MB off a package (or even 4MB for that matter, meaning the package as a whole is a lot lot bigger in comparison).

Even with such data plans, the player may have to download several MBs worth of content to begin with, and just the act of playing the game online will easily reach the MBs of data transferred over time, especially in modded servers with a lot of stuff going on and being replicated to the player.
Even the act of opening this forum and viewing a random thread may hit you with MBs worth of downloads automatically performed by the browser, especially in images and such.

That is only to say that you may be forcing yourself to go through a lot of work and pain for virtually no difference for the player, just my 2 cents.

User avatar
sektor2111
Godlike
Posts: 4709
Joined: Sun May 09, 2010 6:15 pm
Location: On the roof.

Re: Fast UZ compressor, drop-in replacement UCC.exe for compress/decompress unreal .uz files

Post by sektor2111 » Sun Sep 06, 2020 10:14 pm

Usually when I want to visit some new server, I'm using my client loaded with XC_Engine. There is reported how much content I have to download. Me one... I'm not interested to download more than 100 MB, I'm not kidding here. I'm there for playing not to watch downloading screens. If redirect is very far, I cannot bet on the best ever download speed. These issues for me are based on patience and data load on drives - last time I was in troubles due to the hard-drive charge vs exFat access - I solved problem but... I'm not hunting files for the moment... With regard to Internet chapter I can pick one of two ISP services without any traffic restriction. Today I truly tested this new UZ compression... Indeed... it's flawless with less data downloaded and then it leads to a faster join into party.

Beer and happiness because of Buggie man.
Edit: Buffer Tests
Buggie wrote:
Sun Sep 06, 2020 6:21 pm
Default one (and biggest possible), used by UCC is 0x40000 which 262144. So your compression is little worst against default UCC.
Ahem... then I want this "worst" as long as 32k buffer has generated a smaller file than plain ucc.exe and then... 64k buffer went with a smaller UZ file than 32k.
In next moments (I was about to sleep on the keyboard) I tested 128k. Surprise... compression went bigger than for 64k... oops. Too much is already... too much. And then... for 256k I got exactly the same compression level like ucc.exe, files were the same until last byte. I stick with 64k.
Last edited by sektor2111 on Sun Sep 06, 2020 10:34 pm, edited 4 times in total.

Post Reply