Fast UZ compressor, drop-in replacement UCC.exe for compress/decompress unreal .uz files

[Buggie]

If a small file does damage you won't have it in redirect. Here is the Plain Problem OVER KNOWN by all old players and ignored by "admins". If client starts downloading a single file directly from server, next files are downloaded from SERVER TOO unless client knows what to do, HIT F10 and then reconnect or else every single next downloaded file will happen at a doggy slow speed directly from server (except XC or 469 well configured servers of course ).

Smallest files can be last in ServerPackages list. Look like download happens in this order and this not break anything if last files served directly.

Ahem... then I want this "worst" as long as 32k buffer has generated a smaller file than plain ucc.exe and then... 64k buffer went with a smaller UZ file than 32k.
In next moments (I was about to sleep on the keyboard) I tested 128k. Surprise... compression went bigger than for 64k... oops. Too much is already... too much. And then... for 256k I got exactly the same compression level like ucc.exe, files were the same until last byte. I stick with 64k.

Depends from input data. Not a stable across all files.

[sektor2111]

Look like download happens in this order and this not break anything if last files served directly.

Agree here, good point.
In other hand... I won't use tiny files in any server which I do, unless is about ServerActors without a client-download purpose. Even these "Actors" after all are having a good uz compression and are not causing issues.

The bad-ass solution for a small package...
Copying two random pages of WindowsXP help content as an innocent comment embedded in package. Will generate a longer package-file with lousy spam data preventing UZ file to be bigger than package...

[Buggie]

Currently I check v469 download and can confirmed: if previous file download failed via HTTPDownload all other files can be downloaded from it.
But in v436 - no. If HTTPDownload failed download one file, all next will not use HTTPDownload anymore.

"WindowsXP help content" is not a good candidate for this. You want good compression, so just make comment with repeatable same symbol.
I think 1024 times will be enough.
RLE eat this highly repeatable block easily. And you win 1024 bytes, which obviously enough for make .uz smaller from .u.

-----

New version 1.5:

Add allowbad parameter.
Improve output.

https://github.com/SeriousBuggie/UZ/releases/latest

Code: Select all

        allowbad        Allows the creation of .uz files that cause crash on clients below version 469.

Code: Select all

uz compress relicsbindings.u
Compressing relicsbindings.u (582 bytes)
stage 0: 0.000000 secs
stage 1: 0.000000 secs
stage 2: 0.000000 secs
stage 3: 0.000000 secs
Total: 0.000000 secs
Skipped: This .uz file cause crash on clients below 469. Use the 'allowbad' parameter to create this .uz file anyway.
Such files should be the last in the ServerPackages list to avoid speed drops for all subsequent files on older clients.

-----

Also if you want better compression you can consider possibility usage 'newformat' option. IDK from which UT version it supported, but v436 unpack this UZ fine:

Code: Select all

uz compress CTF-Blice-Alpha.unr buffer=0x4000
The BWT buffer size is limited to 16384 (0x4000) bytes
Compressing CTF-Blice-Alpha.unr (87328898 bytes)
stage 0: 4.290000 secs
stage 1: 114.052002 secs
stage 2: 2.870000 secs
stage 3: 3.822000 secs
Total: 125.034004 secs
Compressed CTF-Blice-Alpha.unr -> CTF-Blice-Alpha.unr.uz (21.135%)

Code: Select all

uz compress CTF-Blice-Alpha.unr newformat buffer=0x4000
The BWT buffer size is limited to 16384 (0x4000) bytes
Compressing CTF-Blice-Alpha.unr (87328898 bytes)
stage 0: 4.275000 secs
stage 1: 112.290001 secs
stage 2: 2.792000 secs
stage 3: 1.124000 secs
stage 4: 2.465000 secs
Total: 122.945999 secs
Compressed CTF-Blice-Alpha.unr -> CTF-Blice-Alpha.unr.uz (17.041%)

Code: Select all

ut\system\ucc decompress CTF-Blice-Alpha.unr.uz
stage 0: 1.810315 secs
stage 1: 1.387295 secs
stage 2: 2.902404 secs
stage 3: 1.134453 secs
stage 4: 3.841292 secs
Total: 11.075759 secs
Decompressed CTF-Blice-Alpha.unr.uz -> CTF-Blice-Alpha.unr

[Buggie]

New version v2.0

- Significantly speed up compress. Up to 100-700 times faster in compare with UCC v436.
- Full buffer size by default.

https://github.com/SeriousBuggie/UZ/releases/latest

Compared:

Code: Select all

ucc compress ..\Maps\MH-EscapeFromNaZaliBeta2.unr
Compressing ..\Maps\MH-EscapeFromNaZaliBeta2.unr (72873863 bytes)
stage 0: 2.386206 secs
stage 1: 8357.626953 secs
stage 2: 0.823047 secs
stage 3: 1.207998 secs
Total: 8362.044204 secs
Compressed ..\Maps\MH-EscapeFromNaZaliBeta2.unr -> ..\Maps\MH-EscapeFromNaZaliBeta2.unr.uz (6%)

and

Code: Select all

uz compress MH-EscapeFromNaZaliBeta2.unr
Compressing MH-EscapeFromNaZaliBeta2.unr (72873863 bytes)
stage 0: 3.229000 secs
stage 1: 6.880000 secs
stage 2: 0.749000 secs
stage 3: 1.076000 secs
Total: 11.934000 secs
Compressed MH-EscapeFromNaZaliBeta2.unr -> MH-EscapeFromNaZaliBeta2.unr.uz (6%)

in 700 times faster.
Both file are same. Compressed at full buffer size (0x40000).
And famous CTF-Blice-Alpha.unr:

Code: Select all

uz compress CTF-Blice-Alpha.unr buffer=0x40000
The BWT buffer size is limited to 262144 (0x40000) bytes
Compressing CTF-Blice-Alpha.unr (87328898 bytes)
stage 0: 4.336000 secs
stage 1: 18.954000 secs
stage 2: 2.698000 secs
stage 3: 3.713000 secs
Total: 29.701000 secs
Compressed CTF-Blice-Alpha.unr -> CTF-Blice-Alpha.unr.uz (20.459%)

Code: Select all

uz compress CTF-Blice-Alpha.unr buffer=0x40000 newformat
The BWT buffer size is limited to 262144 (0x40000) bytes
Compressing CTF-Blice-Alpha.unr (87328898 bytes)
stage 0: 4.290000 secs
stage 1: 18.938999 secs
stage 2: 2.714000 secs
stage 3: 1.077000 secs
stage 4: 2.106000 secs
Total: 29.125999 secs
Compressed CTF-Blice-Alpha.unr -> CTF-Blice-Alpha.unr.uz (14.904%)

[sektor2111]

Yep... the two maps which I tested have results as follows:
- good speed and good compression- using plain compression commands;
- better compression and speed at 64k buffer - no jokes.

MH-Sk_Godz has 70.186.745 bytes
Plain compress command has generated a file having 36.857.754 bytes.
Compressing with 64k buffer has generated a file having 36.458.052 bytes.

I've noticed similar results with BotPack.u file. File was compressed faster and better with a 64k buffer.

PS: I think is time to conform file relicsbindings.u with another one which is not crushing non-goty clients...

[Buggie]

Code: Select all

uz compress CTF-Blice-Alpha.unr buffer=0x40000
The BWT buffer size is limited to 262144 (0x40000) bytes
Compressing CTF-Blice-Alpha.unr (87328898 bytes)
stage 0: 4.524000 secs
stage 1: 20.046000 secs
stage 2: 2.901000 secs
stage 3: 4.149000 secs
Total: 31.620001 secs
Compressed CTF-Blice-Alpha.unr -> CTF-Blice-Alpha.unr.uz (20.459%)

Code: Select all

uz compress CTF-Blice-Alpha.unr buffer=0x20000
The BWT buffer size is limited to 131072 (0x20000) bytes
Compressing CTF-Blice-Alpha.unr (87328898 bytes)
stage 0: 4.492000 secs
stage 1: 17.363001 secs
stage 2: 2.792000 secs
stage 3: 3.978000 secs
Total: 28.625002 secs
Compressed CTF-Blice-Alpha.unr -> CTF-Blice-Alpha.unr.uz (20.329%)

Code: Select all

uz compress CTF-Blice-Alpha.unr buffer=0x10000
The BWT buffer size is limited to 65536 (0x10000) bytes
Compressing CTF-Blice-Alpha.unr (87328898 bytes)
stage 0: 4.555000 secs
stage 1: 15.257000 secs
stage 2: 2.870000 secs
stage 3: 4.461000 secs
Total: 27.143000 secs
Compressed CTF-Blice-Alpha.unr -> CTF-Blice-Alpha.unr.uz (20.343%)

Code: Select all

uz compress AS-HiSpeed.unr buffer=0x40000
The BWT buffer size is limited to 262144 (0x40000) bytes
Compressing AS-HiSpeed.unr (4418189 bytes)
stage 0: 0.234000 secs
stage 1: 0.998000 secs
stage 2: 0.203000 secs
stage 3: 0.281000 secs
Total: 1.716000 secs
Compressed AS-HiSpeed.unr -> AS-HiSpeed.unr.uz (28.849%)

Code: Select all

uz compress AS-HiSpeed.unr buffer=0x20000
The BWT buffer size is limited to 131072 (0x20000) bytes
Compressing AS-HiSpeed.unr (4418189 bytes)
stage 0: 0.234000 secs
stage 1: 0.904000 secs
stage 2: 0.219000 secs
stage 3: 0.281000 secs
Total: 1.638000 secs
Compressed AS-HiSpeed.unr -> AS-HiSpeed.unr.uz (29.021%)

Code: Select all

uz compress AS-HiSpeed.unr buffer=0x10000
The BWT buffer size is limited to 65536 (0x10000) bytes
Compressing AS-HiSpeed.unr (4418189 bytes)
stage 0: 0.218000 secs
stage 1: 0.811000 secs
stage 2: 0.188000 secs
stage 3: 0.281000 secs
Total: 1.498000 secs
Compressed AS-HiSpeed.unr -> AS-HiSpeed.unr.uz (28.983%)

So best buffer size depends from input data.

Conform packages do not like anticheats. So be careful with that.

[Buggie]

New version v2.1:

- Faster compress and decompress.

https://github.com/SeriousBuggie/UZ/releases/latest

Comparison:
v2.0:

Code: Select all

uz compress E1M4.umx
Compressing E1M4.umx (98087852 bytes)
stage 0: 5.211000 secs
stage 1: 15.693000 secs
stage 2: 27.176001 secs
stage 3: 15.023000 secs
Total: 63.103001 secs
Compressed E1M4.umx -> E1M4.umx.uz (95%)

Code: Select all

uz decompress E1M4.umx.uz
stage 0: 8.190000 secs
stage 1: 13.275000 secs
stage 2: 2.886000 secs
stage 3: 4.134000 secs
Total: 28.485001 secs
Decompressed E1M4.umx.uz -> E1M4.umx

v2.1:

Code: Select all

uz compress E1M4.umx
Compressing E1M4.umx (98087852 bytes)
stage 0: 1.591000 secs
stage 1: 15.600000 secs
stage 2: 13.338000 secs
stage 3: 7.722000 secs
Total: 38.250999 secs
Compressed E1M4.umx -> E1M4.umx.uz (95%)

Code: Select all

uz decompress E1M4.umx.uz
stage 0: 4.711000 secs
stage 1: 7.987000 secs
stage 2: 3.011000 secs
stage 3: 3.104000 secs
Total: 18.813000 secs
Decompressed E1M4.umx.uz -> E1M4.umx

[Buggie]

New version v2.2:

- Faster decompress.

https://github.com/SeriousBuggie/UZ/releases/latest

Comparison:
v2.1:

Code: Select all

uz decompress E1M4.umx.uz
stage 0: 4.711000 secs
stage 1: 7.987000 secs
stage 2: 3.011000 secs
stage 3: 3.104000 secs
Total: 18.813000 secs
Decompressed E1M4.umx.uz -> E1M4.umx

v2.2:

Code: Select all

uz decompress E1M4.umx.uz
stage 0: 3.417000 secs
stage 1: 7.691000 secs
stage 2: 2.403000 secs
stage 3: 0.421000 secs
Total: 13.931999 secs
Decompressed E1M4.umx.uz -> E1M4.umx

UCC v436:

Code: Select all

ucc decompress E1M4.umx.uz
stage 0: 9.780995 secs
stage 1: 14.849395 secs
stage 2: 3.004905 secs
stage 3: 4.983224 secs
Total: 32.618519 secs
Decompressed E1M4.umx.uz -> E1M4.umx

[Buggie]

New version v3.0:

- Add multi-threading.
- Faster compress/decompress.

https://github.com/SeriousBuggie/UZ/releases/latest

Code: Select all

uz compress CTF-Blice-Alpha.unr
Used 8 threads.
Compressing CTF-Blice-Alpha.unr (87328898 bytes)
stage 0: 0.577000 secs
stage 1: 5.320000 secs
stage 2: 2.075000 secs
stage 3: 1.279000 secs
Total: 9.251000 secs
Compressed CTF-Blice-Alpha.unr -> CTF-Blice-Alpha.unr.uz (20%)

Code: Select all

uz compress E1M4.umx
Used 8 threads.
Compressing E1M4.umx (98087852 bytes)
stage 0: 0.702000 secs
stage 1: 3.666000 secs
stage 2: 13.244000 secs
stage 3: 7.629000 secs
Total: 25.240999 secs
Compressed E1M4.umx -> E1M4.umx.uz (95%)

Code: Select all

uz compress AS-HiSpeed.unr
Used 8 threads.
Compressing AS-HiSpeed.unr (4418189 bytes)
stage 0: 0.031000 secs
stage 1: 0.328000 secs
stage 2: 0.140000 secs
stage 3: 0.093000 secs
Total: 0.592000 secs
Compressed AS-HiSpeed.unr -> AS-HiSpeed.unr.uz (29%)

Code: Select all

uz decompress CTF-Blice-Alpha.unr.uz
Used 8 threads.
stage 0: 0.796000 secs
stage 1: 1.279000 secs
stage 2: 0.624000 secs
stage 3: 0.374000 secs
Total: 3.073000 secs
Decompressed CTF-Blice-Alpha.unr.uz -> CTF-Blice-Alpha.unr

Code: Select all

uz decompress E1M4.umx.uz
Used 8 threads.
stage 0: 3.417000 secs
stage 1: 7.769000 secs
stage 2: 1.342000 secs
stage 3: 0.422000 secs
Total: 12.950000 secs
Decompressed E1M4.umx.uz -> E1M4.umx

Code: Select all

uz decompress AS-HiSpeed.unr.uz
Used 8 threads.
stage 0: 0.062000 secs
stage 1: 0.093000 secs
stage 2: 0.032000 secs
stage 3: 0.031000 secs
Total: 0.218000 secs
Decompressed AS-HiSpeed.unr.uz -> AS-HiSpeed.unr

[sektor2111]

Did I say that you are a genius ?

[Buggie]

New version v3.1:

• Faster compress/decompress.

https://github.com/SeriousBuggie/UZ/releases/latest

Code: Select all

uz compress CTF-Blice-Alpha.unr
Used 8 threads.
Compressing CTF-Blice-Alpha.unr (87328898 bytes)
stage 0: 0.530000 secs
stage 1: 4.197000 secs
stage 2: 1.763000 secs
stage 3: 1.248000 secs
Total: 7.738000 secs
Compressed CTF-Blice-Alpha.unr -> CTF-Blice-Alpha.unr.uz (20%)

Code: Select all

uz compress E1M4.umx
Used 8 threads.
Compressing E1M4.umx (98087852 bytes)
stage 0: 0.562000 secs
stage 1: 3.822000 secs
stage 2: 10.234000 secs
stage 3: 7.098000 secs
Total: 21.716000 secs
Compressed E1M4.umx -> E1M4.umx.uz (95%)

Code: Select all

uz compress AS-HiSpeed.unr
Used 8 threads.
Compressing AS-HiSpeed.unr (4418189 bytes)
stage 0: 0.031000 secs
stage 1: 0.250000 secs
stage 2: 0.125000 secs
stage 3: 0.109000 secs
Total: 0.515000 secs
Compressed AS-HiSpeed.unr -> AS-HiSpeed.unr.uz (29%)

Code: Select all

uz decompress CTF-Blice-Alpha.unr.uz
Used 8 threads.
stage 0: 0.842000 secs
stage 1: 0.936000 secs
stage 2: 0.609000 secs
stage 3: 0.312000 secs
Total: 2.699000 secs
Decompressed CTF-Blice-Alpha.unr.uz -> CTF-Blice-Alpha.unr

Code: Select all

uz decompress E1M4.umx.uz
Used 8 threads.
stage 0: 3.557000 secs
stage 1: 4.821000 secs
stage 2: 1.341000 secs
stage 3: 0.421000 secs
Total: 10.139999 secs
Decompressed E1M4.umx.uz -> E1M4.umx

Code: Select all

uz decompress AS-HiSpeed.unr.uz
Used 8 threads.
stage 0: 0.062000 secs
stage 1: 0.078000 secs
stage 2: 0.047000 secs
stage 3: 0.016000 secs
Total: 0.203000 secs
Decompressed AS-HiSpeed.unr.uz -> AS-HiSpeed.unr

[Barbie]

<joke>Imagine what happens if there will be a version 15: Compression and decompression within a second and compression size of any file to 1 Byte. </joke>

[darksonny]

Hi
im trying to use with cmd mode, how should be called to decompress an uz file? Example:

uz decompress C:\Folderuz\CTF-myMap.unr.uz

Is this the correct as explained in your readme? I got this changelog results:

"Uninitialized: Log file open, 09/19/20 18:02:37
Uninitialized: Realloc: 0 162 
Uninitialized: Realloc: 0 14 
Uninitialized: Realloc: 5497888 14 
Uninitialized: Realloc: 5454520 0 
Uninitialized: Realloc: 0 6 
Uninitialized: Used 8 threads.
Uninitialized: Realloc: 0 132 
Uninitialized: Realloc: 0 132 
Uninitialized: Unknown command 'UZ'. Try 'UZ.exe help'.
Uninitialized: Realloc: 5498056 0 
Uninitialized: Realloc: 5497912 0 
Uninitialized: Realloc: 0 0 
Uninitialized: Realloc: 0 0 
Uninitialized: Realloc: 5440376 0 
Uninitialized: Realloc: 5497888 0 
Uninitialized: Log file closed, 09/19/20 18:02:37"

Am I missing something? Thanks in advanced

[Buggie]

You something miss in your description of issue steps, because for me all works as:
uz decompress U:\T\9\9\CTF-Blice-Alpha.unr.uz

Code: Select all

Used 8 threads.
stage 0: 0.843000 secs
stage 1: 0.936000 secs
stage 2: 0.593000 secs
stage 3: 0.343000 secs
Total: 2.715000 secs
Decompressed U:\T\9\9\CTF-Blice-Alpha.unr.uz -> U:\T\9\9\CTF-Blice-Alpha.unr

[darksonny]

You something miss in your description of issue steps, because for me all works as:
..

Im not sure what I am missing in this decompression, I have this log and I cant decompress. How do you call uz.exe with cmd.exe?

Img attached: