Critical Error on client while redirected download

[Barbie]

Some tests have shown that there is a problem if a compressed file is bigger than the uncompressed. Therefore I put variant amount of comment into the UC file, compiled and compressed them, restarted the UT server, cleaned the clients UT cache and tried to connect. Depending on file sizes I even got two different issues (Server refused to send/Critical Error):

Smaller compressed one:
SBNonFallingMantaV0.u.uz: 924
SBNonFallingMantaV0.u: 957
-> Client downloads and joins

Equal size:
SBNonFallingMantaV0.u.uz: 895
SBNonFallingMantaV0.u: 895
-> Client downloads and joins

Bigger compressed one:
SBNonFallingMantaV0.u.uz: 888
SBNonFallingMantaV0.u: 887
-> '192.168.1.155/MH-Ziki+SBFix1?Name=Ken?Class=Botpack.TMale2?team=0?skin=SoldierSkins.blkt?Face=SoldierSkins.Othello?Checksum=NoChecksum' failed; Downloading package 'SBNonFallingMantaV0' failed: Server refused to send 'SBNonFallingMantaV0'

SBNonFallingMantaV0.u.uz: 885
SBNonFallingMantaV0.u: 884
-> '192.168.1.155/MH-Ziki+SBFix1?Name=Ken?Class=Botpack.TMale2?team=0?skin=SoldierSkins.blkt?Face=SoldierSkins.Othello?Checksum=NoChecksum' failed; Downloading package 'SBNonFallingMantaV0' failed: Server refused to send 'SBNonFallingMantaV0'

SBNonFallingMantaV0.u.uz: 883
SBNonFallingMantaV0.u: 878
-> Critical Error

Some other test with sizes below that leads also to Critical Error.

Servers UnrealTournament.ini settings in [IpDrv.TcpNetDriver]:

Spoiler

Code: Select all

AllowDownloads=False
ConnectionTimeout=15.0
InitialConnectTimeout=120.0
AckTimeout=1.0
KeepAliveTime=0.2
MaxClientRate=200000
SimLatency=0
RelevantTimeout=5.0
SpawnPrioritySeconds=1.0
ServerTravelPause=4.0
NetServerMaxTickRate=20
LanServerMaxTickRate=35
MaxDownloadSize=1
DownloadManagers=IpDrv.HTTPDownload
DownloadManagers=Engine.ChannelDownload

[sektor2111]

Checksum=NoChecksum' failed;

What's this if you don't mind ? What content have those files ? The same BRAIN x.xx ?

I could see at a moment UC files corrupted unable to be compiled because of some nasty invisible bytes added screwing syntax. Forcing a compilation there might do nasty things - I'm guessing only.

However I'm curious how you get such small files. How much script is there ? Only defaultproperties changed ? hmmm......
Is not much simple a package having some content for at least 10kb size.

Those messages are new to me - I never see them. My MaxDownloadSize=0 - Unlimited and explanations

When game is hosted in point A and redirect is hosted in point B (blacklisted and firewall-ed or redirect HOST is DOWN for many reasons), client won't get nothing EVER. You should leave them last resort to gain files from server if redirect fails.
At least I don't have these issues with XC_Engine - if client fails redirect (as described today for tech reasons) server is pumping files in client faster than at game-speed.

From my old experience redirect and game-server should be better to have almost the same location (the same ISP if possible) to prevent switching many hops.

[Barbie]

Checksum=NoChecksum' failed;

What's this if you don't mind ? What content have those files ? The same BRAIN x.xx ?

I'm sorry, I don't get what you want to say here.

However I'm curious how you get such small files. How much script is there ?

See this thread for packages content; it's only a hand full of lines. Usually I embed such small stuff into MyLevel but if MunsterHunt.u is used, UnrealEd's "Compile Changed Scripts" fails. So I decided for an external package. (In the meanwhile I have embedded it into the map to avoid clients redirected download problems, but before I can compile scripts in UnrealEd I have to remove all used stuff of MonsterHunt.u, save the map, close UnrealEd, reopen the map in UnrealEd, add my script, compile it and then add on all removed MonsterHunt.u-stuff. Just wasted working time...)

When machine is hosted in point A and redirect is hosted in point B

Both are running on the same host.

At least I don't have these issues with XC_Engine

Have you ever run into this special case with compressed size > uncompressed size? And it's not a problem of the server, it's a problem of the client.

[sektor2111]

Usually I embed such small stuff into MyLevel but if MunsterHunt.u is used, UnrealEd's "Compile Changed Scripts" fails

Issue number 1 when people are doing MH stuff - credits for trash goes at Shrimp Kenneth Watson. Never use MonsterHunt.u file used in servers for working around MH - you need the MonsterHunt.u file for coders (I have one from Kelly) but I can do a place-holder any time or perhaps works v5.04 which is not funked up for no reason.

Have you ever run into this special case with compressed size > uncompressed size? And it's not a problem of the server, it's a problem of the client.

okay, I'll try to see if I don't have such things already or I'll write a small null thing just for testing purpose because you awake my curiosity.

Edit:
NICE FIND, BARBIE....
It's the same problem with this >

relicsbindings.zip

(1.71 KiB) Downloaded 98 times

However... such stuff won't be a warm welcome for player. Looks like happens at small files. Is this issue fixed in v451 ?? (client).

[Barbie]

Never use MonsterHunt.u file used in servers for working around MH - you need the MonsterHunt.u file for coders

I cannot remember exactly what MonsterHunt version I installed years ago on my developing PC (not server). I've found Help\ReleaseNotes.htm which mentions 'Kenneth "Shrimp" Watson' and it says it is "Release 6". The size of System\MonsterHunt.u is 1251524 and the MD5 sum 570f03b201d4bf1928438ac2381d3a0c.

[sektor2111]

Shrimp has source-code on his site (or still had it ?). If not look for source-code at Medor and compile a clean MH file ONLY in your modding copy of game. That clean build won't disappoint you. Or use alternate PM.

Edit > I have to scan now how many files are tiny as a louse and used in Levels and their alias in redirect, LOL. Perhaps Not today.

[Chris]

When the file is compiled, does it run properly?
The Huffman algorithm is used for lossless text compression pretty much.
A "compressed" file can become bigger than its original if there isn't much to compress, instead you get the opposite effect with a bunch of overhead data.
Try compressing a zip file over and over for example and you'll see what I mean.
One thing to test;
Take the original u. file, compress it, then decompress it again (all using ucc) and compare the results. If the data is any different then there is a bug in the .uz codec.

[Barbie]

It's the same problem with this > relicsbindings.zip

Here also is the size of the compressed file (relicsbindings.u.uz, 669 bytes) bigger than uncompressed file (relicsbindings.u, 564 bytes).

Looks like happens at small files.

The probability of having a compression rate of above 100% raises with the smallness of files (assuming "normal" content). I bet the problem also occurs at any size if you take incompressible content for the file.

[sektor2111]

With your permission I'll develop a tutorial for myself:
sektor-dumbass to do >
- get rid of all UZ files from redirect and server - Now I know why some quality german gaming services were setting up

Code: Select all

[IpDrv.HTTPDownload]
RedirectToURL=some_tricky_link_right_to_game_folders 
ProxyServerHost=
ProxyServerPort=3128
UseCompression=False

UseCompression=False, I believe those guys have respectful years of experience in gaming > UT.
They were knowing this issue.

Given XC_Engine I can remove redirect for [IpDrv.HTTPDownload] but I can keep it for LZMA.

Code: Select all

[XC_IpDrv.XC_HTTPDownLoad]
RedirectToURL=SomeAdress
ProxyServerHost=
ProxyServerPort=3128
UseCompression=True

So LZMA being another stuff (of course I'll do tests) won't suffer... Else I can quit using any compression. Right these days I was thinking at tiny stuff, so I was about to do trash for myself.
I will release a bunch of space wasted with files, even I had that above file in redirect. If such file is in a Level will mock clients. Seeking small louses in redirect and server will take time, so I prefer no UZ compression for redirect with meaning NO Redirect.
Thank you, Barbie for this grave find and Thank you, Higor for XC stuff which will solve this problem. And, thanks again EPIC for that line 324 which no one could mess it much better.

[JackGriffin]

Double check that your cache is completely cleared before you go testing all this. I've went down this road a couple of times trying to suss out why a file would be on the redirect as one size then decompress to another. It ended up being
-the file was lurking in my cache in a couple of different size versions
-the redirect file suffered from the capitalization issue on linux redirects and it was sending a different file than I assumed I was getting.

I've done a shit ton of compressing and decompressing and I've never seen the engine fail at the process and do a bad version. I'd look around for other things to pin it on.

[Barbie]

Double check [...]
-the file was lurking in my cache [...]
-the redirect file suffered from the capitalization issue on linux redirects [...]

Additionally I work in a mixed Windows/Samba (development), Linux (test server in a VM) and again Windows/Samba (test client in a VM) networked environment where even some more caching and locking problems can come in, I know that. But I'm quite sure that I can exclude these as a pitfall, because different files lead to different result groups and this won't happen if always the same file were used.
And of course I cleaned the clients cache (and I mentioned it in the description in series of tests).

I've done a shit ton of compressing and decompressing and I've never seen the engine fail at the process

But that is not the point. Compressing and decompression works fine, even if the compressed file has a ratio > 100%. The problem is the behaviour of the client if compression ratio is greater than 100%.

[JackGriffin]

I didn't deeply read the thread, just kind of skimmed it. I see the issue now and yeah this is odd. BTW, here's an MH that will compile for you to use

recompiled.zip

(363.36 KiB) Downloaded 96 times

Just put it into your folder to use when you need to compile an MH dependent mod then replace it with the stock one when you are done. The zip is passworded only because I don't want to post something that creates a bunch of version mismatches when people who don't understand what it is end up posting it to servers and redirects. This monsterhunt.u is made from Shrimp's complete source code (just goes to show why it's never a bad idea to ask ) If you want to use it just PM me and I'll give you the password.

[sektor2111]

Simple test - file involved "relicsbindings.u" uploaded previously. Add it in serverpackages and remove it from client (if exists). Fire server, join, download, "ENJOY"... I sent to Barbie a PM.

About what you were saying for any mismatch. If I have files only in cache I can assure you I won't lose them. Why ?
1) I'm not purging/uncaching cache;
2) I have big UXX files marked Read-Only - for avoiding 3 years of downloads and deleting them by any sort of "mod" able to handle them (a backup is a bless after all).

Last advice: Files under 1 kb can be marked Read-Only too - and keep them in cache if are not a stock (except skins and player-tools) - THIS is a default rule for client, ignoring it makes UT to punish you.

[Barbie]

Another candidate crossed my way, found in map "MH-GiantBoss1":
SizeOf(Blank.umx): 481
SizeOf(Blank.umx.uz): 553
which leads to a compression ratio of 114% and therefore causes a GPF on clients while downloading.

My solution was changing the music file to "null48.umx" in the map and saving it with a different name.

[sektor2111]

The question is if worth effort to create a list with these lousy files... else now I'm totally convinced to forget UZ.