XC_Engine [20] - XC_Core [7b] - XC_IpDrv

[sektor2111]

So, client needs decompress feature installed (I guess Engine tweaking to gain LZMA).

I want to try new Clocking. Sure natives described doesn't need number assigned for function? Last time did not compile without number.

Edit: Yes

Code: Select all

History: UPackage::GetDllExport <- UFunction::Bind <- Function/Operator <- FScriptCompiler::CompileDeclaration <- FScriptCompiler::CompileStatement <- FirstPass <- TryCompile <- FScriptCompiler::CompileScript <- 

[Higor]

Just directly link to XC_Core.u by using the following macro, you may remove the XC_Core dependancy anytime with a new recompile.

Code: Select all

#obj load file=XC_Core.u package=XC_Core

EDIT:
LZMA compressor/decompressors are in XC_Core.dll, no external linkage.
Also, The compressor uses multiple threads.

[sektor2111]

Worked using native number. Results are different than from AppSeconds() - probably more accurate:

Code: Select all

ScriptLog: Time required for Path Seeking: 0.000226
ScriptLog: Time required for Path Seeking: 0.000201
ScriptLog: Time required for Path Seeking: 0.000208
ScriptLog: Time required for Path Seeking: 0.000291
ScriptLog: Time required for Path Seeking: 0.000276
ScriptLog: Time required for Path Seeking: 0.000229
ScriptLog: Time required for Path Seeking: 0.000213

[sektor2111]

I have compressed all stuff in "lzma" format (amazing speed). Now I'm waiting hints to set up LZMA redirection SERVER-CLIENT. UZ stuff has to be moved ?

[Higor]

I first have to crate the Download, NetConnection and Channel variants, as well as a proper IpDrv driver based on those.
An attempt to port the decompressor to Linux was... annoying.

The whole decompressor code built just fine, but Unreal's GFileManager cannot create a proper file reader on both Win32 and Linux, I solved it on windows by disabling code optimization on the commandlet, but doesn't work on Linux.
Right when I get to port some alien code, it's Unreal's systems that fail on me.

[Higor]

Finally got the LZMA decompressor working on Linux, now to remake the download managers.

The download managers are sent by the server in a prioritized list, which their proper parameters.
On 100% cases that list consists on IpDrv.HTTPDownload and Engine.ChannelDownload.
Tried to load an experimental HTTP download manager but nothing's being sent to the client about it, I guess it's a load or dependancy issue... more stuff to look into.

[sektor2111]

Client has to install decompressor binary in order to work if you don't find the way to inject it. Anyway my last modifications into mods includes BAT files to strip codes (have source_code) and compress them in both UZ and LZMA after compiling. I'm agree with LZMA because it can be compressed faster resulting in smaller file and less data sent + less HDD space used. I'll see how many people are ready to switch from default method.

[Higor]

XC_Core Alpha build 3

XC_Core_alpha3.7z

(55.45 KiB) Downloaded 60 times

NOTE:
Won't include the linux binary until I rewrite the file manager, that thing is completely broken on that platform for some reason.
LZMACompress commandlet can't take '/' characters, use '\' instead.

Created enhanced internal channel download (only tested in v436, but has v451 compatibility code):
- Not affected by (!Closing) assertion bug: http://aluigi.altervista.org/adv/unreaload-adv.txt
- Can send .lzma and .uz files if they're in the same folder as the package.
- Can request and decompress .lzma and .uz files on clients.
- Can decompress UT2k4 UZ2 files (server still doesn't send those).
Instructions can be found on a text file in the zipped file.

PD:
All the client has to do to enable this thing is put the XC_Core files in /System and join a properly setup server.

[sektor2111]

- Can send .lzma and .uz files if they're in the same folder as the package.

I'm confused here. Do I need or No longer need redirect location?

EDIT:
I figured how things works, I have to still test something and I'll be back immediately.

EDIT2:
Back with conclusions. I'll describe my understanding based on results Point to Point...

Point 1) Many thanks, Higor for these dedicated works: XC_Engine, XC_Core. Why ? Will be posted bellow...

Point 2) Admin using XC_Engine doesn't need to pay an additional redirect blabbering with more or less dumb hosts and their admins. Server need to not clamp bandwidth for player-slot and neither to use an amount of monthly data limit, game-servers should be unlimited - Yes, drop redirect in Trash-Box.

Point 3) If admin install both XC_Core + XC_Engine and have lzma + uz files in location as original uncompressed files (directly in server) then server will be ready to send compressed content lzma ( or fallback to uz ? - this is not tested yet ).

Point 4) If PLAYER has dropped XC_Engine + XC_Core into client game and join to a server with these XC files added then will get files lzma in a cute speed sent by server. Good...

Point 5) If server has XC Files but has claimed "RedirectToURL=http://.........." then all content is downloaded as UZ normally from redirect location - so they can be combined, if redirect bandwidth seems to mock player, admin can remove redirect temporary and game-server will act like a redirect...

Point 6) My bad goes here. I did not test if files are partially compressed: 1 file lzma, 2 uz, 5 files uncompressed. Actually I don't care about this test as long as I have ALL files compressed and I cannot fail this chapter (I'm a noob PC operator from 1995 familiarized a bit with MS-DOS). Anyone doing such tests feel free to describe how things work for such case.

My conclusion: I can remove redirect service.

[Higor]

XC_Core alpha 4, with experimental XC_IpDrv.
Changes:
- No more Cache00NN.tmp files cluttering on the client's folder.
- HTTP Download, attempts to download LZMA, UZ and non-compressed files on same session if one fails.
- Expermiental XC_TCPNetDriver (far from finished) usable on both clients and servers.

XC_Core_IpDrv_alpha4.7z

(72.79 KiB) Downloaded 58 times

EDIT:
Will try and setup this on Chamberly's server around the weekend, of course I'm not gonna LZMA batch compress 50gb of data, but will start with some basic stuff.

Warning, this hasn't been throughly tested in
version 440-451 of the game, but it 'should'
work as well.


You can LZMA compress using a XC_Core commandlet:
UCC LZMACompress ..\Maps\CTF-Coret.unr

You can LZMA decompress using 7zip, WinRar or:
UCC LZMADecompress ..\Maps\CTF-Coret.unr.lzma


=====================================
XC_Core's channel download.

You need to precompress the files but once done
it's possible to use XC_Core's internal channel
to send compressed files.

XC_Core will automatically lookup for compressed
files right next to the uncompressed files.
Simply run the compress commandlets on the files
you want to test and leave them there for the
game to find and load.

Example:
..\Maps\CTF-Coret.unr
..\Maps\CTF-Coret.unr.uz
..\Maps\CTF-Coret.unr.uz2 (planned)
..\Maps\CTF-Coret.unr.lzma

** This channel downloader isn't affected by
the !Closing assertion crash.

=====================================
XC_IpDrv's HTTP download.

You need to upload the compressed files into the
redirect as you would normally do, only that
this time you'll be uploading both .UZ and .LZMA
files.

This HTTP Downloader has double fallback lookup,
meaning that if a file fails to download, it
will query the next one the following way:
- RedirAddr:Port\Package.ext.lzma
- RedirAddr:Port\Package.ext.uz
- RedirAddr:Port\Package.ext

** If compression is disabled serverside, both
.LZMA and .UZ won't be queried.

** This HTTP downloader displays the correct
download percentage in the game screen.

=====================================
SERVER SETUP

Extract all XC_Core files in your
UT/System folder

Go to your designated UnrealTournament.ini and
add an extra DownloadManager as shown below:

[IpDrv.TcpNetDriver] or [XC_IpDrv.XC_TcpNetDriver]
AllowDownloads=True
... other settings here
DownloadManagers=XC_IpDrv.XC_HTTPDownload
DownloadManagers=IpDrv.HTTPDownload
DownloadManagers=XC_Core.XC_ChannelDownload
DownloadManagers=Engine.ChannelDownload

[XC_IpDrv.XC_HTTPDownLoad]
RedirectToURL=http://address:port/path/
ProxyServerHost=
ProxyServerPort=
UseCompression=True

=====================================
CLIENT SETUP

Extract all XC_Core files in your
UT/System folder

=====================================
ALTERNATE NET DRIVER (client and server)

Very experimental, can be used to host or join
games, so far I haven't experienced weird
effects, but I don't recommend it's usage yet.

[Engine.Engine]
...
NetworkDevice=XC_IpDrv.XC_TcpNetDriver

This net driver's data polling is not yet
complete and some exploits may remain.
Current features:
- ICMP Unreachable protection.
- Less CPU usage with XC_Engine's Connection handler

[sektor2111]

XC_IpDrv fault
It doesn't seems to work in attempt to join to a server using a host-name. Random example "unreal://MyAnimalPlace.sub_domain.tld:7777/". Client is crashing. Using LAN, it works. Other tests I did not perform as long as I don't have multiple internet connections available.

[Higor]

I see, either way it isn't required to use XC_IpDrv as the main net driver for a UT client.
The enhanced HTTP Download will still run if you have both XC_Core and XC_IpDrv in your System folder.

[Higor]

This happens in XC_Engine 14 alpha when joining a server that doesn't have XC_Core based download managers setup:

Code: Select all

DevNet: PendingLevel received: USES GUID=1E90ACC111D1ED664544279700005453 PKG=Palettes FLAGS=1 SIZE=421024 GEN=1 FNAME=Palettes.utx
DevNet: PendingLevel received: DLMGR CLASS=IpDrv.HTTPDownload PARAMS=http://192.168.1.2:27010/ COMPRESSION=1
DevNet: PendingLevel received: DLMGR CLASS=Engine.ChannelDownload PARAMS=Enabled COMPRESSION=0
DevNet: PendingLevel received: WELCOME LEVEL=DM-ZepTepi][ LONE=0
DevNet: Welcomed by server: LEVEL=DM-ZepTepi][ LONE=0
Log: Decoded IP 192.168.1.2 (192.168.1.2)
DevNet: Receiving package 'DM-ZepTepi]['
XC_Engine: Replacing HTTP download manager with LZMA enhanced version.
XC_Engine: Cancelling current download and restarting...
Log: Decoded IP 192.168.1.2 (192.168.1.2)
DevNet: Receiving package 'DM-ZepTepi]['
DevNet: LZMA file not found in redirect, attempting to download UZ
DevNet: Compressed filesize: 788604
Warning: stage 0: 0.296267 secs
Warning: InData size: 0 - OutData size: 2006919
Warning: stage 1: 0.207367 secs
Warning: InData size: 2006919 - OutData size: 2006919
Log: DecompressLength: 262144 0x002DF460
Log: DecompressLength: 262144 0x002DF460
Log: DecompressLength: 262144 0x002DF460
Log: DecompressLength: 262144 0x002DF460
Log: DecompressLength: 262144 0x002DF460
Log: DecompressLength: 262144 0x002DF460
Log: DecompressLength: 262144 0x002DF460
Log: DecompressLength: 171807 0x002DF460
Warning: stage 2: 0.279554 secs
Warning: InData size: 2006919 - OutData size: 2006815
Warning: stage 3: 0.173687 secs
Warning: InData size: 2006815 - OutData size: 0
Warning: Total: 0.956876 secs

In short,
- XC_Core is now a dependancy of XC_Engine and is necessary for it to run.
- XC_IpDrv is optional and XC_Engine will attempt to replace the download manager clientside to LZMA downloader if possible, also restarting download if necessary.

This means that any client running XC_Engine v14 can make full use of redirects with LZMA files (or multiple fallback feature) on them without requiring a server from setting up anything other than it's default redirect.

================================
XC_Core on linux still giving some issues, the XC_ChannelDownload starts up and is able to open files now, but the server crashes after sending the initial data bunch of the file, crash occurs in its Engine code instead of a XC_Core function, what makes it an effect of something that XC_Core changes and is way more difficult to debug...

================================
In the meantime I'll run with this XC_Engine client for a week or so to test it's stability, it's had lots of internal rewrites and v451 client still has an annoying exception window when you close the game, basically preventing a clean shutdown.

[sektor2111]

If you can make a better deal for a clean client - "convince client" to get lzma without redirect that would be cool.
Aside note... In the past 3-4 years ago or even more I was looking over redirect options, to be honest except hosting files in your own yard I did not see other good things (only restrictions from any kind). Speaking about the same machine without to affect NetSpeed from game is better to send files VIA XC_Engine and if lzma way is possible then is perfect. So to speak this is helpful enough. We have connections now days not old 64 k DSL lines hosting servers intended to prevent any lag.

Other thing, some lists with spam related craps, advertising related and so on, might include whatever redirect WEB place, then client will download files FORCED from game-server as long as redirect is "black-listed" and firewall-ed - happened to me by example. If client can join to a game-server for sure will be able to get files from there too (even from one "the second" machine hosting files nearby machine game-server) in the same location. From me I think a hard work to redirect player is not really No. 1 priority. If client (noob) is able to get files in a faster way lzma and/or uz - mission is successful. For me dealing with redirect is a kid toy, I simply cannot fail any file to be redirected, just sending small content to player is the deal, to not mention my gratitude for new format as long as compression is insane faster than moooing with uz (some files reaching at 400 seconds in UZ process). A high loaded server needs hours for processing UZ compression. LZMA is truly nice - let's just send that to player and we are done.

Once completed redirect's mission probably it would be good to hook decorations and other Engine classes messed up as you did with bRocketTick and SetEnemy if a header to point at replaced classes screwed cannot be delivered. Else you can check what is going on in Lightning chapter as long as a movable light works cute in SP but not exist in Client joined in server. I can even show you a small "map" with a light ported by a mover how works LOCAL and what stupid garbage is ON-LINE...

[Higor]

XC_Core base component released.
Check first post for download and documentation.

==============================
==============================

Holy shit, UTPG apparently did some more undocumented changes, this time with either the FArchive structs or the bit writers/readers.

Gonna leave this here just in case.

Code: Select all

class ENGINE_API FOutBunch : public FBitWriter // Size = 108 // Size  = 116 in v440
{
public:
	// Variables.
	FOutBunch*		Next; //Starting offset of +8 in v440
	UChannel*		Channel;
	FTime			Time;
	UBOOL			ReceivedAck;
	INT				ChIndex;
	INT				ChType;
	INT				ChSequence;
	INT				PacketId;
	BYTE			bOpen;
	BYTE			bClose;
	BYTE			bReliable;

	// Functions.
	FOutBunch();
	FOutBunch( UChannel* InChannel, UBOOL bClose );
	FArchive& operator<<( FName& Name );
	FArchive& operator<<( UObject*& Object );
};

=================================
BTW I noticed that XC_Engine is creating a hook that shouldn't be creating on clients (Level vtable replacement), causing a kick in ACEv09 servers, should be fixed for v14 release...
It it's current state it's possible to open the file channels on v451 servers, gonna have to test on Linux.