Turning a game server into a redirect. [Testing stage]

[Higor]

So I came up with this idea:

- Open a HTTP channel in the server, serialize the file using UBinary and send the data.
- Internal firewall: allow downloads from an IP only if a player has challenged for a client connection in the last minute.

What to research
- Can redirect switch from compressed to non-compressed if a compressed file isn't found?
- Maybe limit file sending to files loaded in the PackageMap object?

Other stuff to do:
- Give UBinary the ability to tell a file's size without opening it.
- Give XC_GameEngine the ability to transfer actors from one level to another? (prevent connection interruption during map load).

================
Project data/tests

- AutoRedirect.u started within UnrealEd.
- Edited UWeb to compile AutoRedirect with ability to change file seeking directory.
- XC_GameEngine feature: deny sending file to IP's not registered as incoming clients (PreLoginHook).
- WebApplication functional.
- UWeb based file sending.
- Supports automatic HTTPDownload.RedirectToURL setup.
- WanIP can be manually input or autodetected from ACE (must have bCacheWANIP=True).

================

Input and ideas greatly appreciated.

[Wises]

don't really understand your Jargon being a nub (myself) however I googled the first foreign word I cam accross being "Serialize" and found this to be of interest;
http://www.polycount.com/forum/showthread.php?t=122099

could be a good start *shrugs*

also I like where this is going ... just hope that it doesn't interfere with your other 150 series-linked projects.
working everything into XC_GameEngine.Engine seems to be the right direction and over the next few years I think you could probably fix most/all of the issues that have stunted development in this game.

Honestly GREAT JOB!!

EDIT: WAIT !.. WHAT?

if I read the thread title correctly , what you intend to do is .. make the Actual game server the redirect_Server as well?
if so .. VERY VERY VERY INGENIOUS!!!

so if i understand some of the Jibberish (coders talk) you are Planning on how to compress and send files on a Seperate HTTP Channel (Like curent MapVotes) to Clients which would be Decompressed client_Side and extracted to /Cache/ folder for GamePlay?

um.. WOW! that would save all sorts of BS in the Redirect Arena.. no need for domains , etc.. wicked.

[Higor]

I wouldn't venture into opening CPU threads in the game, so I'll stick with pre-compressed packages.
So in your UT, you'll end up having a new folder:
- System/
- Maps/
- Textures/
...
- Redirect/

BTW I may have found the way to reduce workload... UWeb has a already a binary file send implementation...

[Chamberly]

Yes!

Would be plenty of uz checking to see if uz file exist without the client having to reconnect for faster download maybe.

A specific idea on the Cache, if there are more than 1 different file by the hash/size, what actually caused the finding the correct file for it to be loaded on the map to play on the server? This would probably sort out on the front end of redirect to check such file for a map. Or any similar idea.

Edit: Will check for redirect from DemoManager.

[Higor]

Experiment done...

- UWeb does do proper file transfers.
- Managed to download a map in 0.1 second from my online UT to my Ferbotz UT. (local test lol)
- UWeb cannot seek outside of the /Web/ directory, so your redirect must be in /Web/Redirect/
- Due to the above limitation, it's best leaving the redirect as UZ compressed-only.

I can make the mini-firewall, autosetup (so no webserver config needs to be edited, like CMV), autoWAN (for dynamic ip servers) and any other stuff needed.

[noccer]

Experiment done...

- UWeb does do proper file transfers.
- Managed to download a map in 0.1 second from my online UT to my Ferbotz UT. (local test lol)
- UWeb cannot seek outside of the /Web/ directory, so your redirect must be in /Web/Redirect/

lol... I ever wondered if this works... LOL

[UT99.org]

Looks like you're getting somewhere. The language files are only relevent for the russian version. Anth went native for demo manager you might want to skip that if you can't get the code. Nexgen has some tcp plugin support. It would be nice if while you're at it at some point add sceenshot sending. (I have fears the mod would be abused by making individual player's lag with repetitive usage by admins but we'll see where this goes)

[Higor]

I got most of it working, and my UT can now redirect UZ files from a variable folder (not limited to Web now).
Problem is how UWeb works, when sending a binary file, it will hang the game until the file's finish transferring.
This is VERY BAD.

The only solution is to give a couple of extra capabilities to UBinary and then use it to buffer by 16 kb per frame or smth (~500-1000 kb/s depending on server).
Maybe add a setting for both kb/frame and active UBinary serializers at a time.

[Wises]

thinking perhaps a client-side mod which could deal with transfer's esp mismatched files and you know... how it reports to the client / they need to figgure out what file to remove from system..

perhaps some kind of auto-overwrite so if they were to connect to say server a.) and download WhoPushedMe.u v1 , then moved to Server b.) which attempted to send whoPushedme v2... then the client-side patch could just go meh.. overwrite current whoPushedMe v1 with v2.. and vice versa if/when they went back to server a.)

something like that.. very interesting none the less..

having the check for client connect before handshaking? and setting up transfers is pretty awesome concept.. I see that you are already thinking way ahead for when the idiots come to exploiting server's and trying to mess with server bandwidths etc...

[Higor]

On the other side, using UBinary to serialize a 15mb file doesn't do the job either, server still hangs (stupid TCPLink function only lets you input 255 bytes at a time...).

Ended up dealing with this directly using XC_GameEngine.
Changed a player's CurrentNetSpeed to 1000000 before said player gets to connect, then restored to original value after join.
Did this on one of my tests and the results were satisfactory, but there's an actual limit to how much speed you can get from this, on my tests it ended up being something between 100 kb/s to 200 kb/s.

[Higor]

Hello ladies and gentlemen, I just got an idea.

Ever since I added some atomic operations into XC_Core's headers, I may have a solution for this.
Sure, needs native code but I can totally open a CPU thread with a UScript call, load the file without hanging the server and then join the thread with the main game thread once file is in memory and send the file to client at a desired speed.

Let's assume we use XC_IpDrv:
- I implement a XC_TCPLink actor (as in IpDrv) but with an extra function that allows the user to load up a file to memory using a different CPU thread.
- When file finished loading, a flag is set and when UScript code passes over said flag, an 'event FileLoaded(string FileName)' is called on the XC_TCPLink actor.
** We assume precompressed via UZ, LZMA or not. Client has control over this
- The file is put in a dynamic array of bytes so that it's loaded once, helpful for multiple clients.
** As always, the server will only grab files if they're actually loaded into the game!
** An option to unload files if file memory exceeds a certain amount of MB can be possible too.
- Then send all the data as usual from the server, alternatively a native method could be used to limit the upload speed to a certain amount.
** Probably using a native FileSend() function directly and completely removing the need of doing it manually within uscript.

[Chamberly]

** An option to unload files if file memory exceeds a certain amount of MB can be possible too.

Hmm a thought, if the mapvote have a maplimit (going by detection? or after loading a new map?), said map wouldn't be played again until then so maybe it'll be best to discard those temporary?

[Higor]

I meant, file data stored in memory for quick redirect.
As soon as it hits a limit (let's say, 200MB of RAM), it will start flushing files out of memory to prevent virtual memory issues.

[Chris]

These are great ideas indeed.
I like that you go big with it, really opening up a whole new section for bugfixing and getting rid of certain limitations.
I also have quite a few ideas, most of them which I've already implemented in a private mod.
I have a windows ported version of the linux dirent library that allows much faster filesearch(traverse a directory for example)
that could come in handy.
My knowledge of the engine internals (limited to the public headers) is quite limited compared to yours, but I have quite some skills in many languages, inluding C++ ofcourse.
Would you like to do some teamwork?
I would have no interest in credits or such, more the fun of cooperating in such projects.

Regards, Chris

[Wormbo]

The idea of a redirect server is to keep traffic off the game server. But if you already mess with native stuff, why not just enable LAN mode for the channel download? That would result in maximum possible transfer speed without any secondary files involved.