billybill wrote:The bug I am trying to address is for file mismatches on a large redirect that would have many servers using it.
The closest thing I've done like this is making linux http servers not case sensitive
Since my php knowledge so far doesn't go beyond 'if browser language is' or 'if ?var=parameter1' entered in url then 'display this text' or 'redirect to this url'. I am not sure the best way to approach this. There a couple already existing php scripts but none address this problem
It's not hard to change the redirect URL as soon as a new map loads and before anyone connects. The question is would it be better to have the server owners themselves have an account with the redirect where they can specify which version of a file to use on that particular server, or have a database of maps that use different files
examples
vsCode: Select all
http://redirect.to.url/redirect.php?server=127.0.0.1&port=7777&file=
The problems with letting the users decide for their own servers:Code: Select all
http://redirect.to.url/redirect.php?map=DM-MAPNAME&file=
- Said server(s) would not be able to support more than one mismatch of the same file
- They would have to make an account, set up any mismatch files. Some might view as a pain
The problems with having one big database of maps and their file versions:
- Who gets to decide? or how do you stop people abusing and/or posting incorrect matches?. It's not really viable without being maintained by someone or a group of trusted people
I suppose Ferali's script might avoid all of this by having the redirect server's php script return the map's dependencies either live (or once per map saving to a database) to list the dependent GUIDs used. Ferali please stop me if there is no need for the discussion
brainstorming ideas for redirect
brainstorming ideas for redirect
Last edited by UT99.org on Mon Feb 10, 2014 9:25 am, edited 3 times in total.
Re: brainstorming ideas for redirect
The same old issue I brought up with Unreal's music files, with the exception that the right files were easily recognizable due to being on Unreal CD.
One specific solution for a specific case is refusing to send files referenced by maps.
This was done in AssaultBonusPack with one of its U files, allowing you to play on the servers regardless of your package version, with 2 limitations:
- If you don't have the file, you may need to manually install ASBP.
- If the map used a newer feature that didn't belong to your package, you need to manually update the file.
The ideal procedure is Package Conformation, although it's been rarely used and most fixes are clientside so using an old package on the player's end won't fix the problem.
The best way to deal with this is pretty simple, rename every version (remember pure?), and add your initials to a package name, other than that it's overthinking things to deal with an easily avoidable problem.
(Just don't run maps that use a package with the name... Wood.utx for example, that's the dumbest thing you could do, and believe me it's been done a lot for this specific case).
One specific solution for a specific case is refusing to send files referenced by maps.
This was done in AssaultBonusPack with one of its U files, allowing you to play on the servers regardless of your package version, with 2 limitations:
- If you don't have the file, you may need to manually install ASBP.
- If the map used a newer feature that didn't belong to your package, you need to manually update the file.
The ideal procedure is Package Conformation, although it's been rarely used and most fixes are clientside so using an old package on the player's end won't fix the problem.
The best way to deal with this is pretty simple, rename every version (remember pure?), and add your initials to a package name, other than that it's overthinking things to deal with an easily avoidable problem.
(Just don't run maps that use a package with the name... Wood.utx for example, that's the dumbest thing you could do, and believe me it's been done a lot for this specific case).
Re: brainstorming ideas for redirect
billybill wrote:It's already widespread on older maps and you can't rename those without touching the map itself. I've never heard of this ASBP, is this going along the lines of demo manager and downloading as a client-side mod?
Re: brainstorming ideas for redirect
I'm not sure this can be done? But I haven't fiddled with UT's web interface that much, at least not in this direction. But I do think that it follows the basic HTML rules as to how to send or recieve variables, so you example code may be bugged. It should look more like this:billybill wrote:vsCode: Select all
http://redirect.to.url/redirect?server=127.0.0.1?port=7777?file=
Code: Select all
http://redirect.to.url/redirect?map=DM-MAPNAME?file=
Code: Select all
http://redirect.to.url/redirect?server=127.0.0.1&port=7777&file=
Code: Select all
http://redirect.to.url/redirect?map=DM-MAPNAME&file=
Retired.
Re: brainstorming ideas for redirect
billybill wrote:ah I believe you are correct. minor stuff but thanks for correcting me. fixed up the first post
- Feralidragon
- Godlike
- Posts: 5493
- Joined: Wed Feb 27, 2008 6:24 pm
- Personal rank: Work In Progress
- Location: Liandri
Re: brainstorming ideas for redirect
Well, you can certainly use it to retrieve the package dependencies, but as for the GUIDs used you are not able to get them (the unreal format only stores the package names and class/texture/sound/etc names it depends upon).billybill wrote: I suppose Ferali's script might avoid all of this by having the redirect server's php script return the map's dependencies either live (or once per map saving to a database) to list the dependent GUIDs used. Ferali please stop me if there is no need for the discussion
The max that can be done is to process them once into a database, and at the same time check if every external package.resource referenced in the package exists in the package you already have, just to ensure compatibility.
The correct thing to do imo is to simply rebuild the database with all packages normalized, or in other words, start with the things that have no issues at all (maps, textures, etc), then advance to the problematic ones with multiple versions, and stick with just 1 of them (the best one). Anything that uses other versions should either be modified (with the permission from the author of course) to use the normalized package, or should be discarded (considering that many files are "certified trash", there would be no great loss imo).
However, in the cases that's not a viable option, then all the needed versions should be hosted and served depending on the server the player goes to.
In PHP you can manage this perfectly and seamlessly: all you would have to do is to:
1 - Have a database with the versions per server (there would be generic versions for all, then probably a table with the specific ones per server);
2 - Fetch and stream the right file back to the client.
Both operations are very simple to do, provided that you setup the right headers in the PHP response and as long as the only response bytes you send are only the file itself (to avoid corrupted files in the client, as basically all you need to do is to setup a call a few "header(...)" functions, and doing an "echo" of the file raw bytes). Be careful to NOT have gzip enabled though (to ensure that, there's another function you can call to disable it, although I don't remember the exact one anymore).
This has several advantages: you solve the multiple versions once and for all, fix the Linux file name case-sensitivity, it's easier to balance and scale-out through multiple servers, that way you can structure the file system any way you want to without compromising anything at all and you can even track the connections to know how many requests you're having, from where, for which server, etc... (which is always useful data). And as a special bonus, you can keep the actual structure of the file organization hidden improving overall security (although the same could be achieved with rewrite rules) and you can authenticate certain things.
The only disadvantage in using PHP is that there are perhaps better things to handle static content like this (memory + cpu wise), but in case of a busy server with limited memory, you can send all the files by chunks through PHP at least and UT won't even notice it (since in the end it will receive all as one).
If a native mod was made for the client however, one could go even further and make the client support partial downloads, torrent-like download (by retrieving different parts from different servers if the bandwidth of one is limited) and far better compressed file formats (rar or zip) and better handling of different versions and caching.
Re: brainstorming ideas for redirect
billybill wrote:Okay so based upon all that I came up with a third option, what about a serverside mod that contacts the redirect server, authenticates and sends the GUIDS to the redirectserver how viable would that be?
All the php is above my head a little, did you say cancelling out headers? That's a little more complex than returning the correct file. And, did you say php was going to use a massive amount of resources? What were you comparing it to if I may ask, were you comparing it to using a clientside torrentlike mod? lol only someone like Anth could pull something like that off. I could almost cite the demo manager mod as proof since it's one step away from there with it's multiple redirect server list
I don't think the client would save the header in the file, unless the script was written incorrect. Correct me if I'm wrong though
- Feralidragon
- Godlike
- Posts: 5493
- Joined: Wed Feb 27, 2008 6:24 pm
- Personal rank: Work In Progress
- Location: Liandri
Re: brainstorming ideas for redirect
Well, the server could contact the redirect server first (even without any requests) to register all its used GUIDs, but it would only do it once as a "cold boot", and then it would only register new ones on new file additions.
As for the PHP, I am not saying to "cancel" headers, what I meant is to setup any headers that the server should normally answer with (like the file correct mimetype for example), but it's not complex at all, it's extremely simple.
What I meant with resources, is that depending on which environment you run PHP itself, it may get heavy, For example, if you use Apache, it will consume a fair amount of memory per worker (each worker is pretty much the same as each possible physical HTTP connection to the server), and if you stream the file, you have to load it first into memory in the case of PHP to then "echo" it as the response.
Thus my suggestion that if that's a problem, to send the file by chunks (consuming a marginal amount of memory in the process).
As for the comparison, PHP wasn't made to serve static content (files), thus its limit to do so is far shorter than software made specifically for that (Varnish, Ngnix, etc), but personally I have no experience with either of them so I don't know exactly how you could go on in doing the same thing and in which language. On another hand, PHP + Apache with Varnish or Nginix would probably solve all those problems (PHP as the innermost layer to process non-cached requests, then Varnish to serve the ones already requested and thus cached).
Anyway, the bottom line is that the redirect server could have logic into it which would handle all those issues, in a seamless way for both the real server and the client, and it doesn't take a guru to do it, anyone with minimal knowledge in either PHP or some other equivalent or near-equivalent server-side language can do something like this with ease. All you need to define more than this is the database itself and how you're going to organize the file structure internally (I would suggest MD5-based sharding, for a good distribution and to load off to other servers easily later on without needing to have all the files in each one, if disk space is a problem of course).
As for the PHP, I am not saying to "cancel" headers, what I meant is to setup any headers that the server should normally answer with (like the file correct mimetype for example), but it's not complex at all, it's extremely simple.
What I meant with resources, is that depending on which environment you run PHP itself, it may get heavy, For example, if you use Apache, it will consume a fair amount of memory per worker (each worker is pretty much the same as each possible physical HTTP connection to the server), and if you stream the file, you have to load it first into memory in the case of PHP to then "echo" it as the response.
Thus my suggestion that if that's a problem, to send the file by chunks (consuming a marginal amount of memory in the process).
As for the comparison, PHP wasn't made to serve static content (files), thus its limit to do so is far shorter than software made specifically for that (Varnish, Ngnix, etc), but personally I have no experience with either of them so I don't know exactly how you could go on in doing the same thing and in which language. On another hand, PHP + Apache with Varnish or Nginix would probably solve all those problems (PHP as the innermost layer to process non-cached requests, then Varnish to serve the ones already requested and thus cached).
Anyway, the bottom line is that the redirect server could have logic into it which would handle all those issues, in a seamless way for both the real server and the client, and it doesn't take a guru to do it, anyone with minimal knowledge in either PHP or some other equivalent or near-equivalent server-side language can do something like this with ease. All you need to define more than this is the database itself and how you're going to organize the file structure internally (I would suggest MD5-based sharding, for a good distribution and to load off to other servers easily later on without needing to have all the files in each one, if disk space is a problem of course).
Re: brainstorming ideas for redirect
billybill wrote:I will check out varnish and other web servers, this is now on my 'to do' list which is very long but so long as I don't lose all love for the game I will eventually get around to it
I know I've seen code that fetches MD5 hashes, I can't recall ever seeing a GUID fetcher. Maybe someone can point me to something existing
If it ends up being a DLL file then it could be an ass making it Linux server compatible, unless of course someone with experience like Anth stepped up
It would be easier for me to start with one of the existing scripts as a base and add the database component and accept connections from the server to update. Or I could simply write the UScript component for the existing authors for them to work off of
CTF-w00tEternalCave.unr
^ on the note of the "tool in question"
that's ok with me
from memory it works (has been several years since I've messed with UT Tbh..) other then the odd game occasionally..
but I vaguely remember using it to source some files for I think Noggy's portal map..
which I think I used BB's Method actually.. but it would have been probably faster using UnrealDeps I reckon.
one thing I will say when it comes to servers... I wish admins would do a simple (vanilla connect) to their servers after propagation to check for misdirected files.uxx because these servers probably only get 1/3rd of the traffic they could otherwise get with faster (down/uploads).
@Sam ; ComboGib has one file missing BTW.
here's a simple tip to check server redirected files easily.
after setting up your server with all mods etc , rename your /Cache folder to Cache2.
then reconnect to server... keep eye on downloads and if they slow down then either the file 'Downloading' or the previous (probably tiny) file needs to be Redirected.
this ensures that wanderers will even attempt to see what your server is like.
once there though , is up to you in keeping them there..
that's ok with me
from memory it works (has been several years since I've messed with UT Tbh..) other then the odd game occasionally..
but I vaguely remember using it to source some files for I think Noggy's portal map..
which I think I used BB's Method actually.. but it would have been probably faster using UnrealDeps I reckon.
one thing I will say when it comes to servers... I wish admins would do a simple (vanilla connect) to their servers after propagation to check for misdirected files.uxx because these servers probably only get 1/3rd of the traffic they could otherwise get with faster (down/uploads).
@Sam ; ComboGib has one file missing BTW.
here's a simple tip to check server redirected files easily.
after setting up your server with all mods etc , rename your /Cache folder to Cache2.
then reconnect to server... keep eye on downloads and if they slow down then either the file 'Downloading' or the previous (probably tiny) file needs to be Redirected.
this ensures that wanderers will even attempt to see what your server is like.
once there though , is up to you in keeping them there..
Re: CTF-w00tEternalCave.unr
billybill wrote:Problem is they pay $15 a year for a gameservers redirect thinking they are in safe hands. Reality is gameservers.com host servers for 100s of games and their support for UT is crap. The files only get moved to the redirect after they are requested, and are moved back at some point afterwards. It means if a server has 5 serverpackages the player would have to reconnect 5 times to get a max download speed for each file. Add to that the many maps. It's a joke and not worth $0.15. You can see how it happens though, good advertising, and the admin can then give ftp access to multiple people and think they don't have to worry about redirection ever again. *sigh*Wises wrote: one thing I will say when it comes to servers... I wish admins would do a simple (vanilla connect) to their servers after propagation to check for misdirected files.uxx because these servers probably only get 1/3rd of the traffic they could otherwise get with faster (down/uploads).
- Chamberly
- Godlike
- Posts: 1963
- Joined: Sat Sep 17, 2011 4:32 pm
- Personal rank: Dame. Vandora
- Location: TN, USA
- Contact:
Re: CTF-w00tEternalCave.unr
^ in these cases , perhaps you could register for something like this: https://www.x10hosting.com/
and create a /subfolder/ and upload the utfiles.uz to , then in your UnrealTournament.ini under the RedirectToURL://blah.x10host.com/redirect/
Problems as we all know with redirects is of course file Mismatches etc. So each Server should really have its own /subfolder/ and files.
perhaps...
and create a /subfolder/ and upload the utfiles.uz to , then in your UnrealTournament.ini under the RedirectToURL://blah.x10host.com/redirect/
Problems as we all know with redirects is of course file Mismatches etc. So each Server should really have its own /subfolder/ and files.
perhaps...
Re: CTF-w00tEternalCave.unr
billybill wrote:Yup, I'd advocate anyone with gameservers to instead use the 100mb webspace they give you, and add -lanplay to the commandline.
Of course not going to work with bunny track servers that have 5000 maps, in that case you can get a good one for like $12 a year elsewhere. Compressing and upload the files yourself isn't all that hard... give redirect ftp access to same people you give server file access
Innovative idea by gameservers but terribly executed... It does deserve a better look into. Why wouldn't they have it compress them the moment they're uploaded rather than when they're requested? The client does not wait around for it to be compressed it's only going to create problems
Re: CTF-w00tEternalCave.unr
well in short yes you can get some good webserver space via web for less then $50 year . here is where you can compress and upload your redirect files and host forums etc.
the domain name you can get free for something like .tk
and the rest is pretty easy.
the domain name you can get free for something like .tk
and the rest is pretty easy.