PHP UPackage

Discussions about Coding and Scripting
User avatar
Feralidragon
Godlike
Posts: 5477
Joined: Wed Feb 27, 2008 6:24 pm
Personal rank: Work In Progress
Location: Liandri

PHP UPackage

Post by Feralidragon »

I just want to drop here an update to the PHP UPackage class I made:
uPackage-0.2.zip
Features:
- validate version and signature
- get dependencies
- get external classes
- get internal classes
- get class tree
- get GUID
- get objects (with query)
- get objects properties*
- get textures
- get palettes
- get sounds

* class and structures are not completely supported yet

This could be useful to process UT packages online in a PHP web server or so for any reason (to extract the info and screenshot of a new map uploaded, extract sounds from a package for preview, check whatever properties of an object, etc).
You do not have the required permissions to view the files attached to this post.
ShaiHulud
Adept
Posts: 446
Joined: Sat Dec 22, 2012 6:37 am

Re: PHP UPackage

Post by ShaiHulud »

Excellent, thanks Feralidragon - sorry for the "fluff" post, but wanted to note my appreciation :)
Last edited by ShaiHulud on Sun May 19, 2013 5:27 am, edited 1 time in total.
JackGriffin
Godlike
Posts: 3767
Joined: Fri Jan 14, 2011 1:53 pm
Personal rank: -Retired-

Re: PHP UPackage

Post by JackGriffin »

This is worth a fluff post. It's a big improvement in potential handling of files by webservers.
So long, and thanks for all the fish
User avatar
Feralidragon
Godlike
Posts: 5477
Joined: Wed Feb 27, 2008 6:24 pm
Personal rank: Work In Progress
Location: Liandri

Re: PHP UPackage

Post by Feralidragon »

I had big plans for this class in the past year when I created it, but they didn't come along due to lack of time.
So I decided to take another look at the class, fix the last remaining bugs and release it with some further functions, just in case this class ends up being useful to someone else. :)
User avatar
papercoffee
Godlike
Posts: 10240
Joined: Wed Jul 15, 2009 11:36 am
Personal rank: coffee addicted !!!
Location: Cologne, the city with the big cathedral.

Re: PHP UPackage

Post by papercoffee »

Sooo... this is a UPackage filter system ??
I don't quiet get it. sorry.

It sounds interesting but ...I doesn't understand it.
User avatar
Feralidragon
Godlike
Posts: 5477
Joined: Wed Feb 27, 2008 6:24 pm
Personal rank: Work In Progress
Location: Liandri

Re: PHP UPackage

Post by Feralidragon »

It's a class for PHP developers which is able to read and extract info from UT packages.
Example of usage:
- a PHP developer builds a UT community site, where you can upload files for UT (mods, maps, whatever);
- with this class said developer can also get extended info on those, such as: which external packages it needs to run, if the package is truly a UT package and not a fake one, extract info on which classes it has and which classes it uses from other packages, etc...

In other words, for the normal user/player/admin, what I posted above is simply useless lol, but for a web PHP developer, it may come in handy to do something with files stored in a server or upload of such, as so far I don't know of any PHP script or class which does the same.

I also intend to add extended functionality such the ability to extract textures and sounds later on, not for ripping purposes (although it may get used for such... but then again people can do that already more easily with other tools), but the idea would be to use such extraction to be able to give an automatic preview of the package in a website (a texture package could have 4 of its textures extracted so the user who's going to download it knows the kind of textures to expect from it, the same for sounds, the same for maps if the texture extracted is the map screenshot itself, and who knows, also to do something close to what Google does with color themes in image searching).

Maybe no one is going to use it, but I am actually having some fun in doing it (and in end I may end up doing something with it just for the lulz, mostly to check how it would turn out).
User avatar
Feralidragon
Godlike
Posts: 5477
Joined: Wed Feb 27, 2008 6:24 pm
Personal rank: Work In Progress
Location: Liandri

Re: PHP UPackage

Post by Feralidragon »

UPackage update: 0.2 (see first post)

Well, can you guess what this is?
Screenshot-CTF-Face.png
It's the screenshot of CTF-Face extracted using this PHP class alone.
Change log:
- added: get objects (with query to filter, limit and offset on the amount of objects)
- added: get object properties (such as the description and author of a level by querying the LevelSummary and then getting its properties)
- added: get texture (you can now extract textures, with any mipmap value, however it's saved as .png instead of .bmp/.pcx)
- added: get palette (if you want to know all the colors in a texture)
- added: get sound (to extract sounds)
- major optimizations on the whole class code (it should now process things even faster). The textures are the slowest to process (since it's more tricky since I have to deal with the image at pixel level), but it should be fast enough (I tested to extract 900 textures of one package, most of them 128x128, and it took around 9 seconds to do so, which is an average of 0.01s per texture).

Again, not sure if this is useful for someone, but I am having fun doing it anyway lol
You do not have the required permissions to view the files attached to this post.
User avatar
Dr.Flay
Godlike
Posts: 3339
Joined: Thu Aug 04, 2011 9:26 pm
Personal rank: Chaos Evangelist
Location: Kernow, UK

Re: PHP UPackage

Post by Dr.Flay »

:tu:
I'll have to see if my programmer dude can use this.
User avatar
Dizzy
Experienced
Posts: 109
Joined: Tue May 21, 2013 3:57 pm
Personal rank: Oaf

Re: PHP UPackage

Post by Dizzy »

Seven years later this is still very useful. Thank you Ferali.
Join the BunnyTrack.net Discord chat server: https://www.bunnytrack.net/discord
User avatar
Feralidragon
Godlike
Posts: 5477
Joined: Wed Feb 27, 2008 6:24 pm
Personal rank: Work In Progress
Location: Liandri

Re: PHP UPackage

Post by Feralidragon »

Damn, 7 years, huh?

This is one of those projects I would like to redo someday, and do a more serious package, and do it properly, because this 7-year old code.... let's just say it's not representative of my code nowadays.

But I am glad it ended up being useful for someone, although I am not sure how compatible this code is with the most recent PHP versions, but I guess it still works. :)
Sapphire
Posts: 2
Joined: Tue May 16, 2017 8:17 pm

Re: PHP UPackage

Post by Sapphire »

@Feralidragon I'd like to say a huge thank you as well. I recently became interested in extracting UT package data as I wanted to simplify our (mine/Dizzy's) server upload process for new files. I followed your code very closely (took me a while to get my head around it as I've never done any "low-level" stuff before) and I've created a JavaScript plugin which is virtually the same as your PHP package, plus a few extra functions: https://github.com/bunnytrack/UTPackage.js/

I'm sure in a few years I'll look back at it and also say it's not representative of my then-current programming skills (in fact, I can already see quite a few areas for improvement), but it's not an exaggeration to say that I've already learnt a ton about PHP/JavaScript/file structures/programming/etc etc during the past few weeks doing this. Thanks again! :tu:
User avatar
Feralidragon
Godlike
Posts: 5477
Joined: Wed Feb 27, 2008 6:24 pm
Personal rank: Work In Progress
Location: Liandri

Re: PHP UPackage

Post by Feralidragon »

I'm glad it was that useful to you.

If I am able to someday (soon-ish) finally finish my current PHP framework, I may try to redo this PHP package properly (especially using specific PHP functions actually tailored to this sort of thing that I didn't know existed back then).
arden
Posts: 2
Joined: Thu Dec 29, 2022 3:47 pm

Re: PHP UPackage

Post by arden »

Thanks for this! I was starting to write my own php script and finding it a pain! so I'll give this a go and hopefully have a more stable one. The plan is to index all my UT files and create links (dependency’s) between them, so they can be easily groups and installed together, I'm planning to make the site public so needed everything to be processed server side, via the php scrips.

On a side note, I see the package version is only for 69, do you think you could include support for the other unreal file versions? up to 128 atleast? I'm not sure what version this is but it's what I've been working off https://utfiles.lagout.org/UEditor_Deve ... format.pdf

Do you mind me asking what reference material you used for the file formats?

Also, have you considered putting this up on github, so many good resources have been lost over the years due to sites going offline, would be a pity to loose this, and maybe some outside contributions in the future!

Thanks again!
arden
User avatar
Feralidragon
Godlike
Posts: 5477
Joined: Wed Feb 27, 2008 6:24 pm
Personal rank: Work In Progress
Location: Liandri

Re: PHP UPackage

Post by Feralidragon »

This package is almost 10 years old, and I haven't been maintaining it at all.

I already thought in redoing it completely, since my own PHP skills improved significantly, which also means that the code from this package isn't that great, but I am still in the process of building a different PHP package (unrelated to this) so I never had the opportunity to get into this one again.
arden wrote: Thu Dec 29, 2022 4:02 pm On a side note, I see the package version is only for 69, do you think you could include support for the other unreal file versions? up to 128 atleast? I'm not sure what version this is but it's what I've been working off https://utfiles.lagout.org/UEditor_Deve ... format.pdf

Do you mind me asking what reference material you used for the file formats?
I only focused on UT99, without any intention of supporting the others.
The document you linked is exactly the document I used as a reference, and from there I also did my own reverse engineering to figure other things out at the time.

Having that said, Shrimp created UnrealArchive: https://unrealarchive.org/ , which is a site that is automatically generated times to times from the data from the packages themselves.
In other words, Shrimp has much better and complete scripts/libs of his own to parse packages and retrieve info such as dependencies and other things, and they also support UT2004 and UT3 packages, so you may want to contact him instead so you can get more complete and up-to-date info on how to do it.

I think it's in a different language other than PHP (Delphi I believe, at least I recall the original libs being in that language), but they could offer a good clue on the logic and how these are parsed.
arden wrote: Thu Dec 29, 2022 4:02 pm Also, have you considered putting this up on github, so many good resources have been lost over the years due to sites going offline, would be a pity to loose this, and maybe some outside contributions in the future!
I haven't because I stopped maintaining it completely 10 years ago, and 10 years ago I didn't even use Git, so I never really considered it.
But, you can consider this to be public domain and host it yourself if you like.

It's not the first time something I created gets hosted in github by someone else in order to maintain it, so it's fine, as I don't intend to maintain it myself. :)
User avatar
Shrimp
Skilled
Posts: 227
Joined: Wed Oct 10, 2018 11:15 am
Location: Australia

Re: PHP UPackage

Post by Shrimp »

Feralidragon wrote: Sat Dec 31, 2022 3:27 pm Having that said, Shrimp created UnrealArchive: https://unrealarchive.org/ , which is a site that is automatically generated times to times from the data from the packages themselves.
In other words, Shrimp has much better and complete scripts/libs of his own to parse packages and retrieve info such as dependencies and other things, and they also support UT2004 and UT3 packages, so you may want to contact him instead so you can get more complete and up-to-date info on how to do it.

I think it's in a different language other than PHP (Delphi I believe, at least I recall the original libs being in that language), but they could offer a good clue on the logic and how these are parsed.
Unreal Archive uses "unreal-package-lib". This library is written in Java, and supports UE1, 2, and has basic UE3 support. Unfortunately since it supports all the engines its usage got a bit more complicated and it needs a bit of an API redesign at some point.

The docs directory contains some references including the original PDF everyone uses.

I also made a dependency tool, "unreal-package-dependencies", used by Unreal Archive, which is used to resolve dependencies between packages.

It's all on Github under MIT, so feel free to use it, change it, or rewrite it in your favourite language.
ShrimpWorks
Unreal Archive - preserving over 25 years of user-created content for the Unreal series!