UnrealArchive.org

Share interesting stuff you have found or created yourself.
noccer
Adept
Posts: 376
Joined: Sun Aug 01, 2010 12:15 pm
Personal rank: Proud Terrorist

Re: UnrealArchive.org

Post by noccer »

Damn, that looks great.
As one of my old friends said, sharing is caring.
I dont know what i find best, the design, the mirrors, the self hosting... simply the mirrors are great. And fine that you do not hotlink files!
How did you get all the mappers and map names in the rows? That was some idea i had in mind ages before but was never would be able to do myself :D
A pity i cannot help with files anymore. However, great work and a good way to go <3
Image

>>You can't steal any ip (v4)adresses, there are exactly 4294967296 of them, and they will still exist when you wrote down all of them, or are stored in a (master)servers database ;)<<
User avatar
Feralidragon
Godlike
Posts: 5498
Joined: Wed Feb 27, 2008 6:24 pm
Personal rank: Work In Progress
Location: Liandri

Re: UnrealArchive.org

Post by Feralidragon »

That is really cool, really nice design overall, fast and easy to navigate. :rock:
It still lacks some content, which is to be expected at this stage, but that will certainly come with time and community contributions. :D

I also intend to do something similar, but in a different way and a lot more selective on the actual content to serve, but unfortunately I was unable to advance with it thus far, and for the time being, repositories like this one is something really needed, the more redundancy of these places we have, the less likely it is for anything of worth to be lost forever.

:gj:
User avatar
FraGnBraG
Inhuman
Posts: 930
Joined: Sun Jun 13, 2010 5:13 pm
Personal rank: Good news everyone!
Location: Canada

Re: UnrealArchive.org

Post by FraGnBraG »

Good job over all. :tu:

how did you populate the meta-data? I wonder since - for example - there are several maps with "unknown" for the author (including one of mine), and what looks like a "guess" at release date of maps... long ago there was a euro fella who created a monster site that was highly auto-magic (varpu fyi) ... this makes me think of that a bit...

curious... :)
-=FraGnBraG Level Design=- ***UPDATED! even works on your phone!***
User avatar
Shrimp
Adept
Posts: 305
Joined: Wed Oct 10, 2018 11:15 am
Location: Australia

Re: UnrealArchive.org

Post by Shrimp »

darksonny wrote:Thank you dude, its a huge effort making this nice collection project! and many of the maps does have some picture to preview it before downloading them. Could be nice (maybe this is a wip process or still unfinished task) all maps could have any pic mainly those funny author who posted a nonrelated preview of his map (pictures like bugsbunny or the logo clan, those are they uselees info that does not helps ut fan to decide wether download or not).
The map pics are extracted automarically directly from the Screenshot property set in the map's LevelInfo. Unfortunately if they didn't put a picture in there, there will be no pic. I can't play 30 000 maps and take screenshots of all of them unfortunately :idea:. Sometimes, if the author put some .jpg or other images in the archive, it will automatically use those too, which is why some maps seem to have a lot of screenshots while others only have one.
papercoffee wrote:I noticed you didn't got all mapping contest packs from UT99.org ...I pinned them to our mapping Discord channel.
https://discordapp.com/channels/2267363 ... 7467059205 (the pin symbol on the right upper corner)
If you prefer those downloads via dropbox PM me and I upload them there.
Thanks. I actually downloaded the individual maps when the contest happened, just hadn't added them yet. You'll note that the lead image for "Map packs" on the side actually contains all the contest entries :rock:. Seemed appropriate.

I've added the pack now, but I haven't looked at how to classify BRUT yet so its not there individually but I'll figure it out.
JackGriffin wrote:I probably need to talk with you about uploading my stuff and what gets uploaded. Over the years I've siteripped everything I could get into and downloaded all the files I ever came in contact with. The things explicitly sent to me to private test I've always placed in a distinct folder so I didn't mistakenly share them but the ripped stuff contains all kinds of unreleased content, beta files, test versions, etc. There's for sure things in there that were not meant to be public but I don't know which is which. I just saved everything.

What are your feelings about files like this? Do you want to just mirror everything and remove stuff if the author gets upset? I can work through it and guess at what should be archived safely but if I do that the community will lose out on a bunch of content that's valuable to learn from as well as history of mod development. I also have a lot of anticheat sources and aimbots, etc. What about those? What about bytehacked files? It can all get a little muddy depending on the level of 'community archive' you want to host.

If you want to discuss this privately just drop me an email. I'll explain more in depth what I have saved. The easiest way to visualize it is that many websites owners are gamers and not web designers. The majority of them think that their files are private when they aren't and someone with a crawler is able to grab all over them. Which I did. A lot. :lol2:
As mentioned, thanks for prompting this whole thing. I really got me thinking, and this is currently the result.

I know what you mean about finding stray stuff :D. I found myself in forgotten university FTP servers probably left over from when people used to play UT there, probably accidental Apache auto-index lists, etc, and was able to find stuff :). I can't complain, it was certainly helpful :mrgreen:.

With regards to the files you have though, I'd be interested in grabbing as much of it as possible and then indexing what I can for the archive. If there are random .uz files, or other things that don't look like packaged/released things, my indexer will pass them over, but slurp up everything else it can identify. Its probably easiest if I could download it form you somehow, I haven't had much luck having people upload to me (too slow). I'll PM you this weekend and we can discuss.
Feralidragon wrote: It still lacks some content, which is to be expected at this stage, but that will certainly come with time and community contributions. :D

I also intend to do something similar, but in a different way and a lot more selective on the actual content to serve, but unfortunately I was unable to advance with it thus far, and for the time being, repositories like this one is something really needed, the more redundancy of these places we have, the less likely it is for anything of worth to be lost forever.
As far as lack of content goes, it's definitely lacking a lot of the more recent stuff (and definitely a lot of old still). I've focused on just scraping as many of the existing repositories as possible, which mostly contain things from up to a few years ago. It seems newer stuff is either "released" directly into caches on servers, or as forum attachments/dropbox uploads, which makes collecting it rather difficult.

Once I get uploads figured out, hopefully things will improve and people can submit their own back-catalogs 8).

When you build your own thing, even if it has better curated or more selective content, feel free to make use of the metadata or steal package reading code, images, file lists, etc if it seems to be of use to you.
FraGnBraG wrote:how did you populate the meta-data? I wonder since - for example - there are several maps with "unknown" for the author (including one of mine), and what looks like a "guess" at release date of maps... long ago there was a euro fella who created a monster site that was highly auto-magic (varpu fyi) ... this makes me think of that a bit...
So, for maps, the authors are pulled directly from the LevelInfo within the maps using my Unreal Engine package reading library (https://github.com/shrimpza/unreal-package-lib). "Unknown" authors typically happen if I can't read the properties form LevelInfo for whatever reason (bugs :noidea: :mrgreen:), or there are no names (seem to be a lot of maps people don't fill out that info). For content that's not maps, where no LevelInfo or dedicated "Author" information is available, I try to look in README files, which generally results in a bit of a mess, but does sometimes get it right. Release dates are definitely guesses - the best approach I found was simply to use the modification date of the content/map/etc. So if someone modified your map, or anything else changed the last modification date (like it was zipped by someone whose system time was wrong), you'll get an inaccurate date.

Please let me know if you spot incorrect authors (just send me links to the content page, and the correct author name), I'll happily correct them :help:. This will definitely help for when I build the "Browse by Authors" page.
ShrimpWorks
Unreal Archive - preserving over 25 years of user-created content for the Unreal series!
User avatar
UnrealGGecko
Godlike
Posts: 2975
Joined: Wed Feb 01, 2012 11:26 am
Personal rank: GEx the Gecko
Location: Kaunas, Lithuania

Re: UnrealArchive.org

Post by UnrealGGecko »

I did notice a handful of maps where the author doesn't fill in the LevelInfo>Author tab but instead puts his name on the Screenshot itself.
User avatar
smeerkat
Skilled
Posts: 151
Joined: Sat Apr 04, 2009 6:00 pm
Personal rank: DHR
Location: Canada

Re: UnrealArchive.org

Post by smeerkat »

Bookmarked! Should this be a sticky thread (or added to an existing sticky)? 8)
User avatar
UnrealGGecko
Godlike
Posts: 2975
Joined: Wed Feb 01, 2012 11:26 am
Personal rank: GEx the Gecko
Location: Kaunas, Lithuania

Re: UnrealArchive.org

Post by UnrealGGecko »

smeerkat wrote:Bookmarked! Should this be a sticky thread (or added to an existing sticky)? 8)
Good point, wash your hands before posting from now on :thuup:
Red_Fist
Godlike
Posts: 2207
Joined: Sun Oct 05, 2008 3:31 am

Re: UnrealArchive.org

Post by Red_Fist »

I think you have these but maybe not all, http://ut99ut.weebly.com/

And the only thing I can think for the great website you did, is put an arrow for Next page, so you can just click next not only the letters., and one at the bottom of the page as well.
Binary Space Partitioning
User avatar
Shrimp
Adept
Posts: 305
Joined: Wed Oct 10, 2018 11:15 am
Location: Australia

Re: UnrealArchive.org

Post by Shrimp »

Hey all,

Just a quick update; I haven't abandoned this or decided to ignore everyone with content and suggestions, just haven't had much time to get stuck into doing anything meaningful recently.

@Red_Fist, I imported your maps, thanks.

I also added ~800 additional bits and pieces scraped from the big BeyondUnreal dump from Archive.org.

I've been planning out the submission process, and have a sketch of how that'll work, just need to get stuck into building it.

In the realm of things I did get done, these little preview card things with custom descriptions for posting in Discord, Slack, and other places that support Open Graph. Mostly pointless but fun.

Image

Image
ShrimpWorks
Unreal Archive - preserving over 25 years of user-created content for the Unreal series!
User avatar
Hellkeeper
Inhuman
Posts: 914
Joined: Tue Feb 25, 2014 12:32 pm
Personal rank: Soulless Automaton
Location: France

Re: UnrealArchive.org

Post by Hellkeeper »

Shrimp wrote:Image
By the moustache of Stalin, I still have this installed in UT2004 and still use it sometimes, one of my best memories of the good old days. Almost brings a tear to my eye to see Arkon's weapon.
You must construct additional pylons.
User avatar
Dr.Flay
Godlike
Posts: 3351
Joined: Thu Aug 04, 2011 9:26 pm
Personal rank: Chaos Evangelist
Location: Kernow, UK

Re: UnrealArchive.org

Post by Dr.Flay »

Heh, I still have Arkon weapons installed also :tu:

You should probably add the other UT2004 patches
https://liandri.beyondunreal.com/Unreal ... 04#Patches
User avatar
esnesi
Godlike
Posts: 1050
Joined: Mon Aug 31, 2015 12:58 pm
Personal rank: Dialed in.

Re: UnrealArchive.org

Post by esnesi »

You guys knew https://unrealarchives.com/ right ?
That is not indexed yet.
User avatar
Shrimp
Adept
Posts: 305
Joined: Wed Oct 10, 2018 11:15 am
Location: Australia

Re: UnrealArchive.org

Post by Shrimp »

iSenSe wrote:You guys knew https://unrealarchives.com/ right ?
That is not indexed yet.
unrealarchives.com serves a somewhat different and more focused purpose - primarily a somewhat more hand-curated collection of single player content for Unreal and UT. I also debated long and hard on my choice of name, being so closely related to that existing site.
Dr.Flay wrote:You should probably add the other UT2004 patches
I think the only one missing is the x64 patch, which I'll add. You'll see on the patch page that all variations are listed for download, rather than a separate page for each. I've not included the specialised hardware patches since those devices aren't available any more, but I'll work on updating the contribution guide so someone can add them if they feel there's value in having them.
ShrimpWorks
Unreal Archive - preserving over 25 years of user-created content for the Unreal series!
Skillz
Novice
Posts: 8
Joined: Sun Dec 16, 2018 9:32 pm

Re: UnrealArchive.org

Post by Skillz »

I am so glad someone finally made something like this. Kudos Shrimp. Nice freaking work man.
User avatar
Dr.Flay
Godlike
Posts: 3351
Joined: Thu Aug 04, 2011 9:26 pm
Personal rank: Chaos Evangelist
Location: Kernow, UK

Re: UnrealArchive.org

Post by Dr.Flay »

Worth noting for all, GameFront have now added a re-upload feature on pages with missing files.
They must be the same name and size. No idea if they still have hash info for the lost files (doubt it).
They also provide a contact feature where you can give the admin a link to a valid copy, so they can do the upload.