ZIP and other old standards

[Dr.Flay]

Most of you can skip straight to the PROs and CONs section

Why do we still use a format with a locked development, and no extra abilities ?
Because of a "Catch-22"

It is only the standard in OSs because it was popular, not popular because it is a standard part of an OS.

Spoiler

If RAR had been more popular, then that is what the OSs would all support.
RAR was also developed with OS integration in mind, and Microsoft and others, are welcome (and very capable) of integrating the free RAR.dll and UnRAR.dll into their OSs, just the same as the ZIP.dll.

Until people swap to using modern (and still developed) compression, the OS makers have no need to support it.
Therefore the educated amongst us, have to drag the industry and users forward.
The gap between the power at your finger-tips, and the ability to use it is growing, because too much is hidden from the user.
The industry loves this general level of ignorance, because it means it call sell you stuff you already have, or can have for free.
Look online and you will find many crappy companies selling ZIP-based programs, even though no one theoretically needs them as it is built in to the OS.

I imagine that in 50 years time people will still be using the old dead ZIP format, simply because they don't have to think about it.
This is the same issue as video and audio formats included in any OS that fail to be updated.
People still make GIF and single-pass AVI files because they are "standard" and the default settings work, yet fail to think that new better standards have already arrived (many years ago), but are only not yet in standard use, because of the lack of use. Again "Catch-22".
MP4 video finally became used as a standard because of mobile devices and people finally realised how good it was for streaming.
Most video you see now streaming online or in a flash player is 2-pass VBR H264/AVC MP4.
Basically everyone can play this format on almost anything, but when most people rip a movie they still use the old, non-streaming AVI container format, that does not support AAC audio
WHY ?
Oh something to do with standards.

Why has it taken so long for the world to treat PNG as standard ?

Spoiler

Until recent years developers still had to pay licence, for adding GIF saving to their programs, even though the free replacement (written by the same guy) has been available since 1996.
The Amiga world took to it straight away, as most Amiga users are intelligent and handsome/beautiful with excellent taste in everything. They are also great cooks and lovers.
Every program supported it, and it was added to the OS datatypes, so if your program did not support it natively it did not matter.

The Apple world also took to it, as they are like magpies with a shiny new thing, which is why Mac owners had proper transparent PNG support in their browsers (even Internet Explorer) way before Windows users.

The Linux world took to it, as they are a wise and are used to comparing many choices. Indeed many of the Amiga PNG CoDecs were Linux ports.

In the Windows95 world, they were still trying to catch-up with a new era of multi-tasking.
I remember tying to explain to a guy who taught IT, why multi-tasking was a good thing. He did not get it. He kept insisting "but I can only do 1 thing at a time"
PNG support in the Windows world seems to be held back by peoples experience with it in Photoshop.
This is not based on my opinion, but what I have witnessed with the list of programs and level of support, on the official PNG site.
Every version of photoshop I have used has been very lacking in PNG options, so to your average user they would have no idea of the benefits.
The majority of peoples graphics experience and expectations is based around photoshop, so they will never know any different.
Most users also are unaware that PNG 24/32bit is the replacement for TIFF, so because people don't use it instead of TIFF, it will never replace it. "Catch-22".

I remember the dawn of Flash and Java.

Spoiler

I remember sitting in the boardroom while an argument raged around me, about whether we should be using these non-standard things in our web-sites.
There were (and are still) alternatives. I still prefer REBOL to Java and Perl (mostly because I can understand it better, and it is smaller).
Both these things are still optional extras for a browser, requiring a user to DL something. This is nothing new of difficult for users.
Requiring a user to install RAR or 7zip, is nothing new, special or difficult (especially as they are ad-free sites and programs).
All PCs made by the company I started with, and ever since, have all been pre-installed with WinRAR.

Unless your distro is intended to be used by someone who needs help to operate a PC, I recommend treating ZIP as a curiosity of yester-year, as the type of person getting the file is probably very able to cope.
If it is important that a file be openable by a chimp, then you can always make SFX files with whichever archiver you choose.
Final distro for an end user, and an archive intended for a power user can have different requirements.

PROS

Spoiler

Integrated ZIP:
Almost all OSs have native support. No software necessary to unpack.
New algorithms have been included in Windows Service Packs (sometimes) and full ZIP programs.
Available as a DLL
A Chimpanzee may be able to operate it.

7zip:
Multiple algorithms for different datatypes.
Multiple base routines for different datatypes.
Huge dictionary sizes allow huge compression rates.
Can make custom ZIP files with higher compression.
Simple interface.
Good OS integration.
Easy to make self-extracting files.
Opens the other formats.
Still developed.
Available as a DLL
A small child can operate it.

RAR:
Multiple algorithms for different datatypes.
Large dictionary sizes allow better compression rates.
Simple and customisable interface.
Very good OS integration (format conversion can also be switched on).
Browses the filesystem faster than Windows Explorer.
Integrated Virus checking (Uses your AV via command line).
Easy to make self-extracting files.
Opens the other formats.
Easy format conversion.
Recovery records allow complete reconstruction of damaged files (up to 10% can be missing or damaged).
Very regular development.
Available as a DLL
A small child can operate it.

CONS

Spoiler

Integrated ZIP:
Uses 1 algorithm at a time.
OS integrated versions only get updated in a service pack.
Extra algorithms rarely used due to backwards compatibility with OSs
No recovery records. You can only salvage some of the undamaged data.
Only opens ZIPs
4GB limit (unless you create ZIP64 files)
No virus checking.

7zip:
You have to install it.
Huge dictionary sizes require huge amounts of RAM to pack and unpack. This can make some files unable to unpack on low RAM PCs.
Can make custom ZIP files that only work reliably in 7zip.
The choice of algorithms is confusing, and you can end up with files that only work in 7zip.
The very simple interface is not flexible.
No recovery records. You can only salvage some of the undamaged data.
No virus checking.

RAR:
You have to install it.
You may have to tell it where your AntiVirus is.

In my opinion, I think it is ludicrous that all of the common compression routines are not installed in Windows, and updated each month
But then obviously you are less likely to use one of Microsoft's own formats.

On the Amiga OS, we regularly add and update all the OS CoDecs, compressors and encryptors, so all programs that use them are equally up-to-date.
The Windows and Apple world will never be like that, so we all have to educate each other about "standards" and when to use them.
and what makes a standard, standard ?

If sticking with standard is things like;
Only using ZIP, Only using the software that came in the box, or "pushed" from a website, Not using Hi-Res or S3TC textures in UT, Not using Unreal 227i, and not using transparent PNGs, then the "standard" is rubbish

[Feralidragon]

Isn't that pretty much the same story for every format in existence?
If a format isn't used at a large scale, it's not supported by many, only the ones which cover a significant part of the systems are the ones receiving support.
There are thousands of formats better than their standard counterparts, but if they're not used in a larger scale at any given point, they're not even taken into account.
Plus, some of the new formats were created because of the licensing of the old ones or because some of them weren't open, and the other way around is also true (use old because the new is closed), like the RAR format.

Spoiler

The GIF only remained because there was a way back then to produce a GIF file without violating the license which protected it: basically the uncompressed GIF format (limited to 128 colors though iirc).

Furthermore there's a huge problem with new formats you are somewhat neglecting in your post: the fact that new formats impose new standards, and the biggest problem of all in the whole world of information is that there are too many standards already, and there's always someone or some entity who comes and thinks has a better format and standard than everyone, and in the end only populate the world with more standards, and in the end is a real mess.
That's why I always stick with actual standards which are supported worldwide for a long time, thus what we need is a full on cooperation between the different entities to just use 1 of them and stick with it, and when the need comes, implement a new one also together aimed to expand the previous one and fix any design problems the old one had.

That's what HTML5 is trying to achieve besides adding functionality and fixes, the same with open document formats which MSOffice is also starting to save as, even video had tons of used formats before and people are starting to stick with just a very few (mainly mp4, mkv and avi), etc, etc... and they're working.

As long there isn't an absolute need for a new one or use an already existing one, no one will use it, because believe it or not, adding support for more formats has high costs, it's not that easy to keep things up to date and still working at all, to not mention that the ones making the updates are often the ones screwing everything by deprecating and removing entire features later on which make the previous saved files unreadable, thus there's a need for porting and conversion as well, which makes the whole thing a huge messy snowball of huge costs, monetary and time-wise.
In fact, one of the teams projects in the place I work on is exactly fighting the trend of multi-standardization (can't enter the details, but it's something to solve the problem with having too many formats), and it's a very extensive project dating a couple of years now and which has many implications worldwide, however a whole team is assembled to it alone, and it's not easy at all.

In conclusion: when it comes to ZIP, it is supported in every single platform, even Android packages are stored in ZIP format.
RAR could be supported, but the huge problem with it is that it is *closed*, meaning no one besides the owners know the algorithm to generate it, which also means that the moment they decide to drop the support we're fucking screwed (pardon my English), whereas with ZIP such problem is inexistent since the format is open for anyone to use and create compressors/decompressors for, either now or within 50 years with quantum computing, since the algorithm is always the same and is *known*, and thus only open formats should be supported in a worldwide scale, not the closed ones.

So if someone asked me "should all OSs and other entities support RAR out of the box?", I would answer a straight "NO", as long it keeps being a closed format.
Sure there are closed formats supported in all the OSs out of the box, but I don't agree with them either, but it's understandable why they are there: they became too popular. But yet, flash is also popular and it still has to be installed afterwards in most platforms.

Perhaps I digressed a lot, but this is my take into the issue.

[papercoffee]

Most users also are unaware that PNG 24/32bit is the replacement for TIFF

But I never did see that PNG can support layers. How can I use it? because I love to use PNG files they are much better than every other file type. Except for PSD what is my working file.

7zip:
You have to install it.

I use 7zip portable no real installation needed.

[Dr.Flay]

Nice post. Great reading
My background with the Amiga has "spoiled" me, as I expect more from a computing environment.
We never worried about standards, as you just add new features and everything can use it. Therefore standards become almost meaningless.

Spoiler

Indeed I obviously agree, but I just stuck to 3 common archive formats for simplicity.
And as I point out, a standard is only truly standard if the majority use it, not because it is a proposed standard.
That just makes it common, not best fit for purpose (I simply believe in using the right tool for the right job).
I don't use the most popular phone, I use a sports phone, as it is waterproof and shock-proof.
I don't drive a car, I ride a motor-bike, as Cornwall has narrow bendy roads.
These things are based on my need, in my current environment, with a view to how well it will keep serving me in the future.
They have their PROs and CONs. Everything is a compromise.

It does not seem to be difficult or expensive to add any compression or CoDec support to the AmigaOS, and in all these years there is yet to be a major balls-up that breaks it.
Standards can be implemented in standard, or non-standard ways.
We use a master compression library, that the sub-types (RAR,ZIP,etc.) bolt onto. All software can use the standard master library (we use the same principal for all datatypes, and even the customisable GUI.)
We don't put library files in the program directory. They go in a standard folder, so all programs can use a "shared library". The OS has always been "object oriented", so all Amiga programmers are aware their shared code needs to be robust.

In Windows we find the "standard" implementations of ZIP and PNG etc. multiplied in many program folders. All media suites install their own version of the standards, therefore individually they break the idea of standard.
They are often at different versions that are not backwardly or forwardly compatible.
I know the problems faced by developers with all these competing standards, as they often have to be implemented differently.
The Amiga world got round those problems years ago, by having standards thought about differently.

eg. on the Amiga all AntiVirus can use the same standard virus definitions, CoDecs and file-info libraries. So any AV will open and scan the same files. If you are lacking the ability to open or scan a certain file, you just add it to the OS. You never have to wait for the developers to release a new version of the AV. If 1 AV company picks up a virus before the others it does not matter.
Imagine that on Windows ! Imagine updating your favourite AV with the pooled definitions of all the AV companies, via VirusTotal.
Never going to happen, because on Windows, "standards" compete for each-others territory, not try and find their own place.
It's exactly the same for packers/archivers. We don't use "AmiRAR" "AmiZIP" etc. You choose the archiver software you like, and it will read and write all the formats installed in the OS.
Many programmers include native routines, with fall-back to the OS.
Implementing standards at OS level, means you can easily do non-standard things by bolting together ideas.

The Windows world just makes things hard for its self (one of the reasons I left the industry).
The file-structure of Windows is its biggest enemy regarding standardisation.
Originally all the system files went into the Windows folder. That got too cluttered so they made a System folder. That got too cluttered so they made a System32 folder.
Now they install many shared libraries into their own special folders. The way Visual C and .Net are installed is a total mess.
Windows should have simple separate folders for Devices, Libraries, CoDecs, Compressors, and Datatypes.
So-called "shared" libraries should be easily shared. If something breaks, it is detected, reported and fixed faster.
Often updating a file in windows will only show a problem in 1 program, so until you open that 1, or do that particular task, it will go unnoticed for a while.

As you point out, often the developers are the ones who break their code
Currently there are 7 versions of "libpng" in my Programs folder, all with their own unique names, and I know they are not compatible.
That is because each of them implemented the PNG source in their own special way.
PNG is carefully designed to always be backwards compatible, On the Amiga it is. We never have any problems because the implementation is truly OS standard.
On Windows we have variations of a single standard, so the ZIP or PNG decoding in a VB, VC, or .net etc. program will be standard for that programming environment, not the OS.
There are 18 different ZIP decoder DLLs of the same "standard", under many names on my PC, all in different folders, all compiled differently, but all do the same job.
If it is so standard there only needs to be 1 good 1 installed in the OS. (though I favour 2
Disk and RAM usage is improved massively when people share the same libraries. Old software can be kept uptodate and even support new formats.

Yeah I am aware that almost every skin, theme, and "package" for everything is just a renamed ZIP.
But invariably the software that uses it, will be using its own DLL in its own folder, ignoring any OS support.
So it could really use any they choose.
WinRAR allows you to add all these custom file extensions, so 1 program easily opens them all.

Interesting note about low-colour GIFs. Didn't know about that.
The loop-hole some of the commercial Amiga publishers used, was that you were allowed to give it away for free. They just provided you with a free GIF plugin after you bought the main product.

@ Paper
No, single PNG doesn't do layers, but you can use the Anim format MNG to store the layers.
Unfortunately most developers have not thought to use it that way, and have tried to develop their own Layered-PNGs (which are obviously non-standard and only work in that program).
The data stored in an MNG, can be displayed however the programmer wishes. An MNG is just a collection of images, palettes and timings.
So as a layered-TIFF replacement, you have 32bit alpha channels and as many layers as you want (works great on the Amiga).
ImageMagick can also do it http://www.imagemagick.org
My main problem with PNG, is the lack of EXIF support.

GIF however is better for tiny things like tiny 8 colour smileys, as below a certain size the file-header becomes the bulk of the file.
I lost a competition with a developer at Opera. We were trying to see who could make the smallest files.
We picked a smiley from their site, and we both threw every optimizing trick and tool at it. The image was not allowed to look any different from the original. Eventually the GIF won. There was not much difference but we were just trying to push the limits of GIF and PNG.

That wasn't quite what I meant by install
"Find and download" would have been better ? Just making the difference between pro-actively going to a site, or getting something via an update or pushed via a site (like flash and java etc.)
Either way I heartily approve of using the no-install USB style distros (though you still have to unpack them anyway ).


There is no perfect solution (on Windows), but RAR has the best reliability out of the 3 compared above. If 7zip and ZIP had recovery records, I would actually favour ZIP, as 7zip often makes files that only it can open.
I did not bother with the ACE format, because even though the compression can be better than 7zip, and it has recovery records, WinAce is a horrid piece of software.

[papercoffee]

when you grow up with something like ARJ is nothing too complicated for oneself. If I can learn a new trick am I more than happy.
So ...how can I make a MNG ...I didn't notice anything like this in Photoshop. Is there a additional tool for PS available?


If you say install, will I imagine an actual "install".

[Wises]

i rememver years ago a format for sound compression supposedly better then .mp3 which never seemed to get off the ground.. by the name of .vqf