On the very top of the Zip-Ada to-do list, for a while, was the support for archive containing large (more than 4 GiB data) or numerous (more than 65535) files – the so-called "Zip64" extensions.

It's now gone from the to-do list and done in the project.

The Zip archive format was defined with 32-bit file sizes and 16-bit number of files. At the time (1989) it was a perfectly sound choice, especially given that the software, PKZIP, was written for 16-bit PC's running on MS-DOS. At some point, these limits began to be problematic for archiving or backup tasks. Although the original Zip format is still the most widely used (including many file types without the .zip name extension, such as .docx, .xlsx, .pptx, .jar), the limits become, slowly, more and more frequently an issue over time. Consequently, PKWARE (the firm behind PKZIP) has introduced, around year 2000, a set of format extensions to overcome those limitations. Given the omnipresence of Zip archives and the difficulty to innovate in software, PKWARE did not opt for a fresh, new, simple, but incompatible format. Instead, they designed a set of extensions, allowed by the flexibility of the Zip format. This seems clumsy and over-engineered, but the design is actually quite clever: a program can create a Zip archive, without size limitations, and without checking in advance whether the Zip64 extensions are needed or not. When the archive is finished, either the Zip64 extensions were not needed and the archive is conform to the year 1989 original format, or they were – then, an unpacker for that archive file has to know about Zip64. Thus, the Zip64 design allowed for a progressive adoption of the new format, since the actual need went progressively from very rare to a bit less rare.

How to implement Zip64 in a new software? There are a few options, which you can combine:

1.    Follow the documentation, appnote.txt
2.    Take inspiration from some open-source software like Info-Zip or 7-Zip
3.    Reverse-engineer data (hex dumps)

All three are very time-consuming (especially, the documentation, aimed at maintaining the openness of the Zip format, is "legally" correct but lacks some indications and practical details).
Even widespread, commercial software (with paying customers and many paid programmers behind it) does not support Zip64 (e.g. Microsoft Word for .docx), or took almost two decades to do so (Microsoft Windows for instance). It reveals something about the time and costs related to the implementation…

Fortunately, a very helpful person called Yaakov not only did an implementation from scratch, but also was kind enough to share his experience in an explanatory and colorful way, with a simple example as an illustration. The article is available here: ZIP64 - Go Big Or Go Home.
I show here (with permission) the key diagram of his article:

Click to enlarge

The Zip64 extensions are in dark background and white letters.
The part with the archived (eventually compressed) files is in pink and brown. In that example, there is only one file in the archive.

As you see, an implementation of Zip64 may be quite tricky, but it is feasible, if we have "the big picture" before our eyes.

Alas, things are a little bit more complicated than it appears when reading the article too fast...

We want, with Zip-Ada, to decode and unpack correctly archives not only made by Zip-Ada, but also by Info-Zip's Zip, 7-Zip, WinZip, and so on (the converse is also true: we want the other guys to be able to unpack our Zip-Ada archives).
For that reason, we need to anticipate all possible ways the data headers have been written, not only our choices. In that respect, we have had some headache with the per-entry "size" Zip64 extensions (the parts in brown and dark green in the chart).
The documentation (4.5.3) describes, for that, a record with a variable number of values:

8 bytes    [A] Original uncompressed file size
8 bytes    [B] Size of compressed data
8 bytes    [C] Offset of local header record
4 bytes    [D] Number of the disk on which this file starts

and defines in plain English some rules about the order and in which circumstances the values are present or not.

The documentation requires both [A] and [B] for the Local Header (part in brown in the chart). Let us figure out the explanation to that rule. For a program that creates the archive, it would not be practical to put only the uncompressed size before compressing the data, because the program would have to shift all the compressed content by 8 bytes in case the compressed size (which is known only after the compression) happened to also exceed 2**32 - 2. Hence, exactly and always two values if the uncompressed size (which is usually known in advance) exceeds the limit.

When it is time for the archiving program to write the Central Directory (the part that concludes the archive, in green, light and dark, in the chart), all information about sizes is known. An archiving program may choose to put only the necessary values in 64 bits. However, traditionally, you would expect that the variable record's contents are self-describing and would contain [A], or: [A] and [B], or: [A] and [B] and [C], or: [A] and [B] and [C] and [D], given the record's size.
With Zip64, it is not the case: each field is optional. This micro-optimization is ridiculous given the fact that the size of archive is breaking the 4 GiB limit. To make things worse, the documentation is not clear about that full optionality.
An archiver (7-Zip does so for instance) may put just [A], the uncompressed size of one first large file that is well compressible, or just [C], the offset of one small file following a file with a large size in the archive. Or it could put [A], [B], [C] in both cases (as we do for Zip-Ada's archive creation). So, we've adapted the archive reader to take all possible cases into consideration.

Thanks to Nicolas Brunot for pointing that (hopefully) ultimate difficulty with Zip64.

Latest sources:
 
Link 1: https://github.com/zertovitch/zip-ada/ 
  With git: git clone https://github.com/zertovitch/zip-ada.git
  As Zip-ball: green button "Code", choice: "Download ZIP"
 
Link 2: https://sourceforge.net/p/unzip-ada/code/HEAD/tree/
  With subversion: svn checkout https://svn.code.sf.net/p/unzip-ada/code/ za
  As Zip-ball: button: "Download Snapshot"

QOI (the Quite OK Image Format, home page here) is a very simple raster image format with a decent lossless compression and an extremely good performance - a direct consequence of its simplicity and its compression features.

It was clear from the first sight on that format that it was urgent to add it to GID, the Generic Image Decoder 😎. GID is free, open-source, available on SourceForge here and GitHub here.

Here, a few examples of QOI test images decoded by GID (with its default background for transparency):

The cool ideas in the QOI format are

• a "moving palette" - a list of recently shown colours, that is updated during the encoding or decoding of the pixels; the indexing is done with a hash function
  
• a shortcut encoding for slightly different colours from a pixel to the next
• a shortcut encoding for slightly different brightness from a pixel to the next.

Transparency, in the form of levels from 0 to 255, (the alpha channel) is supported.

Add to it a run-length encoding that fits well surfaces with identical colours and transparency, and the format remains so simple that you can squeeze its detailed specification on a single, readable, A4 page!

Click to enlarge

Uncompressed file: 1'029'744 bytes.

Compressed size (excluding Zip or 7z archive metadata; data is not preprocessed):

Conclusion: the Zip-Ada (current research branch) compresses that data 17.3% better than 7-Zip v.21.02!

The file (zipped to its smallest compressed size, 4.05%) can be downloaded here. It is part of the old Canterbury corpus benchmark file collection (file name: kennedy.xls).

Please don't draw any conclusion: the test data is a relatively small, special binary file with lots of redundancy.
But that result is a hint that some more juice can be extracted from the LZMA format.

The open-source Zip-Ada project can be found here and here.

As a follow-up of another post about converting bash (Linux) or cmd (Windows) scripts to HAC scripts, here is a fresh example.

I needed to improve an existing cmd script for benchmarking compression software, with the possibility of switching various subsets separately: that is re-run this subset of methods, or that other subset, or the full tests (long!), etc.

Of course, if you use a real language instead of command-line interpreter ones, there is an obvious solution: you can define a set and you can programmatically flip the membership switches.

In Ada, it looks like

  type Category is (
    Reduce_Shrink,
    Deflate,
    Deflate_External,
    BZip2_External,
    PPMd_External,
    LZMA_7z,
    LZMA_3,
    TAR,
    Preselection
  );

  cat_set : array (Category) of Boolean;

Below are screenshots of the quick development of bench.adb using the LEA editor, where you can punch F4 to check eventual errors. If there is one, you get instantly to the offending line / column point.

This script is part of the Zip-Ada project and is very helpful for developing and testing new compression methods.

Click to enlarge

Click to enlarge

A rare case where Zip-Ada's LZMA encoder is much better than LZMA SDK's. Rare but still interesting, and with standard LZMA parameters (no specific tuning for that file):

The compressed size with current revision (rev.#882) of Zip-Ada is slightly worse (42,559 bytes).

The file is part of the classic Canterbury Corpus compression benchmark data set.

New release (2.40) of AZip.

The long-awaited Windows Explorer integration is there:

Context menu for a file

Context menu for a folder

This integration is activated upon installation or on demand via the Manage button:

Configuration

This new version is based on the Zip-Ada library v.57 and includes its recent developments.

Enjoy!

Audacity is a free, open source, audio editor, available here.

If you want to backup you Audacity project, you can manually do it with "Save Lossless Copy of Project..." with the name, say, X, which will create X.aup (project file), a folder X_data, and, in there, a file called "Audio Track.wav".

Some drawbacks:

• It is a manual operation.
• It is blocked during playback.
• Envelopes are applied to the "Audio Track.wav" data. So data is altered and no more a real lossless copy of the project. Actually this operation is something between a backup and an export of the project to a foreign format.

A solution: Zip-Ada.

The latest commit (rev. 796) adds to the Preselection method a specific configuration for detecting Audacity files, so they are compressed better than with default settings.

Funny detail: that configuration makes, in most cases, the compression better than the best available compression with 7-Zip (v.19.00, "ultra" mode, .7z archive).

The compressing process is also around twice as fast as 7-Zip in "ultra" mode. This is no magic, since the "LZ" part of the LZMA compression scheme spends less time finding matches, in the chosen configuration for Zip-Ada.

A backup script could look like this (here for Windows' cmd):

rem --------------------------
rem Nice date YYYY-MM-DD_HH.MM
rem --------------------------

set year=%date:~-4,4%

set month=%date:~-7,2%
if "%month:~0,1%" equ " " set month=0%month:~1,1%

set day=%date:~-10,2%
if "%day:~0,1%" equ " " set day=0%day:~1,1%

set hour=%time:~0,2%
if "%hour:~0,1%" equ " " set hour=0%hour:~1,1%

set min=%time:~3,2%

set nice_date=%year%-%month%-%day%_%hour%.%min%

rem --------------------------

set audacity_project=The Cure - A Forest

zipada -ep2 "%audacity_project%_%nice_date%" "%audacity_project%.aup" "%audacity_project%_data\e08\d08\*.au"

You want to copy your Thunderbird profile from machine A to machine B (with all mail accounts, passwords, settings, feeds, newgroups, ...) ? Actually it is very easy. From the user storage (on Windows, %appdata% (you get there with Windows key+R and typing %appdata%)), you copy the entire Thunderbird folder of machine A to the equivalent location on machine B, and that's it. The new active profile will be automatically selected since the file profiles.ini will be overwritten on the way.

Now, if you want or need to use a cloud drive or a USB stick for the operation, it's better to wrap everything in a Zip file (a single file instead of hundreds) to save time. Plus, you can store the Zip file in case of an emergency (losing data on both A and B machines).

With AZip, it's pretty easy: 

• Shut down Thunderbird on both machines.
• On machine A: drag & drop the Thunderbird folder on an empty AZip window.
• Copy or move the Zip file.
• On machine B: extract everything with another drag & drop, from AZip to the Explorer window with the %appdata% path. When asked "Use archive's folder names for output", say "Yes". When asked "Do you want to replace this file ?", say "All".

That's it!

Here a few screenshots:

Folder tree view - click to enlarge

After recompression - click to enlarge

The latest addition to Zip-Ada (commits #792 to 794) is the possibility of writing contents to a Zip file (or more generally, a Zip stream) as an output stream. You (the programmer) don't need to store contents into some buffer and design an input stream in the Zip_Streams.Root_Zipstream_Type'Class type class to read that buffer, as it is the case for Add_Stream in the Zip_Create package.

How does new output stream work in practice? The best way is to show an example. Here is a reduced version of Test_Zip_Entry_Stream (you can find the full version in the test directory of the Zip-Ada project's sources):

      with Zip.Create;
      with Ada.Command_Line, Ada.Text_IO;

      procedure Test_Zip_Entry_Stream is
        use Zip.Create;
        use Ada.Command_Line, Ada.Text_IO;

        Archive_Info  : Zip_Create_Info;
        Archive_File  : aliased Zip_File_Stream;
        Archive_Entry : aliased Zip_Entry_Stream_Type;
        Text : File_Type;
      begin
        Create_Archive (Archive_Info, Archive_File'Unchecked_Access, "test_zes.zip");
        for I in 1 .. Argument_Count loop
          Open (Archive_Entry);
          Open (Text, In_File, Argument (I));
          while not End_Of_File (Text) loop
            String'Write (Archive_Entry'Access, Get_Line (Text));
            Character'Write (Archive_Entry'Access, ASCII.LF);  --  UNIX end-of-line
          end loop;
          Close (Text);
          Close (Archive_Entry, "zes_" & Argument (I), use_clock, Archive_Info);
        end loop;
        Finish (Archive_Info);
      end Test_Zip_Entry_Stream;