Login
IO stream composition and serialization with Ada Utility Library
To be able to provide this IO stream combination, the Ada Utility Library(https://github.com/stcarrez/ada-util) defines two Ada types: the `Input_Stream` and the `Output_Stream` limited interfaces. The `Input_Stream` interface only defines a `Read` procedure and the `Output_Stream` interface defines the `Write`, `Flush` and `Close` procedures. By implementing these interfaces, it is possible to provide stream objects that can be combined together.
[IO Stream Composition and Serialization](Ada/ada-util-streams.png)
The Ada Utility Library(https://github.com/stcarrez/ada-util) provides stream types that implement these interfaces so that it is possible to read or write files, sockets and system pipes. In many cases, the concrete type implements both interfaces so that reading or writing is possible. This is the case for `File_Stream` which allows to read or write on a file, the `Socket_Stream` which handles sockets by using GNAT sockets. The `Pipe_Stream` on its side allows to launch an external program and either read its output, available through the `Input_Stream`, or feed the external program with some input by using the `Output_Stream`.
The Ada Utility Library(https://github.com/stcarrez/ada-util) also provides stream objects that make transformation on the data through various data encoders. The Ada library supports the following encoders:
- Base 16, Base 64,
- AES encryption or decryption,
- LZMA compression or decompression
Other encoders could be added and it is always possible to provide custom transformations by implementing the `Input_Stream` and `Output_Stream` interfaces.
The last part that completes the IO stream framework is the serialization framework. That framework defines and provides interface and types to read or write a CSV, XML, JSON or HTTP form stream. The serialization framework uses either the `Input_Stream` or the `Output_Stream` interfaces to either read or write the content. The serialization framework defines operations in a way that allows to read or write these streams independently of their representation format.
- LZMA Compression
Let's have a look at compressing a file by using the `Util.Streams` framework. First we need a `File_Stream` that is configured to read the file to compress and we need another `File_Stream` configured for writing to save in another file. The first file is opened by using the `Open` procedure and the `In_File` mode while the second one is using `Create` and the `Out_File` mode. The `File_Stream` is using the Ada `Stream_IO` standard package to access files.
```Ada with Util.Streams.Files;
In_Stream : aliased Util.Streams.Files.File_Stream;
Out_Stream : aliased Util.Streams.Files.File_Stream;
In_Stream.Open (Mode => Ada.Streams.Stream_IO.In_File, Name => Source);
Out_Stream.Create (Mode => Ada.Streams.Stream_IO.Out_File, Name => Destination);
```
In the middle of these two streams, we are going to use a `Compress_Stream` whose job is to compress the data and write the compressed result to a target stream. The compression stream is configured by using the `Initialize` procedure and it is configured to write on the `Out_Stream` file stream. The compression stream needs a buffer and its size is configured with the `Size` parameter.
```Ada with Util.Streams.Lzma;
Compressor : aliased Util.Streams.Lzma.Compress_Stream;
Compressor.Initialize (Output => Out_Stream'Unchecked_Access, Size => 32768);
```
To feed the compressor stream with the input file, we are going to use the `Copy` procedure. This procedure reads the content from the `In_Stream` and writes what is read to the `Compressor` stream.
```Ada
Util.Streams.Copy (From => In_Stream, Into => Compressor);
```
Flushing and closing the files is automatically handled by a `Finalize` procedure on the `File_Stream` type.
Complete source example: compress.adb(https://github.com/stcarrez/ada-util/tree/master/samples/compress.adb)
- LZMA Decompression
The LZMA decompression is very close to the LZMA compression but instead it uses the `Decompress_Stream`. The complete decompression method is the following:
```Ada procedure Decompress_File (Source : in String;
Destination : in String) is
In_Stream : aliased Util.Streams.Files.File_Stream;
Out_Stream : aliased Util.Streams.Files.File_Stream;
Decompressor : aliased Util.Streams.Lzma.Decompress_Stream;
begin
In_Stream.Open (Mode => Ada.Streams.Stream_IO.In_File, Name => Source);
Out_Stream.Create (Mode => Ada.Streams.Stream_IO.Out_File, Name => Destination);
Decompressor.Initialize (Input => In_Stream'Unchecked_Access, Size => 32768);
Util.Streams.Copy (From => Decompressor, Into => Out_Stream);
end Decompress_File; ```
Complete source example: decompress.adb(https://github.com/stcarrez/ada-util/tree/master/samples/decompress.adb)
- AES Encryption
Encryption is a little bit more complex due to the encryption key that must be configured. The encryption is provided by the `Encoding_Stream` and it uses a `Secret_Key` to configure the encryption key. The `Secret_Key` is a limited type and it cannot be copied. To build the encryption key, one method consists in using the PBKDF2 algorithm described in RFC 8018(https://tools.ietf.org/html/rfc8018). The user password is passed to the PBKDF2 algorithm configured to use the HMAC-256 hashing. The hash method is called on itself 20000 times in this example to produce the final encryption key.
```Ada with Util.Streams.AES; with Util.Encoders.AES; with Util.Encoders.KDF.PBKDF2_HMAC_SHA256;
Cipher : aliased Util.Streams.AES.Encoding_Stream;
Password_Key : constant Util.Encoders.Secret_Key := Util.Encoders.Create (Password);
Salt : constant Util.Encoders.Secret_Key := Util.Encoders.Create ("fake-salt");
Key : Util.Encoders.Secret_Key (Length => Util.Encoders.AES.AES_256_Length);
...
PBKDF2_HMAC_SHA256 (Password => Password_Key,
Salt => Salt,
Counter => 20000,
Result => Key);
```
The encoding stream is able to produce or consume another stream. For the encryption, we are going to use the first mode and use the `Produces` procedure to configure the encryption to write on the `Out_Stream` file. Once configured, the `Set_Key` procedure must be called with the encryption key and the encryption method. The initial encryption `IV` vector can be configured by using the `Set_IV` procedure (not used by the example). As soon as the encryption key is configured, the encryption can start and the `Cipher` encoding stream can be used as an `Output_Stream`: we can write on it and it will encrypt the content before passing the result to the next stream. This means that we can use the same `Copy` procedure to read the input file and pass it through the encryption encoder.
```Ada
Cipher.Produces (Output => Out_Stream'Unchecked_Access, Size => 32768);
Cipher.Set_Key (Secret => Key, Mode => Util.Encoders.AES.ECB);
Util.Streams.Copy (From => In_Stream, Into => Cipher);
```
Complete source example: encrypt.adb(https://github.com/stcarrez/ada-util/tree/master/samples/encrypt.adb)
- AES Decryption
Decryption is similar but it uses the `Decoding_Stream` type. Below is the complete example to decrypt the file:
```Ada procedure Decrypt_File (Source : in String;
Destination : in String;
Password : in String) is
In_Stream : aliased Util.Streams.Files.File_Stream;
Out_Stream : aliased Util.Streams.Files.File_Stream;
Decipher : aliased Util.Streams.AES.Decoding_Stream;
Password_Key : constant Util.Encoders.Secret_Key := Util.Encoders.Create (Password);
Salt : constant Util.Encoders.Secret_Key := Util.Encoders.Create ("fake-salt");
Key : Util.Encoders.Secret_Key (Length => Util.Encoders.AES.AES_256_Length);
begin
-- Generate a derived key from the password.
PBKDF2_HMAC_SHA256 (Password => Password_Key,
Salt => Salt,
Counter => 20000,
Result => Key);
-- Setup file -> input and cipher -> output file streams.
In_Stream.Open (Ada.Streams.Stream_IO.In_File, Source);
Out_Stream.Create (Mode => Ada.Streams.Stream_IO.Out_File, Name => Destination);
Decipher.Produces (Output => Out_Stream'Access, Size => 32768);
Decipher.Set_Key (Secret => Key, Mode => Util.Encoders.AES.ECB);
-- Copy input to output through the cipher.
Util.Streams.Copy (From => In_Stream, Into => Decipher);
end Decrypt_File; ```
Complete source example: decrypt.adb(https://github.com/stcarrez/ada-util/tree/master/samples/decrypt.adb)
- Stream composition: LZMA > AES
Now, if we want to compress the stream before encryption, we can do this by connecting the `Compressor` to the `Cipher` stream and we only have to use the `Compressor` instead of the `Cipher` in the call to `Copy`.
```Ada
In_Stream.Open (Ada.Streams.Stream_IO.In_File, Source);
Out_Stream.Create (Mode => Ada.Streams.Stream_IO.Out_File, Name => Destination);
Cipher.Produces (Output => Out_Stream'Unchecked_Access, Size => 32768);
Cipher.Set_Key (Secret => Key, Mode => Util.Encoders.AES.ECB);
Compressor.Initialize (Output => Cipher'Unchecked_Access, Size => 4096);
Util.Streams.Copy (From => In_Stream, Into => Compressor);
```
When `Copy` is called, the following will happen:
- first, it reads the `In_Stream` source file,
- the data is written to the `Compress` stream,
- the `Compressor` stream runs the LZMA compression and writes on the `Cipher` stream,
- the `Cipher` stream encrypts the data and writes on the `Out_Stream`,
- the `Out_Stream` writes on the destination file.
Complete source example: lzma_encrypt.adb(https://github.com/stcarrez/ada-util/tree/master/samples/lzma_encrypt.adb)
- More stream composition: LZMA > AES > Base64
We can easily change the stream composition to encode in Base64 after the encryption. We only have to declare an instance of the Base64 `Encoding_Stream` and configure the encryption stream to write on the Base64 stream instead of the output file. The Base64 stream is configured to write on the output stream.
```Ada In_Stream : aliased Util.Streams.Files.File_Stream; Out_Stream : aliased Util.Streams.Files.File_Stream; Base64 : aliased Util.Streams.Base64.Encoding_Stream; Cipher : aliased Util.Streams.AES.Encoding_Stream; Compressor : aliased Util.Streams.Lzma.Compress_Stream;
In_Stream.Open (Ada.Streams.Stream_IO.In_File, Source);
Out_Stream.Create (Mode => Ada.Streams.Stream_IO.Out_File, Name => Destination);
Base64.Produces (Output => Out_Stream'Unchecked_Access, Size => 32768);
Cipher.Produces (Output => Base64'Unchecked_Access, Size => 32768);
Cipher.Set_Key (Secret => Key, Mode => Util.Encoders.AES.ECB);
Compressor.Initialize (Output => Cipher'Unchecked_Access, Size => 4096);
```
Complete source example: lzma_encrypt_b64.adb(https://github.com/stcarrez/ada-util/tree/master/samples/lzma_encrypt_b64.adb)
- Serialization
Serialization is achieved by using the `Util.Serialize.IO` packages and child packages and their specific types. The parent package defines the limited `Output_Stream` interface which inherit from the `Util.Streams.Output_Stream` interface. This allows to define specific operations to write various Ada types but also it provides common set of abstractions that allow to write either a JSON, XML, CSV and FORM (`x-www-form-urlencoded`) formats.
The target format is supported by a child package so that you only have to use the `Output_Stream` type declared in one of the `JSON`, `XML`, `CSV` or `Form` child package and use it transparently. There are some constraint if you want to switch from one output format to another while keeping the same code. These constraints comes from the nature of the different formats: `XML` has a notion of entity and attribute but other formats don't differentiate entities from attributes.
- A `Start_Document` procedure must be called first. Not all serialization method need it but it is required for JSON to produce a correct output.
- A `Write_Entity` procedure writes an XML entity of the given name. When used in JSON, it writes a JSON attribute.
- A `Start_Entity` procedure prepares the start of an XML entity or a JSON structure with a given name.
- A `Write_Attribute` procedure writes an XML attribute after a `Start_Entity`. When used in JSON, it writes a JSON attribute.
- A `End_Entity` procedure terminates an XML entity or a JSON structure that was opened by `Start_Entity`.
- At the end, the `End_Document` procedure must be called to finish correctly the output and terminate the JSON or XML content.
```Ada procedure Write (Stream : in out Util.Serialize.IO.Output_Stream'Class) is begin
Stream.Start_Document;
Stream.Start_Entity ("person");
Stream.Write_Entity ("name", "Harry Potter");
Stream.Write_Entity ("gender", "male");
Stream.Write_Entity ("age", 17);
Stream.End_Entity ("person");
Stream.End_Document;
end Write; ```
- JSON Serialization
With the above `Write` procedure, if we want to produce a JSON stream, we only have to setup a JSON serializer. The JSON serializer is connected to a `Print_Stream` which provides a buffer and helper operations to write some text content. An instance of the `Print_Stream` is declared in `Output` and configured with a buffer size. The JSON serializer is then connected to it by calling the `Initialize` procedure and giving the `Output` parameter.
After writing the content, the JSON is stored in the `Output` print stream and it can be retrieved by using the `To_String` function.
```Ada with Ada.Text_IO; with Util.Serialize.IO.JSON; with Util.Streams.Texts; procedure Serialize is
Output : aliased Util.Streams.Texts.Print_Stream;
Stream : Util.Serialize.IO.JSON.Output_Stream;
begin
Output.Initialize (Size => 10000);
Stream.Initialize (Output => Output'Unchecked_Access);
Write (Stream);
Ada.Text_IO.Put_Line (Util.Streams.Texts.To_String (Output));
end Serialize; ```
The `Write` procedure described above produces the following JSON content:
```C {"person":{"name":"Harry Potter","gender":"male","age": 17}} ```
Complete source example: serialize.adb(https://github.com/stcarrez/ada-util/tree/master/samples/serialize.adb)
- XML Serialization
Switching to an XML serialization is easy: replace `JSON` by `XML` in the package to use the XML serializer instead.
```Ada with Ada.Text_IO; with Util.Serialize.IO.XML; with Util.Streams.Texts; procedure Serialize is
Output : aliased Util.Streams.Texts.Print_Stream;
Stream : Util.Serialize.IO.XML.Output_Stream;
begin
Output.Initialize (Size => 10000);
Stream.Initialize (Output => Output'Unchecked_Access);
Write (Stream);
Ada.Text_IO.Put_Line (Util.Streams.Texts.To_String (Output));
end Serialize; ```
This time, the same `Write` procedure produces the following XML content:
```C <person><name>Harry Potter</name><gender>male</gender><age>17</age></person> ```
Complete source example: serialize_xml.adb(https://github.com/stcarrez/ada-util/tree/master/samples/serialize_xml.adb)
