โŒ About FreshRSS

Normal view

There are new articles available, click to refresh the page.
Before yesterdayNews from the Ada programming language world

Advanced Resource Embedder 1.2.0

23 January 2022 at 09:04
[Embedding SQL in binary](Ada/resource-embedder-bin.png)

The new release contains the following fixes:

- Fix Ada generator to be able to use other binary content types

 such as `System.Storage_Elements.Storage_Array`
- Fix Ada generator to escape special characters in strings

Since the previous version, Fabien Chouteau asked to be able to use an Ada system type to represent a binary content. This is now possible by using the following XML extract for the description to tell the resource embedder how to integrate the file and generate the Ada source code:

```XML <package>

 <resource name='Resources.Help'
           format='binary'
           type='access constant System.Storage_Elements.Storage_Array'>
   <install mode='copy'>
     <fileset dir="help">
       <include name="**/*.txt"/>
     </fileset>
   </install>
 </resource>
 ...

</package> ```

With the above description, the Ada code generator produces the following package specification:

```Ada with System.Storage_Elements; package Resources.Help is

  type Content_Access is access constant System.Storage_Elements.Storage_Array;
  type Name_Access is access constant String;
  type Name_Array is array (Natural range <>) of Name_Access;
  Names : constant Name_Array;
  function Get_Content (Name : String)
     access constant System.Storage_Elements.Storage_Array;

private

  ...

end Resources.Help; ```

This example is available in Embedding help and documentation in Ada (binary)(https://gitlab.com/stcarrez/resource-embedder/tree/master/examples/ada-help-binary). If you prefer to use `String` instead of a `Storage_Array`, have a look at the Embedding help and documentation in Ada(https://gitlab.com/stcarrez/resource-embedder/tree/master/examples/ada-help). Both examples are similar but they are exposing the file using different Ada types.

To install the tool, follow the instructions given in the initial announcement: Advanced Resource Embedder for Ada, C and Go(https://blog.vacs.fr/vacs/blogs/post.html?post=2021/06/11/Advanced-Resource-Embedder).

If you want to know more about the tool, have a look at its documentation:

  • Resource Embedder Guide(https://resource-embedder.readthedocs.io/en/latest/) PDF(https://gitlab.com/stcarrez/resource-embedder/blob/master/docs/are-book.pdf)
  • Man page: are (1)(https://gitlab.com/stcarrez/resource-embedder/blob/master/docs/are.md)

and if you have ideas for improvements, fork the project and submit a pull request!

Advanced Resource Embedder 1.1.0

4 July 2021 at 16:52

The resource embedder allows to embed files in binaries by producing C, Ada or Go source files that contain the original files. The first version of the tool was representing the file content as an array of bytes. In Ada, they are accessed through an `Ada.Streams.Stream_Element_Array` which is not easy to use when you need the content as a string.

The new release introduces the customization of the format for each resource. The format is controlled through the XML description by the `format` attribute. The following data formats are supported:

  • `binary` format provides the file content as a binary data,
  • `string` format provides the file content as string,
  • `lines` format splits the content in several lines and according to a set of customizable rules.

With the `string` format, the Ada code generator can generate the following function:

```

 function Get_Content (Name : in String)
   return access constant String;

```

The `lines` format tells the code generator to represent the content as an array of separate lines. For this integration, some control is available to indicate how the content must be split and optionally apply some filter on the input content. These controls are made within the XML description by using the `line-separator` and `line-filter` description: The `line-separator` indicates the characters that represent a line separation. There can be several `line-separator` definition. The `line-filter` defines a regular expression that when matched must be replaced by an empty string or a specified content. The `line-filter` are applied in the order of the XML definition.

The example below is intended to integrate an SQL scripts with:

  • a separate line for each SQL statement,
  • remove spurious empty lines and SQL comments.

The SQL statements are separated by `;` (semi-colon) and the `line-separator` indicates to split lines on that character. By splitting on the `;` we allow to have an SQL statement on multiple lines in the original SQL source file.

```XML <package>

 <resource name='Scripts'
           format='lines'
           type='access constant String'>
   <line-separator>;</line-separator>
   <!-- Remove new lines -->
   <line-filter>[\r\n]</line-filter>
   <!-- Remove C comments -->
   <line-filter>/\*[^/]*\*/</line-filter>
   <!-- Remove contiguous spaces after C comments removal -->
   <line-filter replace=' '>[ \t][ \t]+</line-filter>
   <install mode='copy' strip-extension='yes'>
     <fileset dir="sql">
       <include name="**/*.sql"/>
     </fileset>
   </install>
 </resource>

</package> ```

Then the first `line-filter` will remove the `r` and `n` characters.

The regular expression `/\*[./]*\*/` matches a C style comment and remove it.

The last `line-filter` replaces multiple tabs and spaces by a single occurrence.

Below is an example of generated code with the above resource description. Each file is accessed through a separate variable whose name is built from the base name of the original file.

[Resource Embedder Overview](Ada/are-scripts.png)

The command used to generate this code was:

``` are lang=Ada -o src var-access content-only rule=package.xml . ```

and the `sql` directory contains only two files: `create-database.sql` and `drop-database.sql`.

The complete example is available for two languages:

To install the tool, follow the instructions given in the initial announcement: Advanced Resource Embedder for Ada, C and Go(https://blog.vacs.fr/vacs/blogs/post.html?post=2021/06/11/Advanced-Resource-Embedder).

If you want to know more about the tool, have a look at its documentation:

  • Resource Embedder Guide(https://resource-embedder.readthedocs.io/en/latest/) PDF(https://gitlab.com/stcarrez/resource-embedder/blob/master/docs/are-book.pdf)
  • Man page: are (1)(https://gitlab.com/stcarrez/resource-embedder/blob/master/docs/are.md)

and if you have ideas for improvements, fork the project and submit a pull request!

Advanced Resource Embedder for Ada, C and Go

11 June 2021 at 12:43

The tool is able to apply some transformations on the collected files:

  • it can run a Javascript minifier such as `closure`,
  • it can compress CSS files by running `yui-compressor`,
  • it can compress files by running `gzip` or another compression tool.

Once these transformations are executed, it invokes a target generator to produce a source file either in C, Ada or Go language. The generated source file can then be used in the final program and taken into account during the compilation process of that program. At the end, the binary will contain the embedded files with their optional transformations.

[Resource Embedder Overview](Ada/resource-embedder.png)

The process to use ARE is simple:

  • You describe the resources that you want to embed.
 The description is either made on command line arguments or by writing an XML file.
 The XML description gives more flexibility as it allows to define a transformation rule that
 must be executed on the original file before being embedded.  This allows to minify a Javascript
 or CSS file, compress some files and even encrypt a file before its integration.
  • You run the ARE command with your target language and rule description and you give the tool
  •  a list of directories that must be scanned to identify the files that must be collected.
     The ARE tool scan the directories according to the patterns that you have given either on
     the command line or in the XML rule description.  After identifying the files, the tool applies
     the rules and execute the transformations.
     The ARE tool then invokes the target language generator that writes one or several files depending
     on the list of resources.
    
  • Once the files are generated, you use them in your program and add them in your build process
  •  as they are now part of your sources.  After building your program, it now embeds the
     resource files that were collected and optionally transformed.
    
      1. Defining resources to embed

    The first step is to write some `package.xml` file which describes a list of resources with their content. The root XML element is *`package`* and each resource is described by a `resource` XML element. The resource is assigned a name that will be used by the code generator. The C generator will use the name as a prefix for the C function and C type declaration. The Ada and Go generator will use that name for the Ada or Go package.

    The following resource definition declares the `Help` resource. It contains an installation rule that will copy the files under the `help` directory in the resource set. Only files matching the `.txt` pattern will be taken into account.

    ```XML <package>

     <resource name='Help'>
       <install mode='copy'>
         <fileset dir="help">
           <include name="**/*.txt"/>
         </fileset>
       </install>
     </resource>
     ...
    

    </package> ```

    The next resource definition will run an external program to get the content that must be embedded. The `man` directory is scanned and it will execute the command `man #{name}` on each filename found. That directory contains the empty files `ls`, `pwd` and `sh` and this will run and embed the man page for these commands.

    ```XML <package>

      ...
     <resource name='Man'>
       <install mode='exec'>
         <command output='#{dst}'>man #{name}</command>
         <fileset dir="man">
           <include name="*"/>
         </fileset>
       </install>
     </resource>
    

    </package> ```

    You may run other commands such as:

    ``` <command>closure charset UTF-8 #{src} js_output_file #{dst}</command> <command>yui-compressor type css charset UTF-8 -o #{dst} #{src}</command> <command output="#{dst}">gzip --no-name -c #{src}</command> ```

      1. Running the Advanced Resource Embedder

    The tool has several options that allows you to control the target code generator and tune the generation according to your needs. Assuming that the files to embed are located in the current directory, you would use the following command to generate C source files in the `src` directory:

    ``` are lang=c -o src rule=package.xml list-access name-access . ```

    For C, this would generate a `src/man.h`, `src/man.c`, `src/help.h` and `src/help.c`. You now have to include these files in the compilation of your program.

    You would use the following command for Go:

    ``` are lang=go rule=package.xml list-access name-access . ```

    and it would generate in `man/man.go` and `help/help.go` the Go source files.

    You would use the following command for Ada:

    ``` are lang=Ada -o src rule=package.xml list-access name-access . ```

    and it would generate `src/man.ads`, `src/man.adb`, `src/help.ads` and `src/help.adb`.

      1. Using the resource

    The code generator emits by default a set of type and function declaration that give access to the resource. For C, the following structure and declaration are generated in the header for the `man` resource:

    ```C struct man_content {

     const unsigned char *content;
     size_t size;
     time_t modtime;
     int format;
    

    };

    extern const struct man_content* man_get_content(const char* name); ```

    But for the Go language, the `man` generated package declares:

    ```Go type Content struct {

       Content  []byte
       Size  int64
       Modtime  int64
       Format   int
    

    } ... func Get_content(name string) (*Content) {...} ```

    And for Ada, it will generate:

    ```Ada package Resources.Man is

      type Content_Access is access constant Ada.Streams.Stream_Element_Array;
      type Name_Access is access constant String;
      type Format_Type is (FILE_RAW, FILE_GZIP);
      type Content_Type is record
         Name    : Name_Access;
         Content : Content_Access;
         Modtime : Interfaces.C.long := 0;
         Format  : Format_Type := FILE_RAW;
      end record;
      Null_Content : constant Content_Type;
    
      type Name_Array is array (Natural range <>) of Name_Access;
      Names : constant Name_Array;
    
      function Get_Content (Name : String) return Content_Type;
    

    private

      ...
    

    end Resources.Man; ```

    You can avoid the type declaration by using the `--no-type-declaration` option and in that case you have to make available these types somehow. In Ada, this can easily be made by providing these types in a parent Ada package (see the Embedding help and documentation in Ada(https://gitlab.com/stcarrez/resource-embedder/tree/master/examples/ada-help) for the example).

    Now with the generated code, you only need to call the generated get content method to obtain the embedded content. Simple!

      1. Examples

    The project proposes several detailed examples to illustrate various integrations.

    This first set of example shows how to you can embed configuration files in a C, Ada or Go program. The Advance Resource Embedder simply puts the configuration files in an array of bytes that can easily be retrieved by a generated function.

    A second set of example is more advanced by the use of an XML file that describes what must be embedded with the transformations that must be made. It creates two distinct resource sets `help` and `man`. The `help` resource set is composed of a set of fixed documentation files provided in the example. The `man` resource set is created by running the `man` Unix command on various names to embed the man page of `ls`, `pwd` and `sh`.

    More specific examples show how to make specific transformations on the files before integrating them:

      1. License considerations

    The generated code produced by Advance Resource Embedder(https://gitlab.com/stcarrez/resource-embedder) is not covered by any license. However, when you integrate some resource files with the tool, the generated code will contain the original file in some binary form and therefore it may be covered by the license associated with these resource files.

    For example, if you include a file covered by the GNU General Public license, then the generated file is covered by the GPL.

      1. How can you get Advanced Resource Embedder?
        1. Use the source Luke!

    Quick instructions to build the tool:

    ``` git clone --recursive https://gitlab.com/stcarrez/resource-embedder.git cd resource-embedder make build test install ```

        1. Debian Packages for x86_64

    You can install ARE by using the Debian 10 and Ubuntu 20.04 or 18.04 packages. First, setup to accept the signed packages:

    ``` wget -O - https://apt.vacs.fr/apt.vacs.fr.gpg.key | sudo apt-key add - ```

    and choose one of the `echo` command according to your Linux distribution:

    Ubuntu 20.04 ``` echo "deb https://apt.vacs.fr/ubuntu-focal focal main" | sudo tee -a /etc/apt/sources.list.d/vacs.list ```

    Ubuntu 18.04 ``` echo "deb https://apt.vacs.fr/ubuntu-bionic bionic main" | sudo tee -a /etc/apt/sources.list.d/vacs.list ```

    Debian 10 ``` echo "deb https://apt.vacs.fr/debian-buster buster main" | sudo tee -a /etc/apt/sources.list.d/vacs.list ```

    Then, launch the apt update command:

    ``` sudo apt-get update ```

    and install the tool using:

    ``` sudo apt-get install -y are ```

      1. Conclusion

    Embedding files, scripts, documentation and other contents in C and Ada is easily done by using the Advance Resource Embedder(https://gitlab.com/stcarrez/resource-embedder). Go developers are already familiar with these mechanisms thanks to the `go:embed` directive. But the tool provides custom transformations that gives more flexibility for the integration.

    Some benefits in embedding contents in final binaries:

    • the content is embedded within the readonly `.rodata` section,
    • you avoid at least 3 system calls to access the content: an `open`, a `read` and a `close`,
    • you reduce the size of your application on the disk: contents are contiguous within the `.rodata` section
     whereas written on a file system they each require full dedicated data blocks (4K in most cases),
    
  • you get a portable `mmap` of your files for free and without pain.
  • If you want to know more about the tool, have a look at its documentation:

    • Resource Embedder Guide(https://resource-embedder.readthedocs.io/en/latest/) PDF(https://gitlab.com/stcarrez/resource-embedder/blob/master/docs/are-book.pdf)
    • Man page: are (1)(https://gitlab.com/stcarrez/resource-embedder/blob/master/docs/are.md)

    and if you have ideas for improvements, fork the project and submit a pull request!

    โŒ
    โŒ