The tool is able to apply some transformations on the collected files:
- it can run a Javascript minifier such as `closure`,
- it can compress CSS files by running `yui-compressor`,
- it can compress files by running `gzip` or another compression tool.
Once these transformations are executed, it invokes a target generator to produce a source file either in C, Ada or Go language. The generated source file can then be used in the final program and taken into account during the compilation process of that program. At the end, the binary will contain the embedded files with their optional transformations.
[Resource Embedder Overview](Ada/resource-embedder.png)
The process to use ARE is simple:
- You describe the resources that you want to embed.
The description is either made on command line arguments or by writing an XML file.
The XML description gives more flexibility as it allows to define a transformation rule that
must be executed on the original file before being embedded. This allows to minify a Javascript
or CSS file, compress some files and even encrypt a file before its integration.
You run the ARE command with your target language and rule description and you give the tool a list of directories that must be scanned to identify the files that must be collected.
The ARE tool scan the directories according to the patterns that you have given either on
the command line or in the XML rule description. After identifying the files, the tool applies
the rules and execute the transformations.
The ARE tool then invokes the target language generator that writes one or several files depending
on the list of resources.
Once the files are generated, you use them in your program and add them in your build process as they are now part of your sources. After building your program, it now embeds the
resource files that were collected and optionally transformed.
- Defining resources to embed
The first step is to write some `package.xml` file which describes a list of resources with their content. The root XML element is *`package`* and each resource is described by a `resource` XML element. The resource is assigned a name that will be used by the code generator. The C generator will use the name as a prefix for the C function and C type declaration. The Ada and Go generator will use that name for the Ada or Go package.
The following resource definition declares the `Help` resource. It contains an installation rule that will copy the files under the `help` directory in the resource set. Only files matching the `.txt` pattern will be taken into account.
```XML <package>
<resource name='Help'>
<install mode='copy'>
<fileset dir="help">
<include name="**/*.txt"/>
</fileset>
</install>
</resource>
...
</package> ```
The next resource definition will run an external program to get the content that must be embedded. The `man` directory is scanned and it will execute the command `man #{name}` on each filename found. That directory contains the empty files `ls`, `pwd` and `sh` and this will run and embed the man page for these commands.
```XML <package>
...
<resource name='Man'>
<install mode='exec'>
<command output='#{dst}'>man #{name}</command>
<fileset dir="man">
<include name="*"/>
</fileset>
</install>
</resource>
</package> ```
You may run other commands such as:
``` <command>closure charset UTF-8 #{src} js_output_file #{dst}</command> <command>yui-compressor type css charset UTF-8 -o #{dst} #{src}</command> <command output="#{dst}">gzip --no-name -c #{src}</command> ```
- Running the Advanced Resource Embedder
The tool has several options that allows you to control the target code generator and tune the generation according to your needs. Assuming that the files to embed are located in the current directory, you would use the following command to generate C source files in the `src` directory:
``` are lang=c -o src rule=package.xml list-access name-access . ```
For C, this would generate a `src/man.h`, `src/man.c`, `src/help.h` and `src/help.c`. You now have to include these files in the compilation of your program.
You would use the following command for Go:
``` are lang=go rule=package.xml list-access name-access . ```
and it would generate in `man/man.go` and `help/help.go` the Go source files.
You would use the following command for Ada:
``` are lang=Ada -o src rule=package.xml list-access name-access . ```
and it would generate `src/man.ads`, `src/man.adb`, `src/help.ads` and `src/help.adb`.
- Using the resource
The code generator emits by default a set of type and function declaration that give access to the resource. For C, the following structure and declaration are generated in the header for the `man` resource:
```C struct man_content {
const unsigned char *content;
size_t size;
time_t modtime;
int format;
};
extern const struct man_content* man_get_content(const char* name); ```
But for the Go language, the `man` generated package declares:
```Go type Content struct {
Content []byte
Size int64
Modtime int64
Format int
} ... func Get_content(name string) (*Content) {...} ```
And for Ada, it will generate:
```Ada package Resources.Man is
type Content_Access is access constant Ada.Streams.Stream_Element_Array;
type Name_Access is access constant String;
type Format_Type is (FILE_RAW, FILE_GZIP);
type Content_Type is record
Name : Name_Access;
Content : Content_Access;
Modtime : Interfaces.C.long := 0;
Format : Format_Type := FILE_RAW;
end record;
Null_Content : constant Content_Type;
type Name_Array is array (Natural range <>) of Name_Access;
Names : constant Name_Array;
function Get_Content (Name : String) return Content_Type;
private
...
end Resources.Man; ```
You can avoid the type declaration by using the `--no-type-declaration` option and in that case you have to make available these types somehow. In Ada, this can easily be made by providing these types in a parent Ada package (see the Embedding help and documentation in Ada(https://gitlab.com/stcarrez/resource-embedder/tree/master/examples/ada-help) for the example).
Now with the generated code, you only need to call the generated get content method to obtain the embedded content. Simple!
- Examples
The project proposes several detailed examples to illustrate various integrations.
This first set of example shows how to you can embed configuration files in a C, Ada or Go program. The Advance Resource Embedder simply puts the configuration files in an array of bytes that can easily be retrieved by a generated function.
A second set of example is more advanced by the use of an XML file that describes what must be embedded with the transformations that must be made. It creates two distinct resource sets `help` and `man`. The `help` resource set is composed of a set of fixed documentation files provided in the example. The `man` resource set is created by running the `man` Unix command on various names to embed the man page of `ls`, `pwd` and `sh`.
More specific examples show how to make specific transformations on the files before integrating them:
- License considerations
The generated code produced by Advance Resource Embedder(https://gitlab.com/stcarrez/resource-embedder) is not covered by any license. However, when you integrate some resource files with the tool, the generated code will contain the original file in some binary form and therefore it may be covered by the license associated with these resource files.
For example, if you include a file covered by the GNU General Public license, then the generated file is covered by the GPL.
- How can you get Advanced Resource Embedder?
- Use the source Luke!
Quick instructions to build the tool:
``` git clone --recursive https://gitlab.com/stcarrez/resource-embedder.git cd resource-embedder make build test install ```
- Debian Packages for x86_64
You can install ARE by using the Debian 10 and Ubuntu 20.04 or 18.04 packages. First, setup to accept the signed packages:
``` wget -O - https://apt.vacs.fr/apt.vacs.fr.gpg.key | sudo apt-key add - ```
and choose one of the `echo` command according to your Linux distribution:
Ubuntu 20.04 ``` echo "deb https://apt.vacs.fr/ubuntu-focal focal main" | sudo tee -a /etc/apt/sources.list.d/vacs.list ```
Ubuntu 18.04 ``` echo "deb https://apt.vacs.fr/ubuntu-bionic bionic main" | sudo tee -a /etc/apt/sources.list.d/vacs.list ```
Debian 10 ``` echo "deb https://apt.vacs.fr/debian-buster buster main" | sudo tee -a /etc/apt/sources.list.d/vacs.list ```
Then, launch the apt update command:
``` sudo apt-get update ```
and install the tool using:
``` sudo apt-get install -y are ```
- Conclusion
Embedding files, scripts, documentation and other contents in C and Ada is easily done by using the Advance Resource Embedder(https://gitlab.com/stcarrez/resource-embedder). Go developers are already familiar with these mechanisms thanks to the `go:embed` directive. But the tool provides custom transformations that gives more flexibility for the integration.
Some benefits in embedding contents in final binaries:
- the content is embedded within the readonly `.rodata` section,
- you avoid at least 3 system calls to access the content: an `open`, a `read` and a `close`,
- you reduce the size of your application on the disk: contents are contiguous within the `.rodata` section
whereas written on a file system they each require full dedicated data blocks (4K in most cases),
you get a portable `mmap` of your files for free and without pain.If you want to know more about the tool, have a look at its documentation:
- Resource Embedder Guide(https://resource-embedder.readthedocs.io/en/latest/) PDF(https://gitlab.com/stcarrez/resource-embedder/blob/master/docs/are-book.pdf)
- Man page: are (1)(https://gitlab.com/stcarrez/resource-embedder/blob/master/docs/are.md)
and if you have ideas for improvements, fork the project and submit a pull request!