<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" xml:id="chap-projects"> <title>Creating and Managing Projects</title> <para> Once Hydra is installed and running, the next step is to add projects to the build farm. We follow the example of the <link xlink:href="http://nixos.org/patchelf.html">Patchelf project</link>, a software tool written in C and using the GNU Build System (GNU Autoconf and GNU Automake). </para> <para> Log in to the web interface of your Hydra installation using the user name and password you inserted in the database (by default, Hydra's web server listens on <link xlink:href="http://localhost:3000/"><literal>localhost:3000</literal></link>). Then follow the "Create Project" link to create a new project. </para> <section> <title>Project Information</title> <para> A project definition consists of some general information and a set of job sets. The general information identifies a project, its owner, and current state of activity. Here's what we fill in for the patchelf project: <screen> Identifier: patchelf </screen> The <emphasis>identifier</emphasis> is the identity of the project. It is used in URLs and in the names of build results. </para> <para> The identifier should be a unique name (it is the primary database key for the project table in the database). If you try to create a project with an already existing identifier you'd get an error message such as: <screen> I'm very sorry, but an error occurred: DBIx::Class::ResultSet::create(): DBI Exception: DBD::SQLite::st execute failed: column name is not unique(19) at dbdimp.c line 402 </screen> So try to create the project after entering just the general information to figure out if you have chosen a unique name. Job sets can be added once the project has been created. <screen> Display name: Patchelf </screen> The <emphasis>display name</emphasis> is used in menus. <screen> Description: A tool for modifying ELF binaries </screen> The <emphasis>description</emphasis> is used as short documentation of the nature of the project. <screen> Owner: eelco </screen> The <emphasis>owner</emphasis> of a project can create and edit job sets. <screen> Enabled: Yes </screen> Only if the project is <emphasis>enabled</emphasis> are builds performed. </para> <para> Once created there should be an entry for the project in the sidebar. Go to the project page for the <link xlink:href="http://localhost:3000/project/patchelf">Patchelf</link> project. </para> </section> <section> <title>Job Sets</title> <para> A project can consist of multiple <emphasis>job sets</emphasis> (hereafter <emphasis>jobsets</emphasis>), separate tasks that can be built separately, but may depend on each other (without cyclic dependencies, of course). Go to the <link xlink:href="http://localhost:3000/project/patchelf/edit">Edit</link> page of the Patchelf project and "Add a new jobset" by providing the following "Information": <screen> Identifier: trunk Description: Trunk Nix expression: release.nix in input patchelfSrc </screen> This states that in order to build the <literal>trunk</literal> jobset, the Nix expression in the file <filename>release.nix</filename>, which can be obtained from input <literal>patchelfSrc</literal>, should be evaluated. (We'll have a look at <filename>release.nix</filename> later.) </para> <para> To realize a job we probably need a number of inputs, which can be declared in the table below. As many inputs as required can be added. For patchelf we declare the following inputs. <screen> patchelfSrc 'Subversion checkout' https://svn.nixos.org/repos/nix/patchelf/trunk nixpkgs 'Subversion checkout' https://svn.nixos.org/repos/nix/nixpkgs/trunk officialRelease Boolean false system String value "i686-linux" </screen> </para> </section> <section> <title>Release Set</title> <!-- TODO --> there must be one primary job check the radio button of exactly one job https://svn.nixos.org/repos/nix/nixpkgs/trunk </section> <section> <title>Building Jobs</title> </section> <section> <title>Build Recipes</title> <para> Build jobs and <emphasis>build recipes</emphasis> for a jobset are specified in a text file written in the <link xlink:href="http://nixos.org/nix/">Nix language</link>. The recipe is actually called a <emphasis>Nix expression</emphasis> in Nix parlance. By convention this file is often called <filename>release.nix</filename>. </para> <para> The <filename>release.nix</filename> file is typically kept under version control, and the repository that contains it one of the build inputs of the corresponding–often called <literal>hydraConfig</literal> by convention. The repository for that file and the actual file name are specified on the web interface of Hydra under the <literal>Setup</literal> tab of the jobset's overview page, under the <literal>Nix expression</literal> heading. See, for example, the <link xlink:href="http://hydra.nixos.org/jobset/patchelf/trunk">jobset overview page</link> of the PatchELF project, and <link xlink:href="https://svn.nixos.org/repos/nix/patchelf/trunk/release.nix"> the corresponding Nix file</link>. </para> <para> Knowledge of the Nix language is recommended, but the example below should already give a good idea of how it works: </para> <example xml:id='ex-hello'> <title><filename>release.nix</filename> file for GNU Hello</title> <programlisting> let pkgs = import <nixpkgs> {}; <co xml:id='ex-hello-co-import-nixpkgs' /> jobs = rec { <co xml:id='ex-hello-co-jobs' /> tarball = <co xml:id='ex-hello-co-tarball' /> pkgs.releaseTools.sourceTarball { <co xml:id='ex-hello-co-source-tarball' /> name = "hello-tarball"; src = <hello>; <co xml:id='ex-hello-co-tarball-args' /> buildInputs = (with pkgs; [ gettext texLive texinfo ]); }; build = <co xml:id='ex-hello-co-build' /> { system ? builtins.currentSystem }: <co xml:id='ex-hello-co-build-args' /> let pkgs = import <nixpkgs> { inherit system; }; in pkgs.releaseTools.nixBuild { <co xml:id='ex-hello-co-nix-build' /> name = "hello"; src = jobs.tarball; configureFlags = [ "--disable-silent-rules" ]; }; }; in jobs <co xml:id='ex-hello-co-body' /> </programlisting> </example> <para> <xref linkend='ex-hello' /> shows what a <filename>release.nix</filename> file for <link xlink:href="http://www.gnu.org/software/hello/">GNU Hello</link> would you like. GNU Hello is representative of many GNU and non-GNU free software projects: <itemizedlist> <listitem>it uses the GNU Build System, namely GNU Autoconf, and GNU Automake; for users, it means it can be installed using the <link xlink:href="http://www.gnu.org/prep/standards/html_node/Managing-Releases.html">usual <literal>./configure && make install</literal> procedure</link>; </listitem> <listitem>it uses Gettext for internationalization;</listitem> <listitem>it has a Texinfo manual, which can be rendered as PDF with TeX.</listitem> </itemizedlist> The file defines a jobset consisting of two jobs: <literal>tarball</literal>, and <literal>build</literal>. It contains the following elements (referenced from the figure by numbers): <calloutlist> <callout arearefs='ex-hello-co-import-nixpkgs'> <para> This defines a variable <varname>pkgs</varname> holding the set of packages provided by <link xlink:href="http://nixos.org/nixpkgs/">Nixpkgs</link>. </para> <para> Since <varname>nixpkgs</varname> appears in angle brackets, there must be a build input of that name in the Nix search path. In this case, the web interface should show a <varname>nixpkgs</varname> build input, which is a checkout of the Nixpkgs source code repository; Hydra then adds this and other build inputs to the Nix search path when evaluating <filename>release.nix</filename>. </para> </callout> <callout arearefs='ex-hello-co-jobs'> <para> This defines a variable holding the two Hydra jobs–an <emphasis>attribute set</emphasis> in Nix. </para> </callout> <callout arearefs='ex-hello-co-tarball'> <para> This is the definition of the first job, named <varname>tarball</varname>. The purpose of this job is to produce a usable source code tarball. </para> </callout> <callout arearefs='ex-hello-co-tarball-args'> <para> The <varname>tarball</varname> jobs expects a <varname>hello</varname> build input to be available in the Nix search path. Again, this input is passed by Hydra and is meant to be a checkout of GNU Hello's source code repository. </para> </callout> <callout arearefs='ex-hello-co-source-tarball'> <para> The <varname>tarball</varname> job calls the <varname>sourceTarball</varname> function, which (roughly) runs <command>autoreconf && ./configure && make dist</command> on the checkout. The <varname>buildInputs</varname> attribute specifies additional software dependencies for the job<footnote><para>The package names used in <varname>buildInputs</varname>–e.g., <varname>texLive</varname>–are the names of the <emphasis>attributes</emphasis> corresponding to these packages in Nixpkgs, specifically in the <link xlink:href="https://svn.nixos.org/repos/nix/nixpkgs/trunk/pkgs/top-level/all-packages.nix"><filename>all-packages.nix</filename></link> file. See the section entitled “Package Naming” in the Nixpkgs manual for more information.</para></footnote>. </para> </callout> <callout arearefs='ex-hello-co-build'> <para> This is the definition of the <varname>build</varname> job, whose purpose is to build Hello from the tarball produced above. </para> </callout> <callout arearefs='ex-hello-co-build-args'> <para> The <varname>build</varname> function takes one parameter, <varname>system</varname>, which should be a string defining the Nix system type–e.g., <literal>"x86_64-linux"</literal>. Additionally, it refers to <varname>jobs.tarball</varname>, seen above. </para> <para> Hydra inspects the formal argument list of the function (here, the <varname>system</varname> argument) and passes it the corresponding parameter specified as a build input on Hydra's web interface. Here, <varname>system</varname> is passed by Hydra when it calls <varname>build</varname>. Thus, it must be defined as a build input of type string in Hydra, which could take one of several values. </para> <para> The question mark after <literal>system</literal> defines the default value for this argument, and is only useful when debugging locally. </para> </callout> <callout arearefs='ex-hello-co-nix-build'> <para> The <varname>build</varname> job calls the <varname>nixBuild</varname> function, which unpacks the tarball, then runs <command>./configure && make && make check && make install</command>. </para> </callout> <callout arearefs='ex-hello-co-body'> <para> Finally, the set of jobs is returned to Hydra, as a Nix attribute set. </para> </callout> </calloutlist> </para> </section> <section> <title>Building from the Command Line</title> <para> It is often useful to test a build recipe, for instance before it is actually used by Hydra, when testing changes, or when debugging a build issue. Since build recipes for Hydra jobsets are just plain Nix expressions, they can be evaluated using the standard Nix tools. </para> <para> To evaluate the <varname>tarball</varname> jobset of <xref linkend='ex-hello' />, just run: <screen> $ nix-build release.nix -A tarball </screen> However, doing this with <xref linkend='ex-hello' /> as is will probably yield an error like this: <screen> error: user-thrown exception: file `hello' was not found in the Nix search path (add it using $NIX_PATH or -I) </screen> The error is self-explanatory. Assuming <filename>$HOME/src/hello</filename> points to a checkout of Hello, this can be fixed this way: <screen> $ nix-build -I ~/src release.nix -A tarball </screen> Similarly, the <varname>build</varname> jobset can be evaluated: <screen> $ nix-build -I ~/src release.nix -A build </screen> The <varname>build</varname> job reuses the result of the <varname>tarball</varname> job, rebuilding it only if it needs to. </para> </section> <section> <title>Adding More Jobs</title> <para> <xref linkend='ex-hello' /> illustrates how to write the most basic jobs, <varname>tarball</varname> and <varname>build</varname>. In practice, much more can be done by using features readily provided by Nixpkgs or by creating new jobs as customizations of existing jobs. </para> <para> For instance, test coverage report for projects compiled with GCC can be automatically generated using the <varname>coverageAnalysis</varname> function provided by Nixpkgs instead of <varname>nixBuild</varname>. Back to our GNU Hello example, we can define a <varname>coverage</varname> job that produces an HTML code coverage report directly readable from the corresponding Hydra build page: <programlisting> coverage = { system ? builtins.currentSystem }: let pkgs = import nixpkgs { inherit system; }; in pkgs.releaseTools.coverageAnalysis { name = "hello"; src = jobs.tarball; configureFlags = [ "--disable-silent-rules" ]; }; </programlisting> As can be seen, the only difference compared to <varname>build</varname> is the use of <varname>coverageAnalysis</varname>. </para> <para> Nixpkgs provides many more build tools, including the ability to run build in virtual machines, which can themselves run another GNU/Linux distribution, which allows for the creation of packages for these distributions. Please see <link xlink:href="https://svn.nixos.org/repos/nix/nixpkgs/trunk/pkgs/build-support/release/">the <filename>pkgs/build-support/release</filename> directory</link> of Nixpkgs for more. The NixOS manual also contains information about whole-system testing in virtual machine. </para> <para> Now, assume we want to build Hello with an old version of GCC, and with different <command>configure</command> flags. A new <varname>build_exotic</varname> job can be written that simply <emphasis>overrides</emphasis> the relevant arguments passed to <varname>nixBuild</varname>: <programlisting> build_exotic = { system ? builtins.currentSystem }: let pkgs = import nixpkgs { inherit system; }; build = jobs.build { inherit system; }; in pkgs.lib.overrideDerivation build (attrs: { buildInputs = [ pkgs.gcc33 ]; preConfigure = "gcc --version"; configureFlags = attrs.configureFlags ++ [ "--disable-nls" ]; }); </programlisting> The <varname>build_exotic</varname> job reuses <varname>build</varname> and overrides some of its arguments: it adds a dependency on GCC 3.3, a pre-configure phase that runs <command>gcc --version</command>, and adds the <literal>--disable-nls</literal> configure flags. </para> <para> This customization mechanism is very powerful. For instance, it can be used to change the way Hello and <emphasis>all</emphasis> its dependencies–including the C library and compiler used to build it–are built. See the Nixpkgs manual for more. </para> </section> </chapter> <!-- Local Variables: indent-tabs-mode: nil ispell-local-dictionary: "american" End: -->