Commit Graph

139 Commits

Author SHA1 Message Date
e499509595 Switch to new Nix bindings, update Nix for that
Implements support for Nix's new Perl bindings[1]. The current state
basically does `openStore()`, but always uses `auto` and doesn't support
stores at other URIs.

Even though the stores are cached inside the Perl implementation, I
decided to instantiate those once in the Nix helper module. That way
store openings aren't cluttered across the entire codebase. Also, there
are two stores used later on - MACHINE_LOCAL_STORE for `auto`,
BINARY_CACHE_STORE for the one from `store_uri` in `hydra.conf` - and
using consistent names should make the intent clearer then.

This doesn't contain any behavioral changes, i.e. the build product
availability issue from #1352 isn't fixed. This patch only contains the
migration to the new API.

[1] https://github.com/NixOS/nix/pull/9863
2024-02-12 18:50:56 +01:00
bab671124d replace nix cat-store with nix store cat
the former was deprecated in favor of the latter
2022-06-30 00:24:09 +02:00
f07fb7d279 LDAP support: include BC support for the YAML based loading
Includes a refactoring of the configuration loader.
2022-02-11 10:49:38 -05:00
845e6d4760 captureStdoutStderr*: move to Hydra::Helper::Exec which helps avoid some environment variable fixation problems 2022-02-09 14:28:50 -05:00
61189ecca9 Helper/Nix: constructRunCommandLogPath: verify uuid is valid
This shouldn't be possible normally, but it is possible to:

    $db->resultset('RunCommandLogs')->new({ uuid => "../etc/passwd" });

if you have access to the `$db`.
2022-01-31 08:58:33 -08:00
e381751564 Helper/Nix: constructRunCommandLogPath: return undef in case of an error
This allows us to give a web request to an invalid UUID a 404.
2022-01-31 08:58:33 -08:00
8eab7b8543 Helper/Nix: constructRunCommandLogPath: take RunCommandLog as input
This way we ensure that it actually exists in the database, rather than
blindly trusting user-generated input.
2022-01-31 08:58:33 -08:00
fc3cf4ecb2 RunCommandLogs: identify and access via uuid
Using a sha1 of the command combined with the build ID is not a
particularly good or unique identifier:

* A build could fail, be restarted, and then succeed -- assuming no
configuration changes, the sha1 hash of the command as well as the build
ID will be the same. This would lead to an overwritten log file.

* Allowing user input to influence filenames is not the best of ideas.
2022-01-31 08:58:33 -08:00
5d3912962b RunCommand: use helper functions to ensure filenames and paths are the same
Otherwise, it's possible someone updates the format in one place but not
the others, leading to broken or incorrect functionality.
2022-01-28 13:03:15 -08:00
3a6c25489c Hydra::Helper::Nix: expose a captureStdoutStderrWithStdin, make it available in tests 2022-01-21 10:40:06 -05:00
d4fe7e55dd Hydra::Helper::Nix: sort exported functions 2022-01-21 10:40:06 -05:00
5c5ecdb9fc Helper::Nix::isLocalStore: a file store is local 2022-01-14 11:28:39 -05:00
e43f487157 perlcritic: return undef in list context too 2021-12-14 10:24:53 -05:00
5fbf1470bd (perl) machines file: support machine lines with multiple spaces between fields 2021-10-24 21:30:53 -04:00
b817124337 Hydra::Helper::Nix::getMachines: add a test
Fix parsing breakage from #1003: assigning the lines to $lines broke chomp and the filters.

This test validates the parsing works as expected, and also fixes
a minor bug where '-' in features isn't pruned, like in the C++
repo.
2021-10-23 22:53:55 -04:00
d98d74fa4f perlcritic: doesn't like prototypes 2021-10-20 13:09:39 -04:00
7095d00608 perlcritic: make all open() calls three-argument
Two-argument calls end up parsing the second argument to guess what
should happen, three-arg parses ... less?
2021-10-20 13:09:39 -04:00
741590c923 perlcritic: explicitly assign the result of readdir/file reads 2021-09-07 21:55:55 -04:00
c603ae35f0 perlcritic: Don't conditionally declare variables 2021-09-07 21:35:01 -04:00
21e1ff0da1 perlcritic: don't open files as bare words 2021-09-06 22:20:17 -04:00
efd1d78b97 Hydra::Helper::Nix: getHydraNotifyPrometheusConfig fixup strict warning around double-declaring cfg 2021-09-06 22:13:33 -04:00
c54629b732 Helper/Nix.pm: perlcritic statsd in scalar context 2021-09-06 22:13:33 -04:00
4677a7c894 perlcritic: use strict, use warnings 2021-09-06 22:13:33 -04:00
24467a7bde Nix::getHydraNotifyPrometheusConfig: print errors if the configuration provided is invalid. 2021-08-24 11:35:38 -04:00
6d7ee27d25 hydra-notify: make the prometheus endpoint configurable, default-off 2021-08-24 10:56:13 -04:00
4632132d02 fixup perlcritic: indirect new
> Subroutine new called using indirect syntax at line 14, column 14.  See page 349 of PBP.  (Severity: 5)
2021-08-20 11:39:19 -04:00
503b0e0b6f Remove unnecessary comment 2021-07-26 17:16:40 +00:00
173ecb5d92 Apply requested changes to Config.pm 2021-07-26 17:16:40 +00:00
4f95774ca3 Factor out common options 2021-07-26 03:43:09 +00:00
e98bd0ec8d Comment around dupe to avoid drift 2021-07-22 23:27:21 -04:00
09ad52ab60 Document getEvals a bit
Also drop the $self parameter.

Co-authored-by: gustavderdrache
2021-06-16 13:15:01 -04:00
5d95abf540 getBuilds: clarify the names of evals vs. the query builder 2021-06-16 11:53:04 -04:00
bf5c76feb6 getEvals: order by the eval table's ID
I broke this when I added `me.` in f1e75c8bff

I added me. to disambiguate `id`, but:

* eval.id works on the per-build page
* me.id works on the other pages
* Just id works everywhere if I drop:

    , prefetch => { evaluationerror => [ ] },

  but this causes a query per row to collect the evaluationerror
  records later, this becomes significantly slow on non-trivial
  datasets.

Using evals->current_source_alias will use the correct alias
whether it is me or eval or something else.
2021-06-16 11:24:30 -04:00
719d0a6134 jobset overview: join jobsets to builds by jobset id 2021-06-01 11:19:33 -04:00
cc9c91fe12 jobsets: put hidden and enabled jobsets at the end
Allows for generally correct zebra striping
2021-03-31 14:33:20 +00:00
a551fba346 statsd: add a chance to set hostname and port in hydra.conf
Co-authored-by: Graham Christensen <graham@grahamc.com>
2021-03-08 10:03:16 -05:00
f1e75c8bff Move evaluation errors from evaluations to EvaluationErrors, a new table
DBIx likes to eagerly select all columns without a way to really tell
it so. Therefore, this splits this one large column in to its own
table.

I'd also like to make "jobsets" use this table too, but that is on hold
to stop the bleeding caused by the extreme amount of traffic this is
causing.
2021-02-01 21:33:14 -05:00
54b8cb188e perl: jobsetevals -> jobset via by jobset_id
Frankly, this was suspiciously little work.
2021-01-26 13:51:39 -05:00
e34d40d4f8 Remove dead method from Nix.pm
This method has been moved to hydra-eval-jobset a long time ago.
2020-06-05 15:01:36 +02:00
f79810bac1 Improve handling of Perl's block eval errors
Taken from `Perl::Critic`:

A common idiom in perl for dealing with possible errors is to use `eval`
followed by a check of `$@`/`$EVAL_ERROR`:

    eval {
        ...
    };
    if ($EVAL_ERROR) {
        ...
    }

There's a problem with this: the value of `$EVAL_ERROR` (`$@`) can change
between the end of the `eval` and the `if` statement. The issue are object
destructors:

    package Foo;

    ...

    sub DESTROY {
        ...
        eval { ... };
        ...
    }

    package main;

    eval {
        my $foo = Foo->new();
        ...
    };
    if ($EVAL_ERROR) {
        ...
    }

Assuming there are no other references to `$foo` created, when the
`eval` block in `main` is exited, `Foo::DESTROY()` will be invoked,
regardless of whether the `eval` finished normally or not. If the `eval`
in `main` fails, but the `eval` in `Foo::DESTROY()` succeeds, then
`$EVAL_ERROR` will be empty by the time that the `if` is executed.
Additional issues arise if you depend upon the exact contents of
`$EVAL_ERROR` and both `eval`s fail, because the messages from both will
be concatenated.

Even if there isn't an `eval` directly in the `DESTROY()` method code,
it may invoke code that does use `eval` or otherwise affects
`$EVAL_ERROR`.

The solution is to ensure that, upon normal exit, an `eval` returns a
true value and to test that value:

    # Constructors are no problem.
    my $object = eval { Class->new() };

    # To cover the possiblity that an operation may correctly return a
    # false value, end the block with &quot;1&quot;:
    if ( eval { something(); 1 } ) {
        ...
    }

    eval {
        ...
        1;
    }
        or do {
            # Error handling here
        };

Unfortunately, you can't use the `defined` function to test the result;
`eval` returns an empty string on failure.

Various modules have been written to take some of the pain out of
properly localizing and checking `$@`/`$EVAL_ERROR`. For example:

    use Try::Tiny;
    try {
        ...
    } catch {
        # Error handling here;
        # The exception is in $_/$ARG, not $@/$EVAL_ERROR.
    };  # Note semicolon.

"But we don't use DESTROY() anywhere in our code!" you say. That may be
the case, but do any of the third-party modules you use have them? What
about any you may use in the future or updated versions of the ones you
already use?
2020-05-26 11:19:43 +02:00
721c764951 Remove Hydra::Helper::nix::txn_do from the Perl code
To quote the function's comment:

  Awful hack to handle timeouts in SQLite: just retry the transaction.
  DBD::SQLite *has* a 30 second retry window, but apparently it
  doesn't work.

Since SQLite is now dropped entirely, this wrapper can be removed
completely.
2020-04-16 00:42:40 +02:00
117b9ecef1 Nix.pm: readNixFile: pass «--experimental-features nix-command»
Declarative jobsets were broken by the Nix update, causing
nix cat-file to break silently.

This commit restores declarative jobsets, based on top of a commit
making it easier to see what broke.
2020-03-03 22:36:21 -05:00
2e225ba7c8 Do not attempt to report dir for grab command failure if not specified. 2019-03-17 23:15:24 -07:00
e523e8a643 Additional helpful information in error messages. 2018-06-29 17:41:23 -07:00
d8d71ea4a3 hydra-server: Use server_store_uri instead of store_uri, if set
The store_uri may contain parameters that cause Nix to fail when
running under the hydra-server user (e.g. the signing key).
2017-10-18 15:33:55 +02:00
bc60fccf78 Remove remaining references to store_mode etc. 2017-10-18 12:23:07 +02:00
4c26cf0c81 getHydraConfig: Allow multiple includes 2017-04-20 08:00:09 -04:00
7e3f12fe10 Merge branch 'decl-project-remote-store' 2017-04-10 11:36:41 -04:00
fec895a642 hydra-server: Support logs in S3 2017-04-05 17:55:56 +02:00
a738f826e8 declarative projects: Pull jobset spec build from the remote store
Fixes #443
2017-04-01 10:38:49 -04:00