104 Commits

Author SHA1 Message Date
Matej Cotman
a551fba346
statsd: add a chance to set hostname and port in hydra.conf
Co-authored-by: Graham Christensen <graham@grahamc.com>
2021-03-08 10:03:16 -05:00
Graham Christensen
f1e75c8bff
Move evaluation errors from evaluations to EvaluationErrors, a new table
DBIx likes to eagerly select all columns without a way to really tell
it so. Therefore, this splits this one large column in to its own
table.

I'd also like to make "jobsets" use this table too, but that is on hold
to stop the bleeding caused by the extreme amount of traffic this is
causing.
2021-02-01 21:33:14 -05:00
Graham Christensen
54b8cb188e
perl: jobsetevals -> jobset via by jobset_id
Frankly, this was suspiciously little work.
2021-01-26 13:51:39 -05:00
Nikola Knezevic
e34d40d4f8 Remove dead method from Nix.pm
This method has been moved to hydra-eval-jobset a long time ago.
2020-06-05 15:01:36 +02:00
Nikola Knezevic
f79810bac1 Improve handling of Perl's block eval errors
Taken from `Perl::Critic`:

A common idiom in perl for dealing with possible errors is to use `eval`
followed by a check of `$@`/`$EVAL_ERROR`:

    eval {
        ...
    };
    if ($EVAL_ERROR) {
        ...
    }

There's a problem with this: the value of `$EVAL_ERROR` (`$@`) can change
between the end of the `eval` and the `if` statement. The issue are object
destructors:

    package Foo;

    ...

    sub DESTROY {
        ...
        eval { ... };
        ...
    }

    package main;

    eval {
        my $foo = Foo->new();
        ...
    };
    if ($EVAL_ERROR) {
        ...
    }

Assuming there are no other references to `$foo` created, when the
`eval` block in `main` is exited, `Foo::DESTROY()` will be invoked,
regardless of whether the `eval` finished normally or not. If the `eval`
in `main` fails, but the `eval` in `Foo::DESTROY()` succeeds, then
`$EVAL_ERROR` will be empty by the time that the `if` is executed.
Additional issues arise if you depend upon the exact contents of
`$EVAL_ERROR` and both `eval`s fail, because the messages from both will
be concatenated.

Even if there isn't an `eval` directly in the `DESTROY()` method code,
it may invoke code that does use `eval` or otherwise affects
`$EVAL_ERROR`.

The solution is to ensure that, upon normal exit, an `eval` returns a
true value and to test that value:

    # Constructors are no problem.
    my $object = eval { Class->new() };

    # To cover the possiblity that an operation may correctly return a
    # false value, end the block with &quot;1&quot;:
    if ( eval { something(); 1 } ) {
        ...
    }

    eval {
        ...
        1;
    }
        or do {
            # Error handling here
        };

Unfortunately, you can't use the `defined` function to test the result;
`eval` returns an empty string on failure.

Various modules have been written to take some of the pain out of
properly localizing and checking `$@`/`$EVAL_ERROR`. For example:

    use Try::Tiny;
    try {
        ...
    } catch {
        # Error handling here;
        # The exception is in $_/$ARG, not $@/$EVAL_ERROR.
    };  # Note semicolon.

"But we don't use DESTROY() anywhere in our code!" you say. That may be
the case, but do any of the third-party modules you use have them? What
about any you may use in the future or updated versions of the ones you
already use?
2020-05-26 11:19:43 +02:00
Maximilian Bosch
721c764951
Remove Hydra::Helper::nix::txn_do from the Perl code
To quote the function's comment:

  Awful hack to handle timeouts in SQLite: just retry the transaction.
  DBD::SQLite *has* a 30 second retry window, but apparently it
  doesn't work.

Since SQLite is now dropped entirely, this wrapper can be removed
completely.
2020-04-16 00:42:40 +02:00
Graham Christensen
117b9ecef1
Nix.pm: readNixFile: pass «--experimental-features nix-command»
Declarative jobsets were broken by the Nix update, causing
nix cat-file to break silently.

This commit restores declarative jobsets, based on top of a commit
making it easier to see what broke.
2020-03-03 22:36:21 -05:00
Kevin Quick
2e225ba7c8
Do not attempt to report dir for grab command failure if not specified. 2019-03-17 23:15:24 -07:00
Kevin Quick
e523e8a643 Additional helpful information in error messages. 2018-06-29 17:41:23 -07:00
Eelco Dolstra
d8d71ea4a3 hydra-server: Use server_store_uri instead of store_uri, if set
The store_uri may contain parameters that cause Nix to fail when
running under the hydra-server user (e.g. the signing key).
2017-10-18 15:33:55 +02:00
Eelco Dolstra
bc60fccf78
Remove remaining references to store_mode etc. 2017-10-18 12:23:07 +02:00
Shea Levy
4c26cf0c81 getHydraConfig: Allow multiple includes 2017-04-20 08:00:09 -04:00
Shea Levy
7e3f12fe10 Merge branch 'decl-project-remote-store' 2017-04-10 11:36:41 -04:00
Eelco Dolstra
fec895a642
hydra-server: Support logs in S3 2017-04-05 17:55:56 +02:00
Shea Levy
a738f826e8 declarative projects: Pull jobset spec build from the remote store
Fixes #443
2017-04-01 10:38:49 -04:00
Eelco Dolstra
81ac547d2b
Move most of AddBuilds to hydra-eval-jobset
Having this stuff in a separate module is a remnant of the time when
hydra-server could add builds to the queue directly. This is no longer
the case.
2017-02-21 17:20:48 +01:00
Eelco Dolstra
f2724d088d Test: Show error output from hydra-queue-runner on timeout 2016-10-06 17:18:59 +02:00
Joe Hermaszewski
02baff987a Enable apache style includes in the config file 2016-06-19 16:59:24 +01:00
Eelco Dolstra
b081133dcc Move GC roots to /nix/var/nix/gcroots/hydra
The uid split a while back caused the web interface to create GC roots
in /nix/var/nix/gcroots/per-user/hydra-www, where they wouldn't be
purged by hydra-update-gc-roots. Thus restarted builds would
accumulate forever. The fix is to keep the roots in a shared directory
with gid=hydra.
2015-12-14 13:31:24 +01:00
Eelco Dolstra
ea80a75204 Move getBaseUrl 2015-11-17 11:21:20 +01:00
Eelco Dolstra
b9ebd38edf Fix restarting builds
Failed paths were not cleared because the $builds query becomes empty
as soon as the "finished" flag has been cleared. So this needs to be
done last.
2015-10-28 14:54:54 +01:00
Eelco Dolstra
4d1816b152 Remove obsolete Builds columns and provide accurate "Running builds"
This removes the "busy", "locker" and "logfile" columns, which are no
longer used by the queue runner. The "Running builds" page now only
shows builds that have an active build step.
2015-10-27 15:37:17 +01:00
Eelco Dolstra
9fb91460e3 Restart jobset evals efficiently
Fixes

  DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR: stack depth limit exceeded
  HINT: Increase the configuration parameter "max_stack_depth" (currently 2048kB), after ensuring the platform's stack depth limit is adequate. [for Statement "UPDATE Builds SET busy = ?, finished = ?, iscachedbuild = ?, locker = ? WHERE ( ( id = ? OR id = ? OR ...
2015-10-27 13:43:19 +01:00
Eelco Dolstra
1a92f971d4 Restart: Clear failed dependencies 2015-10-09 15:39:41 +02:00
Eelco Dolstra
7aa52517e9 Support multiple machines files
This is primarily useful for the Hydra provisioner, which can write
its machines to another file than /etc/nix/machines.
2015-08-25 15:34:53 +02:00
Eelco Dolstra
06d75699a7 Fix restarting a build 2015-07-10 16:56:53 +02:00
Eelco Dolstra
bbee81efae Use triggers for all notifications on Builds table changes 2015-07-08 12:05:32 +02:00
Eelco Dolstra
95c4294560 Allow cancelling builds marked as busy
Note that if there are active build *steps*, this won't cancel them.
2015-07-07 14:08:46 +02:00
Eelco Dolstra
133d298e26 Asynchronously compress build logs 2015-06-19 15:06:12 +02:00
Eelco Dolstra
c974fb893b Support cancelling builds 2015-06-11 18:07:45 +02:00
Eelco Dolstra
c08883966c Use PostgreSQL notifications for queue events
Hydra-queue-runner now no longer polls the queue periodically, but
instead sleeps until it receives a notification from PostgreSQL about
a change to the queue (build added, build cancelled or build
restarted).

Also, for the "build added" case, we now only check for builds with an
ID greater than the previous greatest ID. This is much more efficient
if the queue is large.
2015-06-11 17:41:59 +02:00
Eelco Dolstra
6d738a31bf Keep track of failed paths in the Hydra database
I.e. don't use Nix's failed paths feature anymore. Easier to keep
everything in one place.
2015-06-10 14:57:16 +02:00
Eelco Dolstra
8b12ac1f6d Basic remote building
This removes the need for Nix's build-remote.pl.

Build logs are now written to $HYDRA_DATA/build-logs because
hydra-queue-runner doesn't have write permission to /nix/var/log.
2015-06-09 14:21:21 +02:00
Eelco Dolstra
9282bcb95f Show eval input changes in deterministic order 2015-02-25 13:13:12 +01:00
Eelco Dolstra
0d28e9372b Remove dead code 2014-11-19 14:59:36 +01:00
Eelco Dolstra
42b23133e8 Fix UTF-8 handling of log files 2014-08-13 18:53:29 +02:00
Eelco Dolstra
9334b84d6f Fix registerRoot
Fixes #175.
2014-08-13 16:29:00 +02:00
Eelco Dolstra
69e3aa0438 Write Hydra roots as regular files instead of symlinks
Note that this requires at least NixOS/Nix@1c208f2b7e.
2014-08-01 17:24:55 +02:00
Eelco Dolstra
6b88be040e Make restartBuilds faster 2014-07-18 00:03:26 +02:00
Eelco Dolstra
ae85d2df23 Add a "My jobsets" tab to the dashboard 2013-11-05 16:05:29 +01:00
Eelco Dolstra
cbd56824a2 Use /etc/nix/machines 2013-10-30 11:21:23 +01:00
Eelco Dolstra
c4e39d4769 Add one-shot jobsets
There are jobsets that are evaluated only once, that is, after they've
been evaluated, they're disabled automatically.  This is primarily
useful for doing releases: for instance, doing an evaluation with
"officialRelease" set to "true" should be done only once.
2013-10-11 12:01:52 +02:00
Eelco Dolstra
5294a0a8a0 Register restarted derivations as GC roots 2013-10-04 17:11:42 +02:00
Eelco Dolstra
052bab169d Add a jobset eval action to restart all aborted/cancelled builds 2013-10-04 17:01:47 +02:00
Eelco Dolstra
7818bb75ed Add an action to cancel all builds in a jobset eval 2013-10-04 15:40:43 +02:00
Eelco Dolstra
4ed877360b hydra-queue-runner: Improved scheduling
Each jobset now has a "scheduling share" that determines how much of
the build farm's time it is entitled to.  For instance, if a jobset
has 100 shares and the total number of shares of all jobsets is 1000,
it's entitled to 10% of the build farm's time.  When there is a free
build slot for a given system type, the queue runner will select the
jobset that is furthest below its scheduling share over a certain time
window (currently, the last day).  Withing that jobset, it will pick
the build with the highest priority.

So meta.schedulingPriority now only determines the order of builds
within a jobset, not between jobsets.  This makes it much easier to
prioritise one jobset over another (e.g. nixpkgs:trunk over
nixpkgs:stdenv).
2013-09-21 14:57:01 +00:00
Petr Rockai
dcf386cfed Make getDrvLogPath work with both bucketed and non-bucketed nix logs. 2013-09-07 13:28:53 +02:00
Shea Levy
788c2bebaa Not every buildstep has a defined drvpath
Signed-off-by: Shea Levy <shea@shealevy.com>
2013-09-03 14:43:08 -04:00
Eelco Dolstra
6655ada236 Serve raw uncompressed logs directly 2013-08-30 18:11:03 +00:00
Eelco Dolstra
fd7e37ef89 Try harder to find build logs
Due to the fixed-output derivation hashing scheme, there can be
multiple derivations of the same output path.  But build logs are
indexed by derivation path.  Thus, we may not be able to find the
log of a build or build step using its derivation.  So as a fallback,
Hydra now looks for other derivations with the same output paths.
2013-08-30 13:53:25 +00:00