70 Commits

Author SHA1 Message Date
Graham Christensen
c30f084f32 eval_started event: change interface to traceID\tjobsetID
I was not going to break the interface until I noticed
the current implementation uses the string literal \t.
2022-02-08 09:51:35 -05:00
Graham Christensen
9dc40e0816 evaluator: don't save project, jobset on builds 2022-01-15 15:58:02 -05:00
Graham Christensen
8c3c573953 hydra-eval-jobset: fixup old reference to project / jobset columns 2022-01-15 12:32:16 -05:00
Graham Christensen
4da80e736e hydra-eval-jobset: send notifications when cached queued / finished builds are submitted 2022-01-11 13:28:04 -05:00
Janne Heß
a69693a832
hydra-eval-jobset: Fix two minor bugs
I'm honestly too lazy to create two commits for fixing these one-line
issues so here's one.

The first hunk fixes the name of the projectName input. This is relevant
now because it gets logged and the log message looks stupid when there
is an input without a name.

The second hunk fixes a warning when using declarative non-flake
jobsets. The implementation may look weird but it's just the same as the
logical implication operator of nix.
2022-01-10 18:28:41 +01:00
Janne Heß
f3d77c3e6b
hydra-eval-jobset: Print the jobset that is evaluated
This is useful for systems that use concurrent evals
2022-01-10 13:44:07 +01:00
Graham Christensen
e84bbc7f90 hydra-eval-jobset: notify build_queued 2021-12-21 20:57:53 -05:00
Graham Christensen
4dfe787bc2 perlcritic: each() called at line 752, column 35. The each function may cause undefined behavior when operating on the hash while iterating. Use a foreach loop over the hash's keys or values instead. 2021-12-14 10:16:25 -05:00
Graham Christensen
7dcf6a01c6 JSON -> JSON::MaybeXS 2021-12-13 15:37:56 -05:00
Graham Christensen
2cbeca5c44
Merge pull request #1071 from DeterminateSystems/log-fetches-evals
hydra-eval-jobset: log fetches and evaluations
2021-12-08 16:00:29 -05:00
Graham Christensen
264092169c hydra-eval-jobset: log fetches and evaluations 2021-12-08 09:25:27 -05:00
Cole Helbling
47cabac4bf hydra-eval-jobset: fix use of uninitialized value
This happens with flake jobsets for obvious reasons (namely, that nixexprinput
and nixexprpath may be undefined for a flake jobset).

12:38:59 hydra-evaluator.1    | Use of uninitialized value $args[0] in join or string at /home/vin/workspace/vcs/hydra/src/script/hydra-eval-jobset line 648.
12:38:59 hydra-evaluator.1    | Use of uninitialized value $args[1] in join or string at /home/vin/workspace/vcs/hydra/src/script/hydra-eval-jobset line 648.
2021-12-03 12:12:53 -08:00
Graham Christensen
0f8d02894a hydra-eval-jobset: Scalar value @declInputs[0] better written as $declInputs[0] at hydra-eval-jobset line 570. 2021-10-19 21:52:39 -04:00
Graham Christensen
c603ae35f0 perlcritic: Don't conditionally declare variables 2021-09-07 21:35:01 -04:00
Graham Christensen
c880888f1e File::Slurp -> File::Slurper 2021-09-06 22:13:33 -04:00
Your Name
4677a7c894 perlcritic: use strict, use warnings 2021-09-06 22:13:33 -04:00
Graham Christensen
508d99d611 Join to builds via jobset_id when easy 2021-06-01 11:16:47 -04:00
Janne Heß
3c86083d21
Fixup #717 "Add the project name to declarative inputs"
```
Mar 10 16:22:35 hydra-b hydra-evaluator[41419]: DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::Pg::st execute failed: ERROR:  null value in column "type" violates not-null constraint
Mar 10 16:22:35 hydra-b hydra-evaluator[41419]: DETAIL:  Failing row contains (62358, projectName, 0, null, null, null, hackworthltd, null, , null). [for Statement "INSERT INTO jobsetevalinputs ( altnr, dependency, eval, name, path, revision, sha256hash, type, uri, value) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )" with ParamValues: 1='0', 2=undef, 3='62358', 4='projectName', 5='', 6=undef, 7=undef, 8=undef, 9=undef, 10='hackworthltd'] at /nix/store/cmqblv437mp57yz5lwvkzcqca4ldf3r5-hydra-0.1.20210308.ebf1cd2/bin/.hydra-eval-jobset-wrapped line 793
Mar 10 16:22:35 hydra-b hydra-evaluator[25828]: evaluation of jobset ‘hackworthltd:.jobsets (jobset#1)’ failed with exit code 1
```

Use the abstraction for creating inputs for simulating the project
name input.

Co-authored-by: Graham Christensen <graham@grahamc.com>
2021-03-16 09:52:36 -04:00
Janne Heß
9e018d5443
Add the project name to declarative inputs
This allows for more generic declarative configurations which can be
shared between projects.
2021-03-08 17:36:52 +01:00
Matej Cotman
a551fba346
statsd: add a chance to set hostname and port in hydra.conf
Co-authored-by: Graham Christensen <graham@grahamc.com>
2021-03-08 10:03:16 -05:00
Graham Christensen
f1e75c8bff
Move evaluation errors from evaluations to EvaluationErrors, a new table
DBIx likes to eagerly select all columns without a way to really tell
it so. Therefore, this splits this one large column in to its own
table.

I'd also like to make "jobsets" use this table too, but that is on hold
to stop the bleeding caused by the extreme amount of traffic this is
causing.
2021-02-01 21:33:14 -05:00
Graham Christensen
54b8cb188e
perl: jobsetevals -> jobset via by jobset_id
Frankly, this was suspiciously little work.
2021-01-26 13:51:39 -05:00
Graham Christensen
9516b256f1
Normalize nixexpr{input,path} from builds to jobsetevals.
Duplicating this data on every record of the builds table cost
approximately 4G of duplication.

Note that the database migration included took about 4h45m on an
untuned server which uses very slow rotational disks in a RAID5 setup,
with not a lot of RAM. I imagine in production it might take an hour
or two, but not 4. If this should become a chunked migration, I can do
that.

Note: Because of the question about chunked migrations, I have NOT
YET tested this migration thoroughly enough for merge.
2021-01-22 09:10:18 -05:00
Graham Christensen
086eed5147
hydra-eval-jobs: write evaluation errorMsg to the jobseteval table 2021-01-21 13:10:41 -05:00
Eelco Dolstra
be709d450b Fix sysbuild
596f4cf4b9
2020-10-22 13:27:52 +02:00
Janne Heß
24acb9d6bb
Fix non-static declarative jobsets
With the current implementation, if ANY hash was found inside the decl
spec, the spec would be treated as static. This is problematic since
`inputs` is a hash and hence any configuration would be handled as a
static one.
This fixes the code to match the documentation and only switch to static
processing when ALL values are hashes.
2020-09-13 18:21:38 +02:00
Graham Christensen
7f16c0d243
declarative projects: support fully static, declarative configuration 2020-09-02 12:35:41 -04:00
Eelco Dolstra
750e2e618a
Merge pull request #770 from NixOS/remove-jobs
Remove the Jobs table
2020-06-01 10:25:41 +02:00
Nikola Knezevic
7148923d30 Make --no-allow-import-from-derivation configurable in hydra-eval-jobset
When deploying Hydra different than hydra.nixos.org one may encounter a problem
as building any job that uses IFD fails with:

May 22 19:41:07 hydra hydra-evaluator[6960]: error: "attempted to realize '/nix/store/1jm02mfiv58rpy8zrx95cpqxzsp64ssh-source.drv' during evaluation but 'allow-import-from-derivation' is false"
May 22 19:41:07 hydra hydra-evaluator[6960]: error: "attempted to realize '/nix/store/av3jr8ix4qcadq2wm3y3hplvxwzlhl4y-source.drv' during evaluation but 'allow-import-from-derivation' is false"
May 22 19:41:07 hydra hydra-evaluator[6960]: error: "attempted to realize
'/nix/store/2jm02mfiv58rpy8zrx95cpqxzsp64ssh-source.drv' during evaluation but
'allow-import-from-derivation' is false"

The recent change enforced passing `--no-allow-import-from-derivation`
to `hydra-eval-job` unconditionally. This change makes it configurable and
defaults to **NOT PASSING IT** -- most of the deployments allow IFDs.

The configuration option is called `allow_import_from_derivation` and
defaults to `true`. It is interpreted as a boolean, with only true option being
`true`.
2020-05-28 10:15:46 +02:00
Eelco Dolstra
8adb433e3b
Remove the Jobs table
This table has been superfluous for a long time.
2020-05-27 20:09:36 +02:00
Eelco Dolstra
e379628db0
hydra-eval-jobset: Pass --no-allow-import-from-derivation
https://github.com/NixOS/nixpkgs/issues/87592
2020-05-12 15:17:50 +02:00
Maximilian Bosch
721c764951
Remove Hydra::Helper::nix::txn_do from the Perl code
To quote the function's comment:

  Awful hack to handle timeouts in SQLite: just retry the transaction.
  DBD::SQLite *has* a 30 second retry window, but apparently it
  doesn't work.

Since SQLite is now dropped entirely, this wrapper can be removed
completely.
2020-04-16 00:42:40 +02:00
Eelco Dolstra
4b5bb4e760
Merge remote-tracking branch 'origin/master' into flake 2020-03-04 15:28:23 +01:00
Graham Christensen
994430b94b
treewide: allow nix command 2020-03-03 22:52:20 -05:00
Eelco Dolstra
2a50daa377
Update aggregate handling
(cherry picked from commit cf961ac8933b76d9f0a3ac3eb49cc08879f5bcc9)
2020-02-20 10:13:39 +01:00
Eelco Dolstra
15187b059b
Remove hydra-eval-guile-jobs
This hasn't been used in a long time (Guix uses its own CI system),
and it probably doesn't work anymore.

(cherry picked from commit 23c9ca3e94669087d463642baea0cf35a0b8d72f)
2020-02-20 09:58:12 +01:00
Eelco Dolstra
6f1d68bda4
Revert "hydra-eval-jobs -> nix eval-hydra-jobs"
This reverts commit 345512a6d0743eaac9d451833ecc1a71d12809c9.
2020-02-19 20:36:52 +01:00
Eelco Dolstra
cf961ac893
Update aggregate handling 2020-02-17 16:33:25 +01:00
Eelco Dolstra
345512a6d0
hydra-eval-jobs -> nix eval-hydra-jobs 2020-02-15 15:59:34 +01:00
Eelco Dolstra
23c9ca3e94
Remove hydra-eval-guile-jobs
This hasn't been used in a long time (Guix uses its own CI system),
and it probably doesn't work anymore.
2020-02-15 15:59:34 +01:00
Eelco Dolstra
881b7449fd Merge remote-tracking branch 'origin/master' into flake 2020-02-11 14:23:16 +01:00
Graham Christensen
f0f41eaaff
LatestSucceededForJob{,set}: Filter with jobset_id 2020-02-11 07:06:20 -05:00
Eelco Dolstra
100e09a5b3
Merge remote-tracking branch 'origin/master' into flake
Also update flake.lock
2020-02-10 17:58:10 +01:00
Graham Christensen
f3a561aecd
Builds: populate Builds.jobset_id in hydra-eval-jobset 2020-02-10 11:43:02 -05:00
Graham Christensen
624f1d8d2d
Jobs: populate Jobs.jobset_id field when writing from hydra-eval-jobset 2020-02-10 11:43:02 -05:00
Eelco Dolstra
55b0afa08f
Merge remote-tracking branch 'origin/master' into flake 2019-11-07 18:42:15 +01:00
Bas van Dijk
840e99f859 hydra-eval-jobset: $firstOutput is not used so can be removed 2019-11-05 13:58:40 +01:00
Eelco Dolstra
a816730d8b uri -> url 2019-10-23 20:31:27 +02:00
Eelco Dolstra
2de52d8538
Merge remote-tracking branch 'origin/master' into flake 2019-08-15 13:56:00 +02:00
Eelco Dolstra
92d8d6baa5
Avoid fetching Projects/Jobsets just to get the name column
In particular, doing a 'select * from Jobsets where ...' must be
avoided, because the 'errormsg' column can be very big.
2019-08-13 18:18:25 +02:00