ahuston-0/hydra
1
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity
3,847 Commits 3 Branches 0 Tags
Commit Graph

218 Commits

Author SHA1 Message Date
Graham Christensen
5de08d412e queue metrics: refactor the metrics into a struct 2022-04-06 20:00:30 -04:00
Graham Christensen
46f52b4c4e bring back the working version Cole made 2022-04-06 15:49:38 -04:00
Cole Helbling
5bff730f2c WIP: I love it when they delete the assignment operator :) 2022-04-06 11:41:40 -07:00
Cole Helbling
edf3c348f2 hydra-queue-runner: make entire address configurable 2022-04-06 10:59:45 -07:00
Cole Helbling
33bc60b83c hydra-queue-runner: move exporter back to State::run 
It's (arguably) better than risking pinning the thread at 100% due to
the busy `while` loop.
 2022-04-06 10:49:14 -07:00
Cole Helbling
8c5636fe18
hydra-queue-runner: use port 9198 by default 
Co-authored-by: Graham Christensen <graham@grahamc.com>
 2022-04-02 17:32:14 -07:00
Cole Helbling
4789eba92c hydra-queue-runer: split metrics functionality into its own function 2022-03-29 10:55:28 -07:00
Cole Helbling
928b3b8268 hydra-queue-runner: fix priority of flag over config file 2022-03-29 10:42:07 -07:00
Cole Helbling
5ddb9a98ca fixup! hydra-queue-runner: log message before and after exporter is started 2022-03-29 08:47:41 -07:00
Cole Helbling
905a7a7beb hydra-queue-runner: read metrics port from queue_runner_metrics_port config 2022-03-29 08:46:43 -07:00
Cole Helbling
9cdc5aceed hydra-queue-runner: log message before and after exporter is started 
This way, if something goes wrong between the two, it's easier to narrow
down where the issue lies.
 2022-03-29 08:41:19 -07:00
Cole Helbling
8503a7917b fixup! hydra-queue-runner: make registry member of State, configurable metrics port 2022-03-22 13:38:13 -07:00
Cole Helbling
c0f826b92d hydra-queue-runner: get the listening port from the exposer itself 
Otherwise, when the port is randomly chosen (e.g. by specifying no port,
or a port of 0), it will just show that the port is 0 and not the port
that is actually serving the metrics.
 2022-03-14 08:41:45 -07:00
Cole Helbling
52a29d43e6 hydra-queue-runner: make registry member of State, configurable metrics port 
Thanks to the updated prometheus-cpp library, specifying a port of 0
will cause it to pick a random (available) port -- ideal for tests.
 2022-03-11 11:58:10 -08:00
Cole Helbling
3bf31bd6a6 hydra-queue-runner: add simple "up" exporter 
There are probably better ways to achieve this (and will likely need to
be refactored a bit to support further metrics).
 2022-03-10 12:36:58 -08:00
Graham Christensen
4acaf9c8b0 hydra-queue-runner: don't dispatch until the machines parser has completed one run 
Periodically, I have seen tests fail because of out of order queue runner behavior:

    checking the queue for builds > 0...
    loading build 1 (tests:basic:empty_dir)
    aborting unsupported build step '...-empty-dir.drv' (type 'x86_64-linux')
    marking build 1 as failed
    adding new machine ‘localhost’

This patch should prevent the dispatcher from running before any machines are
made available.
 2022-02-10 10:54:30 -05:00
regnat
abff212d06 Use system-features from the Nix conf in the default machine file 
Fix #936
 2021-04-28 11:43:04 +02:00
Eelco Dolstra
a7d8ee98da Fix build 2021-02-22 15:10:24 +01:00
Maximilian Bosch
9cc76f6d69
Fix build with latest Nix 
Recently a few internal APIs have changed[1]. The `outputPaths` function
has been removed and a lot of data structures are modeled with
`std::optional` which broke compilation.

This patch updates the code in `hydra-queue-runner` accordingly to make
sure that Hydra compiles again.

[1] https://github.com/NixOS/nix/pull/3883
 2020-09-26 23:37:39 +02:00
Eelco Dolstra
f8e15bc311 Revive putBytes 2020-08-04 18:25:21 +02:00
Eelco Dolstra
4b5813051b
unsigned long long -> uint64_t 2020-08-04 11:38:22 +02:00
Eelco Dolstra
7d3ba616a9
Fix build 2020-08-04 11:33:29 +02:00
Eelco Dolstra
a0e24f446b
Remove unused getMemSize() function 2020-07-27 20:40:57 +02:00
Eelco Dolstra
d4e4be4fd1
Remove SHA-1 hash from BuildProducts 
SHA-1 is deprecated and it will be expensive to compute with the
streaming NAR handler.
 2020-07-27 18:24:10 +02:00
Eelco Dolstra
cbcf6359b4
Remove TokenServer in preparation of making NAR copying O(1) memory 2020-07-27 14:57:22 +02:00
Eelco Dolstra
7985757a1d
Fix build 2020-07-08 12:50:02 +02:00
Eelco Dolstra
bb32aafa4a
Fix build 2020-06-23 13:56:44 +02:00
Graham Christensen
7b705758ec
Merge pull request #732 from Ma27/fix-build 
Fix build against latest Nix
 2020-04-09 09:02:45 -04:00
Bas van Dijk
6e358189ad Separate the build IDs in the build_finished payload with tabs 
hydra-notify splits the payload on tabs so we shouldn't separate the
IDs with spaces.
 2020-04-08 12:05:25 +02:00
Maximilian Bosch
2f9d422172
Fix build against latest Nix 2020-04-07 13:55:38 +02:00
Kevin Quick
a055796ef5
Merge branch 'master' into libpqxx_undeprecate 2020-04-01 11:54:41 -07:00
Eelco Dolstra
6c0bb909df
Merge pull request #704 from basvandijk/sleep-after-catch-exception 
hydra-queue-runner: sleep 5s after handling an exception
 2020-04-01 13:04:31 +02:00
Eelco Dolstra
9727892b61 Don't spam the journal with hydra-queue-runner status dumps 
(cherry picked from commit 15ae932488512ba235ed2f6f841cc5eb56ba9314)
 2020-03-31 22:19:07 +02:00
Eelco Dolstra
ccd046ca3d Keep track of the number of unsupported steps 
(cherry picked from commit 45ffe578b695f9de101b30d44d46f12aa0654f10)
 2020-03-31 22:19:03 +02:00
Eelco Dolstra
4417f9f260 Abort unsupported build steps 
If we don't see machine that supports a build step for
'max_unsupported_time' seconds, the step is aborted. The default is 0,
which is appropriate for Hydra installations that don't provision
missing machines dynamically.

(cherry picked from commit f5cdbfe21d930db43d3812c7d8e87746d6378ef9)
 2020-03-31 22:19:01 +02:00
Eelco Dolstra
e4f5156c41
Build against nix-master 
(cherry picked from commit e7f2139e251cb73195eea6fb84e2a6167b4db968)
 2020-02-20 10:24:04 +01:00
Bas van Dijk
08bfff9d13 hydra-queue-runner: sleep 5s after handling an exception 
instead of immediately calling `readMachinesFiles` again which could
immediately throw another exception again.
 2020-01-14 13:34:35 +01:00
Kevin Quick
7bb6b5e206
Update libpqxx usage to move away from deprecated API interactions. 2020-01-11 22:38:40 -08:00
Hamish Mackenzie
c40c887e50
Fixes for macOS 
Building on macOS with the latest nixpkgs master and NixOS/nixpkgs#77147
fails.  It seems some `std::experimental` (optional) for instance are
not available as `experimental`, but are in `std`.  Also `toJSON` is
missing for `atomic< unsigned long long >`.
 2020-01-07 12:38:06 +13:00
Eelco Dolstra
554bb7d9ce
hydra-queue-runner: Don't pass IN_SYSTEMD to child processes 
This caused local builds to have journal priority prefixes
(e.g. '<3>').
 2019-09-25 17:25:07 +02:00
Eelco Dolstra
d08cfa48d7
Add a 'step_started' notification 2019-08-13 18:18:24 +02:00
Eelco Dolstra
7114d2aceb
Separate payload elements using \t 2019-08-13 18:18:24 +02:00
Eelco Dolstra
2946899504
Turn hydra-notify into a daemon 
It now receives notifications about started/finished builds/steps via
PostgreSQL. This gets rid of the (substantial) overhead of starting
hydra-notify for every event. It also allows other programs (even on
other machines) to listen to Hydra notifications.
 2019-08-13 18:18:21 +02:00
Michael Bishop
3ad091faf3
allow using a shorter context and increase hydra-notify debug 
(cherry picked from commit 1c76ad393669af2f728fd519a050f417319412a6)
 2019-03-20 15:22:24 -04:00
Eelco Dolstra
4e27796eba
Allow setting GC_INITIAL_HEAP_SIZE for hydra-eval-jobs 
This cannot be done in the hydra-evaluator systemd unit, since then
every other Nix process (e.g. hydra-evaluator and nix-prefetch-*) will
also allocate the specified heap size, probably leading to OOM.
 2018-05-16 14:14:53 +02:00
Eelco Dolstra
68afa2bf6f
Dump more system info in /queue-runner-status 2018-03-07 10:06:56 +01:00
Eelco Dolstra
e9670641ec
Distinguish build step states 
The web interface now shows whether a build step is connecting,
copying inputs/outputs, building, etc.
 2017-12-07 15:35:31 +01:00
Eelco Dolstra
457483ba0e
Don't lock the BuildSteps table when inserting 
Instead, optimistically insert a row and retry if there is a conflict.
 2017-12-07 14:41:29 +01:00
Eelco Dolstra
eef0d8861b
Remove test line 2017-10-19 13:17:29 +02:00
Eelco Dolstra
b04dc6c76e
Fix root creation when the root already exists but is owned by another user 2017-10-19 12:28:38 +02:00