ahuston-0/hydra
1
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity
3,564 Commits 6 Branches 0 Tags
a14501c61625e7275d5aef734c9f580c82b7c08f
Commit Graph

3564 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Eelco Dolstra
133d298e26 Asynchronously compress build logs 2015-06-19 15:06:12 +02:00
Eelco Dolstra
8e408048e2 Create build step for non-top-level cached failures 
This fixes the missing build step on failures like

  http://hydra.nixos.org/build/23222231
 2015-06-19 11:33:15 +02:00
Eelco Dolstra
77c8bfd392 Improve logging for aborts 2015-06-19 10:37:22 +02:00
Eelco Dolstra
8db1ae2855 Less verbosity 2015-06-18 17:43:13 +02:00
Eelco Dolstra
89b629eeb1 Fix finishing steps that are not top-level of any build 2015-06-18 17:37:35 +02:00
Eelco Dolstra
9cdbff2fdf Handle concurrent finishing of the same build 
There is a slight possibility that the queue monitor and a builder
thread simultaneously decide to mark a build as finished. That's fine,
as long as we ensure the DB update is idempotent (as ensured by doing
"update Builds set finished = 1 ... where finished = 0").
 2015-06-18 17:12:51 +02:00
Eelco Dolstra
948473c909 Fix race between the queue monitor and the builder threads 2015-06-18 16:30:28 +02:00
Eelco Dolstra
9c03b11ca8 Simplify retry handling 2015-06-18 14:51:50 +02:00
Eelco Dolstra
e039f5f840 Create failed build steps for cached failures 2015-06-18 04:35:37 +02:00
Eelco Dolstra
92ea800cfb Set finishedInDB in a few more places 2015-06-18 04:19:21 +02:00
Eelco Dolstra
47367451c7 hydra-queue-runner: Set isCachedBuild 2015-06-18 03:28:58 +02:00
Eelco Dolstra
8257812d0a Acquire exclusive table lock earlier 2015-06-18 02:44:29 +02:00
Eelco Dolstra
69be3cfe93 hydra-queue-runner: Handle status queries on the main thread 
Doing it on the queue monitor thread was problematic because
processing the queue can take a while.
 2015-06-18 01:57:01 +02:00
Eelco Dolstra
a40ca6b76e hydra-queue-runner: Improve dispatcher 
We now take the machine speed factor into account, just like
build-remote.pl.
 2015-06-18 01:52:20 +02:00
Eelco Dolstra
3855131185 hydra-queue-runner: Improve SSH flags 2015-06-18 00:50:48 +02:00
Eelco Dolstra
f57d0b0c54 hydra-queue-runner: Maintain count of active build steps 2015-06-18 00:24:56 +02:00
Eelco Dolstra
59dae60558 hydra-queue-runner: More stats 2015-06-17 22:38:12 +02:00
Eelco Dolstra
ec8e8edc86 hydra-queue-runner: Handle $HYDRA_DBI 2015-06-17 22:11:01 +02:00
Eelco Dolstra
4d9c74335d Add forgotten file 2015-06-17 21:39:28 +02:00
Eelco Dolstra
ce9e859a9c hydra-queue-runner: Implement --unlock 2015-06-17 21:35:20 +02:00
Eelco Dolstra
ca48818b30 Fix remote building 2015-06-17 17:28:59 +02:00
Eelco Dolstra
11be780948 Handle failure with output 2015-06-17 17:11:42 +02:00
Eelco Dolstra
b1a75c7f63 getQueuedBuilds(): Handle dependent builds first 
If a build A depends on a derivation that is the top-level derivation
of some build B, then we should process B before A (meaning we
shouldn't make the derivation runnable before B has been
added). Otherwise, the derivation will be "accounted" to A rather than
B (so the build step will show up in the wrong build).
 2015-06-17 14:46:02 +02:00
Eelco Dolstra
c6d504edbb Handle SSH hosts without a @ 2015-06-17 13:49:18 +02:00
Eelco Dolstra
745efce828 hydra-queue-runner: Implement timeouts 
Also, keep track of timeouts in the database as a distinct build
status.
 2015-06-17 13:32:33 +02:00
Eelco Dolstra
2da4987bc2 Don't lock the CPU 2015-06-17 11:48:38 +02:00
Eelco Dolstra
b91a616520 Automatically retry aborted builds 
Aborted builds are now put back on the runnable queue and retried
after a certain time interval (currently 60 seconds for the first
retry, then tripled on each subsequent retry).
 2015-06-17 11:45:20 +02:00
Eelco Dolstra
e02654b3a0 Prefer cached failure over unsupported system type 2015-06-16 18:00:39 +02:00
Eelco Dolstra
a984c0badc Merge branch 'master' into build-ng 2015-06-15 18:21:07 +02:00
Eelco Dolstra
42e7301c08 Add status dump facility 
Doing

  $ psql hydra -c 'notify dump_status'

will cause hydra-queue-runner to dump some internal status info on
stderr.
 2015-06-15 18:20:14 +02:00
Eelco Dolstra
dd104f14ea Make the queue monitor more robust, and better debug output 2015-06-15 16:54:52 +02:00
Eelco Dolstra
147eb4fd15 Support requiredSystemFeatures 2015-06-15 16:33:50 +02:00
Eelco Dolstra
508ab7f8a2 Tweak build steps 2015-06-15 15:48:05 +02:00
Eelco Dolstra
21aaa0596b Fail builds with previously failed steps early 2015-06-15 15:31:42 +02:00
Eelco Dolstra
c00bf7cd1a Check non-runnable steps for unsupported system type 2015-06-15 15:13:03 +02:00
Eelco Dolstra
5019fceb20 Add a error type for "unsupported system type" 2015-06-15 15:07:04 +02:00
Eelco Dolstra
541fbd62cc Immediately abort builds that require an unsupported system type 2015-06-15 14:51:49 +02:00
Eelco Dolstra
bf87d3a6ed Use stable Nix 2015-06-15 14:51:38 +02:00
Eelco Dolstra
d6354cbe1f Fix Perl error in tests.api 
Doing "su hydra" causes Perl to be invoked with a wrong PERL5LIB
(pointing to root's profile), leading to "Can't locate strict.pm".
 2015-06-15 11:56:51 +02:00
Eelco Dolstra
abca7a87da Cleanup 2015-06-12 18:15:58 +02:00
Eelco Dolstra
f06ec78859 Handle building from a dirty Git tree 2015-06-12 18:15:58 +02:00
Eelco Dolstra
18e0a62e24 Disable 32-bit builds again 
They're failing consistently:

http://hydra.nixos.org/job/hydra/master/build.i686-linux/all
 2015-06-12 15:35:14 +02:00
Eelco Dolstra
f9cd5adae8 Queue monitor: Get only the fields we need 2015-06-11 18:09:50 +02:00
Eelco Dolstra
c974fb893b Support cancelling builds 2015-06-11 18:07:45 +02:00
Eelco Dolstra
c08883966c Use PostgreSQL notifications for queue events 
Hydra-queue-runner now no longer polls the queue periodically, but
instead sleeps until it receives a notification from PostgreSQL about
a change to the queue (build added, build cancelled or build
restarted).

Also, for the "build added" case, we now only check for builds with an
ID greater than the previous greatest ID. This is much more efficient
if the queue is large.
 2015-06-11 17:41:59 +02:00
Eelco Dolstra
672bbb1c67 hydra-update-gc-roots: Get Builds and BuildOutputs in the same query 
This greatly reduces the number of roundtrips to the database.
 2015-06-11 14:10:21 +02:00
Eelco Dolstra
9989a90e99 Keep the most recent successful build of current jobs 
Fixes #140.
 2015-06-11 14:10:21 +02:00
Eelco Dolstra
d72a88b562 Don't try to handle SIGINT 
It just makes things unnecessarily complicated. We can just exit
without cleaning anything up, since the only thing to do is unmark
builds and build steps as busy. But we can do that by having systemd
call "hydra-queue-runner --unlock" from ExecStopPost.
 2015-06-10 15:55:46 +02:00
Eelco Dolstra
a4fb93c119 Lock builds for a shorter amount of time 2015-06-10 15:36:21 +02:00
Eelco Dolstra
6d738a31bf Keep track of failed paths in the Hydra database 
I.e. don't use Nix's failed paths feature anymore. Easier to keep
everything in one place.
 2015-06-10 14:57:16 +02:00