ahuston-0/hydra
1
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity
4,216 Commits 3 Branches 0 Tags
Commit Graph

442 Commits

Author SHA1 Message Date
Eelco Dolstra
2b76094a23 S3BinaryCacheStore::isValidPath(): Do a GET instead of HEAD 2016-02-19 17:41:11 +01:00
Eelco Dolstra
bd76f9120a Cache .narinfo lookups 2016-02-19 16:19:40 +01:00
Eelco Dolstra
a0f74047da Keep some statistics for the binary cache stores 2016-02-19 14:24:23 +01:00
Eelco Dolstra
dc4a00347d Use a single BinaryCacheStore for all threads 
This will make it easier to do caching / keep stats. Also, we won't
have S3Client's connection pooling if we create multiple S3Client
instances.
 2016-02-18 17:31:19 +01:00
Eelco Dolstra
00a7be13a2 Make queue runner internal status available under /queue-runner-status 2016-02-18 17:11:46 +01:00
Eelco Dolstra
8c9fc677c1 Typo 2016-02-18 16:43:24 +01:00
Eelco Dolstra
db3fcc0f5e Enable substitution on the build machines 
If properly configured, this allows them to get store paths directly
from S3, rather than having to receive them from the queue runner.
 2016-02-18 16:42:05 +01:00
Eelco Dolstra
2d40888e2e Add an S3-backed binary cache store 2016-02-18 16:18:50 +01:00
Eelco Dolstra
0e254ca66d Refactor local binary cache code into a subclass 2016-02-18 14:06:17 +01:00
Eelco Dolstra
a992f688d1 Rename class 2016-02-18 13:02:20 +01:00
Eelco Dolstra
de77cc2910 Rename file 2016-02-18 13:02:20 +01:00
Eelco Dolstra
ce5790285a Merge remote-tracking branch 'origin/master' into binary-cache 2016-02-17 11:54:59 +01:00
Eelco Dolstra
d7a123fcd4 Keep track of the time we spend copying to/from build machines 2016-02-17 10:30:23 +01:00
Eelco Dolstra
25022bf5fd hydra-queue-runner: Support generating a signed binary cache 2016-02-16 16:41:42 +01:00
Eelco Dolstra
744cee134e hydra-queue-runner: Compress binary cache NARs using xz 2016-02-15 21:56:53 +01:00
Eelco Dolstra
2d0dd7fb49 hydra-queue-runner: Write directly to a binary cache 2016-02-15 21:10:29 +01:00
Eelco Dolstra
92d8b59361 Process Nix API changes 2016-02-11 15:59:47 +01:00
Eelco Dolstra
97f8c61928 Fix hydra-queue-runner --build-one 2015-12-29 17:53:33 +01:00
Eelco Dolstra
c087472c71 Remove superfluous "has" function 2015-11-02 14:29:12 +01:00
Eelco Dolstra
2d128d2a6f Don't show redundant "removing machine..." messages 2015-10-30 18:22:43 +01:00
Eelco Dolstra
d8d188301d Fix division-by-zero crash 
Not clear why step_->jobsets was empty...
 2015-10-30 18:01:48 +01:00
Eelco Dolstra
1ff48da3d3 int2String -> std::to_string 2015-10-30 18:01:38 +01:00
Eelco Dolstra
4d1816b152 Remove obsolete Builds columns and provide accurate "Running builds" 
This removes the "busy", "locker" and "logfile" columns, which are no
longer used by the queue runner. The "Running builds" page now only
shows builds that have an active build step.
 2015-10-27 15:37:17 +01:00
Eelco Dolstra
53c80d9526 getQueuedBuilds(): Periodically stop to handle priority bumps 
Previously, priority bumps could take a long time to get noticed if
getQueuedBuilds() was busy processing zillions of queue
additions. (This was made worse by the reintroduction of substitute
checking.)
 2015-10-22 17:00:46 +02:00
Eelco Dolstra
71bf7e02d5 Use nix::willBuildLocally() 2015-10-21 15:44:29 +02:00
Eelco Dolstra
8e8e31ce86 Re-implement log size limits 
The old queue runner already had this. However, we now store "log
limit exceeded" as a separate status code in the database.
 2015-10-06 17:35:08 +02:00
Eelco Dolstra
82504fe010 hydra-queue-runner: Use substitutes 
This allows Hydra to use binaries from available binary caches. It
makes the queue monitor thread quite a bit slower, so if you don't
want to use binary caches, it's better to add "--option
build-use-substitutes false" to the hydra-queue-runner invocation.

Fixed #243.
 2015-10-05 14:57:44 +02:00
Eelco Dolstra
f8141fdc98 Set propagatedFrom for cached failed build steps 2015-09-11 15:55:26 +02:00
Eelco Dolstra
6075ac6fed Remove localhost hack 2015-09-09 16:50:59 +02:00
Eelco Dolstra
ee9bf7ace7 Account steps with preferLocalBuild as a separate system type 
They will show up in machineTypes as (e.g.) x86_64-linux:local instead
of x86_64-linux. This is to prevent the Hydra provisioner from
creating machines for steps that are supposed to be executed locally.
 2015-09-02 13:42:25 +02:00
Eelco Dolstra
7e954aff03 Keep machine stats even when a machine is removed from the machines file 
This is important for the Hydra provisioner, since it needs to be able
to see whether a disabled machine still has jobs running on it.
 2015-09-02 13:31:47 +02:00
Eelco Dolstra
2a7fbd57cc Allow the machines file to specify host public keys 
It's easier for the Hydra provisioner to put host public keys in the
machines file than to separately manage the known_hosts file
(especially when the provisioner runs on a different machine).
 2015-08-26 13:43:02 +02:00
Eelco Dolstra
7aa52517e9 Support multiple machines files 
This is primarily useful for the Hydra provisioner, which can write
its machines to another file than /etc/nix/machines.
 2015-08-25 15:34:53 +02:00
Eelco Dolstra
7a654259ff Wake the dispatcher when the machines file has changed 2015-08-17 15:48:10 +02:00
Eelco Dolstra
092d60735b Keep track of wait time per system type 
I.e., how much time the currently runnable steps per system type have
been waiting. This is useful for deciding whether to provision more
machines.
 2015-08-17 15:45:44 +02:00
Eelco Dolstra
99bfc37764 Don't abort steps that have an unsupported system type 
This is necessary because the required system type can become
available later (e.g. by being provisioned by the
auto-scaler). However, in the future, we may want to fail steps if
they have been unsupported for more than a certain amount of time.
 2015-08-17 15:10:41 +02:00
Eelco Dolstra
ea1eb2e3fb Keep track of requiredSystemFeatures in the machine stats 
For example, steps that require the "kvm" feature may require a
different kind of machine to be provisioned. This can also be used to
require performance-sensitive tests to run on a particular kind of
machine, e.g., by setting requiredSystemFeatures to something like
"ec2-i2.8xlarge".
 2015-08-17 14:37:57 +02:00
Eelco Dolstra
d571e44b86 Keep stats for the Hydra auto scaler 
"hydra-queue-runner --status" now prints how many runnable and running
build steps exist for each machine type. This allows additional
machines to be provisioned based on the Hydra load.
 2015-08-17 13:50:41 +02:00
Eelco Dolstra
d4759c1da2 hydra-queue-runner: Detect changes to the scheduling shares 2015-08-12 13:17:56 +02:00
Eelco Dolstra
576dc0c120 For completeness, re-implement meta.schedulingPriority 2015-08-12 12:05:43 +02:00
Eelco Dolstra
b7965df928 Load the queue in order of global priority 2015-08-11 02:14:34 +02:00
Eelco Dolstra
97f11baa8d Revive jobset scheduling 
(I.e. taking the jobset scheduling share into account.)
 2015-08-11 01:31:56 +02:00
Eelco Dolstra
eb13007fe6 Allow build to be bumped to the front of the queue via the web interface 
Builds now have a "Bump up" action. This will cause the queue runner
to prioritise the steps of the build above all other steps.
 2015-08-10 16:19:47 +02:00
Eelco Dolstra
27182c7c1d Start steps in order of ascending build ID 2015-08-10 16:19:47 +02:00
Eelco Dolstra
593850b956 Fix potential race in dispatcher wakeup 2015-08-10 12:54:55 +02:00
Eelco Dolstra
6a1c950e94 Unindent 2015-08-10 11:33:22 +02:00
Eelco Dolstra
f21b88e388 Remove superfluous check 2015-08-07 04:20:34 +02:00
Eelco Dolstra
f1fbf8c605 Fix race in finishing builds that have been cancelled 2015-08-07 04:18:48 +02:00
Eelco Dolstra
ff3f5eb4d8 Fix remote building on Nix 1.10 2015-07-31 03:41:55 +02:00
Eelco Dolstra
5b9a288123 Workaround for RemoteStore not supporting cmdBuildDerivation yet 2015-07-31 03:39:20 +02:00