ahuston-0/hydra
1
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity

Fix a race that can cause hydra-queue-runner to ignore newly added builds

Browse Source 
As @dtzWill discovered, with the concurrent hydra-evaluator, there can
be multiple active transactions adding builds to the database. As a
result, builds can become visible in a non-monotonically increasing
order, breaking the queue monitor's assumption that build IDs only go
up.

The fix is to have hydra-eval-jobset provide the lowest build ID it
just added in the builds_added notification, and have the queue
monitor check from there.

Fixes #496.
This commit is contained in:
Eelco Dolstra
2017-07-21 14:25:33 +02:00
parent b0432c7762
commit dc5e0b120a
5 changed files with 28 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
src/script/hydra-eval-jobset
View File

@ -25,6 +25,8 @@ STDERR->autoflush(1);
binmode STDERR, ":encoding(utf8)";
my $db = Hydra::Model::DB->new();
my $notifyAdded = $db->storage->dbh->prepare("notify builds_added, ?");
my $config = getHydraConfig();
my $plugins = [Hydra::Plugin->instantiate(db => $db, config => $config)];
@ -720,6 +722,14 @@ sub checkJobsetWrapped {
print STDERR " created new eval ", $ev->id, "\n";
$ev->builds->update({iscurrent => 1});
# Wake up hydra-queue-runner.
my $lowestId;
while (my ($id, $x) = each %buildMap) {
$lowestId = $id if $x->{new} && (!defined $lowestId || $id < $lowestId);
}
$notifyAdded->execute($lowestId) if defined $lowestId;
} else {
print STDERR " created cached eval ", $ev->id, "\n";
$prevEval->builds->update({iscurrent => 1}) if defined $prevEval;