31 Commits

Author SHA1 Message Date
Your Name
de2282bcf4 hydra-notify: print out log lines indicating it is or is not launching the exporter 2021-08-24 10:56:13 -04:00
Your Name
5c1228e141 hydra-notify: pre-declare metrics 2021-08-24 10:56:13 -04:00
Your Name
6d7ee27d25 hydra-notify: make the prometheus endpoint configurable, default-off 2021-08-24 10:56:13 -04:00
Your Name
5d0ad5f649 hydra-notify: initial scratch take of prometheus events 2021-08-24 10:56:12 -04:00
Your Name
6e65c3b320 hydra-notify: fixup printing of build IDs
Used to print:

    sending notifications for build Hydra::Model::DB::Builds=HASH(0x124cf960)->id...

Now it prints:

    sending notifications for build 123...
2021-08-16 16:09:05 -04:00
Your Name
2c50227082 hydra-notify: properly call new_event 2021-08-16 15:52:25 -04:00
Your Name
e572a5e576 hydra-notify: use Hydra::Event 2021-08-16 15:52:14 -04:00
Graham Christensen
fa6d7abc13 hydra-notify: move BuildFinished processing to an Event 2021-08-13 16:51:29 -04:00
Graham Christensen
4a1389e36e hydra-notify: move StepFinished processing to an Event 2021-08-13 16:51:29 -04:00
Graham Christensen
4fdb20d3bd hydra-notify: move BuildStarted processing to an Event 2021-08-13 16:51:29 -04:00
Graham Christensen
10e85e3422 hydra-notify: Create a helper for running each plugin on an event 2021-08-13 16:51:29 -04:00
Graham Christensen
a14c8ad5f8
Merge pull request #995 from DeterminateSystems/declarative-jobsets-plugin
Declarative jobsets: move event handling to a plugin
2021-08-12 15:56:13 -04:00
Graham Christensen
0f958f3425
Merge pull request #997 from DeterminateSystems/abstract-listener
Abstract over postgres' LISTEN/NOTIFY
2021-08-12 14:00:34 -04:00
Graham Christensen
5027003285 Abstract over postgres' LISTEN/NOTIFY
This lets us test the event loop if we wanted, and lets us
test the listening behavior in isolation.
2021-08-12 13:54:05 -04:00
Graham Christensen
593af41808 Declarative jobsets: move event handling to a plugin
Declarative jobsets were sort of tucked in to the event hanlder
itself. It turned out that it could have been implemented as a
plugin without much trouble.
2021-08-12 12:48:18 -04:00
Graham Christensen
9c5f317453 hydra-notify: move buildFinished query in to the function impl
This is more consistent with the other event handlers, of dealing
with IDs and not objects.
2021-08-12 12:30:35 -04:00
Graham Christensen
3fda37f65a
RunCommand: Test 2021-02-24 13:43:25 -05:00
Graham Christensen
648eb980dd
hydra-notify: autoflush stdout too 2020-09-02 12:35:18 -04:00
Nikola Knezevic
7d52946982 Set notificationpendingsince for dependent builds
`build_finished` Postgres event will never be fired for the dependent builds.

For example, on our Hydra, the following query always returns increasing
numbers, even though all notifications have been delivered:

```
hydra=> select count(1) from builds where notificationpendingsince is not null;
 count
-------
  4583
(1 row)
```

Thus, we have to iterate over all dependent builds and mark their
`notificationpendingsince` as `null`, otherwise they will pile up until
the next restart of hydra-notify, when they will get delivered.
2020-05-28 10:45:41 +02:00
Nikola Knezevic
f79810bac1 Improve handling of Perl's block eval errors
Taken from `Perl::Critic`:

A common idiom in perl for dealing with possible errors is to use `eval`
followed by a check of `$@`/`$EVAL_ERROR`:

    eval {
        ...
    };
    if ($EVAL_ERROR) {
        ...
    }

There's a problem with this: the value of `$EVAL_ERROR` (`$@`) can change
between the end of the `eval` and the `if` statement. The issue are object
destructors:

    package Foo;

    ...

    sub DESTROY {
        ...
        eval { ... };
        ...
    }

    package main;

    eval {
        my $foo = Foo->new();
        ...
    };
    if ($EVAL_ERROR) {
        ...
    }

Assuming there are no other references to `$foo` created, when the
`eval` block in `main` is exited, `Foo::DESTROY()` will be invoked,
regardless of whether the `eval` finished normally or not. If the `eval`
in `main` fails, but the `eval` in `Foo::DESTROY()` succeeds, then
`$EVAL_ERROR` will be empty by the time that the `if` is executed.
Additional issues arise if you depend upon the exact contents of
`$EVAL_ERROR` and both `eval`s fail, because the messages from both will
be concatenated.

Even if there isn't an `eval` directly in the `DESTROY()` method code,
it may invoke code that does use `eval` or otherwise affects
`$EVAL_ERROR`.

The solution is to ensure that, upon normal exit, an `eval` returns a
true value and to test that value:

    # Constructors are no problem.
    my $object = eval { Class->new() };

    # To cover the possiblity that an operation may correctly return a
    # false value, end the block with "1":
    if ( eval { something(); 1 } ) {
        ...
    }

    eval {
        ...
        1;
    }
        or do {
            # Error handling here
        };

Unfortunately, you can't use the `defined` function to test the result;
`eval` returns an empty string on failure.

Various modules have been written to take some of the pain out of
properly localizing and checking `$@`/`$EVAL_ERROR`. For example:

    use Try::Tiny;
    try {
        ...
    } catch {
        # Error handling here;
        # The exception is in $_/$ARG, not $@/$EVAL_ERROR.
    };  # Note semicolon.

"But we don't use DESTROY() anywhere in our code!" you say. That may be
the case, but do any of the third-party modules you use have them? What
about any you may use in the future or updated versions of the ones you
already use?
2020-05-26 11:19:43 +02:00
Eelco Dolstra
f49a089fc0
hydra-notify: Don't do an unnecessary fetch of Jobsets 2019-08-13 18:18:24 +02:00
Eelco Dolstra
72c36373bb
hydra-notify: Fix processing notifications 2019-08-13 18:18:24 +02:00
Eelco Dolstra
7114d2aceb
Separate payload elements using \t 2019-08-13 18:18:24 +02:00
Eelco Dolstra
2946899504
Turn hydra-notify into a daemon
It now receives notifications about started/finished builds/steps via
PostgreSQL. This gets rid of the (substantial) overhead of starting
hydra-notify for every event. It also allows other programs (even on
other machines) to listen to Hydra notifications.
2019-08-13 18:18:21 +02:00
Eelco Dolstra
8e17a413f5
hydra-notify step-finished: Don't barf if the step has no log file 2018-08-01 17:17:46 +02:00
Shea Levy
582c399420 Add buildQueued plugin hook 2017-05-24 09:45:31 -04:00
Eelco Dolstra
a55942603a Provide a plugin hook for when build steps finish
Fixes #318.
2016-05-27 14:35:32 +02:00
Shea Levy
4392d3e21d Enable declarative projects.
This allows fully declarative project specifications. This is best
illustrated by example:

* I create a new project, setting the declarative spec file to
  "spec.json" and the declarative input to a git repo pointing
  at git://github.com/shlevy/declarative-hydra-example.git
* hydra creates a special ".jobsets" jobset alongside the project
* Just before evaluating the ".jobsets" jobset, hydra fetches
  declarative-hydra-example.git, reads spec.json as a jobset spec,
  and updates the jobset's configuration accordingly:
{
    "enabled": 1,
    "hidden": false,
    "description": "Jobsets",
    "nixexprinput": "src",
    "nixexprpath": "default.nix",
    "checkinterval": 300,
    "schedulingshares": 100,
    "enableemail": false,
    "emailoverride": "",
    "keepnr": 3,
    "inputs": {
        "src": { "type": "git", "value": "git://github.com/shlevy/declarative-hydra-example.git", "emailresponsible": false },
        "nixpkgs": { "type": "git", "value": "git://github.com/NixOS/nixpkgs.git release-16.03", "emailresponsible": false }
    }
}
* When the "jobsets" job of the ".jobsets" jobset completes, hydra
  reads its output as a JSON representation of a dictionary of
  jobset specs and creates a jobset named "master" configured
  accordingly (In this example, this is the same configuration as
  .jobsets itself, except using release.nix instead of default.nix):
{
    "enabled": 1,
    "hidden": false,
    "description": "js",
    "nixexprinput": "src",
    "nixexprpath": "release.nix",
    "checkinterval": 300,
    "schedulingshares": 100,
    "enableemail": false,
    "emailoverride": "",
    "keepnr": 3,
    "inputs": {
        "src": { "type": "git", "value": "git://github.com/shlevy/declarative-hydra-example.git", "emailresponsible": false },
        "nixpkgs": { "type": "git", "value": "git://github.com/NixOS/nixpkgs.git release-16.03", "emailresponsible": false }
    }
}
2016-05-09 08:54:27 -04:00
Shea Levy
9b37cb89ae Add buildStarted plugin hook 2016-04-12 14:42:01 -04:00
Eelco Dolstra
a6e3cb53b9 Use /usr/bin/env to find perl
This is nicer in nix-shell.
2015-08-17 14:18:20 +02:00
Eelco Dolstra
a317d24b29 hydra-queue-runner: Send build notifications
Since our notification plugins are written in Perl, sending
notification from C++ requires a small Perl helper named
‘hydra-notify’.
2015-06-23 00:14:49 +02:00