128 Commits

Author SHA1 Message Date
Damien Diederen
df7dab1291 GitInput: Include deepClone option in the cache key
Without this commit, two jobsets using the same repository as input,
but different `deepClone` options, end up incorrectly sharing the same
"checkout" for a given (`uri`, `branch`, `revision`) tuple.  The
presence or absence of `.git` is determined by the jobset execution
order.

This patch adds the missing `isDeepClone` boolean to the cache key.

The database upgrade script empties the `CachedGitInputs` table, as we
don't know if existing checkouts are deep clones.  Unfortunately, this
generally forces rebuilds even for correct `deepClone` checkouts, as
the binary contents of `.git` are not deterministic.

Fixes #510
2021-06-19 17:37:40 +02:00
Drew Hess
523d6df5b8
Fix GitHub status update for private flakes.
Also, if the parse fails, don't try to update the GitHub status, as
this will eventually cause rate-limiting.
2021-04-26 01:38:24 +01:00
Graham Christensen
b9bcedbfdb
Merge pull request #596 from kquick/local_inp_url
Update prompt for Local path input to indicate a URL is also valid.
2021-04-14 20:01:58 +00:00
Maximilian Bosch
f9f5ab2fb1
Make gitea public URL configurable
Otherwise, it will be obtained from the jobset input that contains the
URL to the git repo to build.
2021-03-30 23:01:36 +02:00
Maximilian Bosch
eecea56131
Implement VM-test for gitea plugin 2021-03-30 22:35:39 +02:00
Maximilian Bosch
56997d8e8b
Fix error codes for GiteaStatus plugin
* `failure` if a build error occurred, on e.g. an aborted build send
  `error`.
2021-03-30 14:13:46 +02:00
Maximilian Bosch
fef142f13a
Implement simple status notifications for Git repos hosted on gitea 2021-03-30 14:10:21 +02:00
Graham Christensen
2179b4b4b0
RunCommand: emit the finished field as a boolean 2021-03-08 12:11:20 -05:00
Graham Christensen
a756614fa1
RunCommand: pass homepage, description, license, system, and nixname 2021-02-24 16:13:09 -05:00
Silvan Mosberger
58dd7f9ed3
Fix Github status plugin for flakes
If the root flake is a github: one, github status notifications are sent
to it. The githubstatus->inputs configuration isn't used for flakes.
2021-02-06 00:02:30 +01:00
Nathan van Doorn
2742fde8c2
Remove Debug prints from GitLabStatus.pm
These make the hydra-queue-runner logs very noisy even when not using the GitlabStatus plugin.
Also, they shouldn't be necessary except when developing the plugin itself and should have been removed before release.
2020-11-02 10:14:54 +00:00
Casey Ransom
03be8ae7a1 Remove image dependency on hydra.nixos.org
Also standardize the slack image size on 256 pixels.
2020-09-01 15:13:12 -04:00
Eelco Dolstra
d4e4be4fd1
Remove SHA-1 hash from BuildProducts
SHA-1 is deprecated and it will be expensive to compute with the
streaming NAR handler.
2020-07-27 18:24:10 +02:00
Eelco Dolstra
1831866a52
Merge pull request #692 from knl/emit-hostname-with-influxdb-metrics
Add host tag to InfluxDB metrics
2020-06-10 10:57:17 +02:00
Nikola Knezevic
fceaed2b24 Make PathInput plugin cache validity configurable
PathInput plugin keeps a cache of path evaluations. This cache is simple, and
path is not checked more than once every N seconds, where N=30. The caching is
there to avoid expensive calls to `nix-store --add`.

This change makes the validity period configurable. The main use case is
`api-test.pl` which was implemented wrong for a while, as the invocation of
`hydra-eval-jobset` would return the previous evaluation, claiming there are no
changes. The test has been fixed to check better for a new evaluation.
2020-06-04 12:26:47 +02:00
Bas van Dijk
38122544ed GitInput: only convert integer option values to int
The previous code converted option values to ints when the value
contained a digit somewhere. This is too eager since it also converts
strings like `release-0.2` to an int which should not happen.

We now only convert to int when the value is an integer.
2020-05-13 11:41:52 +02:00
Bas van Dijk
f32a2a48d7
Merge pull request #740 from knl/add-githubrefs-plugin
Add GithubRefs plugin
2020-05-08 13:21:45 +02:00
Emery Hemingway
e93c36aab1 SoTest: read credentials from file 2020-04-26 12:12:04 +05:30
Nikola Knezevic
f03e7ef800 Add GithubRefs plugin
This plugin is a counterpart to GithubPulls plugin. Instead of fetching pull
requests, it will fetch all references (branches and tags) that start with a
particular prefix.

The plugin is a copy of GithubPulls plugin with appropriate changes to call the
right API and parse the config matching the need.
2020-04-23 10:45:37 +02:00
Emery Hemingway
a63e349476 Add SoTest plugin
https://opensource.sotest.io/
https://docs.sotest.io/
2020-04-21 15:25:44 +05:30
Maximilian Bosch
721c764951
Remove Hydra::Helper::nix::txn_do from the Perl code
To quote the function's comment:

  Awful hack to handle timeouts in SQLite: just retry the transaction.
  DBD::SQLite *has* a 30 second retry window, but apparently it
  doesn't work.

Since SQLite is now dropped entirely, this wrapper can be removed
completely.
2020-04-16 00:42:40 +02:00
Nikola Knezevic
1bee6e3d8a Add debug logging
This will help us track potential problems with the plugin.
2020-03-26 13:27:44 +01:00
Nikola Knezevic
986fde8888 Refactor code
Extract the conditions before the loop, as they do not change due to channel
definition.
2020-03-26 13:27:44 +01:00
Nikola Knezevic
956f009672 Add documentation for SlackNotification plugin 2020-03-26 13:27:44 +01:00
Nikola Knezevic
01ae944c80 Add host tag to InfluxDB metrics
This should help us discern machines in environments with multiple Hydra deployments.
2020-03-12 14:23:07 +01:00
Eelco Dolstra
c8a4030c5f
Fix error in GitlabStatus plugin
May 15 09:20:10 chef hydra-queue-runner[27523]: Hydra::Plugin::GitlabStatus=HASH(0x519a7b8)->buildFinished: Can't call method "value" on an undefined value at /nix/store/858hinflxcl2jd12wv1r3a8j11ybsf6w-hydra-0.1.2629.89fa829/libexec/hydra/lib/Hydra/Plugin/GitlabStatus.pm line 57.

(cherry picked from commit 438ddf52898cddf100cf59ca91f32581368c9b2e)
2019-08-15 13:55:47 +02:00
Eelco Dolstra
92d8d6baa5
Avoid fetching Projects/Jobsets just to get the name column
In particular, doing a 'select * from Jobsets where ...' must be
avoided, because the 'errormsg' column can be very big.
2019-08-13 18:18:25 +02:00
Eelco Dolstra
16811d3e78
Plugins: Add isEnabled method
Plugins are now disabled at startup time unless there is some relevant
configuration in hydra.conf. This avoids hydra-notify having to do a
lot of redundant work (a lot of plugins did a lot of database queries
*before* deciding they were disabled).

Note: BitBucketStatus users will need to add 'enable_bitbucket_status
= 1' to hydra.conf.
2019-08-13 18:18:25 +02:00
Nikola Knezevic
06bdc8f85c Added the InfluxDBNotification plugin including a NixOS test
This adds a `InfluxDBNotification` plugin which is configured as:

```
<influxdb>
  url = http://127.0.0.1:8086
  db = hydra
</influxdb>
```

which will write a notification for every finished job to the
configured database in InfluxDB looking like:

```
hydra_build_status,cached=false,job=job,jobset=default,project=sample,repo=default,result=success,status=success,system=x86_64-linux build_id="1",build_status=0i,closure_size=584i,duration=0i,main_build_id="1",queued=0i,size=168i 1564156212
```
2019-07-26 17:47:03 +02:00
Robin Gloster
f4e7c104ff
Create a gitlab status plugin
This plugin expects as inputs to a jobset the following:
 - gitlab_status_repo => Name of the repository input pointing to that
   status updates should be POST'ed, i.e. the jobset has a git input
   "nixexprs": "https://gitlab.example.com/project/nixexprs", in which
   case "gitlab_status_repo" would be "nixexprs".
 - gitlab_project_id => ID of the project in Gitlab, i.e. in the above
   case the ID in gitlab of "nixexprs"
2019-04-11 16:40:44 +02:00
Michael Bishop
3ad091faf3
allow using a shorter context and increase hydra-notify debug
(cherry picked from commit 1c76ad393669af2f728fd519a050f417319412a6)
2019-03-20 15:22:24 -04:00
Shea Levy
b298ba4bbe
Merge branch 'pr-gitlabpulls' of git://github.com/nlewo/hydra 2019-01-11 10:08:23 -05:00
Antoine Eiche
d9253543e4 plugin/GitLabPulls: support for using a personal access token (PAT)
In order to access protected or private repositories. Using the target
repository URL along with the merge-request ref instead of the source
repository url and branch is necessary to avoid running into issues if
the source repository is not actually accessible to the user Hydra is
authenticating as.

Thanks Alexei Robyn for this patch.
2018-11-20 16:27:40 +01:00
Rob Vermaas
14d5577bf8 Add duration to Slack notification. 2018-11-20 14:57:50 +01:00
Michael Bishop
7916c6b185 [DEVOPS-1126] throttle github status calls to remain under api ratelimits 2018-11-09 12:06:37 -04:00
Eelco Dolstra
1c44de1779
Merge pull request #567 from phile314/bitbucket_pulls
Add BitBucket pull request support
2018-11-06 10:48:37 +01:00
Antoine Eiche
68aad22d19 Add GitlabPulls input plugin 2018-10-15 14:45:26 +02:00
Kevin Quick
d9515b208c Update prompt for Local path input to indicate a URL is also valid.
The PathInput input for local paths was previously enhanced to allow
URLs for which it would use a nix-prefetch-url operation.  This change
updates the prompt for the declarative input type to indicate this
capability.
2018-09-09 22:08:06 -07:00
Kevin Quick
35bcab74ed Fix darcs input to use darcs-specific SCM cache dir.
Currently re-using the git cache dir which could cause overlap problems.
2018-09-09 22:04:32 -07:00
Nathan van Doorn
a77954be4d Allow for precisely one instance of RunCommand plugin 2018-08-21 15:52:41 +01:00
Eelco Dolstra
e122d3bef3
RunCommand: Return metrics as a float
Apparently, DBIx::Class doesn't handle columns with type 'double
precision' properly.
2018-08-02 12:31:28 +02:00
Eelco Dolstra
6db2cbf094
Add a plugin to execute arbitrary commands when a build finishes
The plugin can be configured using one or more <runcommand> sections
in hydra.conf, e.g.

<runcommand>
  command = echo Build finished
</runcommand>

Optionally, the command can be executed for specific
projects/jobsets/jobs:

  job = patchelf:master:tarball

or

  job = patchelf:*:*

The default is *:*:*.

The command is executed with the environment variable $HYDRA_JSON
pointing to a JSON file containing info about the build, e.g.

  {
    "build": 3772978,
    "buildStatus": 0,
    "drvPath": "/nix/store/9y4h1fyx9pl3ic08i2f09239b90x1lww-patchelf-tarball-0.8pre894_ed92f9f.drv",
    "event": "buildFinished",
    "finished": 1,
    "job": "tarball",
    "jobset": "master",
    "metrics": [
      {
        "name": "random1",
        "unit": null,
        "value": "20282"
      },
      {
        "name": "random2",
        "unit": "KiB",
        "value": "6664"
      }
    ],
    "outputs": [
      {
        "name": "out",
        "path": "/nix/store/39h5xciz5pnh1aypmr3rpdx0536y5s2w-patchelf-tarball-0.8pre894_ed92f9f"
      }
    ],
    "products": [
      {
        "defaultPath": "",
        "fileSize": 148216,
        "name": "patchelf-0.8pre894_ed92f9f.tar.gz",
        "path": "/nix/store/39h5xciz5pnh1aypmr3rpdx0536y5s2w-patchelf-tarball-0.8pre894_ed92f9f/tarballs/patchelf-0.8pre894_ed92f9f.tar.gz",
        "productNr": 4,
        "sha1hash": "9f27d18382436a7f743f6c2f6ad66e1b536ab4c8",
        "sha256hash": "b04faef2916c411f10711b58ea26965df7cb860ca33a87f1e868051b874c44b3",
        "subtype": "source-dist",
        "type": "file"
      },
      {
        "defaultPath": "",
        "fileSize": 121279,
        "name": "patchelf-0.8pre894_ed92f9f.tar.bz2",
        "path": "/nix/store/39h5xciz5pnh1aypmr3rpdx0536y5s2w-patchelf-tarball-0.8pre894_ed92f9f/tarballs/patchelf-0.8pre894_ed92f9f.tar.bz2",
        "productNr": 3,
        "sha1hash": "7a664841fb779dec19023be6a6121e0398067b7c",
        "sha256hash": "c81e36099893f541a11480f869fcdebd2fad3309900519065c8745f614dd024a",
        "subtype": "source-dist",
        "type": "file"
      },
      {
        "defaultPath": "README",
        "fileSize": null,
        "name": "",
        "path": "/nix/store/39h5xciz5pnh1aypmr3rpdx0536y5s2w-patchelf-tarball-0.8pre894_ed92f9f",
        "productNr": 2,
        "sha1hash": null,
        "sha256hash": null,
        "subtype": "readme",
        "type": "doc"
      },
      {
        "defaultPath": "",
        "fileSize": 6230,
        "name": "README",
        "path": "/nix/store/39h5xciz5pnh1aypmr3rpdx0536y5s2w-patchelf-tarball-0.8pre894_ed92f9f/README",
        "productNr": 1,
        "sha1hash": "dc6bb09093183ab52d7e6a35b72d179869bd6fbf",
        "sha256hash": "5371aee9de0216b3ea2d5ea869da9d5ee441b99156a99055e7e11e7a705f7920",
        "subtype": "readme",
        "type": "doc"
      }
    ],
    "project": "patchelf",
    "startTime": 1533137091,
    "stopTime": 1533137094,
    "timestamp": 1533136076
  }

So for example, the following command:

  command = echo Build $(jq -r .build $HYDRA_JSON) \($(jq -r .project $HYDRA_JSON):$(jq -r .jobset $HYDRA_JSON):$(jq -r .job $HYDRA_JSON)\) finished, metrics: $(jq -r '.metrics[].value' $HYDRA_JSON).

will print

  Build 3772978 (patchelf:master:tarball) finished, metrics: 20282 6664.
2018-08-01 19:43:50 +02:00
Philipp Hausmann
f738f6feb3 Add BitBucket pull request support 2018-06-22 11:22:54 +02:00
Michael Bishop
4015bca225
fix a stray newline in a path 2018-03-16 13:04:48 -03:00
Eelco Dolstra
02888105a8
Add a hydra.conf option to enable email notification
Note that it's disabled by default.
2018-03-12 14:04:23 +01:00
Philipp Hausmann
3cf2ffe367 [ Fix #531 ] Trim github pr plugin store paths.
nix-store --add returns a trailing LF which caused wrong include
paths for the github PR plugin JSON input.
2018-02-12 17:59:56 +01:00
Eelco Dolstra
8913c682cf
Doh 2017-12-25 15:02:17 +01:00
Eelco Dolstra
100249c066
Use git rev-list --count 2017-11-14 16:18:54 +01:00
Shea Levy
a6d9201947 GithubStatus: Allow config to specify the integration description and context 2017-07-24 15:44:10 -04:00
Eelco Dolstra
064fd20b6f Merge pull request #494 from Chakerbh/master
Use build id for bitbucket Key
2017-07-11 17:41:22 +02:00