ahuston-0/hydra
1
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity

Add a plugin for backing up builds in s3

Browse Source 
In your hydra config, you can add an arbitrary number of <s3config>
sections, with the following options:

* name (required): Bucket name
* jobs (required): A regex to match job names (in project:jobset:job
  format) that should be backed up to this bucket
* compression_type: bzip2 (default), xz, or none
* prefix: String to prepend to all hydra-created s3 keys (if this is
  meant to represent a directory, you should include the trailing slash,
  e.g. "cache/"). Default "".

After each build with an output (i.e. successful or failed-with-output
builds), the output path and its closure are uploaded to the bucket as
.nar files, with corresponding .narinfos to enable use as a binary
cache.

This plugin requires that s3 credentials be available. It uses
Net::Amazon::S3, which as of this commit the nixpkgs version can
retrieve s3 credentials from the AWS_ACCESS_KEY_ID and
AWS_SECRET_ACCESS_KEY environment variables, or from ec2 instance
metadata when using an IAM role.

This commit also adds a hydra-s3-backup-collect-garbage program, which
uses hydra's gc roots directory to determine which paths are live, and
then deletes all files except nix-cache-info and any .nar or .narinfo
files corresponding to live paths. hydra-s3-backup-collect-garbage
respects the prefix configuration option, so it won't delete anything
outside of the hierarchy you give it, and it has the same credential
requirements as the plugin. Probably a timer unit running the garbage
collection periodically should be added to hydra-module.nix

Note that two of the added tests fail, due to a bug in the interaction
between Net::Amazon::S3 and fake-s3. Those behaviors work against real
s3 though, so I'm committing this even with the broken tests.

Signed-off-by: Shea Levy <shea@shealevy.com>
This commit is contained in:
Shea Levy
2013-09-03 10:53:56 -04:00
committed by Eelco Dolstra
parent 4705af48b8
commit 74388353b5
8 changed files with 303 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

41
release.nix
View File

@ -100,6 +100,7 @@ in rec {
TextDiff
TextTable
XMLSimple
NetAmazonS3
nix git
];
};
@ -143,6 +144,7 @@ in rec {
''; # */
meta.description = "Build of Hydra on ${system}";
passthru.perlDeps = perlDeps;
});
@ -211,4 +213,43 @@ in rec {
$machine->mustSucceed("perl ${./tests/api-test.pl} >&2");
'';
});
tests.s3backup = genAttrs' (system:
with import <nixos/lib/testing.nix> { inherit system; };
let hydra = builtins.getAttr system build; in # build."${system}"
simpleTest {
machine =
{ config, pkgs, ... }:
{ services.postgresql.enable = true;
services.postgresql.package = pkgs.postgresql92;
environment.systemPackages = [ hydra pkgs.rubyLibs.fakes3 ];
virtualisation.memorySize = 2047;
boot.kernelPackages = pkgs.linuxPackages_3_10;
virtualisation.writableStore = true;
networking.extraHosts = ''
127.0.0.1 hydra.s3.amazonaws.com
'';
};
testScript =
''
$machine->waitForJob("postgresql");
# Initialise the database and the state.
$machine->mustSucceed
( "createdb -O root hydra"
, "psql hydra -f ${hydra}/libexec/hydra/sql/hydra-postgresql.sql"
, "mkdir /var/lib/hydra"
, "mkdir /tmp/jobs"
, "cp ${./tests/s3-backup-test.pl} /tmp/s3-backup-test.pl"
, "cp ${./tests/api-test.nix} /tmp/jobs/default.nix"
);
# start fakes3
$machine->mustSucceed("fakes3 --root /tmp/s3 --port 80 &>/dev/null &");
$machine->waitForOpenPort("80");
$machine->mustSucceed("cd /tmp && LOGNAME=root AWS_ACCESS_KEY_ID=foo AWS_SECRET_ACCESS_KEY=bar HYDRA_DBI='dbi:Pg:dbname=hydra;user=root;' HYDRA_CONFIG=${./tests/s3-backup-test.config} perl -I ${hydra}/libexec/hydra/lib -I ${hydra.perlDeps}/lib/perl5/site_perl ./s3-backup-test.pl >&2");
'';
});
}