5d3912962b
RunCommand: use helper functions to ensure filenames and paths are the same
...
Otherwise, it's possible someone updates the format in one place but not
the others, leading to broken or incorrect functionality.
2022-01-28 13:03:15 -08:00
4a441b54ce
hydra-module: /var/lib/hydra -> ${baseDir}
2022-01-28 13:03:15 -08:00
14090fbb86
runcommand-log.tt: init
2022-01-28 13:03:15 -08:00
796ce165d4
RunCommand: Allow displaying command output
2022-01-28 13:03:15 -08:00
4cb5e6cd94
RunCommand: Capture the output of the commands
2022-01-28 13:00:17 -08:00
ef362e92d1
GithubStatus: try pushing statuses for cached buildqueued/buildfinished events
2022-01-25 12:42:28 -05:00
001539c3d2
Merge pull request #1127 from NixOS/grahamc-patch-1
...
Create a bug report issue template
2022-01-25 09:57:17 -05:00
0a5f625746
Update bug_report.md
2022-01-24 20:36:08 -05:00
f6e86efc9f
Merge pull request #1091 from Ma27/ssh-remote-store-location
...
hydra-queue-runner: support store URIs declaring an alternate store location
2022-01-24 14:10:54 -05:00
3a4ea6e563
Merge pull request #1124 from obsidiansystems/simplify--closure-of-path-set
...
simplify, `computeFSClosure` can take a set now
2022-01-24 14:09:35 -05:00
bb68b56f61
Merge pull request #1133 from helsinki-systems/doc/config-format
...
doc: Document the file format of the config
2022-01-21 20:49:18 -05:00
c280692f91
Merge pull request #1126 from DeterminateSystems/build-localhost-paths
...
build-remote: copy missing paths from the binary cache to localhost
2022-01-21 16:16:33 -05:00
44cd890ae3
Merge pull request #1130 from DeterminateSystems/prompt-password
...
hydra-create-user: support prompting for password
2022-01-21 15:38:39 -05:00
ba96a13407
Record metrics when getting the closure to localhost
2022-01-21 15:38:05 -05:00
7e9e82398d
build-remote: copy missing paths from the binary cache to localhost
...
In a Hydra instance I saw:
possibly transient failure building ‘/nix/store/X.drv’ on ‘localhost’:
dependency '/nix/store/Y' of '/nix/store/Y.drv' does not exist,
and substitution is disabled
This is confusing because the Hydra in question does have substitution enabled.
This instance uses:
keep-outputs = true
keep-derivations = true
and an S3 binary cache which is not configured as a substituter in the nix.conf.
It appears this instance encountered a situation where store path Y was built
and present in the binary cache, and Y.drv was GC rooted on the instance,
however Y was not on the host.
When Hydra would try to build this path locally, it would look in the binary
cache to see if it was cached:
(nix)
439 bool valid = isValidPathUncached(storePath);
440
441 if (diskCache && !valid)
442 // FIXME: handle valid = true case.
443 diskCache->upsertNarInfo(getUri(), hashPart, 0);
444
445 return valid;
Since it was cached, the store path was considered Valid.
The queue monitor would then not put this input in for substitution, because
the path is valid:
(hydra)
470 if (!destStore->isValidPath(*i.second.path(*localStore, step->drv->name, i.first))) {
471 valid = false;
472 missing.insert_or_assign(i.first, i.second);
473 }
Hydra appears to correctly handle the case of missing paths that need
to be substituted from the binary cache already, but since most
Hydra instances use `keep-outputs` *and* all paths in the binary cache
originate from that machine, it is not common for a path to be cached
and not GC rooted locally.
I'll run Hydra with this patch for a while and see if we run in to the
problem again.
A big thanks to John Ericson who helped debug this particular issue.
2022-01-21 15:26:45 -05:00
952f629b7c
Test the queue runner in the scenario where a dependency is available in the cache but GC'd locally, where we're building locally
2022-01-21 15:26:45 -05:00
5c3e48fd0d
CliRunners: decode UTF8 before printing stderr/stdout
...
Fixes yath output. It used to say:
step â is now runnable
Now it says:
step ‘/run...2ipqz6hbc41m4c5w5bkq-dependent-job.drv’ is now runnable
2022-01-21 15:24:33 -05:00
e4407f8c93
HydraTestContext: expose the nix state dir
2022-01-21 15:12:10 -05:00
56308dbb05
doc: Document the file format of the config
2022-01-21 20:48:50 +01:00
da1af1ce68
Docs: use hydra-create-user --password-prompt
2022-01-21 13:05:12 -05:00
e351054f61
Merge pull request #1129 from DeterminateSystems/fixup-argon2
...
Fixup argon2 instructions in hydra-create-user
2022-01-21 13:01:37 -05:00
61325853a6
Merge pull request #1132 from DeterminateSystems/ldap-role-match
...
LDAP support: require the prefix 'hydra_' to match documentation
2022-01-21 12:58:35 -05:00
0eeced7f08
hydra-create-user: Warn that creating users with a plaintext password is deprecated
2022-01-21 12:56:15 -05:00
98928a4125
fixups
2022-01-21 12:52:06 -05:00
a888a57baf
tests.ldap: verify the hydra_ prefix is required
2022-01-21 12:46:02 -05:00
76fbde6d6b
Set noecho when reading passwords
2022-01-21 11:11:09 -05:00
8ba4ae461e
Merge pull request #1131 from kenranunderscore/fix-invalid-yaml
...
Fix invalid YAML in documentation
2022-01-21 11:03:28 -05:00
b8f72d7ff2
LDAP support: require the prefix 'hydra_' to match documentation
2022-01-21 10:48:04 -05:00
bb893d0bd5
hydra-create-user: support prompting for passwords
...
I'm not sure this is a good implementation as-is. It does work,
but the password gets echo'd to the screen. I tried to use IO::Prompt
but IO::Prompt really seems to want to read the password from ARGV.
2022-01-21 10:40:56 -05:00
3a6c25489c
Hydra::Helper::Nix: expose a captureStdoutStderrWithStdin, make it available in tests
2022-01-21 10:40:06 -05:00
d4fe7e55dd
Hydra::Helper::Nix: sort exported functions
2022-01-21 10:40:06 -05:00
4945306a2b
hydra-create-user: make docs about using --password-hash better
2022-01-21 10:39:22 -05:00
4476aba5f7
Fix invalid YAML in documentation
2022-01-21 16:38:59 +01:00
12d0d0c176
hydra-create-user: use test_context over test_init
2022-01-21 10:37:48 -05:00
bb9864bad7
hydra-create-user.t: rename the various users based on their password type
2022-01-21 09:12:47 -05:00
46b8f7cce8
Create a bug report issue template
2022-01-20 20:10:28 -05:00
e7a1ae87aa
simplify, computeFSClosure can take a set now
2022-01-20 14:53:01 -05:00
ed1b532b74
Merge pull request #1093 from DeterminateSystems/builds-jobset-project
...
Builds: drop the jobset and project columns
2022-01-17 10:11:55 -05:00
8c50cd06e4
machines: ensure the jobset name is present
2022-01-15 17:11:08 -05:00
c8dc6a9419
Plugins: get project and jobset information from the project and jobset tables
2022-01-15 15:58:02 -05:00
9dc40e0816
evaluator: don't save project, jobset on builds
2022-01-15 15:58:02 -05:00
c539deea99
builds: add a build->project func to get the project via the jobset
2022-01-15 15:58:02 -05:00
f120909547
builds: drop project, jobset columns
...
Indexes were haphazardly dropped.
2022-01-15 15:58:02 -05:00
1caff3a250
Merge pull request #1117 from DeterminateSystems/project-jobset/queue-runner
...
queue-runner: track jobsets by ID
2022-01-15 15:57:14 -05:00
9671d4d135
Merge pull request #1119 from DeterminateSystems/project-jobset/update-gc-roots
...
Project jobset: update-gc-roots
2022-01-15 15:57:06 -05:00
f1a608ac35
Merge pull request #1118 from DeterminateSystems/project-jobset/declarativejobsets
...
Project jobset: Update DeclarativeJobsets to not use the project or jobsets columns on Bulids
2022-01-15 15:56:48 -05:00
7544d4ff47
hydra-update-gc-roots: get project and jobset information from the project and jobset tables
2022-01-15 14:26:45 -05:00
52dda56b99
hydra-update-gc-roots: add a very basic test validating successful execution
2022-01-15 14:20:11 -05:00
72c3110002
queue-runner: track jobsets by ID
2022-01-15 14:06:00 -05:00
17c6bd4fd8
DeclarativeJobsets: get the jobset name from the jobset table
2022-01-15 13:46:32 -05:00
