Commit Graph

4204 Commits

Author SHA1 Message Date
20f5a2120c Use ServeProto::Serialise<ServeProto::BuildOptions> 2023-12-10 13:24:17 -05:00
b56d2383c1 Do not attempt to speak a newer version of the protocol
Both sides need to agree on a version (with `std::min`) for anything to
work. Somehow... we've never done this.

With this comment, the next commit succeeds. Without this commit, the
next commit fails. This is because the next commit exposes serializers
which do different things for proto version 2.7, and we're currently
requesting 2.6.

Opened https://github.com/NixOS/nix/issues/9584 to track this issue
2023-12-10 13:24:17 -05:00
2bd67562b5 Merge pull request from obsidiansystems/serve-proto-build-options
Use `ServeProto::BuildOption`
2023-12-10 13:23:43 -05:00
69a5b00e60 Use ServeProto::BuildOption
More deduplication with Nix.
2023-12-10 13:01:00 -05:00
1d80b72ffb flake.lock: Update Nix master
Flake lock file updates:

• Updated input 'nix':
    'github:NixOS/nix/c8458bd731eb1c74159bebe459ea00165e056b65' (2023-12-09)
  → 'github:NixOS/nix/b7e016ab2464ad2e7e2e856ad0f173157135aae0' (2023-12-10)
2023-12-10 13:00:47 -05:00
105fd18fee Merge pull request from NixOS/use-UnkeyedValidPathInfo-serializer
Use `ServeProto::Serialise<UnkeyedValidPathInfo>` for `QueryValidPaths`
2023-12-10 11:52:56 -05:00
f6f817926a std::move the into the path info map 2023-12-09 12:12:00 -05:00
d0d3b0a298 Use ServeProto::Serialise<UnkeyedValidPathInfo> for QueryValidPaths
Companion to already-merged https://github.com/NixOS/nix/pull/9560
2023-12-09 12:08:04 -05:00
3f932a6731 build-remote: Use std::map<StorePath, UnkeyedValidPathInfo>
It is less denormalized
2023-12-09 11:59:09 -05:00
aaa0e128c1 flake.lock: Update Nix master
Flake lock file updates:

• Updated input 'nix':
    'github:NixOS/nix/c3827ff6348a4d5199eaddf8dbc2ca2e2ef46ec5' (2023-12-07)
  → 'github:NixOS/nix/c8458bd731eb1c74159bebe459ea00165e056b65' (2023-12-09)
2023-12-09 11:59:02 -05:00
4515b5aa17 Merge pull request from NixOS/master
Mere `master` into `nix-next`
2023-12-09 11:53:58 -05:00
411e4d0c24 Let tests themselves intentionally leak temp dir ()
* Let tests themselves intentionally leak temp dir

By default Yath will clean up temporary files, so the result is the
same. But `--keep-dirs` can be passed to `yath test` telling Yath to
*not* clean them up instead. This is very useful for debugging.

* Update t/lib/HydraTestContext.pm

Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2023-12-08 16:30:31 +00:00
831021808c Merge pull request from obsidiansystems/use-build-result-serialiser
Use factored-out `BuildResult` serializer
2023-12-08 11:25:05 -05:00
2ee0068fdc Do not copy for both stores for now
It has a performance cost, and as the comment says we should be doing
the better solution. We want to land this preparatory change on prod
while the rest is still on staging, so we should just skip it for now.

Skipping it will not affect regular fixed-output and input-addressed
derivations, which are the only ones prod would deal with upon getting
this code.

The main CA derivations support branch will revert this commit so it
still works.
2023-12-07 15:05:03 -05:00
31ea6458ca Merge remote-tracking branch 'upstream/master' into ca-derivations-prep 2023-12-07 15:01:35 -05:00
20c8263e3c Update to Nix master
The point of this branch is to always track Nix master, so we are
proactively ready to upgrade to the next Nix release when it is ready.

Flake lock file updates:

• Updated input 'nix':
    'github:NixOS/nix/50f8f1c8bc019a4c0fd098b9ac674b94cfc6af0d' (2023-11-27)
  → 'github:NixOS/nix/c3827ff6348a4d5199eaddf8dbc2ca2e2ef46ec5' (2023-12-07)
• Added input 'nix/libgit2':
    'github:libgit2/libgit2/45fd9ed7ae1a9b74b957ef4f337bc3c8b3df01b5' (2023-10-18)
2023-12-07 13:11:31 -05:00
91bbd5366f Merge pull request from NixOS/nixpkgs-newer-23.05
flake.lock: Update Nixpkgs
2023-12-07 13:10:08 -05:00
a45a27851b flake.lock: Update Nixpkgs
This will be required for upgrading Nix beyond 2.19.

Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/ef0bc3976340dab9a4e087a0bcff661a8b2e87f3' (2023-06-21)
  → 'github:NixOS/nixpkgs/e9f06adb793d1cca5384907b3b8a4071d5d7cb19' (2023-12-03)
2023-12-07 12:21:18 -05:00
6a54ab24e2 Use factored-out BuildResult serializer
For the record, here is the Nix 2.19 version:
https://github.com/NixOS/nix/blob/2.19-maintenance/src/libstore/serve-protocol.cc,
which is what we would initially use.

It is a more complete version of what Hydra has today except for one
thing: it always unconditionally sets the start/stop times.

I think that is correct at the other end seems to unconditionally
measure them, but just to be extra careful, I reproduced the old
behavior of falling back on Hydra's own measurements if `startTime` is
0.

The only difference is that the fallback `stopTime` is now measured from
after the entire `BuildResult` is transferred over the wire, but I think
that should be negligible if it is measurable at all. (And remember,
this is fallback case I already suspect is dead code.)
2023-12-07 02:00:22 -05:00
58707438ba Merge pull request from obsidiansystems/substitute-flag
`copyClosureTo`: Use `SubstituteFlag` instead of `bool`
2023-12-07 00:23:54 -05:00
86cd5e9076 copyClosureTo: Use SubstituteFlag instead of bool
This matches Nix (in the same serialization logic in
`src/libstore/legacy-ssh-store.cc`) and adds clarity.
2023-12-07 00:18:50 -05:00
11f8030b0f Add comment from GitHub about adding to store as code comment 2023-12-06 17:59:25 -05:00
3df8feb3a2 Add TODO about setting null instead of empty string in JSON
An empty string is a sneaky way to avoid hard failures --- things that
expect strings still get strings, but it does conversely open the door
up to soft failures (spooky-action-at-a-distance ones because the string
did not have the expected invariants).

"Fail fast" with null will ultimately make the system more robust, but
force us to fix more things up front, and I don't want to change this
without also fixing those things up front, especially as this commit is
for now just part of the the preparatory PR for which this is dead code.
2023-12-05 11:31:06 -05:00
069b7775c5 hydra-eval-jobs: Ensure we have output path if ca-derivations is disabled
Brought up by @thufschmitt in
https://github.com/NixOS/hydra/pull/1316#discussion_r1415111329 . This
makes this closer to what was originally there --- which just dispatched
off the experimental feature rather than the presence/absense of the
output, too.
2023-12-05 11:26:26 -05:00
e3443cd22a Put back nicer copyClosure instead of manual closure + copy
It looks like we accidentally got the old code back, probably after a merge
conflict resolution.
2023-12-04 17:41:11 -05:00
8046ec2668 Remove unused outputHashes variable
This looks like a stray copy paste.
2023-12-04 16:21:56 -05:00
9ba4417940 Prepare for CA derivation support with lower impact changes
This is just C++ changes without any Perl / Frontend / SQL Schema
changes.

The idea is that it should be possible to redeploy Hydra with these
chnages with (a) no schema migration and also (b) no regressions. We
should be able to much more safely deploy these to a staging server and
then production `hydra.nixos.org`.

Extracted from 

Co-Authored-By: Théophane Hufschmitt <theophane.hufschmitt@tweag.io>
Co-Authored-By: Alexander Sosedkin <monk@unboiled.info>
Co-Authored-By: Andrea Ciceri <andrea.ciceri@autistici.org>
Co-Authored-By: Charlotte 🦝 Delenk Mlotte@chir.rs>
Co-Authored-By: Sandro Jäckel <sandro.jaeckel@gmail.com>
2023-12-04 16:14:47 -05:00
a5d44b60ea Merge pull request from obsidiansystems/split-buildRemote
Split the `buildRemote` function, take 2
2023-12-04 11:37:36 -05:00
363604846a Again, use const in for loop
As requested by @teh. Was lost in merge with master, now added back.
2023-12-04 11:31:05 -05:00
162b538912 Remove unused thisArrow variable 2023-12-04 11:27:39 -05:00
104baef503 Document the connection initialization process 2023-12-04 09:42:04 -05:00
3c5636162a Merge remote-tracking branch 'upstream/master' into split-buildRemote 2023-12-04 09:38:43 -05:00
874fcae1e8 Merge pull request from delroth/queue-runner-perf
queue-runner: only re-sort runnables by prio once per dispatch cycle
2023-12-04 15:27:14 +01:00
4dc8fe0b08 Merge pull request from obsidiansystems/clean-up-deps
Cleanup deps
2023-12-04 15:15:09 +01:00
67eeabd518 Merge remote-tracking branch 'upstream/master' into split-buildRemote 2023-12-04 09:12:58 -05:00
622c25e3c4 Sedding prior to merge 2023-12-04 08:56:06 -05:00
f216bce0e6 Merge pull request from obsidiansystems/2.19
Upgrade to Nix 2.19
2023-12-01 17:07:51 +01:00
4d1c850512 Merge pull request from chayleaf/2.18
support nix 2.18
2023-12-01 15:10:29 +01:00
c922e73c11 Update to Nix 2.19
Flake lock file updates:

• Updated input 'nix':
    'github:NixOS/nix/f5f4de6a550327b4b1a06123c2e450f1b92c73b6' (2023-10-02)
  → 'github:NixOS/nix/50f8f1c8bc019a4c0fd098b9ac674b94cfc6af0d' (2023-11-27)
2023-11-30 15:26:46 -05:00
e172461e55 Use const in for loop
As requested by @teh
2023-11-30 12:19:20 -05:00
0917145622 Make new functions not in header static 2023-11-30 12:19:05 -05:00
2bda7ca642 Further use Machine::Connection to deduplicate 2023-11-30 11:31:58 -05:00
831a2d9bd5 Merge remote-tracking branch 'upstream/master' into split-buildRemote 2023-11-30 11:27:40 -05:00
5db374cb50 Cleanup deps
- `nativeBuildInputs` vs `buildInputs`

- narrow down `with`s for clarity

- use `autoreconfHook` not `bootstrap` script

These sorts of changes have also been done in the Nix repo.
2023-11-30 10:48:17 -05:00
e9da80fff6 support nix 2.18 2023-11-21 18:41:52 +07:00
8f48e4ddec Merge pull request from knedlsepp/fix-mime
Fix MIME types when serving .js and .css to fix rendering of HTML reports
2023-11-17 22:16:27 +01:00
33f8a36736 Merge pull request from stigtsp/crypt-passphrase-argon2-output-len
Set output length of C::P::Argon2 hashes to 16
2023-10-20 16:03:18 +02:00
6a5fb9efae Set output length of C::P::Argon2 hashes to 16
Since the default lengths in Crypt::Passphrase::Argon2 changed from 16
to 32 in in 0.009, some tests that expected the passphrase to be
unchanged started failing.
2023-10-20 00:09:28 +02:00
c1a5ff3959 Merge pull request from nh2/patch-1
`renderInputDiff`: Increase git hash length 6 -> 8
2023-09-09 17:17:43 +02:00
8520ab1391 Merge pull request from sternenseemann/doc-eval-builds
Document all_builds (/eval/{eval-id}/builds) endpoint
2023-09-09 17:11:52 +02:00