Commit Graph

158 Commits

Author SHA1 Message Date
4a4a0f901c Use LegacySSHStore
In https://github.com/NixOS/nix/pull/10748 it is extended with
everything we need.
2025-02-18 14:07:42 -05:00
9a6928d93b Use new CommonSSHStoreConfig::createSSHMaster
This avoids some duplicated code, leveraging the same `StoreReference`
type that also undergirds the machine file dedup we just did prior.

By using `LegacySSHStoreConfig`, we're also taking a baby step towards
using the store interface rather than messing around with the protocol
internals.
2025-02-13 18:13:38 -05:00
cd925e876f Merge branch 'master' into nix-next 2024-05-29 17:05:04 -04:00
5728011da1 queue-runner: try larger pipe buffer sizes
(cherry picked from commit 18466e83261d39b997a73bbd9f0f249c3a91fbeb)
2024-05-23 11:42:35 -04:00
d55bea2a1e Utilize nix::Machine more fully
With https://github.com/NixOS/nix/pull/9839, the `storeUri` field is
much better structured, so we can use it while still opening the SSH
connection ourselves.
2024-05-22 22:02:46 -04:00
71c4e2dc5b Dedup more protocol code
Use https://github.com/NixOS/nix/pull/10749
2024-05-20 18:19:59 -04:00
9e7ac58042 Merge branch 'master' into nix-next 2024-01-24 18:36:03 -05:00
d45e14fd43 Merge pull request #1316 from NixOS/ca-derivations-prep
Prepare for CA derivation support with lower impact changes
2024-01-24 18:12:42 -05:00
9a86da0e7b Merge branch 'master' into nix-next 2024-01-23 15:49:14 -05:00
2e6ee28f9b Machine -> ::Machine so we don't conflict with Nix's 2024-01-23 11:03:19 -05:00
7386caaecf Use Nix's SSHMaster 2024-01-23 10:24:02 -05:00
f1d9230f25 Merge remote-tracking branch 'upstream/master' into nix-next 2024-01-23 01:18:13 -05:00
4e8fbaa3d6 Replace Child with SSHMaster::Connection
Nix defines basically an identical struct for the same purpose, so let's
just use that.
2024-01-23 01:11:46 -05:00
4ac31c89df Use nix::serv_proto::BasicConnection in build_remote.cc
- Use the type itself

  This lays the foundation for being able to dedup the protocol code.

- Use `BasicConnection::handshake`, replacing ours.

- Use `BasicConnection::queryValidPaths`

- Use `BasicConnection::putBuildDerivationRequest`
2024-01-22 14:20:39 -05:00
ebfefb9161 Sync up with some changes done to the main CA branch 2023-12-11 12:46:36 -05:00
8783dd53f6 Merge remote-tracking branch 'upstream/master' into ca-derivations-prep 2023-12-11 12:42:43 -05:00
f3a760ad9c Merge pull request #1324 from obsidiansystems/serve-proto-build-options-serializer
Use `ServeProto::Serialise<ServeProto::BuildOptions>`
2023-12-11 10:45:33 -05:00
8c10331ee8 Fix totalNarSize summation
I accidentally removed it in d0d3b0a298.
2023-12-10 14:05:26 -05:00
20f5a2120c Use ServeProto::Serialise<ServeProto::BuildOptions> 2023-12-10 13:24:17 -05:00
b56d2383c1 Do not attempt to speak a newer version of the protocol
Both sides need to agree on a version (with `std::min`) for anything to
work. Somehow... we've never done this.

With this comment, the next commit succeeds. Without this commit, the
next commit fails. This is because the next commit exposes serializers
which do different things for proto version 2.7, and we're currently
requesting 2.6.

Opened https://github.com/NixOS/nix/issues/9584 to track this issue
2023-12-10 13:24:17 -05:00
69a5b00e60 Use ServeProto::BuildOption
More deduplication with Nix.
2023-12-10 13:01:00 -05:00
f6f817926a std::move the into the path info map 2023-12-09 12:12:00 -05:00
d0d3b0a298 Use ServeProto::Serialise<UnkeyedValidPathInfo> for QueryValidPaths
Companion to already-merged https://github.com/NixOS/nix/pull/9560
2023-12-09 12:08:04 -05:00
3f932a6731 build-remote: Use std::map<StorePath, UnkeyedValidPathInfo>
It is less denormalized
2023-12-09 11:59:09 -05:00
4515b5aa17 Merge pull request #1321 from NixOS/master
Mere `master` into `nix-next`
2023-12-09 11:53:58 -05:00
831021808c Merge pull request #1318 from obsidiansystems/use-build-result-serialiser
Use factored-out `BuildResult` serializer
2023-12-08 11:25:05 -05:00
2ee0068fdc Do not copy for both stores for now
It has a performance cost, and as the comment says we should be doing
the better solution. We want to land this preparatory change on prod
while the rest is still on staging, so we should just skip it for now.

Skipping it will not affect regular fixed-output and input-addressed
derivations, which are the only ones prod would deal with upon getting
this code.

The main CA derivations support branch will revert this commit so it
still works.
2023-12-07 15:05:03 -05:00
31ea6458ca Merge remote-tracking branch 'upstream/master' into ca-derivations-prep 2023-12-07 15:01:35 -05:00
20c8263e3c Update to Nix master
The point of this branch is to always track Nix master, so we are
proactively ready to upgrade to the next Nix release when it is ready.

Flake lock file updates:

• Updated input 'nix':
    'github:NixOS/nix/50f8f1c8bc019a4c0fd098b9ac674b94cfc6af0d' (2023-11-27)
  → 'github:NixOS/nix/c3827ff6348a4d5199eaddf8dbc2ca2e2ef46ec5' (2023-12-07)
• Added input 'nix/libgit2':
    'github:libgit2/libgit2/45fd9ed7ae1a9b74b957ef4f337bc3c8b3df01b5' (2023-10-18)
2023-12-07 13:11:31 -05:00
6a54ab24e2 Use factored-out BuildResult serializer
For the record, here is the Nix 2.19 version:
https://github.com/NixOS/nix/blob/2.19-maintenance/src/libstore/serve-protocol.cc,
which is what we would initially use.

It is a more complete version of what Hydra has today except for one
thing: it always unconditionally sets the start/stop times.

I think that is correct at the other end seems to unconditionally
measure them, but just to be extra careful, I reproduced the old
behavior of falling back on Hydra's own measurements if `startTime` is
0.

The only difference is that the fallback `stopTime` is now measured from
after the entire `BuildResult` is transferred over the wire, but I think
that should be negligible if it is measurable at all. (And remember,
this is fallback case I already suspect is dead code.)
2023-12-07 02:00:22 -05:00
86cd5e9076 copyClosureTo: Use SubstituteFlag instead of bool
This matches Nix (in the same serialization logic in
`src/libstore/legacy-ssh-store.cc`) and adds clarity.
2023-12-07 00:18:50 -05:00
11f8030b0f Add comment from GitHub about adding to store as code comment 2023-12-06 17:59:25 -05:00
8046ec2668 Remove unused outputHashes variable
This looks like a stray copy paste.
2023-12-04 16:21:56 -05:00
9ba4417940 Prepare for CA derivation support with lower impact changes
This is just C++ changes without any Perl / Frontend / SQL Schema
changes.

The idea is that it should be possible to redeploy Hydra with these
chnages with (a) no schema migration and also (b) no regressions. We
should be able to much more safely deploy these to a staging server and
then production `hydra.nixos.org`.

Extracted from #875

Co-Authored-By: Théophane Hufschmitt <theophane.hufschmitt@tweag.io>
Co-Authored-By: Alexander Sosedkin <monk@unboiled.info>
Co-Authored-By: Andrea Ciceri <andrea.ciceri@autistici.org>
Co-Authored-By: Charlotte 🦝 Delenk Mlotte@chir.rs>
Co-Authored-By: Sandro Jäckel <sandro.jaeckel@gmail.com>
2023-12-04 16:14:47 -05:00
363604846a Again, use const in for loop
As requested by @teh. Was lost in merge with master, now added back.
2023-12-04 11:31:05 -05:00
162b538912 Remove unused thisArrow variable 2023-12-04 11:27:39 -05:00
104baef503 Document the connection initialization process 2023-12-04 09:42:04 -05:00
67eeabd518 Merge remote-tracking branch 'upstream/master' into split-buildRemote 2023-12-04 09:12:58 -05:00
622c25e3c4 Sedding prior to merge 2023-12-04 08:56:06 -05:00
c922e73c11 Update to Nix 2.19
Flake lock file updates:

• Updated input 'nix':
    'github:NixOS/nix/f5f4de6a550327b4b1a06123c2e450f1b92c73b6' (2023-10-02)
  → 'github:NixOS/nix/50f8f1c8bc019a4c0fd098b9ac674b94cfc6af0d' (2023-11-27)
2023-11-30 15:26:46 -05:00
e172461e55 Use const in for loop
As requested by @teh
2023-11-30 12:19:20 -05:00
0917145622 Make new functions not in header static 2023-11-30 12:19:05 -05:00
2bda7ca642 Further use Machine::Connection to deduplicate 2023-11-30 11:31:58 -05:00
831a2d9bd5 Merge remote-tracking branch 'upstream/master' into split-buildRemote 2023-11-30 11:27:40 -05:00
e9da80fff6 support nix 2.18 2023-11-21 18:41:52 +07:00
35ccc9ebb2 Fix indentation
Co-authored-by: John Ericson <git@JohnEricson.me>
2023-08-23 17:04:45 +02:00
9f0427385f Apply LTO fix suggested by Ericson2314 2023-08-20 14:55:56 +02:00
b23431a657 Support Nix 2.17 2023-08-04 15:53:48 +02:00
9f69bb5c2c Fix compilation against Nix 2.16 2023-06-23 15:06:55 +02:00
3526d61ff2 Merge remote-tracking branch 'upstream/master' into split-buildRemote 2022-10-25 11:24:54 -04:00