resync hydra with upstream #4

ahuston-0 · 2025-04-09T11:29:59-04:00

ahuston-0 commented

2025-04-09 11:29:59 -04:00

No description provided.

ahuston-0 added 54 commits 2025-04-09 11:30:00 -04:00

Fix rendering of metrics with special characters d22d030503

My main motivation here is to get metrics with brackets to work in order
to support "pytest" test names:

- test_foo.py::test_bar[1]
- test_foo.py::test_bar[2]

I couldn't find an "HTML escape"-style function that would generate
valid html `id` attribute names from random strings, so I went with a
hash digest instead.

Fix race condition in hydra-compress-logs f1deb22c02

Revert "Use LegacySSHStore" 9a5bd39d4c

There were some hangs caused by this. Need to fix them, ideally
reproducing the issue in a test, before trying this again.

This reverts commit 4a4a0f901c.

Merge pull request #1451 from NixOS/revert-to-fix-hangs 97dcdae068

Revert "Use `LegacySSHStore`"

Merge pull request #1450 from NixOS/hydra-compress-race d2db3c7446

Fix race condition in hydra-compress-logs

hydra-eval-jobset: do not wait on n-e-j inside transaction 987dad3371

fixes #1429

hydraTest: remove outdated postgresql version 4bcbed2f1b

error: postgresql_12 has been removed since it reached its EOL upstream

flake.lock: Update feebb61897

Flake lock file updates:

• Updated input 'nix-eval-jobs':
    'github:nix-community/nix-eval-jobs/4b392b284877d203ae262e16af269f702df036bc?narHash=sha256-3wIReAqdTALv39gkWXLMZQvHyBOc3yPkWT2ZsItxedY%3D' (2025-02-14)
  → 'github:nix-community/nix-eval-jobs/f7418fc1fa45b96d37baa95ff3c016dd5be3876b?narHash=sha256-Lo4KFBNcY8tmBuCmEr2XV0IUZtxXHmbXPNLkov/QSU0%3D' (2025-03-26)

Reimplement (named) constituent jobs (+globbing) based on nix-eval-jobs 9911f0107f

Depends on https://github.com/nix-community/nix-eval-jobs/pull/349 & #1421.

Almost equivalent to #1425, but with a small change: when having e.g. an
aggregate job with a glob that matches nothing, the jobset evaluation is
failed now. This was the intended behavior before (hydra-eval-jobset
fails hard if an aggregate is broken), the code-path was never reached
however since the aggregate was never marked as broken in this case
before.

Merge pull request #1455 from qowoz/226-constituent b2fe3f5218

nix-eval-jobs + constituent globs

Merge pull request #1449 from knedlsepp/fix-metrics-rendering-with-special-characters 3b4c4972c2

Fix rendering of metrics with special characters

Merge pull request #1457 from dermetfan/fix-1429 b657bcdfb7

hydra-eval-jobset: do not wait on n-e-j inside transaction

fix development workflow after switching to meson-based build ae18a7b3ae

gitignore hydra-data as created by foreman 3fef32b364

Merge pull request #1458 from NixOS/meson cad08f87d2

docs: fix contribution guide for new meson-based build

flake.lock: Update to nix and nix-eval-jobs 2.27 6b97e3ab7b

Flake lock file updates:

• Updated input 'nix':
    'github:NixOS/nix/e310c19a1aeb1ce1ed4d41d5ab2d02db596e0918?narHash=sha256-q/RgA4bB7zWai4oPySq9mch7qH14IEeom2P64SXdqHs%3D' (2025-02-18)
  → 'github:NixOS/nix/d0f98c76f962147610489e84c10033ca92e9c532?narHash=sha256-u6RhBWQ1XohTZ4Ub5ml1PTcaxQgtqFNng6Sohy1rojw%3D' (2025-04-07)
• Updated input 'nix-eval-jobs':
    'github:nix-community/nix-eval-jobs/f7418fc1fa45b96d37baa95ff3c016dd5be3876b?narHash=sha256-Lo4KFBNcY8tmBuCmEr2XV0IUZtxXHmbXPNLkov/QSU0%3D' (2025-03-26)
  → 'github:nix-community/nix-eval-jobs/62f9c9e8d00d2ff6ab27a6197ab459a8e0808e59?narHash=sha256-PypQspB7h7EENe4RQQUQj2Ay8J1%2BO49AKNO9JbAU4Ek%3D' (2025-04-07)

Fix build (due to C++ API changes) 1cb1e139c4

flake.lock: Update to nix and nix-eval-jobs 2.28 1595064bee

Flake lock file updates:

• Updated input 'nix':
    'github:NixOS/nix/d0f98c76f962147610489e84c10033ca92e9c532?narHash=sha256-u6RhBWQ1XohTZ4Ub5ml1PTcaxQgtqFNng6Sohy1rojw%3D' (2025-04-07)
  → 'github:NixOS/nix/a4962f73b5fc874d4b16baef47921daf349addfc?narHash=sha256-r%2BpsCOW77vTSTNbxTVrYHeh6OgB0QukbnyUVDwg8s4I%3D' (2025-04-07)
• Updated input 'nix-eval-jobs':
    'github:nix-community/nix-eval-jobs/62f9c9e8d00d2ff6ab27a6197ab459a8e0808e59?narHash=sha256-PypQspB7h7EENe4RQQUQj2Ay8J1%2BO49AKNO9JbAU4Ek%3D' (2025-04-07)
  → 'github:nix-community/nix-eval-jobs/cba718bafe5dc1607c2b6761ecf53c641a6f3b21?narHash=sha256-v5n6t49X7MOpqS9j0FtI6TWOXvxuZMmGsp2OfUK5QfA%3D' (2025-04-07)

Fix Nix code 6534a54ee5

Can now at least enter dev shell, but build is still broken.

Fix the build d6a5df25bf

Merge pull request #1460 from NixOS/nix-2.27 257b211832

Nix 2.27

Merge pull request #1461 from NixOS/nix-2.28 1d2d3ae6b7

Nix 2.28

web: include current step status on /machines 0159135fc7

Make "timed out" and "log limit exceeded" builds aborted a816e8e22c

In 73694087a0 I gave builds that failed
because of a timeout or exceeded log limit a stop sign and I stand by
that reasoning: with that it's possible to distinguish between actual
build failures and rather transient things such as timeouts.

Back then I considered it a feature that these are shown in a different
tab, but I don't think that's a good idea anymore. When using a jobset to
e.g. track the regressions from a mass rebuild (like a compiler or gcc
update), "Newly failed builds" should exclusively display regressions (and
flaky builds of course, not much I can do about that).

Also, when a bunch of builds fail in such a jobset because of e.g. a
broken connection to a builder that results in a timeout, I want to be
able to restart them all w/o rebuilding actual regressions.

To make it clear that we not only have "Aborted" builds in the tab, I
renamed the label to "Aborted / Timed out".

Running builds view: show build step names 33b982f408

When using Hydra to build machine configurations, you'll often see
"nixosConfigurations.foo" five times, i.e. for each build step being
run. This isn't very helpful I think because in such a case, a single
build step can also be compiling the Linux kernel.

This change also fetches the `drvpath` and `type` from the `buildsteps`
relation. We're already joining it, so this doesn't make much difference
(confirmed via query logging that this doesn't cause extra SQL queries).

Unfortunately build steps don't have a human readable name, so I'm
deriving it from the drvpath by stripping away the hash (assuming that
it'll never contain a `-` and that `/nix/store/` is used as prefix). I
decided against using the Nix bindings for that to avoid too much
overhead due to store operations for each build step.

Only show stepname if it doesn't equal the name of the drv 9d8f30affe

When building e.g. nixpkgs, the "Running builds" view will mostly look
like this

    hello.x86_64-linux (Build of hello-X.Y)
    exa.x86_64-linux (Build of exa-X.Y)
    ...

This doesn't provide any useful information. Showing the step name only
makes sense if it's not a child of the job's derivation. With this
patch, that information will only be shown if the drv name (i.e. w/o
`/nix/store/` prefix, .drv ext & hash) is not equal to the drv name of
the job itself (build.nixname).

lazy-load evaluation errors 99359c251a

Closes #1362

Stream files from store instead of buffering them abe35881e4

When an artifact is requested from hydra the output is first copied
from the nix store into memory and then sent as a response, delaying
the download and taking up significant amounts of memory.

As reported in https://github.com/NixOS/hydra/issues/1357

Instead of calling a command and blocking while reading in the entire
output, this adds read_into_socket(). the function takes a
command, starting a subprocess with that command, returning a file
descriptor attached to stdout.
This file descriptor is then by responsebuilder of Catalyst to steam
the output directly

(cherry picked from commit 459aa0a5983a0bd546399c08231468d6e9282f54)

readIntoSocket: fix with store URIs containing an & 6133693097

The third argument to `open()` in `-|` mode is passed to a shell if it's
a string. In my case the store URI contains
`?secret-key=${signingKey.directory}/secret&compression=zstd`

For the `nix store cat` case this means that

* until `&` the process will be started in the background. This fails
  immediately because no path to cat is specified.
* `compression=zstd` is a variable assignment
* the `$path` argument to `store cat` is attempted to be executed as
  another command

Passing just the list solves the problem.

(cherry picked from commit 3ee51dbe589458cc54ff753317bbc6db530bddc0)

Merge pull request #1462 from NixOS/web-changes 3fdb18a4bc

A number of Perl-side changes from the hydra.nixos.org branch

Fix evaluation of NixOS tests, avoid with 0d0c4f278b

web: Skip System on /machines 5a9985f96c

It is redundant

queue-runner: limit parallelism of CPU intensive operations 1e2d3211d9

My current theory is that running more parallel xz than available CPU
cores is reducing our overall throughput by requiring more scheduling
overhead and more cache thrashing.

queue-runner: add prom metrics to allow detecting internal bottlenecks efcf6815d9

By looking at the ratio of running vs. waiting for the dispatcher and
the queue monitor, we should get better visibility into what hydra is
currently bottlenecked on.

There are other side effects we can try to measure to get to the same
result, but having a simple way doesn't cost us much.

queue-runner: remove id > X from new builds query d8ffa6b56a

Running the query with/without it shows that it makes no difference to
postgres, since there's an index on finished=0 already. This allows a
few simplifications, but also paves the way towards running multiple
parallel monitor threads in the future.

queue-runner: reduce the time between queue monitor restarts 9265fc5002

This will induce more DB queries (though these are fairly cheap), but at
the benefit of processing bumps within 1m instead of within 10m.

queue runner: introduce some parallelism for remote paths lookup 52a0199a9b

Each output for a given step being ingested is looked up in parallel,
which should basically multiply the speed of builds ingestion by the
average number of outputs per derivation.

queue-runner: switch to pseudorandom ordering of builds processing 8e02589ac8

We don't rely on sequential / monotonic build IDs processing anymore, so
randomizing actually has the advantage of mixing builds for different
systems together, to avoid only one chunk of builds for a single system
getting processed while builders for other systems are starved.

Merge pull request #1463 from NixOS/fix-nixos-tests f02fc5e2ff

Fix evaluation of NixOS tests, avoid `with`

test/gitea: fix eval 78687e23cf

Merge pull request #1465 from NixOS/gitea d4b55f8190

test/gitea: fix eval

Merge pull request #1464 from NixOS/more-hydra.nixos.org-changes 56170dd117

More hydra.nixos.org changes

don't build hydra twice in a pull request + enable merge queue c94ba404fd

Merge pull request #1467 from NixOS/merge-queue 68b2d6da0a

Make github actions ci merge-queue friendly

bump nixpkgs b3a433336e

Add metric for builds waiting for download slot 8a6482bb1c

(cherry picked from commit f23ec71227911891807706b6b978836e4d80edde)

Merge pull request #1468 from NixOS/steps-waiting-for-download-slot e77444da98

Add metric for builds waiting for download slot

Merge pull request #1466 from NixOS/bump-nixpkgs cc4b206d85

Bump nixpkgs

queue-runner: release machine reservation while copying outputs 143a07bff0

This allows for better builder usage when the queue runner is busy. To
avoid running into uncontrollable imbalances between builder/queue
runner, we only release the machine reservation after the local
throttler has found a slot to start copying the outputs for that build.

As opposed to asserting uniqueness to understand resource utilization,
we just switch to using `std::unique_ptr`.

Merge pull request #1469 from NixOS/release-reservations 8350f964ee

queue-runner: release machine reservation while copying outputs

web: replace 'errormsg' with 'errormsg IS NULL' in most cases b4322edd05

This is implement in an extremely hacky way due to poor DBIx feature
support. Ideally, what we'd need is a way to tell DBIx to ignore the
errormsg column unless explicitly requested, and to automatically add a
computed 'errormsg IS NULL' column in others. Since it does not support
that, this commit instead hacks some support via method overrides while
taking care to not break anything obvious.

Merge pull request #1456 from NixOS/hydra.nixos.org-rebased 1c52c4c0ed

web: replace 'errormsg' with 'errormsg IS NULL' in most cases

Queue-runner: Always produce a machines JSON object a5b17d0686

Even if there are no machines, there should at least be an empty object.

Merge pull request #1471 from NixOS/queue-runner-machines-json c8de5b99e3

Queue-runner: Always produce a machines JSON object

ahuston-0 force-pushed feature/resync from 7a0b6b50f3 to 33a935e8ef

2025-04-09 11:31:50 -04:00

Compare

ahuston-0 merged commit 33a935e8ef into add-gitea-pulls

2025-04-09 11:38:41 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: ahuston-0/hydra#4