feature/resync #6

ahuston-0 · 2025-06-01T01:35:59-04:00

ahuston-0 commented

2025-06-01 01:35:59 -04:00

No description provided.

ahuston-0 added 83 commits 2025-06-01 01:36:00 -04:00

Fix rendering of metrics with special characters d22d030503

My main motivation here is to get metrics with brackets to work in order
to support "pytest" test names:

- test_foo.py::test_bar[1]
- test_foo.py::test_bar[2]

I couldn't find an "HTML escape"-style function that would generate
valid html `id` attribute names from random strings, so I went with a
hash digest instead.

Fix race condition in hydra-compress-logs f1deb22c02

Revert "Use LegacySSHStore" 9a5bd39d4c

There were some hangs caused by this. Need to fix them, ideally
reproducing the issue in a test, before trying this again.

This reverts commit 4a4a0f901c.

Merge pull request #1451 from NixOS/revert-to-fix-hangs 97dcdae068

Revert "Use `LegacySSHStore`"

Merge pull request #1450 from NixOS/hydra-compress-race d2db3c7446

Fix race condition in hydra-compress-logs

hydra-eval-jobset: do not wait on n-e-j inside transaction 987dad3371

fixes #1429

hydraTest: remove outdated postgresql version 4bcbed2f1b

error: postgresql_12 has been removed since it reached its EOL upstream

flake.lock: Update feebb61897

Flake lock file updates:

• Updated input 'nix-eval-jobs':
    'github:nix-community/nix-eval-jobs/4b392b284877d203ae262e16af269f702df036bc?narHash=sha256-3wIReAqdTALv39gkWXLMZQvHyBOc3yPkWT2ZsItxedY%3D' (2025-02-14)
  → 'github:nix-community/nix-eval-jobs/f7418fc1fa45b96d37baa95ff3c016dd5be3876b?narHash=sha256-Lo4KFBNcY8tmBuCmEr2XV0IUZtxXHmbXPNLkov/QSU0%3D' (2025-03-26)

Reimplement (named) constituent jobs (+globbing) based on nix-eval-jobs 9911f0107f

Depends on https://github.com/nix-community/nix-eval-jobs/pull/349 & #1421.

Almost equivalent to #1425, but with a small change: when having e.g. an
aggregate job with a glob that matches nothing, the jobset evaluation is
failed now. This was the intended behavior before (hydra-eval-jobset
fails hard if an aggregate is broken), the code-path was never reached
however since the aggregate was never marked as broken in this case
before.

Merge pull request #1455 from qowoz/226-constituent b2fe3f5218

nix-eval-jobs + constituent globs

Merge pull request #1449 from knedlsepp/fix-metrics-rendering-with-special-characters 3b4c4972c2

Fix rendering of metrics with special characters

Merge pull request #1457 from dermetfan/fix-1429 b657bcdfb7

hydra-eval-jobset: do not wait on n-e-j inside transaction

fix development workflow after switching to meson-based build ae18a7b3ae

gitignore hydra-data as created by foreman 3fef32b364

Merge pull request #1458 from NixOS/meson cad08f87d2

docs: fix contribution guide for new meson-based build

flake.lock: Update to nix and nix-eval-jobs 2.27 6b97e3ab7b

Flake lock file updates:

• Updated input 'nix':
    'github:NixOS/nix/e310c19a1aeb1ce1ed4d41d5ab2d02db596e0918?narHash=sha256-q/RgA4bB7zWai4oPySq9mch7qH14IEeom2P64SXdqHs%3D' (2025-02-18)
  → 'github:NixOS/nix/d0f98c76f962147610489e84c10033ca92e9c532?narHash=sha256-u6RhBWQ1XohTZ4Ub5ml1PTcaxQgtqFNng6Sohy1rojw%3D' (2025-04-07)
• Updated input 'nix-eval-jobs':
    'github:nix-community/nix-eval-jobs/f7418fc1fa45b96d37baa95ff3c016dd5be3876b?narHash=sha256-Lo4KFBNcY8tmBuCmEr2XV0IUZtxXHmbXPNLkov/QSU0%3D' (2025-03-26)
  → 'github:nix-community/nix-eval-jobs/62f9c9e8d00d2ff6ab27a6197ab459a8e0808e59?narHash=sha256-PypQspB7h7EENe4RQQUQj2Ay8J1%2BO49AKNO9JbAU4Ek%3D' (2025-04-07)

Fix build (due to C++ API changes) 1cb1e139c4

flake.lock: Update to nix and nix-eval-jobs 2.28 1595064bee

Flake lock file updates:

• Updated input 'nix':
    'github:NixOS/nix/d0f98c76f962147610489e84c10033ca92e9c532?narHash=sha256-u6RhBWQ1XohTZ4Ub5ml1PTcaxQgtqFNng6Sohy1rojw%3D' (2025-04-07)
  → 'github:NixOS/nix/a4962f73b5fc874d4b16baef47921daf349addfc?narHash=sha256-r%2BpsCOW77vTSTNbxTVrYHeh6OgB0QukbnyUVDwg8s4I%3D' (2025-04-07)
• Updated input 'nix-eval-jobs':
    'github:nix-community/nix-eval-jobs/62f9c9e8d00d2ff6ab27a6197ab459a8e0808e59?narHash=sha256-PypQspB7h7EENe4RQQUQj2Ay8J1%2BO49AKNO9JbAU4Ek%3D' (2025-04-07)
  → 'github:nix-community/nix-eval-jobs/cba718bafe5dc1607c2b6761ecf53c641a6f3b21?narHash=sha256-v5n6t49X7MOpqS9j0FtI6TWOXvxuZMmGsp2OfUK5QfA%3D' (2025-04-07)

Fix Nix code 6534a54ee5

Can now at least enter dev shell, but build is still broken.

Fix the build d6a5df25bf

Merge pull request #1460 from NixOS/nix-2.27 257b211832

Nix 2.27

Merge pull request #1461 from NixOS/nix-2.28 1d2d3ae6b7

Nix 2.28

web: include current step status on /machines 0159135fc7

Make "timed out" and "log limit exceeded" builds aborted a816e8e22c

In 73694087a0 I gave builds that failed
because of a timeout or exceeded log limit a stop sign and I stand by
that reasoning: with that it's possible to distinguish between actual
build failures and rather transient things such as timeouts.

Back then I considered it a feature that these are shown in a different
tab, but I don't think that's a good idea anymore. When using a jobset to
e.g. track the regressions from a mass rebuild (like a compiler or gcc
update), "Newly failed builds" should exclusively display regressions (and
flaky builds of course, not much I can do about that).

Also, when a bunch of builds fail in such a jobset because of e.g. a
broken connection to a builder that results in a timeout, I want to be
able to restart them all w/o rebuilding actual regressions.

To make it clear that we not only have "Aborted" builds in the tab, I
renamed the label to "Aborted / Timed out".

Running builds view: show build step names 33b982f408

When using Hydra to build machine configurations, you'll often see
"nixosConfigurations.foo" five times, i.e. for each build step being
run. This isn't very helpful I think because in such a case, a single
build step can also be compiling the Linux kernel.

This change also fetches the `drvpath` and `type` from the `buildsteps`
relation. We're already joining it, so this doesn't make much difference
(confirmed via query logging that this doesn't cause extra SQL queries).

Unfortunately build steps don't have a human readable name, so I'm
deriving it from the drvpath by stripping away the hash (assuming that
it'll never contain a `-` and that `/nix/store/` is used as prefix). I
decided against using the Nix bindings for that to avoid too much
overhead due to store operations for each build step.

Only show stepname if it doesn't equal the name of the drv 9d8f30affe

When building e.g. nixpkgs, the "Running builds" view will mostly look
like this

    hello.x86_64-linux (Build of hello-X.Y)
    exa.x86_64-linux (Build of exa-X.Y)
    ...

This doesn't provide any useful information. Showing the step name only
makes sense if it's not a child of the job's derivation. With this
patch, that information will only be shown if the drv name (i.e. w/o
`/nix/store/` prefix, .drv ext & hash) is not equal to the drv name of
the job itself (build.nixname).

lazy-load evaluation errors 99359c251a

Closes #1362

Stream files from store instead of buffering them abe35881e4

When an artifact is requested from hydra the output is first copied
from the nix store into memory and then sent as a response, delaying
the download and taking up significant amounts of memory.

As reported in https://github.com/NixOS/hydra/issues/1357

Instead of calling a command and blocking while reading in the entire
output, this adds read_into_socket(). the function takes a
command, starting a subprocess with that command, returning a file
descriptor attached to stdout.
This file descriptor is then by responsebuilder of Catalyst to steam
the output directly

(cherry picked from commit 459aa0a5983a0bd546399c08231468d6e9282f54)

readIntoSocket: fix with store URIs containing an & 6133693097

The third argument to `open()` in `-|` mode is passed to a shell if it's
a string. In my case the store URI contains
`?secret-key=${signingKey.directory}/secret&compression=zstd`

For the `nix store cat` case this means that

* until `&` the process will be started in the background. This fails
  immediately because no path to cat is specified.
* `compression=zstd` is a variable assignment
* the `$path` argument to `store cat` is attempted to be executed as
  another command

Passing just the list solves the problem.

(cherry picked from commit 3ee51dbe589458cc54ff753317bbc6db530bddc0)

Merge pull request #1462 from NixOS/web-changes 3fdb18a4bc

A number of Perl-side changes from the hydra.nixos.org branch

Fix evaluation of NixOS tests, avoid with 0d0c4f278b

web: Skip System on /machines 5a9985f96c

It is redundant

queue-runner: limit parallelism of CPU intensive operations 1e2d3211d9

My current theory is that running more parallel xz than available CPU
cores is reducing our overall throughput by requiring more scheduling
overhead and more cache thrashing.

queue-runner: add prom metrics to allow detecting internal bottlenecks efcf6815d9

By looking at the ratio of running vs. waiting for the dispatcher and
the queue monitor, we should get better visibility into what hydra is
currently bottlenecked on.

There are other side effects we can try to measure to get to the same
result, but having a simple way doesn't cost us much.

queue-runner: remove id > X from new builds query d8ffa6b56a

Running the query with/without it shows that it makes no difference to
postgres, since there's an index on finished=0 already. This allows a
few simplifications, but also paves the way towards running multiple
parallel monitor threads in the future.

queue-runner: reduce the time between queue monitor restarts 9265fc5002

This will induce more DB queries (though these are fairly cheap), but at
the benefit of processing bumps within 1m instead of within 10m.

queue runner: introduce some parallelism for remote paths lookup 52a0199a9b

Each output for a given step being ingested is looked up in parallel,
which should basically multiply the speed of builds ingestion by the
average number of outputs per derivation.

queue-runner: switch to pseudorandom ordering of builds processing 8e02589ac8

We don't rely on sequential / monotonic build IDs processing anymore, so
randomizing actually has the advantage of mixing builds for different
systems together, to avoid only one chunk of builds for a single system
getting processed while builders for other systems are starved.

Merge pull request #1463 from NixOS/fix-nixos-tests f02fc5e2ff

Fix evaluation of NixOS tests, avoid `with`

test/gitea: fix eval 78687e23cf

Merge pull request #1465 from NixOS/gitea d4b55f8190

test/gitea: fix eval

Merge pull request #1464 from NixOS/more-hydra.nixos.org-changes 56170dd117

More hydra.nixos.org changes

don't build hydra twice in a pull request + enable merge queue c94ba404fd

Merge pull request #1467 from NixOS/merge-queue 68b2d6da0a

Make github actions ci merge-queue friendly

bump nixpkgs b3a433336e

Add metric for builds waiting for download slot 8a6482bb1c

(cherry picked from commit f23ec71227911891807706b6b978836e4d80edde)

Merge pull request #1468 from NixOS/steps-waiting-for-download-slot e77444da98

Add metric for builds waiting for download slot

Merge pull request #1466 from NixOS/bump-nixpkgs cc4b206d85

Bump nixpkgs

queue-runner: release machine reservation while copying outputs 143a07bff0

This allows for better builder usage when the queue runner is busy. To
avoid running into uncontrollable imbalances between builder/queue
runner, we only release the machine reservation after the local
throttler has found a slot to start copying the outputs for that build.

As opposed to asserting uniqueness to understand resource utilization,
we just switch to using `std::unique_ptr`.

Merge pull request #1469 from NixOS/release-reservations 8350f964ee

queue-runner: release machine reservation while copying outputs

web: replace 'errormsg' with 'errormsg IS NULL' in most cases b4322edd05

This is implement in an extremely hacky way due to poor DBIx feature
support. Ideally, what we'd need is a way to tell DBIx to ignore the
errormsg column unless explicitly requested, and to automatically add a
computed 'errormsg IS NULL' column in others. Since it does not support
that, this commit instead hacks some support via method overrides while
taking care to not break anything obvious.

Merge pull request #1456 from NixOS/hydra.nixos.org-rebased 1c52c4c0ed

web: replace 'errormsg' with 'errormsg IS NULL' in most cases

Queue-runner: Always produce a machines JSON object a5b17d0686

Even if there are no machines, there should at least be an empty object.

Merge pull request #1471 from NixOS/queue-runner-machines-json c8de5b99e3

Queue-runner: Always produce a machines JSON object

Fix compilation with a nix which was compiled withou aws sdk 7e0157e387

Merge pull request #1472 from SuperSandro2000/without-aws-sdk d5ad16abc2

Fix compilation with a nix which was compiled withou aws sdk

Fix displaying eval errors in jobset eval view f1a976d3fd

Quickfix for something that annoyed me once too often.

Specifically, I'm talking about `/eval/1#tabs-errors`.

To not fetch long errors on each request, this is only done on-demand.
I.e., when the tab is opened, an iframe is requested with the errors.
This iframe uses a template for both the jobset view and the jobset-eval
view. It is differentiated by checking if `jobset` or `eval` is defined.

However, the jobset-eval view also has a `jobset` variable in its stash
which means that in both cases the `if` path was used. Since
`jobset.fetcherrormsg` isn't defined in the eval case though, you always
got an empty error.

The band-aid fix is relatively simple: swap if and else: the `eval`
variable is not defined in the stash of the jobset view, so now this is
a useful condition to decide which view we're in.

(cherry picked from commit 70c3d75f73)

re-enable restrict-eval for non-flakes 8d75026513

Merge pull request #1470 from qowoz/eval-view 5f6b075754

Fix displaying eval errors in jobset eval view

web: increase colspan for machine row in machine status cf33a9158a

Merge pull request #1474 from NixOS/machine-status-colspan ffbde9c9e3

web: increase colspan for machine row in machine status

Merge commit from fork 6fcfa9e796

Re-enable restrict-eval for non-flakes

jobset-eval: fix actions not showing up sometimes for new jobs 0ab357e435

New jobs have their "new" status take precedence over them being
"failed" or "queued", which means actions that can act on "failed" or
"queued" jobs weren't shown to the user when they could only act on
"new" jobs.

(cherry picked from commit 9a4a5dd624)

Merge pull request #1478 from qowoz/fix-actions bdde73acbd

jobset-eval: fix actions not showing up sometimes for new jobs

queue runner: attempt at slightly smarter scheduling criteria 720db63d52

Instead of just going for "whatever is the oldest build we know of",
use the following first:

- Is the step more constrained? If so, schedule it first to avoid
  filling up "more desirable" build slots with less constrained builds.

- Does the step have more dependents? If so, schedule it first to try
  and maximize open parallelism and breadth of scheduling options.

(cherry picked from commit b8d03adaf4)

flake.lock: Update 23755bf001

Flake lock file updates:

• Updated input 'nix':
    'github:NixOS/nix/a4962f73b5fc874d4b16baef47921daf349addfc' (2025-04-07)
  → 'github:NixOS/nix/70921714cb3b5e6041b7413459541838651079f3' (2025-04-23)
• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/db8f4fe18ce772a9c8f3adf321416981c8fe9371' (2025-04-07)
  → 'github:NixOS/nixpkgs/eea3403f7ca9f9942098f4f2756adab4ec924b2b' (2025-04-23)

Merge pull request #1480 from NixOS/update-nix b023cc8f87

flake.lock: Update

Use Nix without the flake 89fcb931ce

This is what we do for `nix-eval-jobs` already. It allows for more
fine-grained control over dependencies.

Merge pull request #1481 from NixOS/nix-flake-false 455f1a0665

Use Nix without the flake

hydra: expose nix-cli package 8218a9ad1b

This makes it easier in other packages to get the nix version used to
build Hydra.

Merge pull request #1482 from NixOS/hydra-passthru 8d068fea3e

Expose nix package in hydra package

Merge pull request #1479 from qowoz/queue-runner 1b5c2fb747

queue runner: attempt at slightly smarter scheduling criteria

Add missing slash 2ae27dd20d

error: access to absolute path '/nix/store/sai35xfsrba2a2vasmzxakmn54wdfa13-sourcepackaging' is forbidden in pure evaluation mode (use '--impure' to override)

Merge pull request #1483 from SuperSandro2000/patch-2 183bc39d1a

Add missing slash

Migrate from "gc-" prefixed nix options da1aebe970

These have been deprecated, e.g. gc-keep-outputs is now just
keep-outputs.

Merge pull request #1484 from NixOS/nix-keep-options 35c9264306

Migrate from "gc-" prefixed nix options

doc/manual: correct nginx reverse proxy example 8bb7d27588

- hydra does not remove the base URI from the request before processing
  it, so this must be done in the reverse proxy. in nginx this is done
  by giving proxy_pass a URI rather than a protocol/host/port; see:

  https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass

- proxy_redirect is not correct/required: hydra uses proxy headers to
  correctly form redirects in most cases, and where it doesn't it
  produces local redirects which aren't matched by this directive anyway

flake.lock: Update Nixpkgs to 25.05 8a50488f6c

Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/eea3403f7ca9f9942098f4f2756adab4ec924b2b?narHash=sha256-JT1wMjLIypWJA0N2V27WpUw8feDmTok4Dwkb0oYXDS4%3D' (2025-04-23)
  → 'github:NixOS/nixpkgs/db1aed32009f408e4048c1dd0beaf714dd34ed93?narHash=sha256-8A7HjmnvCpDjmETrZY1QwzKunR63LiP7lHu1eA5q6JI%3D' (2025-05-24)

flake.lock: Update Nix and nix-eval-jobs to 2.29 dafa252d08

Flake lock file updates:

• Updated input 'nix':
    'github:NixOS/nix/70921714cb3b5e6041b7413459541838651079f3?narHash=sha256-ZbB3IH9OlJvo14GlQZbYHzJojf/HCDT38GzYTod8DaU%3D' (2025-04-23)
  → 'github:NixOS/nix/d761dad79c79af17aa476a29749bd9d69747548f?narHash=sha256-rCpANMHFIlafta6J/G0ILRd%2BWNSnzv/lzi40Y8f1AR8%3D' (2025-05-25)
• Updated input 'nix-eval-jobs':
    'github:nix-community/nix-eval-jobs/1260c6599d22dfd8c25fea6893c3d031996b20e1?narHash=sha256-n220U5pjzCtTtOJtbga4Xr/PyllowKw9anSevgCqJEw%3D' (2025-04-11)
  → 'github:nix-community/nix-eval-jobs/d9262e535e35454daebcebd434bdb9c1486bb998?narHash=sha256-AJ22q6yWc1hPkqssXMxQqD6QUeJ6hbx52xWHhKsmuP0%3D' (2025-05-25)

Fix build with Nix 2.29 278a3ebfd5

Merge pull request #1488 from NixOS/nixpkgs-25.05 de3646cb13

flake.lock: Update Nixpkgs to 25.05

Merge pull request #1485 from NixOS/nix-2.29 362524b563

Nix 2.29

Merge pull request #1487 from tomjnixon/reverse_proxy_docs 2e3c168ec4

doc/manual: correct nginx reverse proxy example

ahuston-0 force-pushed feature/resync from 8428fd66a9 to ae8c1554cb

2025-06-01 01:37:32 -04:00

Compare

ahuston-0 merged commit ae8c1554cb into add-gitea-pulls

2025-06-01 01:37:55 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: ahuston-0/hydra#6