Commit Graph

3821 Commits

Author SHA1 Message Date
107d60027f hydra-eval-jobs: Fix unexpected EOF when a top-level attr fails 2021-02-22 16:29:07 +01:00
a7d8ee98da Fix build 2021-02-22 15:10:24 +01:00
34b438ab6e flake.lock: Update
Flake input changes:

* Updated 'nix': 'github:NixOS/nix/8a2ce0f455da32bc20978e68c0aad9efb4560abc' -> 'github:NixOS/nix/548437c2347159c4c79352283dd12ce58324f1d6'
* Removed 'nix/lowdown-src'
2021-02-22 15:03:20 +01:00
45d9a22d73 flake.nix: add perlPackages until they're available from nixpkgs
These packages were added to Nixpkgs in https://github.com/NixOS/nixpkgs/pull/113702.
2021-02-19 17:06:49 -05:00
2240035e20 Run tests with yath
This will let us run tests in parallel, and creates a more Perl-standard
test development experience.
2021-02-19 17:04:19 -05:00
a39b479280 Merge pull request #866 from Infinisil/github-status-flakes
Fix Github status plugin for flakes
2021-02-16 17:00:46 +01:00
150213cbb3 Fix login if Hydra runs behind HTTP proxy with sub-path location 2021-02-07 19:18:29 +01:00
58dd7f9ed3 Fix Github status plugin for flakes
If the root flake is a github: one, github status notifications are sent
to it. The githubstatus->inputs configuration isn't used for flakes.
2021-02-06 00:02:30 +01:00
d8a5892795 Merge pull request #856 from immae/fix_check_jobsets
Fix check in jobsets
2021-02-03 16:25:18 -05:00
339a09f2e4 Fix check in jobsets
The current check happening in jobsets is incorrect.
The wanted constraint is stated as follow :
- If type is 0 (legacy), then the flake field should be null, and
  both nixExprInput and nixExprPath should be non-null
- If type is 1 (flake), then the flake field should be non-null, and
  both nixExprInput and nixExprPath should be null

The current version will not catch (i.e. it will accept) situations
where you have for instance :
type = 1, nixExprPath null, nixExprInput non-null, flake non-null

This commit fixes that.

I split(ted) that into two constraints, to make it more readable and
easier to extend if a new type appears in the future.

The complete query could be instead :
( type = 0
  AND nixExprInput IS NOT NULL AND nixExprPath IS NOT NULL AND flake IS NULL )
OR ( type = 1
  AND nixExprInput IS NULL AND nixExprPath IS NULL AND flake IS NOT NULL )

(but an "OR" cannot be split, hence the other formulation)
2021-02-03 22:14:53 +01:00
bc12fe19f9 Merge pull request #855 from grahamc/jobsetevals-fixups
JobsetEvals: fixup permission references
2021-02-02 11:04:18 -05:00
6de9c6540c Merge pull request #858 from Infinisil/fix-declarative-flakes
Fix transition from declarative non-flake to flake jobset
2021-02-02 11:04:05 -05:00
ac80843e31 Merge pull request #860 from grahamc/eval-error-table
Move evaluation errors from evaluations to EvaluationErrors, a new table
2021-02-02 10:37:04 -05:00
f1e75c8bff Move evaluation errors from evaluations to EvaluationErrors, a new table
DBIx likes to eagerly select all columns without a way to really tell
it so. Therefore, this splits this one large column in to its own
table.

I'd also like to make "jobsets" use this table too, but that is on hold
to stop the bleeding caused by the extreme amount of traffic this is
causing.
2021-02-01 21:33:14 -05:00
1d45b63516 Fix transition from declarative non-flake to flake jobset
The database has these constraints:

    check ((type = 0) = (nixExprInput is not null and nixExprPath is not null)),
    check ((type = 1) = (flake is not null)),

which prevented switching to flakes in a declarative jobspec, since the
nixexpr{path,input} fields were not nulled in such an update

Co-Authored-By: Graham Christensen <graham@grahamc.com>
2021-02-01 18:57:40 +01:00
8d7bfe1706 JobsetEvals: fixup permission references
Going from an eval to a project now requires hopping through the jobset
2021-02-01 10:31:05 -05:00
aaf16d542c Merge pull request #854 from NixOS/limit-query-time
search: limit queries to 20s
2021-01-30 11:55:22 -05:00
91e63fb7da search: limit queries to 20s
Even 20s is really long, but it cuts off queries which are today
running for 500+s.
2021-01-30 11:51:20 -05:00
72e237fb2f Merge pull request #853 from NixOS/search-limit-reqs
search: limit results to 50, default to 10
2021-01-30 08:57:05 -05:00
4f308b1f2f search: limit results to 50, default to 10
This search query is pretty heavy. Defaulting to 500 has caused
Hydra's web UI to appear to be down. Since 500 can take it down, users
probably shouldn't be allowed t ask for that many.
2021-01-30 08:37:57 -05:00
6d047c286f Merge pull request #850 from grahamc/jobset-evals-by-id
Jobset -> JobsetEvals by JobsetEvals.jobset_id
2021-01-28 09:25:18 -05:00
5fcc2018db hydra-evaluator: clean up names, clean up & / * spacing 2021-01-28 09:15:19 -05:00
091d58c128 hydra-dev-server: autoflush stderr/stdout 2021-01-26 13:51:39 -05:00
54b8cb188e perl: jobsetevals -> jobset via by jobset_id
Frankly, this was suspiciously little work.
2021-01-26 13:51:39 -05:00
54341cd9f6 hydra-evaluator: deal in jobset IDs 2021-01-26 13:51:31 -05:00
cb01859718 hydra-evaluator: JobsetName -> JobsetIdentity 2021-01-26 11:50:38 -05:00
705a45df2b hydra-evaluator: reformat readJobsets query 2021-01-26 11:50:37 -05:00
ac3e8a4a59 jobsetevals: refer to jobset by ID 2021-01-26 11:50:37 -05:00
99e3c83358 JobsetEvals: noop: re-run the generator to update the order of fields 2021-01-26 11:50:36 -05:00
bf674a9653 hydra.sql: embed some in-line docs about schema changes 2021-01-26 11:50:36 -05:00
dc5a0d59c5 sql: Stop loading SQL if an error occurs
Otherwise we may go ahead and create DBIx classes for a half-loaded schema.
2021-01-26 11:50:32 -05:00
d0b3f2dac4 Merge pull request #848 from grahamc/normalize-nixexprinputpath
Normalize nixexpr{input,path} from builds to jobsetevals.
2021-01-25 15:22:01 +01:00
9516b256f1 Normalize nixexpr{input,path} from builds to jobsetevals.
Duplicating this data on every record of the builds table cost
approximately 4G of duplication.

Note that the database migration included took about 4h45m on an
untuned server which uses very slow rotational disks in a RAID5 setup,
with not a lot of RAM. I imagine in production it might take an hour
or two, but not 4. If this should become a chunked migration, I can do
that.

Note: Because of the question about chunked migrations, I have NOT
YET tested this migration thoroughly enough for merge.
2021-01-22 09:10:18 -05:00
53c2fc2216 Merge pull request #847 from grahamc/jobsetevals-evaluation-errors
JobsetEvals: record evaluation errors
2021-01-22 15:08:22 +01:00
bd99052a6f tests: create database with the utf-8 locale
Otherwise tests may fail with wide character errors.
2021-01-21 17:08:05 -05:00
c64c4aac4f jobset page: render error labels per eval 2021-01-21 17:08:02 -05:00
805dd6e7ee Evaluation page: render evaluation errors 2021-01-21 13:11:05 -05:00
086eed5147 hydra-eval-jobs: write evaluation errorMsg to the jobseteval table 2021-01-21 13:10:41 -05:00
fb6b10a86c gitignore: artifacts 2021-01-21 13:10:41 -05:00
d9989b7fa1 Schema: add errorMsg, errorTime to JobsetEvals 2021-01-21 13:10:41 -05:00
6bb876cb35 Merge pull request #846 from grahamc/buildoutputs-index-hash-path
BuildOutputs: index path with HASH
2021-01-18 20:01:44 +01:00
bc4b96d053 BuildOutputs: index path with HASH
Looking at AWS' Performance Insights for a Hydra instance, I found
the hydra-queue-runner's query:

    select id, buildStatus, releaseName, closureSize, size
    from Builds b
    join BuildOutputs o on b.id = o.build
    where
      finished = ?
      and (buildStatus = ? or buildStatus = ?)
      and path = $1

was the slowest query by at least 10x. Running an explain on this
showed why:

hydra=> explain select id, buildStatus, releaseName, closureSize, size
    from Builds b join BuildOutputs o on b.id = o.build where
    finished = 1 and (buildStatus = 0 or buildStatus = 6) and
    path = '/nix/store/s93khs2dncf2cy273mbyr4fb4ns3db20-MIDIVisualizer-5.1';

                                                     QUERY PLAN
    ------------------------------------------------------------------------
     Gather  (cost=1000.43..33718.98 rows=2 width=56)
       Workers Planned: 2
       ->  Nested Loop  (cost=0.43..32718.78 rows=1 width=56)
             ->  Parallel Seq Scan on buildoutputs o  (cost=0.00..32710.32
                                                       rows=1
                                                        width=4)
                   Filter: (path = '/nix/store/s93kh...snip...'::text)
             ->  Index Scan using indexbuildsonjobsetidfinishedid on builds b
                                            (cost=0.43..8.45 rows=1 width=56)
                   Index Cond: ((id = o.build) AND (finished = 1))
                   Filter: ((buildstatus = 0) OR (buildstatus = 6))
    (8 rows)

A paralell sequential scan is definitely better than a sequential scan, but the
cost ranging from 0 to 32710 is not great. Looking at the table, I saw the `path`
column is completely unindex:

    hydra=> \d buildoutputs
                Table "public.buildoutputs"
    Column |  Type   | Collation | Nullable | Default
    --------+---------+-----------+----------+---------
    build  | integer |           | not null |
    name   | text    |           | not null |
    path   | text    |           | not null |
    Indexes:
        "buildoutputs_pkey" PRIMARY KEY, btree (build, name)
    Foreign-key constraints:
        "buildoutputs_build_fkey" FOREIGN KEY (build) REFERENCES builds(id)
            ON DELETE CASCADE

Since we always do exact matches on the path and don't care about ordering,
and since the path column is very high cardinality a `hash` index is a
good candidate. Note that I did test a btree index and it performed
similarly well, but slightly worse.

After creating the index (this took about 10 seconds) on a test database:

    create index IndexBuildOutputsPath on BuildOutputs using hash(path);

We get a *significantly* reduced cost:

    hydra=> explain select id, buildStatus, releaseName, closureSize, size
    hydra->     from Builds b join BuildOutputs o on b.id = o.build where
    hydra->     finished = 1 and (buildStatus = 0 or buildStatus = 6) and
    hydra->     path = '/nix/store/s93khs2dncf2cy273mbyr4fb4ns3db20-MIDIVisualizer-5.1';
                                                QUERY PLAN
    -------------------------------------------------------------------------------------------------------
    Nested Loop  (cost=0.43..41.41 rows=2 width=56)
    ->  Index Scan using buildoutputs_path_hash on buildoutputs o  (cost=0.00..16.05 rows=3 width=4)
            Index Cond: (path = '/nix/store/s93khs2dncf2cy273mbyr4fb4ns3db20-MIDIVisualizer-5.1'::text)
    ->  Index Scan using indexbuildsonjobsetidfinishedid on builds b  (cost=0.43..8.45 rows=1 width=56)
            Index Cond: ((id = o.build) AND (finished = 1))
            Filter: ((buildstatus = 0) OR (buildstatus = 6))
    (6 rows)

For direct comparison, the overall query plan was changed:

    From: Gather      (cost=1000.43..33718.98 rows=2 width=56)
    To:   Nested Loop (cost=   0.43.....41.41 rows=2 width=56)

and the query plan for buildoutputs changed from a maximum cost of
32,710 down to 16.

In practical terms, the query's planning and execution time was reduced:

Before (ms) | Try 1   | Try 2   | Try 3
------------+---------+---------+--------
Planning    |   0.898 |   0.416 |   0.383
Execution   | 138.644 | 172.331 | 375.585

After (ms)  | Try 1   | Try 2   | Try 3
------------+---------+---------+--------
Planning    |   0.298 |   0.290 |   0.296
Execution   | 219.625 |   0.035 |   0.034
2021-01-18 11:28:05 -05:00
be0aa7eb85 Merge pull request #841 from pingiun/github-login
Implement GitHub logins
2021-01-05 14:51:51 +01:00
43d662f63a Don't use enable_github_login option after all
Instead the github_client_id option is used to detect if github logins
should be enabled.
2021-01-04 18:09:49 +01:00
c49ca66689 Die when no email is found 2021-01-04 18:09:05 +01:00
20d8134936 Update src/lib/Hydra/Controller/User.pm
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-01-04 17:48:43 +01:00
19f9d8249f Update src/lib/Hydra/Controller/User.pm
Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2021-01-04 17:48:37 +01:00
20d09518f8 Merge pull request #839 from pingiun/shield-io
Add endpoint to generate a shields.io badge
2021-01-04 14:09:09 +01:00
525a229dac Convert validate-openapi to a Hydra job 2021-01-03 18:47:05 +01:00
ce7b23ae09 Disable broken validate-openapi test 2021-01-03 18:40:08 +01:00