diff --git a/doc/manual/src/SUMMARY.md b/doc/manual/src/SUMMARY.md index 357e795a..5ef8ac90 100644 --- a/doc/manual/src/SUMMARY.md +++ b/doc/manual/src/SUMMARY.md @@ -10,6 +10,7 @@ - [RunCommand](./plugins/RunCommand.md) - [Using the external API](api.md) - [Webhooks](webhooks.md) + - [Webhook Authentication Migration Guide](webhook-migration-guide.md) - [Monitoring Hydra](./monitoring/README.md) ## Developer's Guide diff --git a/doc/manual/src/configuration.md b/doc/manual/src/configuration.md index bd8141a3..491376b3 100644 --- a/doc/manual/src/configuration.md +++ b/doc/manual/src/configuration.md @@ -266,6 +266,40 @@ default role mapping: Note that configuring both the LDAP parameters in the hydra.conf and via the environment variable is a fatal error. +Webhook Authentication +--------------------- + +Hydra supports authenticating webhook requests from GitHub and Gitea to prevent unauthorized job evaluations. +Webhook secrets should be stored in separate files outside the Nix store for security using Config::General's include mechanism. + +In your main `hydra.conf`: +```apache + + Include /var/lib/hydra/secrets/webhook-secrets.conf + +``` + +Then create `/var/lib/hydra/secrets/webhook-secrets.conf` with your actual secrets: +```apache + + secret = your-github-webhook-secret + + + secret = your-gitea-webhook-secret + +``` + +For multiple secrets (useful for rotation or multiple environments), use an array: +```apache + + secret = your-github-webhook-secret-prod + secret = your-github-webhook-secret-staging + +``` + +**Important**: The secrets file should have restricted permissions (e.g., 0600) to prevent unauthorized access. +See the [Webhooks documentation](webhooks.md) for detailed setup instructions. + Embedding Extra HTML -------------------- diff --git a/doc/manual/src/webhook-migration-guide.md b/doc/manual/src/webhook-migration-guide.md new file mode 100644 index 00000000..76864eb3 --- /dev/null +++ b/doc/manual/src/webhook-migration-guide.md @@ -0,0 +1,168 @@ +# Webhook Authentication Migration Guide + +This guide helps Hydra administrators migrate from unauthenticated webhooks to authenticated webhooks to secure their Hydra instances against unauthorized job evaluations. + +## Why Migrate? + +Currently, Hydra's webhook endpoints (`/api/push-github` and `/api/push-gitea`) accept any POST request without authentication. This vulnerability allows: +- Anyone to trigger expensive job evaluations +- Potential denial of service through repeated requests +- Manipulation of build timing and scheduling + +## Step-by-Step Migration for NixOS + +### 1. Create Webhook Configuration + +Create a webhook secrets configuration file with the generated secrets: + +```bash +# Create the secrets configuration file with inline secret generation +cat > /var/lib/hydra/secrets/webhook-secrets.conf < + secret = $(openssl rand -hex 32) + + + secret = $(openssl rand -hex 32) + +EOF + +# Set secure permissions +chmod 0600 /var/lib/hydra/secrets/webhook-secrets.conf +chown hydra:hydra /var/lib/hydra/secrets/webhook-secrets.conf +``` + +**Important**: Save the generated secrets to configure them in GitHub/Gitea later. You can view them with: +```bash +cat /var/lib/hydra/secrets/webhook-secrets.conf +``` + +Then update your NixOS configuration to include the webhook configuration: + +```nix +{ + services.hydra-dev = { + enable = true; + hydraURL = "https://hydra.example.com"; + notificationSender = "hydra@example.com"; + + extraConfig = '' + + Include /var/lib/hydra/secrets/webhook-secrets.conf + + ''; + }; +} +``` + +For multiple secrets (useful for rotation or multiple environments), update your webhook-secrets.conf: + +```apache + + secret = your-github-webhook-secret-prod + secret = your-github-webhook-secret-staging + + + secret = your-gitea-webhook-secret + +``` + +### 2. Deploy Configuration + +Apply the NixOS configuration: + +```bash +nixos-rebuild switch +``` + +This will automatically restart Hydra services with the new configuration. + +### 3. Verify Configuration + +Check Hydra's logs to ensure secrets were loaded successfully: + +```bash +journalctl -u hydra-server | grep -i webhook +``` + +You should not see warnings about webhook authentication not being configured. + +### 4. Update Your Webhooks + +#### GitHub +1. Navigate to your repository settings: `https://github.com///settings/hooks` +2. Edit your existing Hydra webhook +3. In the "Secret" field, paste the content of `/var/lib/hydra/secrets/github-webhook-secret` +4. Click "Update webhook" +5. GitHub will send a ping event to verify the configuration + +#### Gitea +1. Navigate to your repository webhook settings +2. Edit your existing Hydra webhook +3. In the "Secret" field, paste the content of `/var/lib/hydra/secrets/gitea-webhook-secret` +4. Click "Update Webhook" +5. Use the "Test Delivery" button to verify the configuration + +### 5. Test the Configuration + +After updating each webhook: +1. Make a test commit to trigger the webhook +2. Check Hydra's logs for successful authentication +3. Verify the evaluation was triggered in Hydra's web interface + +## Troubleshooting + +### 401 Unauthorized Errors + +If webhooks start failing with 401 errors: +- Verify the secret in the Git forge matches the file content exactly +- Check file permissions: `ls -la /var/lib/hydra/secrets/` +- Ensure no extra whitespace in secret files +- Check Hydra logs for specific error messages + +### Webhook Still Unauthenticated + +If you see warnings about unauthenticated webhooks after configuration: +- Verify the configuration syntax in your NixOS module +- Ensure the NixOS configuration was successfully applied +- Check that the webhook-secrets.conf file exists and is readable by the Hydra user +- Verify the Include path is correct in your hydra.conf +- Check the syntax of your webhook-secrets.conf file + +### Testing Without Git Forge + +You can test webhook authentication using curl: + +```bash +# Read the secret +SECRET=$(cat /var/lib/hydra/secrets/github-webhook-secret) + +# Create test payload +PAYLOAD='{"ref":"refs/heads/main","repository":{"clone_url":"https://github.com/test/repo.git"}}' + +# Calculate signature +SIGNATURE="sha256=$(echo -n "$PAYLOAD" | openssl dgst -sha256 -hmac "$SECRET" | cut -d' ' -f2)" + +# Send authenticated request +curl -X POST https://your-hydra/api/push-github \ + -H "Content-Type: application/json" \ + -H "X-Hub-Signature-256: $SIGNATURE" \ + -d "$PAYLOAD" +``` + +For Gitea (no prefix in signature): +```bash +# Read the secret +SECRET=$(cat /var/lib/hydra/secrets/gitea-webhook-secret) + +# Create test payload +PAYLOAD='{"ref":"refs/heads/main","repository":{"clone_url":"https://gitea.example.com/test/repo.git"}}' + +# Calculate signature +SIGNATURE=$(echo -n "$PAYLOAD" | openssl dgst -sha256 -hmac "$SECRET" | cut -d' ' -f2) + +# Send authenticated request +curl -X POST https://your-hydra/api/push-gitea \ + -H "Content-Type: application/json" \ + -H "X-Gitea-Signature: $SIGNATURE" \ + -d "$PAYLOAD" +``` diff --git a/doc/manual/src/webhooks.md b/doc/manual/src/webhooks.md index 674e1064..7a211788 100644 --- a/doc/manual/src/webhooks.md +++ b/doc/manual/src/webhooks.md @@ -3,6 +3,58 @@ Hydra can be notified by github or gitea with webhooks to trigger a new evaluation when a jobset has a github repo in its input. +## Webhook Authentication + +Hydra supports webhook signature verification for both GitHub and Gitea using HMAC-SHA256. This ensures that webhook +requests are coming from your configured Git forge and haven't been tampered with. + +### Configuring Webhook Authentication + +1. **Create webhook configuration**: Generate and store webhook secrets securely: + ```bash + # Create directory and generate secrets in one step + mkdir -p /var/lib/hydra/secrets + cat > /var/lib/hydra/secrets/webhook-secrets.conf < + secret = $(openssl rand -hex 32) + + + secret = $(openssl rand -hex 32) + + EOF + + # Set secure permissions + chmod 0600 /var/lib/hydra/secrets/webhook-secrets.conf + chown hydra:hydra /var/lib/hydra/secrets/webhook-secrets.conf + ``` + +2. **Configure Hydra**: Add the following to your `hydra.conf`: + ```apache + + Include /var/lib/hydra/secrets/webhook-secrets.conf + + ``` + +3. **Configure your Git forge**: View the generated secrets and configure them in GitHub/Gitea: + ```bash + grep "secret =" /var/lib/hydra/secrets/webhook-secrets.conf + ``` + +### Multiple Secrets Support + +Hydra supports configuring multiple secrets for each platform, which is useful for: +- Zero-downtime secret rotation +- Supporting multiple environments (production/staging) +- Gradual migration of webhooks + +To configure multiple secrets, use array syntax: +```apache + + secret = current-webhook-secret + secret = previous-webhook-secret + +``` + ## GitHub To set up a webhook for a GitHub repository go to `https://github.com///settings` @@ -10,11 +62,16 @@ and in the `Webhooks` tab click on `Add webhook`. - In `Payload URL` fill in `https:///api/push-github`. - In `Content type` switch to `application/json`. -- The `Secret` field can stay empty. +- In the `Secret` field, enter the content of your GitHub webhook secret file (if authentication is configured). - For `Which events would you like to trigger this webhook?` keep the default option for events on `Just the push event.`. Then add the hook with `Add webhook`. +### Verifying GitHub Webhook Security + +After configuration, GitHub will send webhook requests with an `X-Hub-Signature-256` header containing the HMAC-SHA256 +signature of the request body. Hydra will verify this signature matches the configured secret. + ## Gitea To set up a webhook for a Gitea repository go to the settings of the repository in your Gitea instance @@ -22,6 +79,23 @@ and in the `Webhooks` tab click on `Add Webhook` and choose `Gitea` in the drop - In `Target URL` fill in `https:///api/push-gitea`. - Keep HTTP method `POST`, POST Content Type `application/json` and Trigger On `Push Events`. +- In the `Secret` field, enter the content of your Gitea webhook secret file (if authentication is configured). - Change the branch filter to match the git branch hydra builds. Then add the hook with `Add webhook`. + +### Verifying Gitea Webhook Security + +After configuration, Gitea will send webhook requests with an `X-Gitea-Signature` header containing the HMAC-SHA256 +signature of the request body. Hydra will verify this signature matches the configured secret. + +## Troubleshooting + +If you receive 401 Unauthorized errors: +- Verify the webhook secret in your Git forge matches the content of the secret file exactly +- Check that the secret file has proper permissions (should be 0600) +- Look at Hydra's logs for specific error messages +- Ensure the correct signature header is being sent by your Git forge + +If you see warnings about webhook authentication not being configured: +- Configure webhook authentication as described above to secure your endpoints diff --git a/src/lib/Hydra/Controller/API.pm b/src/lib/Hydra/Controller/API.pm index 9f8b7cba..1c1f78b6 100644 --- a/src/lib/Hydra/Controller/API.pm +++ b/src/lib/Hydra/Controller/API.pm @@ -12,6 +12,8 @@ use DateTime; use Digest::SHA qw(sha256_hex); use Text::Diff; use IPC::Run qw(run); +use Digest::SHA qw(hmac_sha256_hex); +use String::Compare::ConstantTime qw(equals); sub api : Chained('/') PathPart('api') CaptureArgs(0) { @@ -274,13 +276,84 @@ sub push : Chained('api') PathPart('push') Args(0) { ); } +sub verifyWebhookSignature { + my ($c, $platform, $header_name, $signature_prefix) = @_; + + # Get secrets from config + my $webhook_config = $c->config->{webhooks} // {}; + my $platform_config = $webhook_config->{$platform} // {}; + my $secrets = $platform_config->{secret}; + + # Normalize to array + $secrets = [] unless defined $secrets; + $secrets = [$secrets] unless ref($secrets) eq 'ARRAY'; + + # Trim whitespace from secrets + my @secrets = grep { defined && length } map { s/^\s+|\s+$//gr } @$secrets; + + if (@secrets) { + my $signature = $c->request->header($header_name); + + if (!$signature) { + $c->log->warn("Webhook authentication failed for $platform: Missing signature from IP " . $c->request->address); + $c->response->status(401); + $c->stash->{json} = { error => "Missing webhook signature" }; + $c->forward('View::JSON'); + return 0; + } + + # Get the raw body content from the buffered PSGI input + # For JSON requests, Catalyst will have already read and buffered the body + my $input = $c->request->env->{'psgi.input'}; + $input->seek(0, 0); + local $/; + my $payload = <$input>; + $input->seek(0, 0); # Reset for any other consumers + + unless (defined $payload && length $payload) { + $c->log->warn("Webhook authentication failed for $platform: Empty request body from IP " . $c->request->address); + $c->response->status(400); + $c->stash->{json} = { error => "Empty request body" }; + $c->forward('View::JSON'); + return 0; + } + + my $valid = 0; + for my $secret (@secrets) { + my $expected = $signature_prefix . hmac_sha256_hex($payload, $secret); + if (equals($signature, $expected)) { + $valid = 1; + last; + } + } + + if (!$valid) { + $c->log->warn("Webhook authentication failed for $platform: Invalid signature from IP " . $c->request->address); + $c->response->status(401); + $c->stash->{json} = { error => "Invalid webhook signature" }; + $c->forward('View::JSON'); + return 0; + } + + return 1; + } else { + $c->log->warn("Webhook authentication failed for $platform: Unable to validate signature from IP " . $c->request->address . " because no secrets are configured"); + $c->response->status(401); + $c->stash->{json} = { error => "Invalid webhook signature" }; + $c->forward('View::JSON'); + return 0; + } +} + sub push_github : Chained('api') PathPart('push-github') Args(0) { my ($self, $c) = @_; $c->{stash}->{json}->{jobsetsTriggered} = []; + return unless verifyWebhookSignature($c, 'github', 'X-Hub-Signature-256', 'sha256='); + my $in = $c->request->{data}; - my $owner = $in->{repository}->{owner}->{name} or die; + my $owner = ($in->{repository}->{owner}->{name} // $in->{repository}->{owner}->{login}) or die; my $repo = $in->{repository}->{name} or die; print STDERR "got push from GitHub repository $owner/$repo\n"; @@ -297,6 +370,9 @@ sub push_gitea : Chained('api') PathPart('push-gitea') Args(0) { $c->{stash}->{json}->{jobsetsTriggered} = []; + # Note: Gitea doesn't use sha256= prefix + return unless verifyWebhookSignature($c, 'gitea', 'X-Gitea-Signature', ''); + my $in = $c->request->{data}; my $url = $in->{repository}->{clone_url} or die; $url =~ s/.git$//; diff --git a/t/Hydra/Controller/API/checks.t b/t/Hydra/Controller/API/checks.t index e4c72ff2..81223e34 100644 --- a/t/Hydra/Controller/API/checks.t +++ b/t/Hydra/Controller/API/checks.t @@ -6,6 +6,7 @@ use Catalyst::Test (); use HTTP::Request; use HTTP::Request::Common; use JSON::MaybeXS qw(decode_json encode_json); +use Digest::SHA qw(hmac_sha256_hex); sub is_json { my ($response, $message) = @_; @@ -21,7 +22,13 @@ sub is_json { return $data; } -my $ctx = test_context(); +my $ctx = test_context(hydra_config => qq| + + + secret = test + + +|); Catalyst::Test->import('Hydra'); # Create a user to log in to @@ -188,16 +195,20 @@ subtest "/api/push-github" => sub { my $jobsetinput = $jobset->jobsetinputs->create({name => "src", type => "git"}); $jobsetinput->jobsetinputalts->create({altnr => 0, value => "https://github.com/OWNER/LEGACY-REPO.git"}); + my $payload = encode_json({ + repository => { + owner => { + name => "OWNER", + }, + name => "LEGACY-REPO", + } + }); + my $signature = "sha256=" . hmac_sha256_hex($payload, 'test'); + my $req = POST '/api/push-github', "Content-Type" => "application/json", - "Content" => encode_json({ - repository => { - owner => { - name => "OWNER", - }, - name => "LEGACY-REPO", - } - }); + "X-Hub-Signature-256" => $signature, + "Content" => $payload; my $response = request($req); ok($response->is_success, "The API enpdoint for triggering jobsets returns 200."); @@ -214,16 +225,20 @@ subtest "/api/push-github" => sub { emailoverride => "" }); + my $payload = encode_json({ + repository => { + owner => { + name => "OWNER", + }, + name => "FLAKE-REPO", + } + }); + my $signature = "sha256=" . hmac_sha256_hex($payload, 'test'); + my $req = POST '/api/push-github', "Content-Type" => "application/json", - "Content" => encode_json({ - repository => { - owner => { - name => "OWNER", - }, - name => "FLAKE-REPO", - } - }); + "X-Hub-Signature-256" => $signature, + "Content" => $payload; my $response = request($req); ok($response->is_success, "The API enpdoint for triggering jobsets returns 200."); diff --git a/t/Hydra/Controller/API/webhooks.t b/t/Hydra/Controller/API/webhooks.t new file mode 100644 index 00000000..eed6dfc0 --- /dev/null +++ b/t/Hydra/Controller/API/webhooks.t @@ -0,0 +1,209 @@ +use strict; +use warnings; +use Setup; +use Test2::V0; +use Test2::Tools::Subtest qw(subtest_streamed); +use HTTP::Request; +use HTTP::Request::Common; +use JSON::MaybeXS qw(decode_json encode_json); +use Digest::SHA qw(hmac_sha256_hex); + +# Create webhook configuration +my $github_secret = "github-test-secret-12345"; +my $github_secret_alt = "github-alternative-secret"; +my $gitea_secret = "gitea-test-secret-abcdef"; + +# Create a temporary directory first to get the path +use File::Temp; +my $tmpdir = File::Temp->newdir(CLEANUP => 0); +my $tmpdir_path = $tmpdir->dirname; + +# Write webhook secrets configuration before creating test context +mkdir "$tmpdir_path/hydra-data"; + +# Create webhook secrets configuration file +my $webhook_config = qq| + + secret = $github_secret + secret = $github_secret_alt + + + secret = $gitea_secret + +|; +write_file("$tmpdir_path/hydra-data/webhook-secrets.conf", $webhook_config); +chmod 0600, "$tmpdir_path/hydra-data/webhook-secrets.conf"; + +# Create test context with webhook configuration using include +my $ctx = test_context( + tmpdir => $tmpdir, + hydra_config => qq| + + Include $tmpdir_path/hydra-data/webhook-secrets.conf + +| +); + +# Import Catalyst::Test after test context is set up +require Catalyst::Test; +Catalyst::Test->import('Hydra'); + +# Create a project and jobset for testing +my $user = $ctx->db()->resultset('Users')->create({ + username => "webhook-test", + emailaddress => 'webhook-test@example.org', + password => '' +}); + +my $project = $ctx->db()->resultset('Projects')->create({ + name => "webhook-test", + displayname => "webhook-test", + owner => $user->username +}); + +my $jobset = $project->jobsets->create({ + name => "test-jobset", + nixexprinput => "src", + nixexprpath => "default.nix", + emailoverride => "" +}); + +my $jobsetinput = $jobset->jobsetinputs->create({name => "src", type => "git"}); +$jobsetinput->jobsetinputalts->create({altnr => 0, value => "https://github.com/owner/repo.git"}); + +# Create another jobset for Gitea +my $jobset_gitea = $project->jobsets->create({ + name => "test-jobset-gitea", + nixexprinput => "src", + nixexprpath => "default.nix", + emailoverride => "" +}); + +my $jobsetinput_gitea = $jobset_gitea->jobsetinputs->create({name => "src", type => "git"}); +$jobsetinput_gitea->jobsetinputalts->create({altnr => 0, value => "https://gitea.example.com/owner/repo.git"}); + +subtest "GitHub webhook authentication" => sub { + my $payload = encode_json({ + repository => { + owner => { name => "owner" }, + name => "repo" + } + }); + + subtest "without authentication - properly rejects" => sub { + my $req = POST '/api/push-github', + "Content-Type" => "application/json", + "Content" => $payload; + + my $response = request($req); + is($response->code, 401, "Unauthenticated request is rejected"); + + my $data = decode_json($response->content); + is($data->{error}, "Missing webhook signature", "Proper error message for missing signature"); + }; + + subtest "with valid signature" => sub { + my $signature = "sha256=" . hmac_sha256_hex($payload, $github_secret); + + my $req = POST '/api/push-github', + "Content-Type" => "application/json", + "X-Hub-Signature-256" => $signature, + "Content" => $payload; + + my $response = request($req); + is($response->code, 200, "Valid signature is accepted"); + + if ($response->code != 200) { + diag("Error response: " . $response->content); + } + + my $data = decode_json($response->content); + is($data->{jobsetsTriggered}, ["webhook-test:test-jobset"], "Jobset was triggered with valid authentication"); + }; + + subtest "with invalid signature" => sub { + my $signature = "sha256=" . hmac_sha256_hex($payload, "wrong-secret"); + + my $req = POST '/api/push-github', + "Content-Type" => "application/json", + "X-Hub-Signature-256" => $signature, + "Content" => $payload; + + my $response = request($req); + is($response->code, 401, "Invalid signature is rejected"); + + my $data = decode_json($response->content); + is($data->{error}, "Invalid webhook signature", "Proper error message for invalid signature"); + }; + + subtest "with second valid secret (multiple secrets configured)" => sub { + my $signature = "sha256=" . hmac_sha256_hex($payload, $github_secret_alt); + + my $req = POST '/api/push-github', + "Content-Type" => "application/json", + "X-Hub-Signature-256" => $signature, + "Content" => $payload; + + my $response = request($req); + is($response->code, 200, "Second valid secret is accepted"); + }; +}; + +subtest "Gitea webhook authentication" => sub { + my $payload = encode_json({ + repository => { + owner => { username => "owner" }, + name => "repo", + clone_url => "https://gitea.example.com/owner/repo.git" + } + }); + + subtest "without authentication - properly rejects" => sub { + my $req = POST '/api/push-gitea', + "Content-Type" => "application/json", + "Content" => $payload; + + my $response = request($req); + is($response->code, 401, "Unauthenticated request is rejected"); + + my $data = decode_json($response->content); + is($data->{error}, "Missing webhook signature", "Proper error message for missing signature"); + }; + + subtest "with valid signature" => sub { + # Note: Gitea doesn't use sha256= prefix + my $signature = hmac_sha256_hex($payload, $gitea_secret); + + my $req = POST '/api/push-gitea', + "Content-Type" => "application/json", + "X-Gitea-Signature" => $signature, + "Content" => $payload; + + my $response = request($req); + is($response->code, 200, "Valid signature is accepted"); + + if ($response->code != 200) { + diag("Error response: " . $response->content); + } + + my $data = decode_json($response->content); + is($data->{jobsetsTriggered}, ["webhook-test:test-jobset-gitea"], "Jobset was triggered with valid authentication"); + }; + + subtest "with invalid signature" => sub { + my $signature = hmac_sha256_hex($payload, "wrong-secret"); + + my $req = POST '/api/push-gitea', + "Content-Type" => "application/json", + "X-Gitea-Signature" => $signature, + "Content" => $payload; + + my $response = request($req); + is($response->code, 401, "Invalid signature is rejected"); + + my $data = decode_json($response->content); + is($data->{error}, "Invalid webhook signature", "Proper error message for invalid signature"); + }; +}; + +done_testing;