t: increase timeouts for slow commands with high load

We've seen many fails on ofborg, at lot of them ultimately appear to come down to
a timeout being hit, resulting in something like this:

Failure executing slapadd -F /<path>/slap.d -b dc=example -l /<path>/load.ldif.

Hopefully this resolves it for most cases.
I've done some endurance testing and this helps a lot.
some other commands also regularly time-out with high load:

- hydra-init
- hydra-create-user
- nix-store --delete

This should address most issues with tests randomly failing.

Used the following script for endurance testing:

```

import os
import subprocess

run_counter = 0
fail_counter = 0

while True:
    try:
        run_counter += 1
        print(f"Starting run {run_counter}")
        env = os.environ
        env["YATH_JOB_COUNT"] = "20"
        result = subprocess.run(["perl", "t/test.pl"], env=env)
        if (result.returncode != 0):
            fail_counter += 1
        print(f"Finish run {run_counter}, total fail count: {fail_counter}")
    except KeyboardInterrupt:
        print(f"Finished {run_counter} runs with {fail_counter} fails")
        break
```

In case someone else wants to do it on their system :).
Note that YATH_JOB_COUNT may need to be changed loosely based on your
cores.
I only have 4 cores (8 threads), so for others higher numbers might
yield better results in hashing out unstable tests.
This commit is contained in:
Rick van Schijndel
2024-07-31 01:19:40 +02:00
parent d7986226f0
commit 578a3d2292
5 changed files with 14 additions and 14 deletions

View File

@ -28,7 +28,7 @@ subtest "hydra-init upgrades user's password hashes from sha1 to sha1 inside Arg
$janet->setPassword("foobar");
is($alice->password, "8843d7f92416211de9ebb963ff4ce28125932878", "Alices's sha1 is stored in the database");
my ($res, $stdout, $stderr) = captureStdoutStderr(5, ("hydra-init"));
my ($res, $stdout, $stderr) = captureStdoutStderr(30, ("hydra-init"));
if ($res != 0) {
is($stdout, "");
is($stderr, "");
@ -55,7 +55,7 @@ subtest "hydra-init upgrades user's password hashes from sha1 to sha1 inside Arg
};
subtest "Running hydra-init don't break Alice or Janet's passwords" => sub {
my ($res, $stdout, $stderr) = captureStdoutStderr(5, ("hydra-init"));
my ($res, $stdout, $stderr) = captureStdoutStderr(30, ("hydra-init"));
is($res, 0, "hydra-init should exit zero");
my $updatedAlice = $db->resultset('Users')->find({ username => "alice" });