t: increase timeouts for slow commands with high load
We've seen many fails on ofborg, at lot of them ultimately appear to come down to a timeout being hit, resulting in something like this: Failure executing slapadd -F /<path>/slap.d -b dc=example -l /<path>/load.ldif. Hopefully this resolves it for most cases. I've done some endurance testing and this helps a lot. some other commands also regularly time-out with high load: - hydra-init - hydra-create-user - nix-store --delete This should address most issues with tests randomly failing. Used the following script for endurance testing: ``` import os import subprocess run_counter = 0 fail_counter = 0 while True: try: run_counter += 1 print(f"Starting run {run_counter}") env = os.environ env["YATH_JOB_COUNT"] = "20" result = subprocess.run(["perl", "t/test.pl"], env=env) if (result.returncode != 0): fail_counter += 1 print(f"Finish run {run_counter}, total fail count: {fail_counter}") except KeyboardInterrupt: print(f"Finished {run_counter} runs with {fail_counter} fails") break ``` In case someone else wants to do it on their system :). Note that YATH_JOB_COUNT may need to be changed loosely based on your cores. I only have 4 cores (8 threads), so for others higher numbers might yield better results in hashing out unstable tests.
This commit is contained in:
@ -92,7 +92,7 @@ sub new {
|
||||
$opts{'before_init'}->($self);
|
||||
}
|
||||
|
||||
expectOkay(5, ("hydra-init"));
|
||||
expectOkay(30, ("hydra-init"));
|
||||
|
||||
return $self;
|
||||
}
|
||||
|
@ -70,7 +70,7 @@ sub add_user {
|
||||
my $email = $opts{'email'} // "$name\@example";
|
||||
my $password = $opts{'password'} // rand_chars();
|
||||
|
||||
my ($res, $stdout, $stderr) = captureStdoutStderr(1, ("slappasswd", "-s", $password));
|
||||
my ($res, $stdout, $stderr) = captureStdoutStderr(5, ("slappasswd", "-s", $password));
|
||||
if ($res) {
|
||||
die "Failed to execute slappasswd ($res): $stderr, $stdout";
|
||||
}
|
||||
@ -178,7 +178,7 @@ sub start {
|
||||
sub validateConfig {
|
||||
my ($self) = @_;
|
||||
|
||||
expectOkay(1, ("slaptest", "-u", "-F", $self->{"_slapd_dir"}));
|
||||
expectOkay(5, ("slaptest", "-u", "-F", $self->{"_slapd_dir"}));
|
||||
}
|
||||
|
||||
sub _spawn {
|
||||
@ -218,7 +218,7 @@ sub load_ldif {
|
||||
|
||||
my $path = "${\$self->{'_tmpdir'}}/load.ldif";
|
||||
write_file($path, $content);
|
||||
expectOkay(1, ("slapadd", "-F", $self->{"_slapd_dir"}, "-b", $suffix, "-l", $path));
|
||||
expectOkay(5, ("slapadd", "-F", $self->{"_slapd_dir"}, "-b", $suffix, "-l", $path));
|
||||
$self->validateConfig();
|
||||
}
|
||||
|
||||
|
Reference in New Issue
Block a user