ahuston-0/hydra
1
0
Fork 0
Code Issues Pull Requests 1 Packages Projects Releases Wiki Activity

queue-runner: limit parallelism of CPU intensive operations

Browse Source 
My current theory is that running more parallel xz than available CPU
cores is reducing our overall throughput by requiring more scheduling
overhead and more cache thrashing.
This commit is contained in:
Pierre Bourdon
2024-04-11 15:03:23 +02:00
committed by John Ericson
parent 5a9985f96c
commit 1e2d3211d9
4 changed files with 28 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
src/hydra-queue-runner/hydra-queue-runner.cc
View File

@@ -85,6 +85,7 @@ State::State(std::optional<std::string> metricsAddrOpt)
: config(std::make_unique<HydraConfig>())
, maxUnsupportedTime(config->getIntOption("max_unsupported_time", 0))
, dbPool(config->getIntOption("max_db_connections", 128))
, localWorkThrottler(config->getIntOption("max_local_worker_threads", std::min(maxSupportedLocalWorkers, std::max(4u, std::thread::hardware_concurrency()) - 2)))
, maxOutputSize(config->getIntOption("max_output_size", 2ULL << 30))
, maxLogSize(config->getIntOption("max_log_size", 64ULL << 20))
, uploadLogsToBinaryCache(config->getBoolOption("upload_logs_to_binary_cache", false))