Managing Process Priority in Cloud Mac CI

When the same cloud Mac runs Xcode builds, tests, log compression, and artifact validation at the same time, the most common problem is not a single process running completely out of control. Instead, several supporting jobs that could finish later compete with the critical build for CPU, disk, and memory bandwidth. The result is often inconsistent compilation times, test timeouts, or a sluggish remote desktop. The first step is therefore not to lower every process priority indiscriminately, but to distinguish the critical path from background work.

Establish a comparable process baseline first

Run a representative build with no other jobs active, then repeat it while compression, validation, and indexing run concurrently. For both runs, record the build duration, failure stage, and the CPU usage, memory usage, state, and nice value of the main processes.

mkdir -p "$HOME/ci-observe"

ps -axo pid,ppid,user,%cpu,%mem,state,nice,etime,command \
  | sort -k4 -nr \
  | head -n 40 \
  > "$HOME/ci-observe/processes.txt"

iostat -w 2 -c 15 > "$HOME/ci-observe/iostat.txt"
vm_stat 2 15 > "$HOME/ci-observe/vm-stat.txt"

Sampling should cover the actual workspace and realistic dependency volume, but do not mix the initial dependency download with stable builds in the same baseline. If the build takes longer while iostat remains consistently busy and the compression process consumes substantial CPU, there is a reasonable basis for adjusting supporting jobs. If the main issue is memory pressure or excessive test concurrency, changing priorities alone will usually not help.

Priority is a scheduling hint, not a resource limit. It can influence which process runs first, but it cannot replace concurrency control or guarantee that a job receives a fixed share of CPU time.

Classify jobs by their role in the critical path

CI processes can be divided into three categories. The first includes compilation, linking, testing, and artifact export. These jobs determine whether the pipeline completes and should retain normal priority. The second includes log compression, checksum generation, and cleanup of old artifacts. These jobs must finish, but they usually do not need to run ahead of the build. The third includes editor indexing, interactive analysis, and temporary diagnostics, which should run only when needed.

Do not create blanket rules based on tool names

The same tool can belong to different paths. For example, tar is critical when it unpacks a required dependency before a build, but it is a background task when it compresses logs afterward. The relevant questions are whether the job blocks the next step, whether its failure must stop the pipeline, and whether the current task uses its output immediately.

Define these classifications in the pipeline configuration instead of relying on an operator to apply them manually. This makes it clear during script reviews which commands have been deprioritized and prevents new jobs from inheriting an unsuitable policy by default.

Wrap background jobs with nice and taskpolicy

nice adjusts a process's scheduling priority value. A positive value means the process is more willing to yield execution time. A regular user can lower the priority of a process they start, but cannot arbitrarily raise it to a higher priority. On macOS, taskpolicy -b places a command under a background policy whose effects are not limited to CPU scheduling. It should therefore be used only for tasks that have been confirmed not to block the critical path.

The following script runs checksum generation in the background at a lower priority, keeps the Xcode build under the normal policy, and preserves the exit status of each job separately:

#!/bin/zsh

set -u
mkdir -p Artifacts Logs

taskpolicy -b nice -n 10 /bin/sh -c \
  'find Artifacts -type f -print0 | xargs -0 shasum -a 256 > Logs/checksums.txt' &
audit_pid=$!

xcodebuild \
  -workspace App.xcworkspace \
  -scheme App \
  -configuration Release \
  -destination 'generic/platform=iOS' \
  build > Logs/xcodebuild.log 2>&1
build_rc=$?

wait "$audit_pid"
audit_rc=$?

if (( build_rc != 0 )); then
  exit "$build_rc"
fi

exit "$audit_rc"

Do not apply taskpolicy -b to the entire CI Runner. Doing so also deprioritizes the compiler, test processes, and child processes. The whole pipeline becomes slower without making the responsible job easy to identify. Keep the wrapper as close as possible to the individual background command.

Use QoS correctly in application code

If supporting work is performed by an in-house Swift tool, assign queue QoS in the code rather than trying to infer the process's role after launch. Work initiated by the user whose result they are waiting for can use .userInitiated. Non-urgent work such as organizing logs or inventorying caches can use .utility. Reserve .background for tasks the user is not waiting for and whose delayed completion will not affect the current result.

let queue = DispatchQueue(
    label: "com.surevm.ci.artifact-audit",
    qos: .utility
)

queue.async {
    auditArtifacts()
}

Do not set every queue to .userInteractive in an attempt to make everything faster. Excessively high QoS causes supporting work to compete with critical tasks for scheduler resources. It can also produce priority inversion, where a high-priority task waits for a lock held by a low-priority queue. Shared locks, serial queues, and synchronous calls should all be included in the review.

Verify the results and define rollback conditions

After making the adjustment, repeat the build at least three times with identical inputs and compare the median duration, failure point, and resource samples. A single faster run does not prove that the policy is effective. Also confirm that background jobs such as log compression and checksum generation finish completely. A successful critical build must not be used to conceal a failed supporting job.

During validation, confirm each of the following:

• Compilation, linking, and test processes do not inherit the background policy.
• A failed background job returns a nonzero exit code.
• The number of pipelines running concurrently on the Runner does not exceed the node's capacity.
• The same code, dependency lockfile, and Xcode version are used before and after the change.
• Remote desktop responsiveness is not the only criterion; process and I/O samples must also be retained.
• After a node restart, the priority wrappers are still applied automatically by the pipeline script.

If deprioritized supporting jobs accumulate into the next build, the problem has shifted from resource contention to insufficient throughput. At that point, reduce the number of background jobs, narrow their processing scope, or move them to a separate post-build stage instead of continuing to increase the nice value. Check the SureVM control panel for the specific node configurations currently available. Regardless of the selected configuration, identifying the critical path before changing scheduling policy is usually easier to validate and roll back than making a global adjustment.

Frequently asked questions