Atomic Job Locks for Duplicate-Proof Cloud Mac CI Builds

When the same commit is triggered simultaneously by a Webhook retry, a manual rerun, and a scheduled job, two Xcode builds can enter the same workspace within seconds of each other. They may not fail immediately. More often, both processes modify DerivedData and overwrite the archive directory until they produce an intermittent error that is difficult to reproduce. On a cloud Mac that stays online continuously, preventing these duplicate triggers is often more important than handling an isolated command failure.

First identify the resource that actually requires mutual exclusion

Do not start by placing a global lock on the entire node. The lock scope should be determined by the shared write target, not by the repository name. Two jobs should use the same lock key whenever they share any of the following resources:

• generated files or dependency directories in the same workspace;
• the same DerivedData path;
• the same archive, export, or upload directory;
• a simulator device set that can only be accessed serially;
• the same final deployment target.

Conversely, test shards with separate checkouts, DerivedData directories, and simulator device sets can run in parallel under different lock keys. A good lock key combines the repository, Scheme, and action, such as client-release-archive. Do not simply include the branch name, because two branches may still compete for the same archive target.

A lock protects shared side effects, not the CPU. Bypassing a necessary lock to increase concurrency usually just turns waiting time into cleanup time.

Use mkdir as an atomic contention point

The mkdir command included with macOS provides a simple and reliable contention point: when multiple processes try to create the same path at the same time, only one succeeds. Unlike checking whether a file exists before writing a PID, this approach has no race window between the check and the write.

The following script creates a random owner token for the lock and updates a heartbeat every 15 seconds. On exit, only the process whose token still matches may remove the lock, preventing an old job from accidentally deleting a newer job’s lock.

#!/bin/bash
set -euo pipefail

LOCK_ROOT="${HOME}/Library/Caches/ci-locks"
LOCK_KEY="client-release-archive"
LOCK_DIR="${LOCK_ROOT}/${LOCK_KEY}.lock"
TOKEN="$(uuidgen)"

mkdir -p "$LOCK_ROOT"

if ! mkdir "$LOCK_DIR" 2>/dev/null; then
  echo "build lock is already held"
  test -f "$LOCK_DIR/owner" && cat "$LOCK_DIR/owner"
  exit 75
fi

printf 'token=%s
pid=%s
started=%s
' \
  "$TOKEN" "$$" "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" > "$LOCK_DIR/owner"
touch "$LOCK_DIR/heartbeat"

heartbeat() {
  while true; do
    touch "$LOCK_DIR/heartbeat"
    sleep 15
  done
}

heartbeat &
HEARTBEAT_PID=$!

cleanup() {
  kill "$HEARTBEAT_PID" 2>/dev/null || true
  if grep -Fq "token=${TOKEN}" "$LOCK_DIR/owner" 2>/dev/null; then
    rm -rf "$LOCK_DIR"
  fi
}

trap cleanup EXIT
trap 'exit 130' INT TERM HUP

xcodebuild \
  -workspace Client.xcworkspace \
  -scheme Client \
  -configuration Release \
  -derivedDataPath "$PWD/.ci/DerivedData" \
  archive

Exit code 75 clearly indicates that the job is temporarily unavailable for execution. The queue layer may retry it after a delay, but it should not replay the job indefinitely within the same second. Every path must be quoted, and the lock root should be located somewhere writable by the current execution account and safe from repository cleanup commands.

Do not use PID alone to identify stale locks

A PID file is useful for diagnostics, but it cannot prove ownership by itself. macOS may reuse a PID, and after a node restart, the number in an old file may belong to an entirely unrelated process. A cleanup process should therefore check at least three conditions together:

1. Whether the heartbeat modification time exceeds the pipeline’s maximum execution duration;
2. Whether the queue has marked the job as completed or timed out;
3. Whether related build processes are still writing to the protected directory.

A stale heartbeat only means that the situation needs investigation; it does not mean the lock can safely be taken over. If a build is paused because disk I/O is blocked, the process may still hold files. Forcing a second build to start at that point will increase the scope of the damage. A safer design delegates cleanup to an independent recovery job instead of allowing a job waiting for the lock to delete its predecessor’s state.

Add observable fields to the lock

The owner file can also record the commit hash, job number, Scheme, and working directory, but it should not contain access tokens, signing material, or other sensitive values. Failure logs should include at least the lock key, start time, and job number so engineers can distinguish normal queueing from an abnormal exit in the previous run.

Treat waiting, timeouts, and cancellation separately

Mutual exclusion does not mean every later job must fail. Real-world queues generally use one of three strategies:

• release archives: keep the first job and make later jobs wait;
• branch validation: cancel the older job for the same branch and keep the latest commit;
• scheduled builds: skip the current run if a job with the same key is already running.

Whichever strategy is used, waiting must have an upper limit. A wait timeout should terminate only the waiting job and must not remove the holder’s lock. When canceling the holder, first send a normal termination signal so that trap can complete cleanup. Only if the process cannot exit should cleanup move to a manual or independent recovery workflow.

Avoid performing unrelated work while holding the lock. Code checkout, read-only validation, and parameter preparation can happen before lock acquisition. Acquire the lock only when the job reaches the stage that actually modifies shared directories. This shortens the critical section and reduces queue congestion.

Run failure drills before production rollout

First, trigger the script twice in succession in a non-release job and confirm that only one process enters xcodebuild. Then separately simulate normal completion, a command failure, and receipt of a termination signal, checking each time whether the lock directory is removed. Finally, leave an old lock in place manually and verify that the waiting job only reports the contention without attempting cleanup on its own.

Use the following acceptance checklist:

• the lock key maps precisely to the shared resource;
• lock acquisition relies on a single atomic operation;
• the random owner token is verified before unlocking;
• EXIT, INT, TERM, and HUP all have cleanup paths;
• the heartbeat is used only for diagnostics and never triggers lock takeover by itself;
• logs identify the job holding the lock without exposing sensitive data;
• a wait timeout does not affect the running build;
• independent jobs use separate workspaces and output directories.

Once these checks are complete, duplicate triggers become explainable queueing events instead of hidden races that randomly corrupt build state. If more concurrency is needed later, first separate shared directories and deployment targets, then refine the lock keys rather than removing mutual exclusion outright.

Frequently asked questions