Isolate Temporary Directories and Caches in Cloud Mac CI

When the same cloud Mac runs multiple CI jobs in succession, the hardest failures to reproduce are often caused not by code defects, but by temporary files left behind by the previous job. Stale sockets, unreleased locks, partially written caches, or export directories with reused names can cause the next build of the same commit to produce different results. A dedicated physical node prevents interference from other accounts, but it does not automatically isolate concurrent jobs within the same account. The pipeline must therefore establish explicit filesystem boundaries.

First identify directories that leak state between jobs

Do not assume that an empty workspace means the environment has been fully reset. macOS tools store state outside the workspace, so troubleshooting should cover at least these four categories:

• temporary files, sockets, and partial downloads under TMPDIR;
• Xcode DerivedData, indexes, and build databases;
• the Clang module cache and per-job SwiftPM caches;
• export, log, and test attachment directories created by scripts.

Run env | sort before and after a normal build to record the actual HOME, TMPDIR, and workspace paths in use. Then compare the sizes of the relevant directories with du -sh instead of starting with recursive deletion. If failures occur only under concurrency, look first for fixed filenames, fixed ports, and fixed output paths.

The goal of isolation is not to delete every cache before every run. It is to give each job explicit ownership of its writable directories and clean up only the content that job created.

Create a unique root directory for every job

The job identifier should come from a unique number provided by the CI system. If no reliable identifier is available, use mktemp to generate one. Do not rely on the branch name alone, because the same branch may trigger multiple builds at the same time.

#!/bin/bash
set -euo pipefail

job_id="${CI_JOB_ID:-manual}"
job_root="$(mktemp -d "${TMPDIR:-/tmp}/surevm-ci.${job_id}.XXXXXX")"

export TMPDIR="${job_root}/tmp"
export DERIVED_DATA="${job_root}/DerivedData"
export MODULE_CACHE="${job_root}/ModuleCache"
export RESULT_BUNDLE="${job_root}/Results/Test.xcresult"

mkdir -p "$TMPDIR" "$DERIVED_DATA" "$MODULE_CACHE" "$(dirname "$RESULT_BUNDLE")"

cleanup() {
  local status=$?
  rm -rf "$job_root"
  exit "$status"
}

trap cleanup EXIT INT TERM

A directory created by mktemp will not normally collide with one created by another job. The cleanup function saves the exit status before removing the directory, then exits with the original status so that a cleanup command cannot hide a test failure. Quote path variables to prevent workspace names containing spaces from being split into multiple arguments.

Do not override HOME blindly

Pointing HOME at the job directory may appear to provide complete isolation, but it can also disconnect the current user from Keychain, tool preferences, and completed system initialization. For ordinary compilation, individual package managers can be assigned separate caches. For signing workflows, it is usually better to preserve HOME and isolate only build artifacts and caches that can be moved safely.

Wire Xcode write paths explicitly

Setting TMPDIR alone does not relocate DerivedData. Pass critical paths directly to the build command so the logs clearly show which directories the current job used.

xcodebuild \
  -workspace App.xcworkspace \
  -scheme App \
  -configuration Debug \
  -derivedDataPath "$DERIVED_DATA" \
  -clonedSourcePackagesDirPath "${job_root}/SourcePackages" \
  COMPILER_INDEX_STORE_ENABLE=NO \
  CLANG_MODULE_CACHE_PATH="$MODULE_CACHE" \
  test \
  -resultBundlePath "$RESULT_BUNDLE"

If CI does not need code indexing, disable the index store to reduce unrelated writes. Test results should remain inside the job root, but if attachments must be uploaded after a failure, archive them before cleanup begins. A more reliable sequence is: run the tests, copy diagnostic files to the pipeline artifact area, verify the copy, and then exit.

Keep shared caches read-only and job caches writable

When dependencies genuinely need to be reused, treat a validated shared cache as a seed. Copy or restore it into the job’s own directory at startup, write only to the job-local copy during the build, and update the shared version in a separate step after success. Never allow two xcodebuild processes to write to the same module cache or build database concurrently.

Preserve evidence after abnormal termination

Using trap 'rm -rf ...' EXIT directly removes the failure state as soon as the job exits. In practice, generate a small diagnostic report before cleanup that includes the exit status, available disk space, directory sizes, and any related processes that are still running. Then decide which files to retain. Do not print the complete environment to the logs, because it may contain tokens or signing-related values.

The following can run before cleanup:

{
  echo "exit_status=$status"
  df -h "$job_root"
  du -sh "$job_root"/* 2>/dev/null || true
  find "$job_root" -type s -print
} > "${ARTIFACT_DIR}/job-cleanup.txt"

ARTIFACT_DIR should be outside the job root, with the pipeline responsible for uploading and periodically cleaning it. If the failed state must be retained, compress the diagnostic directory before deleting the original, and redact sensitive information from logs and paths.

Verify isolation with residue audits

After implementing the changes, run the same commit twice in succession. Before the second run starts, confirm that no directory remains for the previous job identifier. For concurrent runs, also verify that the two jobs use completely different TMPDIR, DerivedData, and result bundle paths.

The acceptance checklist can be fixed at five items: every job has a unique root directory; all Xcode paths are specified explicitly; shared caches receive no concurrent writes; exit traps run after both success and failure; and diagnostic artifacts are copied before cleanup. If disk usage continues to grow, use job identifiers to trace tool directories that remain outside the boundary instead of broadening the deletion scope.

On SureVM dedicated physical Macs, this approach is especially effective for long-running build nodes. It preserves the machine-level toolchain while confining each job’s mutable state to directories that can be tracked and removed. The final test is simple: whether the previous job succeeded, failed, or was interrupted, it must not change the input conditions of the next job.

Frequently asked questions