Reproducible Swift macro expansion tests on a cloud Mac

After a macro implementation was merged, every local test passed, but the next build on a cloud Mac generated members in a different order and placed a diagnostic one line off. The final output still compiled, yet the API shown in code review was no longer the same. Swift macros run during compilation, and their output depends on the macro implementation, the SwiftSyntax dependency, and the active toolchain. They therefore cannot be tested like ordinary functions alone.

Define the contract to lock down first

Regression tests should cover three layers. The first compares the complete macro-expanded source to catch changes in members, access levels, attributes, and formatting. The second verifies errors, warnings, and fix-its, especially the token targeted by each diagnostic. The third covers compilation and runtime behavior to confirm that the generated code can be called in practice.

Do not treat “the tests compile” as proof that the macro output has not changed. A macro’s public contract is often the exact shape of the source it generates.

Start with the smallest possible input to establish a baseline, then add empty declarations, generics, nested types, existing members with the same name, and invalid arguments. Each test case should verify only one rule so that a failure can quickly be attributed to either an implementation regression or a toolchain change.

Lock down generated source with expansion tests

A macro package’s test target can import SwiftSyntaxMacrosTestSupport and use assertMacroExpansion to specify both the input and the expected source. The following test assumes that TraceIDMacro adds a string property to a struct:

import SwiftSyntaxMacros
import SwiftSyntaxMacrosTestSupport
import XCTest
@testable import TraceIDMacros

private let macros: [String: Macro.Type] = [
    "TraceID": TraceIDMacro.self
]

final class TraceIDMacroTests: XCTestCase {
    func testAddsTraceID() {
        assertMacroExpansion(
            """
            @TraceID
            struct Job {}
            """,
            expandedSource:
            """
            struct Job {
                let traceID: String
            }
            """,
            macros: macros
        )
    }
}

Keep the expected text complete rather than merely checking whether traceID appears. Partial string assertions can miss duplicate members, incorrect indentation, and unintended additions to the interface. If the macro accepts arguments, add separate cases for default values, explicit values, and invalid expressions.

Separate error paths into dedicated tests

Do not mix invalid input with successful expansion in the same test. Assert the diagnostic text, line and column, and fix-its separately, and name each test after the condition that triggers it. This way, changing one error message does not force the team to reapprove an entire successful expansion snapshot.

Record the toolchain fingerprint

A cloud Mac on BookaMac can run the same pipeline over the long term, but a stable node does not mean the toolchain will never change. Every test run should archive the selected Xcode path, Swift version, host architecture, and dependency lock file together.

set -euo pipefail
export LANG=C
export LC_ALL=C

mkdir -p artifacts
xcode-select -p | tee artifacts/xcode-path.txt
xcrun swift --version | tee artifacts/swift-version.txt
uname -m | tee artifacts/host-arch.txt
cp Package.resolved artifacts/Package.resolved
xcrun swift test 2>&1 | tee artifacts/swift-test.log

Setting LANG and LC_ALL keeps diagnostic text from changing with the language environment of the account running the tests. Using xcrun ensures that tests run with the currently selected Xcode toolchain rather than another swift binary that happens to appear in the shell path. If the repository has no Package.resolved, the script should explicitly skip the copy instead of creating an empty file that falsely suggests the dependencies were locked.

Isolate concurrency and shared state

If a macro implementation reads environment variables, the current directory, the current time, or temporary files, its tests can easily become nondeterministic under parallel execution. A more reliable rule is for the macro to derive its output solely from the syntax tree and explicit arguments. When external data is genuinely required, move the reading logic into a separate type and inject fixed values during tests.

First make the smallest test set pass in serial mode, then enable parallel testing. If failures appear only in parallel, check for fixed filenames, shared cache directories, global mutable variables, and test cleanup order before adding retries. Retries merely turn race conditions into intermittent green runs.

Macro tests should not modify the real project source either. Write temporary input to a separate directory for each test case and clean it up afterward. On failure, retain only sanitized input, expansion output, and logs. This preserves reproducibility without allowing artifacts from one run to affect the next.

Upgrade the toolchain safely

Before upgrading Xcode or SwiftSyntax, run the full test suite with the old toolchain and save the baseline. Then switch toolchains and run the same command. Review differences in this order:

1. Confirm the Xcode path and Swift version actually selected.
2. Check whether Package.resolved changed as expected.
3. Separate formatting changes such as whitespace and indentation from semantic changes to members, types, and access levels.
4. Rerun behavior tests to confirm that the generated interface remains callable.
5. Update the expected expansions only after the review is complete.

If every test case shows similar formatting differences, investigate printer rules or toolchain changes first. If only one edge case changes, the macro implementation itself is the more likely cause. Do not refresh snapshots with a bulk replacement, or a genuine API regression may be buried among large volumes of textual changes.

The final artifacts should include three categories: machine-readable test results, actual-versus-expected diffs for expansion failures, and the toolchain fingerprint. When the next cloud Mac build fails, the investigation no longer has to begin with “the environment may be different.” Instead, the team can identify exactly which contract layer changed and the version in which the change first appeared.

Frequently asked questions