-
Notifications
You must be signed in to change notification settings - Fork 24
[do not merge] Use SnapshotPreviews from Cameroncooke/snapshots ci branch
#780
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
cameroncooke
wants to merge
13
commits into
main
Choose a base branch
from
cameroncooke/snapshots-ci
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+432
−8
Open
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
594b128
Try out snapshots upload flow
NicoHinderling c140e23
add a ls step
NicoHinderling 811838a
dont continue on error
NicoHinderling dc36282
update
NicoHinderling 0fea4c1
try uploading via fastlane + clean up log output
NicoHinderling 709f559
add xcpretty and pre-boot the simulator
NicoHinderling 2abfe82
try new script approach
NicoHinderling 5739478
tweaks
NicoHinderling dffa253
tweak
NicoHinderling 458d330
Use SnapshotPreviews from `cameroncooke/snapshot-ci` branch
cameroncooke e52fefa
Upload workflow to support new SnapshotPreviews image exports
cameroncooke f620d22
Enable new workflow
cameroncooke f5c416a
fix: use absolute path for snapshot export dir to avoid test bootstra…
cameroncooke File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| scheme: HackerNews | ||
| test_target: HackerNewsTests/HackerNewsSnapshotTest | ||
| bundle_id: com.emergetools.hackernews | ||
| max_retries: 1 | ||
|
|
||
| destinations: | ||
| - name: iPhone 17 Pro Max | ||
| sdk: iphonesimulator | ||
| xcode_flags: TARGETED_DEVICE_FAMILY=1 | ||
| enabled: true | ||
| - name: iPad Air 11-inch (M3) | ||
| sdk: iphonesimulator | ||
| xcode_flags: TARGETED_DEVICE_FAMILY=1,2 | ||
| enabled: false |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,270 @@ | ||
| #!/usr/bin/env bash | ||
| # | ||
| # TODO: Potentially put this behind a CLI | ||
| # | ||
| # | ||
| # Benefits pulled from Emerge's internal iOS snapshot infrastructure: | ||
| # | ||
| # - Crash report collection: after any xcodebuild failure, captures .ips files from | ||
| # ~/Library/Logs/DiagnosticReports scoped to the test run window | ||
| # - Retry on crash: distinguishes crash failures from legitimate test failures and | ||
| # retries automatically (configurable via max_retries) | ||
| # - Zero-size image detection: scans extracted PNGs with sips after xcresulttool | ||
| # export to catch renders that "succeed" but produce empty/corrupt images | ||
| # - Multi-device orchestration: runs against multiple simulators (iPhone, iPad, OS | ||
| # versions) and collects results per-device before aggregating | ||
| # - Permission reset before run: calls xcrun simctl privacy reset all before each | ||
| # run to prevent permission dialogs from blocking renders | ||
| # - Config file support: YAML config for scheme, test target, bundle ID, and | ||
| # destinations — no bash knowledge required for customers | ||
| # - Simulator lifecycle management: boots simulators, waits for ready state, and | ||
| # tears down cleanly between runs | ||
| # - Structured summary manifest: emits a JSON manifest with per-device image counts, | ||
| # attempt numbers, and any captured crash report filenames | ||
| # | ||
| # Potential follow-ups (not yet implemented): | ||
| # - Regex-based exclusion filtering: filter extracted PNGs by filename pattern before | ||
| # upload, driven by exclusion rules in the config file | ||
| # - Per-test timing: extract per-test duration from xcresult and include in the | ||
| # manifest so Sentry can surface slow previews | ||
| # - Image metadata enrichment: emit a sidecar JSON per image with device name, OS | ||
| # version, orientation, and color scheme for richer Sentry UI context | ||
| # - Sharding: split the test suite across multiple CI machines by enumerating test | ||
| # method names (via xcodebuild build-for-testing + xctest list), accepting | ||
| # shard/num_shards config params, and merging per-shard image directories | ||
| # afterward; requires the customer's test target to expose one method per preview | ||
| set -euo pipefail | ||
|
|
||
| CONFIG_FILE=".github/emerge-snapshots.yml" | ||
| OUTPUT_DIR="snapshot-images" | ||
| XCRESULT_DIR=".." | ||
|
|
||
| while [[ $# -gt 0 ]]; do | ||
| case "$1" in | ||
| --config) CONFIG_FILE="$2"; shift 2 ;; | ||
| --output-dir) OUTPUT_DIR="$2"; shift 2 ;; | ||
| --xcresult-dir) XCRESULT_DIR="$2"; shift 2 ;; | ||
| --help|-h) echo "Usage: emerge-snapshots [--config PATH] [--output-dir PATH] [--xcresult-dir PATH]"; exit 0 ;; | ||
| *) echo "Unknown argument: $1"; exit 1 ;; | ||
| esac | ||
| done | ||
| [[ -f "$CONFIG_FILE" ]] || { echo "Config not found: $CONFIG_FILE"; exit 1; } | ||
|
|
||
| parse_config() { | ||
| ruby -ryaml -e " | ||
| c = YAML.load_file(ARGV[0]) | ||
| %w[scheme test_target bundle_id].each { |k| abort(\"#{k} required\") unless c[k] } | ||
| puts \"SCHEME=#{c['scheme']}\" | ||
| puts \"TEST_TARGET=#{c['test_target']}\" | ||
| puts \"BUNDLE_ID=#{c['bundle_id']}\" | ||
| puts \"MAX_RETRIES=#{c.fetch('max_retries', 1)}\" | ||
| puts '---' | ||
| (c['destinations'] || []).select { |d| d['enabled'] }.each do |d| | ||
| puts [d['name'], d['sdk'], d['xcode_flags']].join('|') | ||
| end | ||
| " "$1" | ||
| } | ||
|
|
||
| config_output=$(parse_config "$CONFIG_FILE") | ||
| eval "$(echo "$config_output" | awk '/^---$/{exit} {print}')" | ||
| DESTINATIONS=() | ||
| while IFS= read -r line; do | ||
| DESTINATIONS+=("$line") | ||
| done < <(echo "$config_output" | awk '/^---$/{found=1; next} found{print}') | ||
|
|
||
| [[ ${#DESTINATIONS[@]} -gt 0 ]] || { echo "No enabled destinations in config"; exit 1; } | ||
|
|
||
| shopt -s nullglob | ||
|
|
||
| TOTAL_IMAGE_COUNT=0 | ||
| declare -a DEVICE_SUMMARIES=() | ||
| declare -a CAPTURED_CRASH_REPORTS=() | ||
|
|
||
| boot_simulator() { | ||
| local device_name="$1" | ||
| xcrun simctl boot "$device_name" || true | ||
| xcrun simctl bootstatus "$device_name" -b | ||
| CURRENT_UDID=$(xcrun simctl list devices booted --json \ | ||
| | jq -r '.devices | to_entries[] | .value[] | select(.state == "Booted") | .udid' \ | ||
| | head -1) | ||
| } | ||
|
|
||
| reset_simulator_permissions() { | ||
| local udid="$1" | ||
| local bundle_id="$2" | ||
| if [ -z "$udid" ]; then | ||
| echo "Warning: UDID empty, skipping permission reset" | ||
| return 0 | ||
| fi | ||
| xcrun simctl privacy "$udid" reset all "$bundle_id" | ||
| } | ||
|
|
||
| run_tests() { | ||
| local device_name="$1" | ||
| local sdk="$2" | ||
| local xcode_flags="$3" | ||
| local result_path="$4" | ||
|
|
||
| rm -rf "$result_path" | ||
| touch /tmp/snapshot_test_start_marker | ||
| TEST_START_EPOCH=$(date +%s) | ||
|
|
||
| set +e | ||
| set -o pipefail | ||
| xcodebuild test \ | ||
| -scheme "$SCHEME" \ | ||
| -sdk "$sdk" \ | ||
| -destination "platform=iOS Simulator,name=${device_name}" \ | ||
| -only-testing:"$TEST_TARGET" \ | ||
| -resultBundlePath "$result_path" \ | ||
| $xcode_flags \ | ||
| ONLY_ACTIVE_ARCH=YES \ | ||
| SUPPORTS_MACCATALYST=NO \ | ||
| | xcpretty | ||
| local exit_code=$? | ||
| set -e | ||
| return $exit_code | ||
| } | ||
|
|
||
| collect_crash_reports() { | ||
| local dest_dir="$1" | ||
| [ -d ~/Library/Logs/DiagnosticReports ] || return 0 | ||
|
|
||
| local count=0 | ||
| mkdir -p "$dest_dir" | ||
|
|
||
| while IFS= read -r -d '' ips_file; do | ||
| cp "$ips_file" "$dest_dir/" | ||
| CAPTURED_CRASH_REPORTS+=("$(basename "$ips_file")") | ||
| count=$((count + 1)) | ||
| done < <(find ~/Library/Logs/DiagnosticReports -name "*.ips" -newer /tmp/snapshot_test_start_marker -print0 2>/dev/null) | ||
|
|
||
| echo "Crash reports collected: $count" | ||
| return $count | ||
| } | ||
|
|
||
| extract_images() { | ||
| local result_path="$1" | ||
| local dest_dir="$2" | ||
| xcrun xcresulttool export attachments \ | ||
| --path "$result_path" \ | ||
| --output-path "$dest_dir" | ||
| } | ||
|
|
||
| check_zero_size_images() { | ||
| local dir="$1" | ||
| command -v sips &>/dev/null || return 0 | ||
|
|
||
| for f in "$dir"/*.png; do | ||
| local width | ||
| width=$(sips -g pixelWidth "$f" | awk '/pixelWidth/{print $2}') | ||
| if [ -z "$width" ] || [ "$width" -eq 0 ]; then | ||
| echo "Error: Zero-size image detected: $f" | ||
| exit 1 | ||
| fi | ||
| done | ||
| } | ||
|
|
||
| run_device() { | ||
| local device_name="$1" | ||
| local sdk="$2" | ||
| local xcode_flags="$3" | ||
| local label="$4" | ||
|
|
||
| boot_simulator "$device_name" | ||
| reset_simulator_permissions "$CURRENT_UDID" "$BUNDLE_ID" | ||
|
|
||
| local attempt=1 | ||
| local exit_code=0 | ||
| local xcresult="" | ||
|
|
||
| while true; do | ||
| xcresult="$XCRESULT_DIR/SnapshotResults-${label}-attempt${attempt}.xcresult" | ||
| run_tests "$device_name" "$sdk" "$xcode_flags" "$xcresult" || exit_code=$? | ||
|
|
||
| if [ "$exit_code" -eq 0 ]; then | ||
| break | ||
| fi | ||
|
|
||
| local crash_count=0 | ||
| collect_crash_reports "$OUTPUT_DIR/crash-reports" || crash_count=$? | ||
|
|
||
| if [ "$crash_count" -gt 0 ] && [ "$attempt" -le "$MAX_RETRIES" ]; then | ||
| echo "Crash detected (attempt $attempt), retrying..." | ||
| attempt=$((attempt + 1)) | ||
| exit_code=0 | ||
| continue | ||
| else | ||
| exit 1 | ||
| fi | ||
| done | ||
|
|
||
| extract_images "$xcresult" "$XCRESULT_DIR/snapshots-$label" | ||
| check_zero_size_images "$XCRESULT_DIR/snapshots-$label" | ||
|
|
||
| local copied=0 | ||
| for png in "$XCRESULT_DIR/snapshots-$label/"*.png; do | ||
| cp "$png" "$OUTPUT_DIR/" | ||
| copied=$((copied + 1)) | ||
| done | ||
|
|
||
| TOTAL_IMAGE_COUNT=$((TOTAL_IMAGE_COUNT + copied)) | ||
| DEVICE_SUMMARIES+=("${label}|${device_name}|${copied}|${attempt}") | ||
| } | ||
|
|
||
| emit_summary() { | ||
| echo "" | ||
| echo "=== Snapshot Capture Summary ===" | ||
| printf "%-35s %-8s %-9s\n" "Device" "Images" "Attempts" | ||
| printf "%-35s %-8s %-9s\n" "------" "------" "--------" | ||
|
|
||
| local devices_json="[]" | ||
| for entry in ${DEVICE_SUMMARIES[@]+"${DEVICE_SUMMARIES[@]}"}; do | ||
| IFS='|' read -r label name count attempts <<< "$entry" | ||
| printf "%-35s %-8s %-9s\n" "$name" "$count" "$attempts" | ||
| if command -v jq &>/dev/null; then | ||
| devices_json=$(echo "$devices_json" | jq \ | ||
| --arg label "$label" \ | ||
| --arg name "$name" \ | ||
| --argjson count "$count" \ | ||
| --argjson attempts "$attempts" \ | ||
| '. + [{"label": $label, "name": $name, "image_count": $count, "attempts": $attempts}]') | ||
| fi | ||
| done | ||
|
|
||
| echo "--------------------------------" | ||
| echo "Total images: $TOTAL_IMAGE_COUNT" | ||
|
|
||
| if ! command -v jq &>/dev/null; then | ||
| echo "Warning: jq not available, skipping manifest write" | ||
| return 0 | ||
| fi | ||
|
|
||
| local crash_json="[]" | ||
| for cr in ${CAPTURED_CRASH_REPORTS[@]+"${CAPTURED_CRASH_REPORTS[@]}"}; do | ||
| crash_json=$(echo "$crash_json" | jq --arg cr "$cr" '. + [$cr]') | ||
| done | ||
|
|
||
| jq --null-input \ | ||
| --arg ts "$(date -u +"%Y-%m-%dT%H:%M:%SZ")" \ | ||
| --argjson devices "$devices_json" \ | ||
| --argjson total "$TOTAL_IMAGE_COUNT" \ | ||
| --argjson crashes "$crash_json" \ | ||
| '{ | ||
| generated_at: $ts, | ||
| devices: $devices, | ||
| total_image_count: $total, | ||
| crash_reports_captured: $crashes | ||
| }' > "$OUTPUT_DIR/snapshot-manifest.json" | ||
| } | ||
|
|
||
| mkdir -p "$OUTPUT_DIR" | ||
|
|
||
| for dest in "${DESTINATIONS[@]}"; do | ||
| IFS='|' read -r device_name sdk xcode_flags <<< "$dest" | ||
| label=$(echo "$device_name" | tr ' (),' '-' | tr '[:upper:]' '[:lower:]' | tr -s '-') | ||
| run_device "$device_name" "$sdk" "$xcode_flags" "$label" | ||
| done | ||
|
|
||
| emit_summary | ||
| echo "Done. $TOTAL_IMAGE_COUNT images in $OUTPUT_DIR/" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| name: Sentry iOS Upload (Snapshots) | ||
|
|
||
| on: | ||
| push: | ||
| branches: [main] | ||
| pull_request: | ||
| branches: [main] | ||
| paths: [ios/**, .github/workflows/ios*] | ||
|
|
||
| jobs: | ||
| upload_sentry_snapshots: | ||
| runs-on: macos-26 | ||
|
|
||
| defaults: | ||
| run: | ||
| working-directory: ./ios | ||
|
|
||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v6 | ||
| with: | ||
| fetch-depth: 0 | ||
|
|
||
| - name: Set up Ruby env | ||
| uses: ruby/setup-ruby@v1 | ||
| with: | ||
| ruby-version: 3.3.10 | ||
| bundler-cache: true | ||
|
|
||
| - name: Setup gems | ||
| run: exec ../.github/scripts/ios/setup.sh | ||
|
|
||
| - name: Cache Swift Package Manager | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: | | ||
| ~/Library/Caches/org.swift.swiftpm | ||
| ~/Library/Developer/Xcode/DerivedData/HackerNews-*/SourcePackages | ||
| key: ${{ runner.os }}-spm-${{ hashFiles('**/Package.resolved') }} | ||
| restore-keys: ${{ runner.os }}-spm- | ||
|
|
||
| - name: Boot simulator | ||
| run: xcrun simctl boot "iPhone 17 Pro Max" || true | ||
|
|
||
| - name: Generate snapshot images (iPhone) | ||
| env: | ||
| EMERGE_DEFAULT_DEVICE_NAME: iPhone 17 Pro Max | ||
| run: | | ||
| set -o pipefail && TEST_RUNNER_SNAPSHOTS_EXPORT_DIR="$PWD/snapshot-images" xcodebuild test \ | ||
| -scheme HackerNews \ | ||
| -sdk iphonesimulator \ | ||
| -destination 'platform=iOS Simulator,name=iPhone 17 Pro Max' \ | ||
| -only-testing:HackerNewsTests/HackerNewsSnapshotTest \ | ||
| -resultBundlePath ../SnapshotResults-iphone.xcresult \ | ||
| ONLY_ACTIVE_ARCH=YES \ | ||
| TARGETED_DEVICE_FAMILY=1 \ | ||
| SUPPORTS_MACCATALYST=NO \ | ||
| | xcpretty | ||
|
|
||
| # - name: Generate snapshot images (iPad) | ||
| # env: | ||
| # EMERGE_DEFAULT_DEVICE_NAME: iPhone 17 Pro Max | ||
| # run: | | ||
| # set -o pipefail && TEST_RUNNER_SNAPSHOTS_EXPORT_DIR="snapshot-images/" xcodebuild test \ | ||
| # -scheme HackerNews \ | ||
| # -sdk iphonesimulator \ | ||
| # -destination 'platform=iOS Simulator,name=iPad Air 11-inch (M3)' \ | ||
| # -only-testing:HackerNewsTests/HackerNewsSnapshotTest \ | ||
| # -resultBundlePath ../SnapshotResults-ipad.xcresult \ | ||
| # ONLY_ACTIVE_ARCH=YES \ | ||
| # TARGETED_DEVICE_FAMILY="1,2" \ | ||
| # SUPPORTS_MACCATALYST=NO \ | ||
| # | xcpretty | ||
|
|
||
| - name: List generated images | ||
| run: | | ||
| echo "Generated snapshot images:" | ||
| ls -1 snapshot-images/ | ||
| echo "Total: $(ls -1 snapshot-images/ | wc -l | tr -d ' ') images" | ||
|
|
||
| - name: Upload snapshots to Sentry | ||
| env: | ||
| SENTRY_SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_SENTRY_AUTH_TOKEN }} | ||
| run: bundle exec fastlane ios upload_sentry_preview_snapshots |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.