Current system health

All systems operational

Live health information for distribution, metadata, media, APIs, partner services, and internal infrastructure.

Live summary

523 monitored components online

Last refreshed moments ago

Component directory

523 services shown

Manual component overrides, maintenance notices, and incident records are administered through the internal operations dashboard. The base operational state is Online.

Incident history

Resolution reports

42 structured reports spanning 2022–2026, with impact summaries, root-cause findings, corrective actions, affected components, and response timelines.

Showing 42 of 42 matching reports

SR-2026-009minorResolvedScenario record

Analytics, Reporting & Search

Catalog search index refresh delay

Recently updated releases could take longer than normal to appear in nonessential catalog search results. Delivery and release processing continued normally.

Resolution time

Jul 18, 2026, 3:19 PM UTC

1h 14m

Impact

Recently updated releases could take longer than normal to appear in nonessential catalog search results. Delivery and release processing continued normally.

Root cause

A scheduled search-index compaction job exceeded its expected memory budget and paused incremental refresh workers.

Resolution

The compaction job was restarted with a reduced batch size, queued index updates were replayed, and search freshness returned to its normal target.

Corrective action

Added batch-size guardrails, memory-pressure alerts, and an automatic fallback to incremental-only indexing during compaction.

Affected components

Release Search IndexArtist Search IndexFull-Text Search Service

Response timeline

Investigating · Jul 18, 2026, 2:05 PM UTC

Operations began reviewing analytics, reporting & search telemetry and affected processing queues.

Identified · Jul 18, 2026, 2:20 PM UTC

A scheduled search-index compaction job exceeded its expected memory budget and paused incremental refresh workers.

Monitoring · Jul 18, 2026, 2:57 PM UTC

The compaction job was restarted with a reduced batch size, queued index updates were replayed, and search freshness returned to its normal target. Teams monitored recovery indicators before closing the report.

Resolved · Jul 18, 2026, 3:19 PM UTC

Normal performance was confirmed. Follow-up action: Added batch-size guardrails, memory-pressure alerts, and an automatic fallback to incremental-only indexing during compaction.

SR-2026-008majorResolvedScenario record

DDEX Delivery Endpoints

Partner acknowledgement receipt backlog

Delivery acknowledgements for a subset of partner routes were delayed, while outbound DDEX packages continued to queue safely without data loss.

Resolution time

Jun 29, 2026, 5:53 AM UTC

3h 33m

Impact

Delivery acknowledgements for a subset of partner routes were delayed, while outbound DDEX packages continued to queue safely without data loss.

Root cause

A receipt-normalization worker entered a retry loop after receiving an unexpected acknowledgement envelope variant.

Resolution

The envelope was isolated, the parser compatibility rule was expanded, and the receipt backlog was drained in chronological order.

Corrective action

Introduced schema-variant fixtures, retry-loop circuit breakers, and queue-age dashboards for partner acknowledgements.

Affected components

Spotify — Acknowledgement and ReceiptApple Music — Acknowledgement and ReceiptPartner Delivery Receipts

Response timeline

Investigating · Jun 29, 2026, 2:20 AM UTC

Operations began reviewing ddex delivery endpoints telemetry and affected processing queues.

Identified · Jun 29, 2026, 3:03 AM UTC

A receipt-normalization worker entered a retry loop after receiving an unexpected acknowledgement envelope variant.

Monitoring · Jun 29, 2026, 4:49 AM UTC

The envelope was isolated, the parser compatibility rule was expanded, and the receipt backlog was drained in chronological order. Teams monitored recovery indicators before closing the report.

Resolved · Jun 29, 2026, 5:53 AM UTC

Normal performance was confirmed. Follow-up action: Introduced schema-variant fixtures, retry-loop circuit breakers, and queue-age dashboards for partner acknowledgements.

SR-2026-007minorResolvedScenario record

Audio & Media Processing

Artwork color-profile validation slowdown

Artwork validation completed more slowly for newly uploaded releases. Audio intake and metadata delivery were unaffected.

Resolution time

May 11, 2026, 7:33 PM UTC

51 minutes

Impact

Artwork validation completed more slowly for newly uploaded releases. Audio intake and metadata delivery were unaffected.

Root cause

A malformed embedded ICC profile triggered repeated fallback conversions in the image validation worker.

Resolution

The problematic profile path was bypassed, affected files were reprocessed, and validation latency normalized.

Corrective action

Added malformed-profile detection, per-file conversion limits, and dedicated test fixtures for uncommon color profiles.

Affected components

Artwork Color Profile ValidationArtwork Dimension Validation

Response timeline

Investigating · May 11, 2026, 6:42 PM UTC

Operations began reviewing audio & media processing telemetry and affected processing queues.

Identified · May 11, 2026, 6:52 PM UTC

A malformed embedded ICC profile triggered repeated fallback conversions in the image validation worker.

Monitoring · May 11, 2026, 7:18 PM UTC

The problematic profile path was bypassed, affected files were reprocessed, and validation latency normalized. Teams monitored recovery indicators before closing the report.

Resolved · May 11, 2026, 7:33 PM UTC

Normal performance was confirmed. Follow-up action: Added malformed-profile detection, per-file conversion limits, and dedicated test fixtures for uncommon color profiles.

SR-2026-006minorResolvedScenario record

Rights, Royalties & Finance

Royalty statement export queue delay

Some nonurgent statement exports remained queued longer than expected. Earnings calculations and stored balances were not changed.

Resolution time

Apr 23, 2026, 10:53 AM UTC

1h 38m

Impact

Some nonurgent statement exports remained queued longer than expected. Earnings calculations and stored balances were not changed.

Root cause

A large batch of multi-territory exports saturated the document-rendering worker pool.

Resolution

Worker concurrency was temporarily increased and the queued exports were completed in priority order.

Corrective action

Added export-size estimation, fair-queue scheduling, and reserved capacity for interactive export requests.

Affected components

Financial Reporting ExportRoyalty Statement Parsing

Response timeline

Investigating · Apr 23, 2026, 9:15 AM UTC

Operations began reviewing rights, royalties & finance telemetry and affected processing queues.

Identified · Apr 23, 2026, 9:35 AM UTC

A large batch of multi-territory exports saturated the document-rendering worker pool.

Monitoring · Apr 23, 2026, 10:24 AM UTC

Worker concurrency was temporarily increased and the queued exports were completed in priority order. Teams monitored recovery indicators before closing the report.

Resolved · Apr 23, 2026, 10:53 AM UTC

Normal performance was confirmed. Follow-up action: Added export-size estimation, fair-queue scheduling, and reserved capacity for interactive export requests.

SR-2026-005minorResolvedScenario record

Support & Communications

Incident notification email delay

Status updates were visible on the public page, but a subset of email notifications arrived later than normal.

Resolution time

Mar 7, 2026, 9:44 PM UTC

36 minutes

Impact

Status updates were visible on the public page, but a subset of email notifications arrived later than normal.

Root cause

A notification batch exceeded the provider connection pool and waited for available sending slots.

Resolution

The batch was split into smaller segments and remaining notifications were delivered.

Corrective action

Implemented adaptive batch sizing, provider pool telemetry, and an in-app fallback notice when email delivery is delayed.

Affected components

System Email NotificationsIncident Update Publishing

Response timeline

Investigating · Mar 7, 2026, 9:08 PM UTC

Operations began reviewing support & communications telemetry and affected processing queues.

Identified · Mar 7, 2026, 9:16 PM UTC

A notification batch exceeded the provider connection pool and waited for available sending slots.

Monitoring · Mar 7, 2026, 9:33 PM UTC

The batch was split into smaller segments and remaining notifications were delivered. Teams monitored recovery indicators before closing the report.

Resolved · Mar 7, 2026, 9:44 PM UTC

Normal performance was confirmed. Follow-up action: Implemented adaptive batch sizing, provider pool telemetry, and an in-app fallback notice when email delivery is delayed.

SR-2026-004majorResolvedScenario record

Core Platform

Gateway synchronization retry saturation

New release records remained safely stored but synchronized to downstream gateway records more slowly than normal.

Resolution time

Feb 16, 2026, 10:16 AM UTC

4h 46m

Impact

New release records remained safely stored but synchronized to downstream gateway records more slowly than normal.

Root cause

A malformed optional territory value caused repeated per-record synchronization retries without sufficient backoff.

Resolution

The malformed value was normalized, retry queues were reset, and all pending UPC-indexed records were synchronized.

Corrective action

Added payload prevalidation, exponential backoff, dead-letter isolation, and per-record retry ceilings.

Affected components

Gateway SynchronizationSSO Gateway APIRelease Updates

Response timeline

Investigating · Feb 16, 2026, 5:30 AM UTC

Operations began reviewing core platform telemetry and affected processing queues.

Identified · Feb 16, 2026, 6:27 AM UTC

A malformed optional territory value caused repeated per-record synchronization retries without sufficient backoff.

Monitoring · Feb 16, 2026, 8:50 AM UTC

The malformed value was normalized, retry queues were reset, and all pending UPC-indexed records were synchronized. Teams monitored recovery indicators before closing the report.

Resolved · Feb 16, 2026, 10:16 AM UTC

Normal performance was confirmed. Follow-up action: Added payload prevalidation, exponential backoff, dead-letter isolation, and per-record retry ceilings.

SR-2026-003minorResolvedScenario record

Metadata & Identifier Services

ISRC relationship lookup latency

Some recording relationship lookups took longer to complete, while direct ISRC validation remained available.

Resolution time

Jan 28, 2026, 2:19 PM UTC

1h 7m

Impact

Some recording relationship lookups took longer to complete, while direct ISRC validation remained available.

Root cause

A metadata cache shard was rebuilding after routine maintenance and served a higher-than-normal percentage of uncached queries.

Resolution

Cache warming was prioritized for high-frequency recording identifiers and lookup latency returned to normal.

Corrective action

Added prewarming manifests, cache-rebuild progress indicators, and request coalescing for duplicate lookups.

Affected components

ISRC ValidationRecording Entity LookupMetadata Partner Lookup

Response timeline

Investigating · Jan 28, 2026, 1:12 PM UTC

Operations began reviewing metadata & identifier services telemetry and affected processing queues.

Identified · Jan 28, 2026, 1:25 PM UTC

A metadata cache shard was rebuilding after routine maintenance and served a higher-than-normal percentage of uncached queries.

Monitoring · Jan 28, 2026, 1:59 PM UTC

Cache warming was prioritized for high-frequency recording identifiers and lookup latency returned to normal. Teams monitored recovery indicators before closing the report.

Resolved · Jan 28, 2026, 2:19 PM UTC

Normal performance was confirmed. Follow-up action: Added prewarming manifests, cache-rebuild progress indicators, and request coalescing for duplicate lookups.

SR-2026-002majorResolvedScenario record

DSP Delivery Network

YouTube Content ID asset receipt delay

Content ID asset receipts were delayed for a limited batch. Assets remained queued and no duplicate submissions were generated.

Resolution time

Jan 14, 2026, 3:02 AM UTC

2h 22m

Impact

Content ID asset receipts were delayed for a limited batch. Assets remained queued and no duplicate submissions were generated.

Root cause

A partner receipt payload included an unrecognized nested status code that was routed to manual review.

Resolution

The status code was mapped, held receipts were replayed, and asset delivery states were reconciled.

Corrective action

Expanded receipt code coverage and added automated safe handling for unknown nonterminal status values.

Affected components

YouTube Music Status ReceiptsYouTube Content ID — Acknowledgement and ReceiptPartner Delivery Receipts

Response timeline

Investigating · Jan 14, 2026, 12:40 AM UTC

Operations began reviewing dsp delivery network telemetry and affected processing queues.

Identified · Jan 14, 2026, 1:08 AM UTC

A partner receipt payload included an unrecognized nested status code that was routed to manual review.

Monitoring · Jan 14, 2026, 2:19 AM UTC

The status code was mapped, held receipts were replayed, and asset delivery states were reconciled. Teams monitored recovery indicators before closing the report.

Resolved · Jan 14, 2026, 3:02 AM UTC

Normal performance was confirmed. Follow-up action: Expanded receipt code coverage and added automated safe handling for unknown nonterminal status values.

SR-2026-001minorResolvedScenario record

API & Developer Services

Sandbox token exchange latency

Sandbox authorization-code exchanges completed slowly for some developer sessions. Production API authentication was unaffected.

Resolution time

Jan 3, 2026, 5:38 PM UTC

43 minutes

Impact

Sandbox authorization-code exchanges completed slowly for some developer sessions. Production API authentication was unaffected.

Root cause

A sandbox-only signing-key cache was cold after a routine deployment.

Resolution

Signing keys were preloaded across sandbox workers and pending token exchanges completed successfully.

Corrective action

Added deployment-time key prewarming and separate latency alerts for sandbox authentication paths.

Affected components

Sandbox APIOAuth Token ExchangeOAuth PKCE Verification

Response timeline

Investigating · Jan 3, 2026, 4:55 PM UTC

Operations began reviewing api & developer services telemetry and affected processing queues.

Identified · Jan 3, 2026, 5:04 PM UTC

A sandbox-only signing-key cache was cold after a routine deployment.

Monitoring · Jan 3, 2026, 5:25 PM UTC

Signing keys were preloaded across sandbox workers and pending token exchanges completed successfully. Teams monitored recovery indicators before closing the report.

Resolved · Jan 3, 2026, 5:38 PM UTC

Normal performance was confirmed. Follow-up action: Added deployment-time key prewarming and separate latency alerts for sandbox authentication paths.

SR-2025-009minorResolvedScenario record

DDEX Delivery Endpoints

Amazon Music metadata receipt lag

Metadata delivery receipts for a subset of updates appeared later than expected. Outbound delivery remained available.

Resolution time

Dec 9, 2025, 12:53 PM UTC

1h 29m

Impact

Metadata delivery receipts for a subset of updates appeared later than expected. Outbound delivery remained available.

Root cause

Receipt polling was temporarily constrained by a conservative partner-route rate limit.

Resolution

Polling intervals were redistributed and delayed receipts were reconciled without resubmitting releases.

Corrective action

Added route-specific rate-limit forecasting and staggered receipt polling for large update batches.

Affected components

Amazon Music — Acknowledgement and ReceiptAmazon Music Status Receipts

Response timeline

Investigating · Dec 9, 2025, 11:24 AM UTC

Operations began reviewing ddex delivery endpoints telemetry and affected processing queues.

Identified · Dec 9, 2025, 11:42 AM UTC

Receipt polling was temporarily constrained by a conservative partner-route rate limit.

Monitoring · Dec 9, 2025, 12:26 PM UTC

Polling intervals were redistributed and delayed receipts were reconciled without resubmitting releases. Teams monitored recovery indicators before closing the report.

Resolved · Dec 9, 2025, 12:53 PM UTC

Normal performance was confirmed. Follow-up action: Added route-specific rate-limit forecasting and staggered receipt polling for large update batches.

SR-2025-008majorResolvedScenario record

Catalog & Release Operations

CSV ingestion worker backlog

Large catalog CSV uploads remained in processing longer than normal, but files stayed intact and were processed in upload order.

Resolution time

Oct 27, 2025, 9:14 PM UTC

2h 8m

Impact

Large catalog CSV uploads remained in processing longer than normal, but files stayed intact and were processed in upload order.

Root cause

Several unusually large catalog files arrived simultaneously and exhausted the standard parsing worker pool.

Resolution

Additional parsing workers were allocated and the backlog was drained with deduplication checks preserved.

Corrective action

Added file-size-aware queue partitioning, burst capacity, and estimated completion times for large imports.

Affected components

Catalog CSV ParsingRelease Row GroupingRelease Creation

Response timeline

Investigating · Oct 27, 2025, 7:06 PM UTC

Operations began reviewing catalog & release operations telemetry and affected processing queues.

Identified · Oct 27, 2025, 7:32 PM UTC

Several unusually large catalog files arrived simultaneously and exhausted the standard parsing worker pool.

Monitoring · Oct 27, 2025, 8:36 PM UTC

Additional parsing workers were allocated and the backlog was drained with deduplication checks preserved. Teams monitored recovery indicators before closing the report.

Resolved · Oct 27, 2025, 9:14 PM UTC

Normal performance was confirmed. Follow-up action: Added file-size-aware queue partitioning, burst capacity, and estimated completion times for large imports.

SR-2025-007minorResolvedScenario record

Audio & Media Processing

Timed lyrics publishing delay

Validated timed lyrics took longer than normal to move from intake to publishing queues. Audio deliveries were unaffected.

Resolution time

Sep 13, 2025, 8:42 AM UTC

57 minutes

Impact

Validated timed lyrics took longer than normal to move from intake to publishing queues. Audio deliveries were unaffected.

Root cause

A character-normalization edge case caused repeated validation retries on a small group of TTML documents.

Resolution

The normalization rule was corrected and queued lyric documents were revalidated.

Corrective action

Added Unicode edge-case fixtures and a retry cap that routes malformed documents directly to review.

Affected components

Timed Lyrics IntakeTTML Lyrics ValidationKaraoke Lyrics Processing

Response timeline

Investigating · Sep 13, 2025, 7:45 AM UTC

Operations began reviewing audio & media processing telemetry and affected processing queues.

Identified · Sep 13, 2025, 7:56 AM UTC

A character-normalization edge case caused repeated validation retries on a small group of TTML documents.

Monitoring · Sep 13, 2025, 8:25 AM UTC

The normalization rule was corrected and queued lyric documents were revalidated. Teams monitored recovery indicators before closing the report.

Resolved · Sep 13, 2025, 8:42 AM UTC

Normal performance was confirmed. Follow-up action: Added Unicode edge-case fixtures and a retry cap that routes malformed documents directly to review.

SR-2025-006majorResolvedScenario record

DSP Delivery Network

TikTok preview asset processing backlog

Preview asset processing was delayed for selected TikTok-bound releases while metadata delivery continued.

Resolution time

Aug 2, 2025, 6:14 AM UTC

2h 56m

Impact

Preview asset processing was delayed for selected TikTok-bound releases while metadata delivery continued.

Root cause

A media transcode worker group was reduced during maintenance and did not scale back up automatically.

Resolution

Worker capacity was restored and queued preview assets were processed by original priority.

Corrective action

Added post-maintenance capacity verification and automatic minimum-worker enforcement.

Affected components

TikTok Audio DeliveryTikTok — Resource and Asset Delivery

Response timeline

Investigating · Aug 2, 2025, 3:18 AM UTC

Operations began reviewing dsp delivery network telemetry and affected processing queues.

Identified · Aug 2, 2025, 3:53 AM UTC

A media transcode worker group was reduced during maintenance and did not scale back up automatically.

Monitoring · Aug 2, 2025, 5:21 AM UTC

Worker capacity was restored and queued preview assets were processed by original priority. Teams monitored recovery indicators before closing the report.

Resolved · Aug 2, 2025, 6:14 AM UTC

Normal performance was confirmed. Follow-up action: Added post-maintenance capacity verification and automatic minimum-worker enforcement.

SR-2025-005minorResolvedScenario record

Metadata & Identifier Services

UPC validation cache inconsistency

A small number of repeated UPC checks returned inconsistent cached messages, though final checksum decisions remained correct.

Resolution time

Jun 21, 2025, 4:35 PM UTC

45 minutes

Impact

A small number of repeated UPC checks returned inconsistent cached messages, though final checksum decisions remained correct.

Root cause

Two cache versions used different explanatory message formats after a rolling release.

Resolution

The older cache namespace was invalidated and all workers were aligned to the current validation response format.

Corrective action

Versioned validation caches and added cross-worker contract tests for response consistency.

Affected components

UPC ValidationGTIN Checksum Validation

Response timeline

Investigating · Jun 21, 2025, 3:50 PM UTC

Operations began reviewing metadata & identifier services telemetry and affected processing queues.

Identified · Jun 21, 2025, 3:59 PM UTC

Two cache versions used different explanatory message formats after a rolling release.

Monitoring · Jun 21, 2025, 4:21 PM UTC

The older cache namespace was invalidated and all workers were aligned to the current validation response format. Teams monitored recovery indicators before closing the report.

Resolved · Jun 21, 2025, 4:35 PM UTC

Normal performance was confirmed. Follow-up action: Versioned validation caches and added cross-worker contract tests for response consistency.

SR-2025-004majorResolvedScenario record

Audio & Media Processing

Dolby Atmos ADM package verification queue

Atmos package verification took longer than normal for new submissions. Stereo processing and existing deliveries were unaffected.

Resolution time

May 9, 2025, 2:22 AM UTC

3h 51m

Impact

Atmos package verification took longer than normal for new submissions. Stereo processing and existing deliveries were unaffected.

Root cause

An unusually complex set of channel-layout validations increased average verification time and saturated the specialized worker queue.

Resolution

Complex packages were moved to a dedicated queue and standard validations resumed at normal speed.

Corrective action

Added complexity scoring, dedicated high-complexity capacity, and queue-depth alerts for immersive-audio validation.

Affected components

Dolby Atmos ADM IntakeDolby Atmos Metadata ValidationChannel Layout Validation

Response timeline

Investigating · May 8, 2025, 10:31 PM UTC

Operations began reviewing audio & media processing telemetry and affected processing queues.

Identified · May 8, 2025, 11:17 PM UTC

An unusually complex set of channel-layout validations increased average verification time and saturated the specialized worker queue.

Monitoring · May 9, 2025, 1:13 AM UTC

Complex packages were moved to a dedicated queue and standard validations resumed at normal speed. Teams monitored recovery indicators before closing the report.

Resolved · May 9, 2025, 2:22 AM UTC

Normal performance was confirmed. Follow-up action: Added complexity scoring, dedicated high-complexity capacity, and queue-depth alerts for immersive-audio validation.

SR-2025-003minorResolvedScenario record

Analytics, Reporting & Search

Partner directory status cache delay

Recently changed nonessential endpoint labels took longer to appear in public search results. Live component health values remained current.

Resolution time

Mar 19, 2025, 11:16 AM UTC

1h 2m

Impact

Recently changed nonessential endpoint labels took longer to appear in public search results. Live component health values remained current.

Root cause

A static directory cache did not invalidate after a content-only metadata update.

Resolution

The directory cache was purged and rebuilt from the current component registry.

Corrective action

Linked content updates to cache invalidation and added freshness timestamps to directory responses.

Affected components

Status Component SearchPublic Service Notices

Response timeline

Investigating · Mar 19, 2025, 10:14 AM UTC

Operations began reviewing analytics, reporting & search telemetry and affected processing queues.

Identified · Mar 19, 2025, 10:26 AM UTC

A static directory cache did not invalidate after a content-only metadata update.

Monitoring · Mar 19, 2025, 10:57 AM UTC

The directory cache was purged and rebuilt from the current component registry. Teams monitored recovery indicators before closing the report.

Resolved · Mar 19, 2025, 11:16 AM UTC

Normal performance was confirmed. Follow-up action: Linked content updates to cache invalidation and added freshness timestamps to directory responses.

SR-2025-002minorResolvedScenario record

Rights, Royalties & Finance

Territory earnings report generation delay

Some territory-level report views loaded slowly while aggregate earnings totals remained available.

Resolution time

Feb 4, 2025, 5:49 AM UTC

1h 44m

Impact

Some territory-level report views loaded slowly while aggregate earnings totals remained available.

Root cause

A currency reference table refresh held a read lock longer than expected during report aggregation.

Resolution

The refresh was moved to a nonblocking swap operation and pending reports completed.

Corrective action

Implemented atomic reference-table replacement and query-time fallbacks to the last validated currency snapshot.

Affected components

Territory Earnings BreakdownCurrency Normalization

Response timeline

Investigating · Feb 4, 2025, 4:05 AM UTC

Operations began reviewing rights, royalties & finance telemetry and affected processing queues.

Identified · Feb 4, 2025, 4:26 AM UTC

A currency reference table refresh held a read lock longer than expected during report aggregation.

Monitoring · Feb 4, 2025, 5:18 AM UTC

The refresh was moved to a nonblocking swap operation and pending reports completed. Teams monitored recovery indicators before closing the report.

Resolved · Feb 4, 2025, 5:49 AM UTC

Normal performance was confirmed. Follow-up action: Implemented atomic reference-table replacement and query-time fallbacks to the last validated currency snapshot.

SR-2025-001minorResolvedScenario record

Identity, Security & Storage

Organization profile update delay

Organization-domain profile edits were saved but appeared with a short delay in account views. Authentication and permissions were unaffected.

Resolution time

Jan 12, 2025, 6:17 PM UTC

39 minutes

Impact

Organization-domain profile edits were saved but appeared with a short delay in account views. Authentication and permissions were unaffected.

Root cause

A profile cache invalidation event was not emitted for one update path.

Resolution

The missing invalidation event was restored and affected profiles were refreshed.

Corrective action

Added update-path coverage tests and cache-consistency checks after profile writes.

Affected components

Organization Domain ProfileAccount Profile Service

Response timeline

Investigating · Jan 12, 2025, 5:38 PM UTC

Operations began reviewing identity, security & storage telemetry and affected processing queues.

Identified · Jan 12, 2025, 5:46 PM UTC

A profile cache invalidation event was not emitted for one update path.

Monitoring · Jan 12, 2025, 6:05 PM UTC

The missing invalidation event was restored and affected profiles were refreshed. Teams monitored recovery indicators before closing the report.

Resolved · Jan 12, 2025, 6:17 PM UTC

Normal performance was confirmed. Follow-up action: Added update-path coverage tests and cache-consistency checks after profile writes.

SR-2024-008majorResolvedScenario record

Catalog & Release Operations

DDEX ERN parser compatibility issue

A subset of ERN messages using an uncommon namespace pattern required manual review before release records could be created.

Resolution time

Nov 18, 2024, 11:28 AM UTC

3h 16m

Impact

A subset of ERN messages using an uncommon namespace pattern required manual review before release records could be created.

Root cause

The namespace resolver expected a canonical prefix even though the incoming XML used a valid alternate prefix.

Resolution

Namespace handling was updated to resolve by URI, and held messages were reprocessed successfully.

Corrective action

Added alternate-prefix fixtures and namespace-agnostic extraction tests for supported DDEX message families.

Affected components

Release CreationTerritory ExtractionAI Parsing Runtime

Response timeline

Investigating · Nov 18, 2024, 8:12 AM UTC

Operations began reviewing catalog & release operations telemetry and affected processing queues.

Identified · Nov 18, 2024, 8:51 AM UTC

The namespace resolver expected a canonical prefix even though the incoming XML used a valid alternate prefix.

Monitoring · Nov 18, 2024, 10:29 AM UTC

Namespace handling was updated to resolve by URI, and held messages were reprocessed successfully. Teams monitored recovery indicators before closing the report.

Resolved · Nov 18, 2024, 11:28 AM UTC

Normal performance was confirmed. Follow-up action: Added alternate-prefix fixtures and namespace-agnostic extraction tests for supported DDEX message families.

SR-2024-007minorResolvedScenario record

DSP Delivery Network

Deezer territory update receipt delay

Territory update receipts were delayed for a limited batch. Existing availability remained unchanged until confirmation.

Resolution time

Oct 3, 2024, 9:51 PM UTC

1h 11m

Impact

Territory update receipts were delayed for a limited batch. Existing availability remained unchanged until confirmation.

Root cause

A route-specific acknowledgement poller restarted during routine infrastructure maintenance.

Resolution

The poller resumed from its stored cursor and all pending receipts were reconciled.

Corrective action

Added cursor checkpoint validation and automatic restart verification for route pollers.

Affected components

Deezer Territory RoutingDeezer Status Receipts

Response timeline

Investigating · Oct 3, 2024, 8:40 PM UTC

Operations began reviewing dsp delivery network telemetry and affected processing queues.

Identified · Oct 3, 2024, 8:54 PM UTC

A route-specific acknowledgement poller restarted during routine infrastructure maintenance.

Monitoring · Oct 3, 2024, 9:30 PM UTC

The poller resumed from its stored cursor and all pending receipts were reconciled. Teams monitored recovery indicators before closing the report.

Resolved · Oct 3, 2024, 9:51 PM UTC

Normal performance was confirmed. Follow-up action: Added cursor checkpoint validation and automatic restart verification for route pollers.

SR-2024-006majorResolvedScenario record

Audio & Media Processing

High-resolution audio integrity scan slowdown

Large high-resolution masters remained in validation longer than usual. Files were retained securely and no audio data was modified.

Resolution time

Aug 26, 2024, 3:08 PM UTC

2h 13m

Impact

Large high-resolution masters remained in validation longer than usual. Files were retained securely and no audio data was modified.

Root cause

Integrity scans were running in a low-throughput compatibility mode after a scanner update.

Resolution

The scanner configuration was corrected and queued masters were rescanned at normal throughput.

Corrective action

Added throughput benchmarks to scanner deployments and an automatic rollback threshold for validation speed.

Affected components

High-Resolution Audio IntakeAudio File Integrity ScanSample Rate Validation

Response timeline

Investigating · Aug 26, 2024, 12:55 PM UTC

Operations began reviewing audio & media processing telemetry and affected processing queues.

Identified · Aug 26, 2024, 1:22 PM UTC

Integrity scans were running in a low-throughput compatibility mode after a scanner update.

Monitoring · Aug 26, 2024, 2:28 PM UTC

The scanner configuration was corrected and queued masters were rescanned at normal throughput. Teams monitored recovery indicators before closing the report.

Resolved · Aug 26, 2024, 3:08 PM UTC

Normal performance was confirmed. Follow-up action: Added throughput benchmarks to scanner deployments and an automatic rollback threshold for validation speed.

SR-2024-005minorResolvedScenario record

Third Party and Partners

Metadata partner search rate-limit response

Broad metadata searches were temporarily slowed by upstream request limits. Exact identifier lookups remained available.

Resolution time

Jul 9, 2024, 2:05 AM UTC

48 minutes

Impact

Broad metadata searches were temporarily slowed by upstream request limits. Exact identifier lookups remained available.

Root cause

A scheduled catalog enrichment task produced more broad searches than forecast.

Resolution

The enrichment task was throttled and interactive searches were assigned a separate request budget.

Corrective action

Separated background and interactive quotas and added predictive request-volume controls.

Affected components

Metadata Partner SearchMetadata Partner Rate Limit

Response timeline

Investigating · Jul 9, 2024, 1:17 AM UTC

Operations began reviewing third party and partners telemetry and affected processing queues.

Identified · Jul 9, 2024, 1:27 AM UTC

A scheduled catalog enrichment task produced more broad searches than forecast.

Monitoring · Jul 9, 2024, 1:51 AM UTC

The enrichment task was throttled and interactive searches were assigned a separate request budget. Teams monitored recovery indicators before closing the report.

Resolved · Jul 9, 2024, 2:05 AM UTC

Normal performance was confirmed. Follow-up action: Separated background and interactive quotas and added predictive request-volume controls.

SR-2024-004minorResolvedScenario record

Analytics, Reporting & Search

Delivery activity report freshness delay

Historical delivery activity reports were behind current processing by approximately one reporting cycle. Live status indicators remained current.

Resolution time

May 30, 2024, 5:37 PM UTC

1h 34m

Impact

Historical delivery activity reports were behind current processing by approximately one reporting cycle. Live status indicators remained current.

Root cause

An aggregation job skipped a partition after detecting a transient checksum mismatch.

Resolution

The partition was verified, replayed, and reporting freshness returned to normal.

Corrective action

Added automatic partition retry after checksum verification and clearer separation of live versus historical freshness metrics.

Affected components

Delivery Activity ReportingPlatform Status Reporting

Response timeline

Investigating · May 30, 2024, 4:03 PM UTC

Operations began reviewing analytics, reporting & search telemetry and affected processing queues.

Identified · May 30, 2024, 4:22 PM UTC

An aggregation job skipped a partition after detecting a transient checksum mismatch.

Monitoring · May 30, 2024, 5:09 PM UTC

The partition was verified, replayed, and reporting freshness returned to normal. Teams monitored recovery indicators before closing the report.

Resolved · May 30, 2024, 5:37 PM UTC

Normal performance was confirmed. Follow-up action: Added automatic partition retry after checksum verification and clearer separation of live versus historical freshness metrics.

SR-2024-003majorResolvedScenario record

DDEX Delivery Endpoints

TIDAL high-resolution asset validation hold

A small batch of high-resolution assets was held before partner delivery while format declarations were revalidated.

Resolution time

Apr 14, 2024, 9:09 AM UTC

2h 47m

Impact

A small batch of high-resolution assets was held before partner delivery while format declarations were revalidated.

Root cause

Container metadata and declared bit depth disagreed on several files, triggering a safety hold.

Resolution

The files were reanalyzed, valid assets were released, and mismatched assets were returned with precise correction guidance.

Corrective action

Added preflight comparison of container metadata, declared specifications, and actual decoded stream properties.

Affected components

TIDAL — Resource and Asset DeliveryHigh-Resolution Audio IntakeBit Depth Validation

Response timeline

Investigating · Apr 14, 2024, 6:22 AM UTC

Operations began reviewing ddex delivery endpoints telemetry and affected processing queues.

Identified · Apr 14, 2024, 6:55 AM UTC

Container metadata and declared bit depth disagreed on several files, triggering a safety hold.

Monitoring · Apr 14, 2024, 8:19 AM UTC

The files were reanalyzed, valid assets were released, and mismatched assets were returned with precise correction guidance. Teams monitored recovery indicators before closing the report.

Resolved · Apr 14, 2024, 9:09 AM UTC

Normal performance was confirmed. Follow-up action: Added preflight comparison of container metadata, declared specifications, and actual decoded stream properties.

SR-2024-002minorResolvedScenario record

Support & Communications

Technical documentation search indexing lag

New documentation sections were available by direct link but did not immediately appear in documentation search.

Resolution time

Feb 23, 2024, 12:09 AM UTC

59 minutes

Impact

New documentation sections were available by direct link but did not immediately appear in documentation search.

Root cause

The documentation crawler deferred a reindex after detecting a temporary sitemap fetch error.

Resolution

The sitemap was refetched and the documentation index was rebuilt.

Corrective action

Added sitemap retry with cached fallback and visible documentation-index freshness reporting.

Affected components

Technical DocumentationAPI Reference DocumentationFull-Text Search Service

Response timeline

Investigating · Feb 22, 2024, 11:10 PM UTC

Operations began reviewing support & communications telemetry and affected processing queues.

Identified · Feb 22, 2024, 11:22 PM UTC

The documentation crawler deferred a reindex after detecting a temporary sitemap fetch error.

Monitoring · Feb 22, 2024, 11:51 PM UTC

The sitemap was refetched and the documentation index was rebuilt. Teams monitored recovery indicators before closing the report.

Resolved · Feb 23, 2024, 12:09 AM UTC

Normal performance was confirmed. Follow-up action: Added sitemap retry with cached fallback and visible documentation-index freshness reporting.

SR-2024-001minorResolvedScenario record

Catalog & Release Operations

Release detail status display mismatch

A small number of detail pages displayed a stale status label after a successful delivery update. Underlying records were correct.

Resolution time

Jan 6, 2024, 2:16 PM UTC

32 minutes

Impact

A small number of detail pages displayed a stale status label after a successful delivery update. Underlying records were correct.

Root cause

A detail-view cache key omitted the release update timestamp.

Resolution

The cache key was corrected and affected detail views were invalidated.

Corrective action

Added status-consistency checks between list and detail views and included record versioning in cache keys.

Affected components

Release Detail ServiceRelease Status Derivation

Response timeline

Investigating · Jan 6, 2024, 1:44 PM UTC

Operations began reviewing catalog & release operations telemetry and affected processing queues.

Identified · Jan 6, 2024, 1:52 PM UTC

A detail-view cache key omitted the release update timestamp.

Monitoring · Jan 6, 2024, 2:06 PM UTC

The cache key was corrected and affected detail views were invalidated. Teams monitored recovery indicators before closing the report.

Resolved · Jan 6, 2024, 2:16 PM UTC

Normal performance was confirmed. Follow-up action: Added status-consistency checks between list and detail views and included record versioning in cache keys.

SR-2023-008majorResolvedScenario record

DSP Delivery Network

Release takedown acknowledgement delay

Takedown requests were submitted, but confirmation receipts were delayed for selected routes. Requests were not duplicated.

Resolution time

Dec 12, 2023, 8:39 AM UTC

4h 4m

Impact

Takedown requests were submitted, but confirmation receipts were delayed for selected routes. Requests were not duplicated.

Root cause

The acknowledgement reconciler paused after detecting an out-of-order receipt sequence.

Resolution

Receipt ordering rules were corrected and pending confirmations were reconciled against original request IDs.

Corrective action

Added out-of-order receipt support, idempotency checks, and route-specific reconciliation alerts.

Affected components

Spotify Update and TakedownApple Music Update and TakedownPartner Delivery Receipts

Response timeline

Investigating · Dec 12, 2023, 4:35 AM UTC

Operations began reviewing dsp delivery network telemetry and affected processing queues.

Identified · Dec 12, 2023, 5:24 AM UTC

The acknowledgement reconciler paused after detecting an out-of-order receipt sequence.

Monitoring · Dec 12, 2023, 7:26 AM UTC

Receipt ordering rules were corrected and pending confirmations were reconciled against original request IDs. Teams monitored recovery indicators before closing the report.

Resolved · Dec 12, 2023, 8:39 AM UTC

Normal performance was confirmed. Follow-up action: Added out-of-order receipt support, idempotency checks, and route-specific reconciliation alerts.

SR-2023-007minorResolvedScenario record

Metadata & Identifier Services

Artist relationship expansion slowdown

Complex artist relationship queries took longer than normal. Direct artist and release lookups remained available.

Resolution time

Oct 29, 2023, 7:33 PM UTC

1h 13m

Impact

Complex artist relationship queries took longer than normal. Direct artist and release lookups remained available.

Root cause

A highly connected entity generated a large uncached relationship graph.

Resolution

The query was paginated internally and frequently requested relationship paths were cached.

Corrective action

Added graph-size limits, progressive relationship loading, and high-cardinality query telemetry.

Affected components

Artist Credit ExpansionRelationship Expansion

Response timeline

Investigating · Oct 29, 2023, 6:20 PM UTC

Operations began reviewing metadata & identifier services telemetry and affected processing queues.

Identified · Oct 29, 2023, 6:35 PM UTC

A highly connected entity generated a large uncached relationship graph.

Monitoring · Oct 29, 2023, 7:11 PM UTC

The query was paginated internally and frequently requested relationship paths were cached. Teams monitored recovery indicators before closing the report.

Resolved · Oct 29, 2023, 7:33 PM UTC

Normal performance was confirmed. Follow-up action: Added graph-size limits, progressive relationship loading, and high-cardinality query telemetry.

SR-2023-006minorResolvedScenario record

Analytics, Reporting & Search

Audit feed event ordering issue

Recent audit events briefly appeared out of sequence in one interface. Event records and timestamps remained intact.

Resolution time

Sep 5, 2023, 9:49 AM UTC

41 minutes

Impact

Recent audit events briefly appeared out of sequence in one interface. Event records and timestamps remained intact.

Root cause

The compact feed sorted equal-millisecond timestamps by arrival order instead of event identifier.

Resolution

A deterministic secondary sort by event identifier was added and the feed refreshed.

Corrective action

Standardized audit sorting across all views and added equal-timestamp test coverage.

Affected components

Admin Audit FeedFull Audit Log ReportingAudit Log

Response timeline

Investigating · Sep 5, 2023, 9:08 AM UTC

Operations began reviewing analytics, reporting & search telemetry and affected processing queues.

Identified · Sep 5, 2023, 9:16 AM UTC

The compact feed sorted equal-millisecond timestamps by arrival order instead of event identifier.

Monitoring · Sep 5, 2023, 9:37 AM UTC

A deterministic secondary sort by event identifier was added and the feed refreshed. Teams monitored recovery indicators before closing the report.

Resolved · Sep 5, 2023, 9:49 AM UTC

Normal performance was confirmed. Follow-up action: Standardized audit sorting across all views and added equal-timestamp test coverage.

SR-2023-005majorResolvedScenario record

Catalog & Release Operations

Territory availability cache stale after update

New territory settings were stored correctly but some cached availability views reflected the previous territory set.

Resolution time

Jul 17, 2023, 4:50 AM UTC

1h 58m

Impact

New territory settings were stored correctly but some cached availability views reflected the previous territory set.

Root cause

A territory-only update did not trigger the same cache invalidation path as a full metadata update.

Resolution

Territory caches were invalidated and availability views were rebuilt from current release records.

Corrective action

Unified cache invalidation for all delivery-affecting metadata changes and added territory-specific regression tests.

Affected components

Territory ExtractionApple Music Availability FeedSpotify Availability Feed

Response timeline

Investigating · Jul 17, 2023, 2:52 AM UTC

Operations began reviewing catalog & release operations telemetry and affected processing queues.

Identified · Jul 17, 2023, 3:16 AM UTC

A territory-only update did not trigger the same cache invalidation path as a full metadata update.

Monitoring · Jul 17, 2023, 4:15 AM UTC

Territory caches were invalidated and availability views were rebuilt from current release records. Teams monitored recovery indicators before closing the report.

Resolved · Jul 17, 2023, 4:50 AM UTC

Normal performance was confirmed. Follow-up action: Unified cache invalidation for all delivery-affecting metadata changes and added territory-specific regression tests.

SR-2023-004minorResolvedScenario record

API & Developer Services

Track upload URL generation delay

Some signed upload URLs took longer to generate, while existing upload URLs remained valid.

Resolution time

May 26, 2023, 3:23 PM UTC

1h 4m

Impact

Some signed upload URLs took longer to generate, while existing upload URLs remained valid.

Root cause

A storage permission policy refresh temporarily increased authorization checks.

Resolution

The refreshed policy was cached across workers and pending URL requests completed.

Corrective action

Added policy preloading and separate latency monitoring for signed URL generation.

Affected components

Track Upload URL EndpointFile Upload Authorization

Response timeline

Investigating · May 26, 2023, 2:19 PM UTC

Operations began reviewing api & developer services telemetry and affected processing queues.

Identified · May 26, 2023, 2:32 PM UTC

A storage permission policy refresh temporarily increased authorization checks.

Monitoring · May 26, 2023, 3:04 PM UTC

The refreshed policy was cached across workers and pending URL requests completed. Teams monitored recovery indicators before closing the report.

Resolved · May 26, 2023, 3:23 PM UTC

Normal performance was confirmed. Follow-up action: Added policy preloading and separate latency monitoring for signed URL generation.

SR-2023-003minorResolvedScenario record

DSP Delivery Network

YouTube channel mapping lookup timeout

A subset of channel mapping searches timed out and required retry. Existing channel mappings were unaffected.

Resolution time

Apr 2, 2023, 11:23 PM UTC

37 minutes

Impact

A subset of channel mapping searches timed out and required retry. Existing channel mappings were unaffected.

Root cause

A background preference-refresh task competed with interactive mapping searches for the same connection pool.

Resolution

Interactive requests were moved to a reserved pool and timed-out lookups succeeded on retry.

Corrective action

Separated background refresh traffic and added pool-saturation protection for interactive lookups.

Affected components

Partner Search APIThird Party and Partners Artist PreferencesYouTube Music Metadata Delivery

Response timeline

Investigating · Apr 2, 2023, 10:46 PM UTC

Operations began reviewing dsp delivery network telemetry and affected processing queues.

Identified · Apr 2, 2023, 10:54 PM UTC

A background preference-refresh task competed with interactive mapping searches for the same connection pool.

Monitoring · Apr 2, 2023, 11:12 PM UTC

Interactive requests were moved to a reserved pool and timed-out lookups succeeded on retry. Teams monitored recovery indicators before closing the report.

Resolved · Apr 2, 2023, 11:23 PM UTC

Normal performance was confirmed. Follow-up action: Separated background refresh traffic and added pool-saturation protection for interactive lookups.

SR-2023-002majorResolvedScenario record

Catalog & Release Operations

Delivery log timezone normalization defect

A subset of imported delivery timestamps displayed with an incorrect offset, while delivery status values remained accurate.

Resolution time

Feb 15, 2023, 1:42 PM UTC

2h 31m

Impact

A subset of imported delivery timestamps displayed with an incorrect offset, while delivery status values remained accurate.

Root cause

One accepted timestamp format was interpreted in local time instead of UTC during normalization.

Resolution

Affected timestamps were recalculated from source records and the parser was corrected.

Corrective action

Added explicit timezone parsing, source-format fixtures, and timestamp round-trip validation.

Affected components

Delivery Timestamp MappingDelivery Activity Reporting

Response timeline

Investigating · Feb 15, 2023, 11:11 AM UTC

Operations began reviewing catalog & release operations telemetry and affected processing queues.

Identified · Feb 15, 2023, 11:41 AM UTC

One accepted timestamp format was interpreted in local time instead of UTC during normalization.

Monitoring · Feb 15, 2023, 12:57 PM UTC

Affected timestamps were recalculated from source records and the parser was corrected. Teams monitored recovery indicators before closing the report.

Resolved · Feb 15, 2023, 1:42 PM UTC

Normal performance was confirmed. Follow-up action: Added explicit timezone parsing, source-format fixtures, and timestamp round-trip validation.

SR-2023-001minorResolvedScenario record

Identity, Security & Storage

Partner authorization token refresh delay

A small number of background partner jobs waited for token refresh before continuing. No credentials were exposed.

Resolution time

Jan 9, 2023, 4:17 AM UTC

52 minutes

Impact

A small number of background partner jobs waited for token refresh before continuing. No credentials were exposed.

Root cause

Refresh requests converged at the same expiration boundary and temporarily serialized on a signing lock.

Resolution

Refresh requests were jittered and queued jobs resumed after tokens were renewed.

Corrective action

Added expiration jitter, proactive refresh windows, and lock-contention alerts.

Affected components

OAuth Token ExchangeService Role AuthenticationThird Party and Partners Authentication

Response timeline

Investigating · Jan 9, 2023, 3:25 AM UTC

Operations began reviewing identity, security & storage telemetry and affected processing queues.

Identified · Jan 9, 2023, 3:35 AM UTC

Refresh requests converged at the same expiration boundary and temporarily serialized on a signing lock.

Monitoring · Jan 9, 2023, 4:01 AM UTC

Refresh requests were jittered and queued jobs resumed after tokens were renewed. Teams monitored recovery indicators before closing the report.

Resolved · Jan 9, 2023, 4:17 AM UTC

Normal performance was confirmed. Follow-up action: Added expiration jitter, proactive refresh windows, and lock-contention alerts.

SR-2022-008majorResolvedScenario record

Identity, Security & Storage

Managed storage migration window extension

A planned nonessential storage migration ran longer than expected. Existing public assets remained available while new background copies queued safely.

Resolution time

Dec 3, 2022, 12:42 PM UTC

5h 12m

Impact

A planned nonessential storage migration ran longer than expected. Existing public assets remained available while new background copies queued safely.

Root cause

Checksum verification took longer on several large media objects than the migration estimate allowed.

Resolution

The migration window was extended, checksums completed, and queued copies finalized without data loss.

Corrective action

Improved migration sizing with object-level throughput sampling and added automatic window-extension notices.

Affected components

Storage Object URLsManaged File StoragePublic CDN Delivery

Response timeline

Investigating · Dec 3, 2022, 7:30 AM UTC

Operations began reviewing identity, security & storage telemetry and affected processing queues.

Identified · Dec 3, 2022, 8:32 AM UTC

Checksum verification took longer on several large media objects than the migration estimate allowed.

Monitoring · Dec 3, 2022, 11:08 AM UTC

The migration window was extended, checksums completed, and queued copies finalized without data loss. Teams monitored recovery indicators before closing the report.

Resolved · Dec 3, 2022, 12:42 PM UTC

Normal performance was confirmed. Follow-up action: Improved migration sizing with object-level throughput sampling and added automatic window-extension notices.

SR-2022-007minorResolvedScenario record

Third Party and Partners

ISRC lookup integration rate limit

Bulk ISRC enrichment slowed temporarily while direct release processing continued.

Resolution time

Oct 18, 2022, 5:25 PM UTC

1h 23m

Impact

Bulk ISRC enrichment slowed temporarily while direct release processing continued.

Root cause

A batch enrichment task exceeded the conservative external lookup budget.

Resolution

The task was throttled and remaining identifiers were processed within the available request budget.

Corrective action

Added adaptive rate control, request deduplication, and cached reuse of recent identifier results.

Affected components

ISRC ValidationMetadata Partner Rate LimitMetadata Partner Lookup

Response timeline

Investigating · Oct 18, 2022, 4:02 PM UTC

Operations began reviewing third party and partners telemetry and affected processing queues.

Identified · Oct 18, 2022, 4:19 PM UTC

A batch enrichment task exceeded the conservative external lookup budget.

Monitoring · Oct 18, 2022, 5:00 PM UTC

The task was throttled and remaining identifiers were processed within the available request budget. Teams monitored recovery indicators before closing the report.

Resolved · Oct 18, 2022, 5:25 PM UTC

Normal performance was confirmed. Follow-up action: Added adaptive rate control, request deduplication, and cached reuse of recent identifier results.

SR-2022-006majorResolvedScenario record

Rights, Royalties & Finance

Royalty territory report processing delay

Territory-level report exports completed later than normal. Source statements and totals remained unchanged.

Resolution time

Sep 1, 2022, 7:51 AM UTC

2h 7m

Impact

Territory-level report exports completed later than normal. Source statements and totals remained unchanged.

Root cause

A reporting query generated an inefficient plan after a statistics refresh.

Resolution

The query plan was stabilized and delayed reports were regenerated.

Corrective action

Added plan regression checks and pinned tested query strategies for large territory datasets.

Affected components

Territory Earnings BreakdownFinancial Reporting Export

Response timeline

Investigating · Sep 1, 2022, 5:44 AM UTC

Operations began reviewing rights, royalties & finance telemetry and affected processing queues.

Identified · Sep 1, 2022, 6:09 AM UTC

A reporting query generated an inefficient plan after a statistics refresh.

Monitoring · Sep 1, 2022, 7:13 AM UTC

The query plan was stabilized and delayed reports were regenerated. Teams monitored recovery indicators before closing the report.

Resolved · Sep 1, 2022, 7:51 AM UTC

Normal performance was confirmed. Follow-up action: Added plan regression checks and pinned tested query strategies for large territory datasets.

SR-2022-005minorResolvedScenario record

Audio & Media Processing

Artwork CDN propagation delay

New artwork was accepted but took longer to appear at a subset of edge locations.

Resolution time

Jul 24, 2022, 8:02 PM UTC

46 minutes

Impact

New artwork was accepted but took longer to appear at a subset of edge locations.

Root cause

One cache region delayed invalidation after an artwork replacement.

Resolution

The affected cache region was purged and the current artwork propagated globally.

Corrective action

Added regional propagation checks and automatic re-purge when edge versions disagree.

Affected components

Artwork IntakeContent Delivery CachePublic CDN Delivery

Response timeline

Investigating · Jul 24, 2022, 7:16 PM UTC

Operations began reviewing audio & media processing telemetry and affected processing queues.

Identified · Jul 24, 2022, 7:25 PM UTC

One cache region delayed invalidation after an artwork replacement.

Monitoring · Jul 24, 2022, 7:48 PM UTC

The affected cache region was purged and the current artwork propagated globally. Teams monitored recovery indicators before closing the report.

Resolved · Jul 24, 2022, 8:02 PM UTC

Normal performance was confirmed. Follow-up action: Added regional propagation checks and automatic re-purge when edge versions disagree.

SR-2022-004minorResolvedScenario record

Catalog & Release Operations

Duplicate ingestion job detection delay

A repeated upload remained queued for review longer than expected. No duplicate release record was published.

Resolution time

Jun 10, 2022, 12:09 PM UTC

1h 31m

Impact

A repeated upload remained queued for review longer than expected. No duplicate release record was published.

Root cause

The deduplication fingerprint was generated after the job entered the parsing queue instead of before enqueue.

Resolution

The job was safely cancelled and fingerprint generation was moved ahead of queue insertion.

Corrective action

Added prequeue duplicate detection and explicit duplicate-job status messages.

Affected components

UPC DeduplicationISRC DeduplicationRelease Row Grouping

Response timeline

Investigating · Jun 10, 2022, 10:38 AM UTC

Operations began reviewing catalog & release operations telemetry and affected processing queues.

Identified · Jun 10, 2022, 10:56 AM UTC

The deduplication fingerprint was generated after the job entered the parsing queue instead of before enqueue.

Monitoring · Jun 10, 2022, 11:42 AM UTC

The job was safely cancelled and fingerprint generation was moved ahead of queue insertion. Teams monitored recovery indicators before closing the report.

Resolved · Jun 10, 2022, 12:09 PM UTC

Normal performance was confirmed. Follow-up action: Added prequeue duplicate detection and explicit duplicate-job status messages.

SR-2022-003minorResolvedScenario record

Core Platform

Release detail cache refresh delay

Recently edited release details could require a manual refresh to display the latest values. Saved data remained current.

Resolution time

Apr 29, 2022, 12:03 AM UTC

58 minutes

Impact

Recently edited release details could require a manual refresh to display the latest values. Saved data remained current.

Root cause

A cache invalidation event was dropped during a brief client reconnection.

Resolution

The affected query keys were invalidated and detail pages refreshed from source records.

Corrective action

Added reconnect-time invalidation replay and stale-data indicators for release detail views.

Affected components

Release Detail ServiceReact Query Cache Layer

Response timeline

Investigating · Apr 28, 2022, 11:05 PM UTC

Operations began reviewing core platform telemetry and affected processing queues.

Identified · Apr 28, 2022, 11:17 PM UTC

A cache invalidation event was dropped during a brief client reconnection.

Monitoring · Apr 28, 2022, 11:46 PM UTC

The affected query keys were invalidated and detail pages refreshed from source records. Teams monitored recovery indicators before closing the report.

Resolved · Apr 29, 2022, 12:03 AM UTC

Normal performance was confirmed. Follow-up action: Added reconnect-time invalidation replay and stale-data indicators for release detail views.

SR-2022-002majorResolvedScenario record

Catalog & Release Operations

Release status derivation backlog

Some release summaries displayed Pending longer than expected after partner delivery receipts arrived.

Resolution time

Mar 12, 2022, 4:43 PM UTC

2h 53m

Impact

Some release summaries displayed Pending longer than expected after partner delivery receipts arrived.

Root cause

Status-derivation jobs were waiting behind a large catalog normalization batch.

Resolution

Status jobs were moved to a priority queue and affected release summaries were recalculated.

Corrective action

Separated delivery-state work from bulk catalog normalization and reserved capacity for receipt-driven updates.

Affected components

Release Status DerivationDelivery Status EnginePlatform Status Reporting

Response timeline

Investigating · Mar 12, 2022, 1:50 PM UTC

Operations began reviewing catalog & release operations telemetry and affected processing queues.

Identified · Mar 12, 2022, 2:25 PM UTC

Status-derivation jobs were waiting behind a large catalog normalization batch.

Monitoring · Mar 12, 2022, 3:51 PM UTC

Status jobs were moved to a priority queue and affected release summaries were recalculated. Teams monitored recovery indicators before closing the report.

Resolved · Mar 12, 2022, 4:43 PM UTC

Normal performance was confirmed. Follow-up action: Separated delivery-state work from bulk catalog normalization and reserved capacity for receipt-driven updates.

SR-2022-001minorResolvedScenario record

Support & Communications

Public service notice publishing delay

A routine maintenance notice appeared later than its scheduled publication time. Service health data remained accessible.

Resolution time

Jan 20, 2022, 3:22 AM UTC

1h 8m

Impact

A routine maintenance notice appeared later than its scheduled publication time. Service health data remained accessible.

Root cause

The notice scheduler used an incorrect timezone offset for one publication rule.

Resolution

The schedule was corrected and the notice was published with accurate timing.

Corrective action

Standardized schedules on UTC storage with explicit display-zone conversion and prepublication checks.

Affected components

Public Service NoticesMaintenance NoticesStatus Page

Response timeline

Investigating · Jan 20, 2022, 2:14 AM UTC

Operations began reviewing support & communications telemetry and affected processing queues.

Identified · Jan 20, 2022, 2:28 AM UTC

The notice scheduler used an incorrect timezone offset for one publication rule.

Monitoring · Jan 20, 2022, 3:02 AM UTC

The schedule was corrected and the notice was published with accurate timing. Teams monitored recovery indicators before closing the report.

Resolved · Jan 20, 2022, 3:22 AM UTC

Normal performance was confirmed. Follow-up action: Standardized schedules on UTC storage with explicit display-zone conversion and prepublication checks.

Incident history

Completed maintenance windows

4 maintenance windows that have concluded, retained here as part of the status history. Affected services have returned to their normal operating state.

AppleCompleted

Apple Podcasts Connect and Analytics maintenance

Apple Podcasts Connect and Apple Podcasts Analytics are within an extended scheduled-maintenance period. Apple indicates normal operation is expected to be restored on August 8, 2026.

Apple Podcasts ConnectApple Podcasts Connect Analytics
Online
Starts
Jul 30, 2026, 2:00 PM PDT
Jul 30, 2026, 5:00 PM EDT
Expected restoration
Aug 8, 2026, 8:00 AM PDT
Aug 8, 2026, 11:00 AM EDT
Maximum expected duration
Up to 210 hours

Apple timeline

Maintenance progression

Maintenance begins

Jul 30, 2026, 2:00 PM PDT · Jul 30, 2026, 5:00 PM EDT

Apple begins the scheduled maintenance window. Apple Podcasts Connect and Analytics access may become unavailable.

Access interruption expected

Jul 30, 2026, 2:05 PM PDT · Jul 30, 2026, 5:05 PM EDT

Podcast creators may be unable to sign in to Apple Podcasts Connect or view Analytics during the maintenance period.

Mid-window monitoring

Jul 30, 2026, 3:00 PM PDT · Jul 30, 2026, 6:00 PM EDT

SSO continues monitoring Apple service availability and partner delivery acknowledgements while maintenance is in progress.

Apple published maintenance window begins

Aug 8, 2026, 6:00 AM PDT · Aug 8, 2026, 9:00 AM EDT

Apple’s published August 8 maintenance window begins at 6:00 a.m. PDT. Connect and Analytics access may be unavailable for up to two hours.

Normal operation expected

Aug 8, 2026, 8:00 AM PDT · Aug 8, 2026, 11:00 AM EDT

Apple indicates normal Apple Podcasts Connect and Analytics operation is expected to be restored by August 8 at approximately 8:00 a.m. PDT.

AppleCompleted

iTunes Connect maintenance for Apple Music and Apple Books

The iTunes Connect publishing portal used for Apple Music and Apple Books is within an extended scheduled-maintenance period. Transporter deliveries are expected to remain available, although processing delays may occur. Apple indicates normal operation is expected to be restored on August 8, 2026.

Apple iTunes Connect — Music PublishingApple iTunes Connect — Books Publishing
Online
Starts
Jul 30, 2026, 2:00 PM PDT
Jul 30, 2026, 5:00 PM EDT
Expected restoration
Aug 8, 2026, 8:00 AM PDT
Aug 8, 2026, 11:00 AM EDT
Maximum expected duration
Up to 210 hours

Apple timeline

Maintenance progression

Publishing portal maintenance begins

Jul 30, 2026, 2:00 PM PDT · Jul 30, 2026, 5:00 PM EDT

Apple begins maintenance affecting iTunes Connect access for Apple Music and Apple Books publishing workflows.

Connect access unavailable

Jul 30, 2026, 2:05 PM PDT · Jul 30, 2026, 5:05 PM EDT

Users may be unable to access the iTunes Connect publishing portal or complete portal-based changes during the window.

Transporter remains available

Jul 30, 2026, 2:10 PM PDT · Jul 30, 2026, 5:10 PM EDT

New music and book deliveries sent through Transporter are expected to continue, but Apple may process them more slowly until maintenance completes.

Delivery processing monitored

Jul 30, 2026, 3:00 PM PDT · Jul 30, 2026, 6:00 PM EDT

SSO monitors Apple Music and Apple Books delivery receipts, processing acknowledgements, and any downstream delays.

Apple published maintenance window begins

Aug 8, 2026, 6:00 AM PDT · Aug 8, 2026, 9:00 AM EDT

Apple’s published August 8 maintenance window begins at 6:00 a.m. PDT for iTunes Connect publishing services used by Apple Music and Apple Books.

Normal operation expected

Aug 8, 2026, 8:00 AM PDT · Aug 8, 2026, 11:00 AM EDT

Apple indicates normal iTunes Connect access and publishing-portal processing are expected to be restored by August 8 at approximately 8:00 a.m. PDT.

SSOCompleted

SSO Gateway Analytics services: limited performance

SSO Gateway analytics services are operating in a limited state while SSO engineers and developers complete backend and server updates related to new integrations, server rehauls, and merges to new server grids. Related UI work has been completed; the limited performance is scoped to analytics endpoints and reporting surfaces.

Release Search IndexArtist Search IndexUPC Search IndexISRC Search IndexDistributor Search IndexPartial Match SearchCatalog Activity ReportingDelivery Activity ReportingPlatform Status ReportingUpload Queue ReportingIngestion Results ReportingRecent Uploads FeedAdmin Audit FeedFull Audit Log ReportingIncident ReportingStatus Component SearchOperational SummaryAffected Service Counts
Online
Starts
Aug 2, 2026, 5:30 PM PDT
Aug 2, 2026, 8:30 PM EDT
Expected restoration
Aug 13, 2026, 9:00 PM PDT
Aug 14, 2026, 12:00 AM EDT
Maximum expected duration
Up to 268 hours

Apple timeline

Maintenance progression

Limited-performance window begins

Aug 2, 2026, 5:30 PM PDT · Aug 2, 2026, 8:30 PM EDT

SSO Gateway analytics endpoints move into a limited-performance state as backend and server-grid work begins.

UI work complete

Aug 2, 2026, 6:00 PM PDT · Aug 2, 2026, 9:00 PM EDT

Gateway UI updates related to this maintenance have been completed; remaining impact is limited to analytics services.

Backend and server rehaul underway

Aug 6, 2026, 5:00 PM PDT · Aug 6, 2026, 8:00 PM EDT

Engineering continues backend updates, new integration wiring, server rehauls, and merges to the new server grids.

Final verification

Aug 13, 2026, 5:00 AM PDT · Aug 13, 2026, 8:00 AM EDT

SSO engineers verify analytics endpoints against the rebuilt server grids ahead of restoration.

Full restoration expected

Aug 13, 2026, 9:00 PM PDT · Aug 14, 2026, 12:00 AM EDT

SSO Gateway analytics services are expected to return to normal performance as the maintenance window closes.

SSOCompleted

SSO Gateway platform maintenance: music videos, podcasts, advanced DDEX tooling

SSO Gateway platform services outside of music delivery are in scheduled maintenance while SSO engineers and developers complete major updates. UI work has been completed; maintenance covers music videos, podcasts, and advanced DDEX services (DDEX file tools and DDEX file ingestion via PATH and APIs from SSO Digital Solutions subscriptions) as backend and server updates, new integrations, server rehauls, and merges to new server grids are performed.

SSO Gateway APIGateway SynchronizationMusic Video IntakeVideo File ValidationMotion Artwork ProcessingApple Podcasts ConnectApple Podcasts Connect Analytics
Online
Starts
Aug 2, 2026, 5:30 PM PDT
Aug 2, 2026, 8:30 PM EDT
Expected restoration
Aug 13, 2026, 9:00 PM PDT
Aug 14, 2026, 12:00 AM EDT
Maximum expected duration
Up to 268 hours

Apple timeline

Maintenance progression

Maintenance begins

Aug 2, 2026, 5:30 PM PDT · Aug 2, 2026, 8:30 PM EDT

SSO Gateway platform maintenance begins for music videos, podcasts, and advanced DDEX tooling. Music delivery is excluded and continues operating.

UI updates completed

Aug 2, 2026, 6:00 PM PDT · Aug 2, 2026, 9:00 PM EDT

UI updates tied to this maintenance have been completed; remaining work is backend-only.

Backend and server-grid work continues

Aug 6, 2026, 5:00 PM PDT · Aug 6, 2026, 8:00 PM EDT

SSO engineers continue backend updates, new integration wiring, server rehauls, and merges to the new server grids.

Verification phase

Aug 13, 2026, 5:00 AM PDT · Aug 13, 2026, 8:00 AM EDT

Engineering verifies DDEX file tools, PATH/API ingestion, music video, and podcast workflows against the rebuilt infrastructure.

Restoration expected

Aug 13, 2026, 9:00 PM PDT · Aug 14, 2026, 12:00 AM EDT

SSO Gateway platform services are expected to return to normal operation as the maintenance window closes.

450+ Global Platforms

DDEX Delivery Endpoints

Representative streaming, social, retail, metadata, fitness, and commercial-music destinations monitored across the delivery network.

View complete endpoint directory
SpotifyApple MusiciTunes StoreAmazon MusicYouTube MusicYouTube Content IDTikTokTikTok Commercial Music LibraryDeezerTIDALPandoraSoundCloudAudiomackAnghamiBoomplayJioSaavnGaanaHungama MusicKKBOXLINE MUSICAWARakuten MusicRecoChokuTencent MusicQQ MusicKugou MusicKuwo MusicNetEase Cloud MusicMelonGenie MusicFLOVIBEBugs!Yandex MusicVK MusicZvukQobuzNapsteriHeartRadioClaro MúsicaJOOXTREBELBeatportBeatsourceTraxsource7digitalBandcampSoundtrack Your BrandTouchTunesSiriusXMPelotonMeta MusicFacebookInstagramSnapchatCapCutShazamGracenoteMusixmatchMixcloudTwitchVevo