RealityKit visionOS Developer

Description and Goals

This skill provides comprehensive guidance for implementing RealityKit-based spatial experiences on visionOS. RealityKit uses an Entity Component System (ECS) architecture where entities are lightweight containers, behavior comes from components, and systems drive per-frame updates.

Goals

• Enable developers to build immersive 3D experiences on visionOS using RealityKit
• Provide clear guidance on when to use each component and system
• Help developers understand ECS patterns and best practices
• Support debugging and optimization of RealityKit scenes
• Ensure proper integration with SwiftUI via RealityView

What This Skill Should Do

When implementing RealityKit features on visionOS, this skill should:

1. Guide component selection - Help you choose the right components for rendering, interaction, physics, audio, and animation needs
2. Provide system implementation patterns - Show how to create custom systems for continuous behavior
3. Offer code examples - Demonstrate common patterns like async asset loading, interactive entities, and custom systems
4. Highlight best practices - Emphasize proper async loading, component registration, and performance considerations
5. Warn about pitfalls - Identify common mistakes like using ARView on visionOS or blocking the main actor

Load the appropriate component or system reference file from the tables below for detailed usage, code examples, and best practices.

Information About the Skill

Core Concepts

Entities and Components

• Entities are lightweight containers; behavior comes from components.
• Prefer composition over inheritance and use custom Component + Codable when you need per-entity state.
• Register custom components once with Component.registerComponent() before use.
• Keep entity transforms and component updates on the main actor.

RealityView and Attachments

• Use RealityView to bridge SwiftUI and RealityKit.
• Load assets with Entity(named:) or Entity(contentsOf:) asynchronously and handle errors.
• Always use ViewAttachmentComponent for SwiftUI overlays in 3D and avoid the RealityView attachments closure.

Systems and Queries

• Use a custom System for continuous, per-frame behavior.
• Query entities with EntityQuery + QueryPredicate and process them in update(context:).
• Use SystemDependency to control update order when multiple systems interact.

Components Reference

Use this table to decide which component reference file to load when implementing RealityKit features:

Rendering and Appearance

User Interaction

Anchoring and Spatial

Cameras

Lighting and Shadows

Audio

Animation and Character

Physics and Collision

Portals and Environments

Presentation and UI

Networking and Sync

Systems Reference

Use this reference when implementing custom ECS behavior:

Implementation Patterns

RealityView Async Load

RealityView { content in
    do {
        let entity = try await Entity(named: "Scene")
        content.add(entity)
    } catch {
        print("Failed to load entity: \(error)")
    }
}

Interactive Entity Setup

let entity = ModelEntity(mesh: .generateBox(size: 0.1))
entity.components.set(CollisionComponent(shapes: [.generateBox(size: [0.1, 0.1, 0.1])]))
entity.components.set(InputTargetComponent())
entity.components.set(ManipulationComponent())

Custom System Skeleton

import RealityKit

struct SpinComponent: Component, Codable {
    var speed: Float
}

struct SpinSystem: System {
    static let query = EntityQuery(where: .has(SpinComponent.self))

    init(scene: Scene) {}

    func update(context: SceneUpdateContext) {
        for entity in context.entities(matching: Self.query, updatingSystemWhen: .rendering) {
            guard let spin = entity.components[SpinComponent.self] else { continue }
            entity.transform.rotation *= simd_quatf(angle: spin.speed * Float(context.deltaTime), axis: [0, 1, 0])
        }
    }
}

SpinSystem.registerSystem()

Pitfalls and Checks

• Always load assets asynchronously; avoid blocking the main actor.
• Avoid ARView on visionOS; use RealityView.
• Add CollisionComponent + InputTargetComponent for draggable or tappable entities.
• Never use the RealityView update closure; use a custom System for continuous behavior instead.
• Mesh generation is limited to box, sphere, plane, cylinder, and cone.