1---2name: visual-intelligence3description: Integrate your app with iOS Visual Intelligence for camera-based search and object recognition. Use when…4---56# skills/apple-intelligence/visual-intelligence/SKILL.md
V
SKILL.mdAgent skill
Transport
File
Status
Listed
SKILL.md
Install
The skill file stays in the publisher's repository.
Paste the copied command in a terminal. It installs this SKILL.md into that agent.
shterminal
1npx skills add rshankras/claude-code-apple-skills --skill visual-intelligence -a cursor -y
Paste the copied command in a terminal. It installs this SKILL.md into that agent.
Download skills/apple-intelligence/visual-intelligence/SKILL.md from the publisher, or copy the path and open the repository. PlugMCP lists the skill and does not host the file.
Where does visual-intelligence live?
The publisher keeps it at https://github.com/rshankras/claude-code-apple-skills.
Which agents can use visual-intelligence?
Claude, Cursor, Codex, and Grok Build can read a SKILL.md you add from the publisher's repository. Grok Bot installs it when you send the skill's GitHub folder link in a chat.
From the publisher
SKILL.md
name
visual-intelligence
description
Integrate your app with iOS Visual Intelligence for camera-based search and object recognition. Use when adding visual search capabilities.
Integrate your app with iOS Visual Intelligence to let users find app content by pointing their camera at objects.
When This Skill Activates
User wants camera-based search in their app
User asks about visual search integration
User wants to surface app content in system searches
User needs to handle visual intelligence queries
Overview
Visual Intelligence lets users:
Point camera at objects or use screenshots
System identifies what they're looking at
Your app provides matching content
Results appear in system UI
Your app implements:
IntentValueQuery to receive search requests
AppEntity types for searchable content
Display representations for results
Platform Availability (WWDC26 297)
Visual Intelligence runs on iOS, iPadOS, and macOS — the same entities, query, and OpenIntent code works unchanged on all three. Handle both camera captures of physical objects (iOS) and screenshots of digital media (iPad/Mac) as input.
On Mac, the input pixel buffer can be much larger than what you'd encounter on iPhone — consider whether resizing is necessary before matching.
Quick Start
1. Import Frameworks
import VisualIntelligence
import AppIntents
2. Create App Entity
struct ProductEntity: AppEntity {
var id: String
var name: String
var price: String
var imageName: String
static var typeDisplayRepresentation: TypeDisplayRepresentation {
TypeDisplayRepresentation(
name: LocalizedStringResource("Product"),
numericFormat: "\(placeholder: .int) products"
)
}
var displayRepresentation: DisplayRepresentation {
DisplayRepresentation(
title: "\(name)",
subtitle: "\(price)",
image: .init(named: imageName)
)
}
// Deep link URL
var appLinkURL: URL? {
URL(string: "myapp://product/\(id)")
}
}
3. Create Intent Value Query
struct ProductIntentValueQuery: IntentValueQuery {
func values(for input: SemanticContentDescriptor) async throws -> [ProductEntity] {
// Search using labels
if !input.labels.isEmpty {
return await searchProducts(matching: input.labels)
}
// Search using image
if let pixelBuffer = input.pixelBuffer {
return await searchProducts(from: pixelBuffer)
}
return []
}
private func searchProducts(matching labels: [String]) async -> [ProductEntity] {
// Search your database using provided labels
// Return matching products
}
private func searchProducts(from pixelBuffer: CVReadOnlyPixelBuffer) async -> [ProductEntity] {
// Use image recognition on the pixel buffer
// Return matching products
}
}
SemanticContentDescriptor
The system provides this object with information about what the user is looking at.
Properties
Property
Type
Description
labels
[String]
Classification labels from Visual Intelligence
pixelBuffer
CVReadOnlyPixelBuffer?
Raw image data
Usage Patterns
Label-based Search:
func values(for input: SemanticContentDescriptor) async throws -> [ProductEntity] {
// Labels like "shoe", "sneaker", "Nike" etc.
let labels = input.labels
// Search your content using these labels
return products.filter { product in
labels.contains { label in
product.tags.contains(label.lowercased())
}
}
}
Image-based Search:
func values(for input: SemanticContentDescriptor) async throws -> [ProductEntity] {
guard let pixelBuffer = input.pixelBuffer else {
return []
}
// Convert to CGImage for processing
let ciImage = CIImage(cvPixelBuffer: pixelBuffer)
let context = CIContext()
guard let cgImage = context.createCGImage(ciImage, from: ciImage.extent) else {
return []
}
// Use your ML model or image matching logic
return await imageSearch.findMatches(for: cgImage)
}
Multiple Result Types
Use @UnionValue when your app has different content types.
Rules (WWDC26 297):
An app can have only ONE IntentValueQuery that accepts a SemanticContentDescriptor. All result types must flow through that single query — a @UnionValue enum with one case per entity type.
Every entity type in the union needs its own OpenIntent — without one, results of that type can't appear in image search.
Think beyond pixel matching: an album matched by image similarity can also surface the artist's nearby concerts — be creative about the type of content you return based on the context.
@UnionValue
enum SearchResult {
case product(ProductEntity)
case category(CategoryEntity)
case store(StoreEntity)
}
struct VisualSearchQuery: IntentValueQuery {
func values(for input: SemanticContentDescriptor) async throws -> [SearchResult] {
var results: [SearchResult] = []
// Search products
let products = await productSearch(input.labels)
results.append(contentsOf: products.map { .product($0) })
// Search categories
let categories = await categorySearch(input.labels)
results.append(contentsOf: categories.map { .category($0) })
return results
}
}
Display Representations
Create compelling visual representations for search results.
Result Card Real Estate (WWDC26 297)
The search-result card gives about three lines of text for a title and subtitle, plus a thumbnail image — put the most important identifying info there (album name + artist).
With multiple results the sheet uses a two-column layout: if you initialize DisplayRepresentation with an image URL, serve a thumbnail-sized image, not the full-resolution asset — smaller images load faster.
Exception: a single result renders its image at the full width of the results sheet — don't over-shrink for that case.
// ❌ Full-res image URLs in DisplayRepresentation for multi-result responses
// ✅ Thumbnail-sized images (two-column sheet); full-width only when returning one result
Whether you're searching on device or hitting a server, the same principles apply: return results fast and ranked.
Vision-framework pattern:
Pre-computeGenerateImageFeaturePrintRequest feature prints for your catalog; at query time, convert the pixel buffer via VideoToolbox (VTCreateCGImageFromCVPixelBuffer) and generate just one feature print to compare.
Filter with a maximum distance threshold to drop dissimilar results, sort ascending by distance so the best match is first, and cap the result count. Apple's sample signature: search(matching:limit: Int = 10, maxDistance: Double = 1.0).
Return [] when nothing matches or the pixel buffer is absent — the system handles displaying an empty response. ❌ Don't pad with weak matches.
// ❌ Compute feature prints at query time
// ✅ Pre-compute catalog prints; query = 1 print + threshold + sort + limit
let matches = catalogPrints
.map { entry in (entry, entry.print.distance(to: queryPrint)) }
.filter { $0.1 <= maxDistance }
.sorted { $0.1 < $1.1 }
.prefix(limit)
Vision offers more than feature prints for visual search: text extraction, barcode scanning, face detection, image classification (WWDC26 297).
Deep Linking
Enable users to open specific content from search results.
OpenIntent Rules (WWDC26 297)
Tapping a result runs your OpenIntent for that entity type, and its perform() runs as the app comes to the foreground:
Do navigation in perform(); defer heavy loading until after the view appears.
Take people straight to the content they selected — no intermediate screens.
✅ Reuse the OpenIntent from your existing App Intents adoption — you don't need a separate one just for Visual Intelligence. ❌ Duplicate per-feature OpenIntents.
// ❌ Heavy loading inside OpenIntent.perform (runs during foregrounding)
// ✅ Navigate only; load after the view appears; reuse one OpenIntent everywhere
URL-based Deep Links
struct ProductEntity: AppEntity {
// ... other properties
var appLinkURL: URL? {
URL(string: "myapp://product/\(id)")
}
}
Handle in App
@main
struct MyApp: App {
var body: some Scene {
WindowGroup {
ContentView()
.onOpenURL { url in
handleDeepLink(url)
}
}
}
func handleDeepLink(_ url: URL) {
guard url.scheme == "myapp" else { return }
switch url.host {
case "product":
let id = url.lastPathComponent
navigationState.showProduct(id: id)
default:
break
}
}
}
"More Results" Button
Provide access to additional results beyond the initial set.
struct ViewMoreProductsIntent: AppIntent, VisualIntelligenceSearchIntent {
static var title: LocalizedStringResource = "View More Products"
@Parameter(title: "Semantic Content")
var semanticContent: SemanticContentDescriptor
func perform() async throws -> some IntentResult {
// Store search context for your app
SearchContext.shared.currentSearch = semanticContent.labels
// Return empty result - system will open your app
return .result()
}
}
semanticContentSearch Schema (WWDC26 297)
The schema-based form: conform to .visualIntelligence.semanticContentSearch and the system supplies the semanticContent property automatically:
@AppIntent(schema: .visualIntelligence.semanticContentSearch)
struct SemanticContentSearchIntent: AppIntent {
static let openAppWhenRun: Bool = true
var semanticContent: SemanticContentDescriptor
func perform() async throws -> some IntentResult {
let results = try await library.search(matching: semanticContent)
await MainActor.run { AppState.shared.openSearch(with: results) }
return .result()
}
}
Rules (WWDC26 297): pre-populate the in-app search view from the captured context (never a blank screen), and use it to expose what the Visual Intelligence sheet can't — filters, categories, the full depth of your content.
Complete Example
import SwiftUI
import AppIntents
import VisualIntelligence
// MARK: - Entities
struct RecipeEntity: AppEntity {
var id: String
var name: String
var cuisine: String
var prepTime: String
var imageName: String
static var typeDisplayRepresentation: TypeDisplayRepresentation {
TypeDisplayRepresentation(
name: LocalizedStringResource("Recipe"),
numericFormat: "\(placeholder: .int) recipes"
)
}
var displayRepresentation: DisplayRepresentation {
DisplayRepresentation(
title: "\(name)",
subtitle: "\(cuisine) · \(prepTime)",
image: .init(named: imageName)
)
}
var appLinkURL: URL? {
URL(string: "recipes://recipe/\(id)")
}
}
// MARK: - Intent Value Query
struct RecipeVisualSearchQuery: IntentValueQuery {
@Dependency var recipeStore: RecipeStore
func values(for input: SemanticContentDescriptor) async throws -> [RecipeEntity] {
// Use labels to find recipes
// Labels might include: "pasta", "tomato", "Italian", etc.
let matchingRecipes = await recipeStore.search(
ingredients: input.labels,
limit: 15
)
return matchingRecipes.map { recipe in
RecipeEntity(
id: recipe.id,
name: recipe.name,
cuisine: recipe.cuisine,
prepTime: recipe.prepTimeFormatted,
imageName: recipe.thumbnailName
)
}
}
}
// MARK: - More Results Intent
struct ViewMoreRecipesIntent: AppIntent, VisualIntelligenceSearchIntent {
static var title: LocalizedStringResource = "View More Recipes"
@Parameter(title: "Semantic Content")
var semanticContent: SemanticContentDescriptor
func perform() async throws -> some IntentResult {
// Save search context
await MainActor.run {
RecipeSearchState.shared.searchTerms = semanticContent.labels
}
return .result()
}
}
// MARK: - Recipe Store
@Observable
class RecipeStore {
private var recipes: [Recipe] = []
func search(ingredients: [String], limit: Int) async -> [Recipe] {
recipes
.filter { recipe in
ingredients.contains { ingredient in
recipe.ingredients.contains { recipeIngredient in
recipeIngredient.lowercased().contains(ingredient.lowercased())
}
}
}
.prefix(limit)
.map { $0 }
}
}
Two integration directions: your app provides results (everything above), and your app receives data Visual Intelligence writes into shared system stores:
Visual Intelligence system action
Store
Your app reads via
Create calendar events — including multiple events at once
If your app already reads from these stores, Visual Intelligence becomes a source of input automatically — zero VI-specific code. One requirement: observe change notifications so VI-created data appears without a relaunch. EventKit pattern from Apple's sample: requestFullAccessToEvents() → fetch with a predicate (a 90-day window) → observe .EKEventStoreChanged notifications and refetch.
Testing
Build and run on physical device
Open Camera or take screenshot
Activate Visual Intelligence
Point at objects relevant to your app
Verify results appear
Test tapping results opens your app correctly
On iPad and Mac, test with screenshots of digital media — the primary entry point there (WWDC26 297)
Checklist
Import VisualIntelligence and AppIntents
Create AppEntity types for searchable content
Implement IntentValueQuery
Handle both labels and pixelBuffer
Create DisplayRepresentation for each entity
Implement deep linking URLs
Handle URLs in app with onOpenURL
Add "More Results" intent if needed — pre-populated, never a blank search screen
Only one SemanticContentDescriptor-accepting IntentValueQuery in the app (WWDC26 297)
OpenIntent per entity type; navigation only in perform(), heavy loading deferred (WWDC26 297)
Thumbnail-sized display images for multi-result responses (WWDC26 297)
Return [] when nothing matches — no weak-match padding (WWDC26 297)
Feature prints pre-computed, results filtered by max distance and sorted best-first (WWDC26 297)
