iOS 27 On-Device AI & Apple Intelligence: Foundation Models and Beyond

Foundation Models is a new Apple framework introduced in iOS 27 that gives developers on-device access to the same Apple Intelligence language model powering system features, enabling text generation, structured output, and tool-calling entirely on-device without a network connection.

iOS 27 opens the Foundation Models framework to third-party LLM providers via a new public LanguageModel protocol, enabling anyone to integrate custom, server-based, or open-source models using the same Swift API as Apple's on-device system model.

Visual Intelligence brings iOS 17's Visual Look Up capabilities to a new developer-facing API surface in iOS 27, letting apps pipe live camera frames or static images through on-device scene understanding to extract subjects, text, barcodes, and rich semantic labels without any cloud round-trip.

App Schemas let developers describe their app's content and actions using pre-defined domain schemas (like the Calendar domain) so Siri can understand, search, and act on app data without custom NLP. Entities conforming to IndexedEntity are donated to Spotlight's semantic index, enabling natural-language queries over app content.

Core AI is a new iOS 27 framework that lets developers bring their own on-device AI models (vision transformers, LLMs, etc.) directly into apps, using the familiar FoundationModels session API. Models run entirely on-device with no cloud dependency, no per-token cost, and no data leaving the device.

SpotlightSearchTool is a new Foundation Models tool that lets a language model directly query your app's Core Spotlight index, enabling conversational, context-aware search over your app's own donated content. Developers can drop it into any LanguageModelSession to get AI-generated answers grounded in app data.

A new Apple framework for measuring the quality and reliability of intelligent features powered by generative AI. It integrates with Swift Testing to let developers define datasets, metrics, and optimization targets that automatically assess probabilistic outputs from language models.

App Schemas let developers map their app's entities and actions to predefined Siri-understandable structures, grouped into App Schema Domains (e.g. messages, photos, mail), so Siri can find content, answer questions, and execute actions in natural language without any custom NLU code.

iOS 27 introduces security patterns and APIs for protecting agentic features built with Foundation Models and App Intents, including lifecycle event modifiers to inject deterministic security checkpoints into LLM-driven agent execution loops.

Dynamic profiles let you switch language models, instructions, and tools within a single LanguageModelSession, enabling multi-agent orchestration patterns like baton-pass and phone-a-friend directly in your app. Combined with session properties and lifecycle modifiers, you can build context-aware, multi-stage AI workflows on-device and in the cloud.

Visual Intelligence now lets apps register as image search providers via App Intents, returning matched entities when users highlight and search images. Introduced in iOS 26 and now expanded to iPadOS and macOS in iOS 27, apps can surface rich, ranked results directly in the Visual Intelligence UI.

Core AI is Apple's new on-device inference framework for iOS 27, giving developers direct access to the same high-performance ML execution engine that powers Apple Intelligence. It supports CPU, GPU, and Neural Engine across all Apple Silicon devices.

iOS 27 opens the Foundation Models framework to third-party LLM providers via a new public LanguageModel protocol, allowing anyone to integrate server-based or local models (e.g. Claude, Gemini, CoreAI, MLX) using the same API as Apple's on-device system model.

macOS 27 ships a pre-installed `fm` command-line tool and a pip-installable Python SDK that expose the on-device Apple Foundation Model outside of Swift apps, enabling shell scripts, automation pipelines, and Python-based ML evaluation workflows.

The Evaluations framework in Xcode 27 lets developers generate and validate synthetic test data at scale using the SampleGenerator API, then run robust evaluations against agentic workflows that involve tool calling — all from Swift code.

iOS 27 expands App Intents to deeply integrate with Siri's on-device language model, enabling natural-language invocation of app actions with richer parameter resolution and multi-step task chaining. Apps can now expose capabilities that Siri understands contextually without rigid phrase matching.

iOS 27 extends App Intents with new APIs for custom Siri dialog responses, interaction donations, OwnershipProvidingEntity for smart confirmations, and IndexedEntityQuery for semantic Spotlight indexing — enabling richer, more personal Siri and Apple Intelligence experiences.