What’s New in App Intents in 2024

Here's my breakdown of the session "What's new in App Intents" from Apple's WWDC'24, covering the new Spotlight features, Entities and Files integrations, Universal Links, and developer improvements.

While watching the App Intents sessions from WWDC’24, I wanted to summarize the contents of each video in a blog post – starting with “What’s new in App Intents,” available from the Apple Developer website and app, or on YouTube:

Here’s the description from Apple (with line breaks added for readability):

Learn about improvements and refinements to App Intents, and discover how this framework can help you expose your app’s functionality to Siri and all-new features.

We’ll show you how to make your entities more meaningful to the platform with the Transferable API, File Representations, new IntentFile APIs, and Spotlight Indexing, opening up powerful functionality in Siri and the Shortcuts app.

Empower your intents to take people deep into your app with URL Representable Entities.

Explore new techniques to model your entities and intents with new APIs for error handling and union values.

Chapters are as follows:

Before I begin, this video also also assumes you’re already aware of the basics of the App Intents API, building off the main sessions from 2022 and 2023 (which you should watch if you haven’t yet).

In many ways, the extended lifespan of this API is why Apple has released other new videos to explain why you should add App Intents and how to design them (which I’ll cover in future posts) in an updated context for 2024.

Table of contents:

Introduction

In reintroducing App Intents in 2024, developer Kenny York starts off the video emphasizing the variety of experience App Intents already powers, including Shortcuts, Spotlight, widgets, the Action button, and the new Apple Pencil squeeze (introduced in May 2024).

This year, those experiences now extend to includes Apple Intelligence and the new Controls experience in Control Center (both covered in future posts).

The App Intents framework itself is also expanding beyond intents and App Shortcuts to emphasize entities, helping the system understand and use them directly when invoked. And finally, developer improvements are designed at making it easier to make App Intents, so they’re not as hard to implement now that developers will be building a lot more.

Spotlight

By default, opening Spotlight on your device already creates a rich experience showing content you might want to see, from suggested apps and actions to a daily summary or recent searches.

Spotlight also helps you search for specific content inside an app (including a new preview experience that I’ve yet to discover how to design on your own).

Last year, Apple improved this experience by allowing developers to create App Shortcuts specifically for the Spotlight experience (as covered in their “Explore enhancement to App Intents” video) and emphasize the most important actions (and entities) from their apps.

IndexedEntity protocol

In order to take this even further, Apple has introduced a new IndexedEntity protocol that lets you index the entities from an app, making it available to the new powerful semantic search capabilities of your device new in iOS 18.

With this protocol, developers can create an index of all the items (or entities) from their app, giving each item a set of attributes, (including keywords), and even assign them priority to match features like favorites lists.

Then, by handing it off to Spotlight when the app is launched or updated, all that data is indexed and searchable, plus more easily matched when queried using natural language.

Entities and Files

In order to add an extra layer of meaning beyond the App Entity and attributes, Apple is allowing developers to declare how their entities can be converted into new formats, which formats their App Intents can accept, and also securely provide files themselves in lieu of direct access.

Transferable AppEntity

Another issue with expanding the AppEntity protocol and making all the items in your apps accessible via AppIntents is what ends up being a format issue – how does data from one app properly convert into the right data for another app? In their example, a trail entity from their app could become a PDF in another when passed via App Intents.

To help with this, Apple now allows developers to extend their entities with Transferable, an API introduced in 2022 designed to convert formats from one type to another.

With Transferable, developers can actually specific which types of data they want their entity to convert to, including fallback options sorted by priority.

For anyone with a long history of Shortcuts and its original independent version Workflow, this acts like a modern-day version of the Content Graph engine, which is what powered Workflows inputs/outputs to intelligently convert data types when needed. Now, it’s built into App Intents, powered by a Swift API, and developers have control over which data types it gets converted into and when.

Improvements to IntentFile

On the flip side, when an App Intent is designed to receive an entity, developers can specify which of these content types it can support, what’s available from the other app, and which one to pick.

Plus, when that content type isn’t supported, it can give the app the original URL of the content and pass it to Transferable for conversion.

FileEntity

Finally, Apple has also addressed limitations for document-based apps where the file itself is the entity, and not a representation of an object from your database – in these cases, the FileEntity API is needed to operate on the actual file itself.

In these cases, FileEntity allows Siri and Shortcuts to securely access the files, perform the operation, and pass it along to the next step using an ID instead of the file itself.

With this method, apps can provide information about their files and other apps can operate on those files, but handled properly through App Intents instead of direct access.

Beyond making it easier to meaningfully search for content within your app, Apple is also improving deep linking to those specific entities using Universal Links.

While the slide above is in full developer-speak, what Apple has introduced this year is the ability to generate and open links to any item, page, or action within your app. In essence, every part of every app is now represented by a URL – which makes sense, since it is a universal resource locator.

Now, App Intents allows URL to redirect to everything section, every thing you see, and every action you can take in your app.

In practice, this will mostly be used by developers as the universal method of accessing said item from another action step in Shortcuts, another app when it is requesting Transferable operate on an entity, or specifying to open content after creating it anew.

However, I’m fascinated by the larger implications here – everything in every app is linkable now… I’ll have to do more research to really flesh out what this means for apps that are also web services.

Developer improvements

Now that developers can use App Intents to index all of their entities, make them meaningful to Spotlight and Siri, and interlink between any entity, action, or section, there’s a lot of App Intents work to be done by developers.

Thankfully, Apple already designed App Intents from the ground up to make the entire framework functional in code only (compared to the Xcode UI for custom intents), extendable by only requiring you to add onto existing App Intents implementations, and, in most cases, already in-use for features like widgets.

This year, they’re building off the improvements of the last few years in two specific ways – a new UnionValue macro that makes multiple parameters or properties representable under one type, and the ability for Xcode 16.0 to generate title strings automatically.

UnionValue

For developers, UnionValue means they can let users choose between more complex data types in a union under a new, single value – Apple describes it as an “or” parameter.

Generated titles

And for developers creating lots of App Intents can avoid duplicating unnecessary work in designing titles for their intents that are already implied in the struct’s property – instead of adding the title “Location” to a location, Xcode can simply figure it out. And when developers still want that control to specify a unique title, they can add it back in.

Framework improvements

Finally, in order to make App Intents work better with the new changes to App Entities, Apple has removed limitations on the interaction between the two and lets you define app entities in a framework and reference them from your app and extension targets.

However, it should be noted that Apple hasn’t expanded this to libraries outside of a framework – for now, libraries are not supported.

Summary

Apple also explicitly mentioned that not all the changes to App Intents are covered in this video, as some changes are directly relevant to other sessions on Apple Intelligence and Controls, and those changes are only covered in those sessions.

Here’s the full list from the graphic above for any developers searching through the App Intents documentation for everything new:

  • IndexedEntity
  • Prebuilt errors
  • Union types /
  • UniqueEntity
  • SetValuelntent
  • ControlConfigurationIntent
  • CameraCapturelntent
  • URLRepresentableEntity
  • URLRepresentableEnum
  • URLRepresentablelntent
  • AudioRecordingIntent
  • Intent confirmation
  • FileEntity
  • Assistantintent
  • AssistantEntity
  • AssistantEnum
  • Flexible default parameter values

Wrap-up

This year, Apple has expanded the App Intents framework to act as an all-encompassing tool from defining intents and building App Shortcuts on top of them to fully indexing all the entities from your apps, making them meaningful, linkable, and transferable, and in general making it easier to develop, integrate, and maintain a balanced App Intents integration with less effort or roadblocks.

Developers
For developers, this means making your app, its objects, capabilities, and interface accessible to the rest of the ecosystem in a thoughtful and secure way.

Users
For users, this means all your apps have added a layer that Apple Intelligence can understand, you can control from within the Shortcuts app, and will continue to be developer against for years to come.

Apple
For Apple, this means that Siri and your devices can finally understand what your apps can do, what you’ve created with them, and the connections between it all – and in a private way that doesn’t allow apps, users, or even themselves to abuse that access.

In many ways, App Intents is a new operating system for apps themselves, letting them talk to the system, tell it what’s possible, and be ready for anything when the time comes.

Apple is also pushing App Intents in a huge way – I’ll cover it in the next session, but they’re declaring “Everything in your app should be an App Intent.” That means this is the API for how your apps work now, and will be going forward.

For most Apple users, they’ll simply experience App Intents by way of better Spotlight, the enhanced Siri integrations, and via features like widgets or the Action button, never having heard of this technology powering it all.

But behind the scenes, App Intents will be working hard to make sure everything functions properly – Siri will have gotten better “overnight,” while Apple has been building up App Intents for years to get us to here.

Developers, are you looking for help with your App Intents implementation?

I’m now available for consulting – email me.

Posts You Might Like