Apple’s Game

This week Apple introduced app extensions into both iOS and OSX. When a 3rd party app wants a particular kind of service (such as photo editing), iOS presents the user with a list of other apps on their device that have the desired extension. Once the user picks one, the extension appears right within the 3rd party app so the user can use it without switching out of their current flow. This allows apps to interoperate in a controlled manner without sacrificing security, privacy, and convenience for the user.

One of the interesting things about this is how the underlying mechanisms actually work - the extensions themselves are entirely self-contained apps in their own right. They are walled off from all other apps - including their own parent app for the most part - and are given a limited view of the outside world that mostly only includes the data necessary to do the type of task the extension was designed to fulfill. This means that the extension apps are, essentially, entirely self-contained. As far as users are concerned, their flow is relatively uninterrupted and they’re able to do what they want when they want without iOS standing too much in the way. It should just work.

Apple also announced that AirPlay will now support direct peer-to-peer connections. This means that latency will be much lower and connections should be more reliable. It also means that your iPhone (for example) will not need to be on the same wifi network as your AppleTV in order to use AirPlay. Lifting that constraint won’t matter much for typical home use, but it’s a huge deal in schools and businesses. It also means it’ll be far easier to play a video from your phone on your friend’s AppleTV without needing to ask to join their wifi network - and, indeed, not even needing to *know* you’d need to join a wifi network in the first place! It’ll just work.

Game controllers for iOS have been around awhile now, but they haven’t taken off yet in a big way. The specification has always included both standalone wireless controllers, and “shell” controllers that wrap around an iPhone or iPod. While a number of games support the game controller input API, it’s been pretty spotty and as a result there aren’t a ton of manufacturers of the controllers out there yet. Even with those downsides, Apple announced this week that their game controller API will now transparently forward controller events from one device to another. What this means is that if you already have a shell-style controller for your iPhone, you can now use your iPhone as a dedicated standalone controller to play games that are running on your iPad or Mac and the game itself doesn’t need to know any different. It just works.

Metal is one more thing Apple introduced us to this week. Metal is an extremely thin layer of software that interfaces between apps and the underlying GPUs. It does the same job that OpenGL ES has been doing for us for years, but Metal is optimized for Apple’s own hardware and software needs while also cutting out a bunch of legacy cruft in the process. This means games that use Metal will use less CPU time communicating with the GPU, which leaves more CPU time left over for running the actual game itself.

Each one of these things is great by itself, but I believe that there’s an even bigger plan hidden in plain sight - and it’ll all land this fall.

Apple now has everything they need to disrupt the game console industry in a way that none of them see coming. I predict that we’ll see a new AppleTV update (and hardware) this fall along with a new app extension type for AirPlay. AirPlay will become about more than just streaming video to your AppleTV - instead that’ll simply be one of the things you can do with it. Apps (mostly games, I suspect) will be able to bundle an AirPlay extension inside - just like how apps can now bundle photo editing or sharing extensions as of iOS 8. The key difference is where the AirPlay extension app actually executes - instead of running on your device itself from within another host app, the AirPlay extension app will be automatically uploaded to whatever AppleTV you are currently AirPlaying with and will run directly on the AppleTV natively instead. This means no video streaming lag and minimal controller lag. Your iPhone would then turn into a generic game controller with onscreen controls or, if you have a physical shell controller attached to your iPhone, it activates that instead. The game controller inputs are then relayed to the AppleTV and thus to the AirPlay extension app using the new game controller forwarding feature.

The AirPlay extension API would probably require that the extension communicate with the outside world primarily by way of the existing game controller API. This means that for any game that adopted the AirPlay extension mechanism, they’d also be required to support physical external game controllers by way of Apple’s API which will increase adoption significantly.

Since the game controller specification also defined wireless game controllers, you could cut out the iPhone middleman entirely, too. Any app that is submitted to the App Store that includes an AirPlay extension could also automatically show up on AppleTV in an AppleTV Store. Users without an iPhone, but with a compatible wireless controller, could purchase games directly from that store and play them on their AppleTV using their controller without even owning any other iOS devices and it’d just work.

If that customer buys an iPhone or iPad later, all the games they purchased on their AppleTV would also work on their shiny new iPhone or iPad because Apple would require all games to support normal touch-only controls - just as they have always done even for apps that support the existing game controller APIs. This would be a huge win for Apple, of course, but also for customers - buy the game once and it’s always yours on any device no matter how you want to experience it. There would be no AppleTV-exclusive games in the AppleTV Store - only normal touch-enabled iOS games that happen to also bundle an AirPlay extension. No fragmentation.

Due to AirPlay now supporting peer-to-peer connections, this means that if you bring your iPhone or iPad with you to a friend’s house (or anywhere with an AppleTV such as a hotel room, school, etc), and you have all of your games in your pocket but can play them on the nearby screen if you want. You can play without needing to purchase the game on that particular AppleTV, without needing to sign in with your iCloud account to access your purchases, without needing to get their wifi password, or indeed without there even needing to *be* a wifi network to join in the first place. All without any hassle. When you go home, you take the game and any earned progress along with you in your pocket.

Thanks to Metal, there’s even a chance Apple could enable all of this for the current generation of AppleTV hardware (while also introducing a beefier A7-powered AppleTV). The reason is that while Apple talked a lot about how Metal increases the amount of CPU available for games, that same tradeoff means older hardware could potentially keep up with newer hardware by making the tradeoff in reverse - the CPU might be slower, but since the overhead of talking to the GPU is now lower, you could achieve a similar performance level on older hardware as an OpenGL ES app running on current hardware. And boom - with a single ecosystem software update, Apple will have dropped an entire gaming console ecosystem into millions of living rooms without anyone having to buy anything new.