Uber Had the Opportunity to Monitor Everything on Your iPhone’s Screen

Daniel Jalkut:

Yesterday, Gizmodo reported that Uber had been granted an entitlement for their iOS app that allowed them to capture an image of an iPhone’s screen at any time, even when the Uber app was not the active app on the phone. This is a big deal, because users don’t typically expect than an iPhone app that is not active might have the ability to eavesdrop on anything they are doing.

I have long felt that the sandboxing infrastructure on both iOS and Mac should be used to more accurately convey to users specifically what the apps they install are capable of doing. Currently the sandboxing system is used primarily to identify to Apple what a specific app’s privileges are. The requested entitlements are used to inform Apple’s decision to approve or reject an app, but the specific list of entitlements is not easily available to users, whose security is actually on the line.

This is absolutely fucking ridiculous. Fuck Uber. Apple should be ashamed for working with them at any level. Allowing an app to covertly record your screen without any prompting is exactly the kind of thing that Apple’s iOS app review process should prevent.

Uber claims they didn’t do anything wrong with this ability, the security researchers told Gizmodo that they didn’t detect anything going on with this code.

There are companies that are less trustworthy than Uber, but few have the opportunity to be as evil on such a large scale. Enabling them to do anything more than operate at a basic level on your platform is a mistake. At this point Apple should block them entirely and attempt to help the Taxi industry to reform and compete with Uber. Not that Apple would ever would, but still that would be the best thing to come out of this. The next best thing would be the improvements to the entitlement system that Jalkut suggests.

I wouldn’t even bother to wonder what Uber are doing on Android, where security is a fucking joke and carriers are still selling devices running ancient versions of that operating system that are affected by dozens of security vulnerabilities. This is especially true for pay-as-you-go phones sold cheaply at places like Walmart, Target, and so on. Those carriers and stores are endangering their customers by continuing to sell these devices.

Apple Special Event September 2017 Notes

Apple held their first iPhone event at their new Steve Jobs Theater on their UFO campus in California today. The event opened with Tim Cook memorializing Jobs. Then he discussed disaster relief projects and encouraged donations. Finally Cook talked about Apple’s new campus. No mentions of offices for the engineers, who have complained that the new campus is almost entirely an open space.

Angela Ahrendts talked about their stores before Cook came back out to start announcing devices and software.

Here’s what they announced.

WatchOS 4
Tim Cook talked about their success then played a video featuring letters from users about how the watch has helped them. Here’s that video:

Jeff Williams reiterated WatchOS 4 updates with better coaching, a redesigned workout app, features for swimmers, and integration with gym equipment. The new OS also gets a new heart rate app and a heart rate complication for the watch faces. The watch will now alert you to an unusually high heart rate when it thinks you’re not active. It’ll also monitor for arrhythmias in a study with Stanford Medicine and the FDA later this year.

WatchOS 4 will be out on September 19th.

Apple Watch Series 3
Here’s their ad for the new Series 3 watch:

It’s a cellular device that takes calls with your regular iPhone’s phone number. Of course it’ll also have data capabilities and will stream music from Apple Music.

There’s a new faster processor and Siri finally speaks on the watch if you want to hear them instead of reading a response. The device also has a barometric altimeter built-in.

The Series 3 Apple Watch retains the form factor of the previous watches. Cellular models of the Series 3 have a red dot on the crown so that you can feel special.

Williams made a very awful joke during this presentation about a potential device with all of this functionality looking like a house arrest bracelet. Not funny.

Obviously Apple is very focused on this being an activity monitoring device. I’m not very active but I still appreciate mine as a device for triaging notifications.

Series 2 watches are getting dropped. The Series 1 will remain in the lineup at $250. The base series 3 watch will be $330 and the Series 3 with cellular capabilities will be $400.

No word on the cost for the cellular plan to go with it, but Williams said they have a special deal worked out with some carriers.

The new Series 3 watches will be available to pre-order on September 15th, they’ll be generally available on September 22nd.

I don’t see much reason to upgrade from my Series 0, but I’m glad they’re continuing to work on improvements.

Apple TV 4K

The Apple TV is one of the most expensive devices you can buy to put under your TV, and it has been a long time since Apple made any changes to the hardware. Eddy Cue’s pitch for this upgrade talked about how the new device also includes support for HDR standards. It also has an A10X Fusion chip, Eddy reminds us that it’s same one as in the iPad Pro. He boasted that it’s twice as fast at general computing tasks — and four times as fast on the GPU side — as the current 1080p 4th generation Apple TV.

Apple’s Aerial screensaver that displays high-resolution footage of interesting places to look at is getting updated with 4K footage.

4K movies will cost the same from Apple’s iTunes store as they currently charge for HD films. Any movies people currently own in HD that have 4K versions will be upgraded to 4K HDR for free.

Cue also said that Netflix and Amazon are on-board for the 4K update. Cue didn’t mention that the Amazon Prime Video app still isn’t available on the Apple TV even though Apple said they had a deal worked out with Amazon earlier this year.

Live news and sports are coming to the TV app on the Apple TV and iOS devices.

That Game Company’s Jenova Chen demoed a new game called Sky for the Apple TV. It’s a fun multiplayer game that has a similar style as their previous games like Journey. It’s temporarily exclusive to the Apple TV and iOS devices and will be out this winter. I didn’t hear anyone mention if the game runs in 4K.

Here’s a trailer for it:

The Apple TV lineup will now be:

  • Old Apple TV 32GB at $150.
  • Apple TV 4K 32GB $180
  • Apple TV 4K 64GB $200

It’ll be up for pre-orders on the 15th and have general availability on the 22nd.

Unfortunately there were no updates to the Apple TV Siri Remote announced, which is a shame because it’s not great for gaming and has some other design issues. Especially regarding picking up the remote and accidentally pressing the wrong button because it’s not easy to tell which orientation you’re holding it in without looking at it.

iPhone 8, 8 Plus

The rumors and leaks were accurate , there were three new iPhones announced today instead of two. An iPhone X, iPhone 8, and iPhone 8 Plus.

Here’s the iPhone 8 ad:

Phil Schiller did his bit to sell the iPhone 8 and 8 Plus. As rumored it has a glass back as well the front. It’ll be in Silver, Space Gray, and Gold. They assure that the glass is reinforced with steel and the “most durable ever” glass.

The display has been updated. It now has the True Tone technology from the iPad Pro that adjusts the display’s color temperature to look correct in different lighting conditions. The iPad Pro still doesn’t have the iPhone’s 3D Touch feature and that makes writing on it a little bit more frustrating when I switch back and forth between my iPhone and iPad.

The iPhone 8 and 8 Plus speakers are louder and more accurate.

There’s a new A11 Bionic chip with six cores inside the iPhone 8. It’s supposedly 25% faster than the A10. Two of the six A11 cores are tuned for performance, and four of the cores are set up for tasks that can run efficiently and don’t require as much performance so that the device can save you some battery life. Those four low-power cores are supposedly 70% faster than the ones in the A10 chip.

The GPU on the 8 is the first one Apple has developed, it’s supposed to be 30% faster than last year’s A10 while taking half the power.

There’s a new image processing engine in the A11 that Schiller says will deliver faster autofocus in low-light conditions among other improvements that should make photos taken with the 8 look sharper and have less noise.

The cameras still shoot 12 megapixels, but Apple says that the sensor is larger and lets in more light. The Plus still has the exclusive on the dual camera setup versus the regular iPhone 8.

Apple’s photos captured with the 8 and 8 Plus’ cameras look fantastic, but they are just demonstrating the difference between an experienced photographer shooting for demonstration purposes and someone taking snapshots. Of course these camera improvements will help with either type of scenario, but they’re not going to turn Frank G. iPhoneOwner into a pro photographer unless they’re interested enough to edit their photos and think about what they’re doing before taking the photo.

Schiller says that there will be a new feature called Portrait Lighting in beta for the iPhone 8 Plus. He promises that this feature will be more than a filter, it’ll present you with different lighting choices for your subject in Portrait Mode within Apple’s Camera app. You’ll be able to edit those photos and choose different lighting scenarios after the fact.

Phil Schiller also promised that these phones have better video quality. You’ll be able to record 4K videos at 60 frames-per-second and if you like slo-mo you’ll be able to record 1080p video at 240 FPS.

Schiller says that the cameras and the A11 are also configured for augmented reality. The demos I have seen for AR on the iPhone 7 and 7 Plus are impressive, it could be useful to have a focus on AR in the hardware as well. Schiller played pre-recorded AR demos of a Warhammer game, an MLB overlay on a live game at a stadium, and an astronomy app called Sky Guide overlaying star charts directly on the sky as viewed through an iPhone.

Directive Games’ Atli Mar demoed their AR game, The Machines. It looked a little bit like an RTS but overlaid on a table. Mar said that there was a strategic advantage to being able to move the game’s camera around by moving the physical phone, but I’m not sure if that lines up with reality. It looked almost as awkward as the AR demo from WWDC but it is very difficult to play a game for an audience and line up your view to present something cinematic, especially in front of a live audience.

The iPhone 8 also has “wireless charging” which is just contact charging, it isn’t truly wireless. It’ll use the Qi charging standard that is already widely supported. It’s good that Apple didn’t make another standard, but they aren’t including a contact charging device in the box.

The iPhone 8 and 8 Plus will be up for pre-order on September 15th and available in-stores on September 22nd. Here are the prices for different configurations:

iPhone 8

  • 64 GB for $700
  • 256 GB for $850

iPhone 8 Plus

  • 64GB for $800
  • 256GB for $950

iOS 11 will be out on September 19th.

iPhone X

Apple’s surprise of the iPhone X was spoiled by leaks and rumors, I was pretty disgusted with some journalists who took umbrage with the person who leaked the information. It’s only in Apple’s promotional interests to keep these things a secret and they aren’t going to affect our national security.

Here’s the iPhone X ad:

 

Phil Schiller called it “the iPhone 10,” skipping 9, I wasn’t sure if that was intentional or not at first, but then they continued it and I guess this is just what it is now.

The X’s OLED display is 5.8 inches diagonally and continues the journey to a rounded rectangular slab that is all display without any border. It’s almost there, there’s just a notch at the top of the phone for the front-facing camera, earpiece, and other sensors. Apple calls this new display a “Super Retina Display” at 2436×1125 pixels and  It’s 458 pixels-per-inch. So you’re getting a display that’s larger than the Plus and has more resolution in a smaller package. It’s also got an HDR standard for films and Apple’s True Tone tech.

Last year we lost the headphone TRSS jack and the home button was replaced with a simulation that I’ve grown accustomed to. This year the home button is just gone. You can wake the phone up by tapping on the display when it is in sleep mode.

iOS has other changes to accommodate the lack of a home button, of course. To get to the home screen from a “locked” iPhone X you now just swipe up from the bottom of the screen. It’s the same gesture when you’re in an app. Pausing mid-swipe, instead of raising your hand off the screen to complete the gesture, will bring up the multitasking switcher to choose from your recent apps.

I’m curious how that will work out with games. This overloading of gestures is a lot for users to take in.

The iPhone X’s sleep/wake switch is larger and can also be used to activate Siri.

Without a home button there isn’t room for a sensor to read your fingerprint for TouchID, so the iPhone X will have a face recognition system that Apple calls FaceID. Phil Schiller assures us that new sensors at the top of the phone in the notch will securely detect your face, even in the dark, to authorize your iPhone to be unlocked or in other places where you use TouchID. Including with Apple Pay.

Phil Schiller’s pitch for FaceID included a terrifying image of a wall of fake faces that Apple’s engineers used to test the face unlocking feature.

Apple are making an API available to developers for accessing the kinect-style data from the sensors on the front of the iPhone. They’re using the sensors internally to make 12 animated emojis that respond to your face and can be recorded along with audio messages to send over iMessage. Snapchat was demonstrated with it for face masking as well. They had a pretty impressive Ultimate Warrior style mask.

Craig Federighi demonstrated the iPhone X and showed that you can access the new iOS 11 Control Center by swiping down from the upper right corner where the connection status indicators are.

Phil Schiller boasted about the iPhone X cameras, they’re even better than the ones in the iPhone 8 Plus with optical image stabilization on both cameras. The front-facing camera is supposed to be better for selfies as well.

The iPhone X is supposed to get 2 more hours of battery life over the base iPhone 7, and have the same Qi charging as the 8.

The iPhone X will be up for pre-order on October 27th and shipping on November 3rd. Here are the prices for different configurations:

  • 64GB for $1000
  • 256GB for $1150

AirPower & Updated AirPods

Apple also has a new oval Qi charging mat coming out next year that will charge the iPhone 8 and X, as well as their Apple Watch and a new case for the AirPods that Phil Schiller said was “optional.” I’m not sure yet if “optional” means a more expensive set of AirPods or a separate purchase after the fact or both. I’d hope I could upgrade my current AirPods with a one-time purchase.

Overall:
I’m kind of disappointed with the pitch for the iPhone 8 versus the iPhone X. I don’t think I would personally want the iPhone 8 or 8 Plus when there’s an iPhone X available. Although I’m sure the iPhone X will not be available in as large a quantity as the 8. I also am not looking forward to an upgrade to the AirPods already. They don’t seem like something that should change as often, but I guess it might end up being similar to the Watch where there are updates but they seem less necessary than the phone.

I’m very disappointed that Apple is calling contact charging “wireless” when there are wires involved and you can’t just hold your iPhone five feet from the AirPower dingus to charge it.

It’s Time For iOS To Allow Apps From Outside the App Store

Recently, Apple started removing VPN apps from their iOS App Store in China in order to comply with local laws. That may be something they have to do as a business, but it’s time to allow apps from developers outside of the App Store. Gruber:

To me, the more interesting question isn’t whether Apple should be selling its products in China, but rather whether Apple should continue to make the App Store the only way to install apps on iOS devices. A full-on “install whatever you want” policy isn’t going to happen, but something like Gatekeeper on MacOS could.

Keep iOS App Store-only by default. Add a preference in Settings to allow apps to be downloaded from “identified developers” (those with an Apple developer certificate) in addition to the App Store. In that scenario, the App Store is no longer a single choke point for all native apps on the device.

The App Store was envisioned as a means for Apple to maintain strict control over the software running on iOS devices. But in a totalitarian state like China (or perhaps Russia, next), it becomes a source of control for the totalitarian regime.

Gruber doesn’t think this will happen, but it should. These pocket computers are supremely important to communications and it’s well past time for Apple to open things up.

Apps Dropping from iOS 11

Cyrus Nemati for Slate in an article titled “iPhone Gamers, Brace Yourselves for the App-ocalypse”:

If you’re an iPhone user still addicted to Flappy Bird, be ready to experience withdrawal symptoms.
When Apple launches iOS 11 in September, the company will drop support for old 32-bit applications—which is most apps released before 2014. Apps that haven’t been updated by their developers to run on the more efficient 64-bit architecture will cease to work

That’s true, but this article is also completely ridiculous in its conclusions. Here’s one of the solutions that Nemati suggests:

One obvious move: Create an emulator. By creating what amounts to simulators of old iOS versions, Apple could keep its past alive indefinitely. It’s something we saw recently when Nintendo released NES Classic, a replica of its stocky first gaming system loaded with 30 cherished early games. At 2.3 million units sold, it’s a good example of an emulator being used to make a quick buck (or more) on the back of nostalgia.

Emulating the Nintendo Entertainment System is easy today on low-powered, cheap, hardware. The Classic Edition kind of made sense for Nintendo as a product, but it is unlikely that Apple would ever release something similar.

Planned obsolescence is Apple’s modus operandi. It needs to keep customers buying new iPhones with updated specs. It’s unfortunate for consumers, who may have grown attached or even paid for these soon-to-be-defunct apps, and a shame for the creators who may see their work disappear.

The idea that Apple wants people to be unhappy with their devices on a schedule is almost as ridiculous as the idea of the iPhone Classic. Would you continue to buy things from a company that make you unhappy? I think you should replace things when they no-longer meet your needs or when there are new features that would dramatically change the experience of using that object in your life.

I agree that there should be some effort to archive these old applications, Apple is rich enough to do something about it, but it is going to be difficult to do that in today’s software environment.

Here are some of the software and hardware hurdles anyone who wanted to archive old iOS software would have to overcome:

  1. What’s the definitive article? With desktop software and older console games you could pack something up in an ISO or a copy of a cartridge’s ROM and say “this is the definitive article.” That’s difficult to do with networked software that is constantly being updated. You might say that the final version of the software released to the public by the original developer is the definitive article, but I bet some people would want a prior version because it did something differently, or the first version because it represented the most raw and original idea the developer had.
  2. An iPhone Classic that is running an out-of-date version of iOS for using old applications would probably be inappropriate for a general audience as it would lack modern security and networking features for accessing any data online. Any exploits that enable jailbreaking also enable bad people to do bad things to that device.
  3. What about the iPad, would there have to be an iPad Classic Edition? An Apple TV Classic Edition? An Apple Watch Classic Edition?
  4. Could you imagine Apple going on stage during an event to try and explain a custom version of the iPhone that exists solely to run historical software? “Here’s the iPhone Classic, it only runs iOS versions up to 10.” It would confuse their messaging around whatever iPhone they’re actually trying to sell. That’s not our problem as users and developers, but it’s another reason why they would be less likely to have anything to do with such an effort.
  5. This old App Store. They’d also have to maintain an older version of their app store and review software updates for those 32bit apps that couldn’t be transitioned to 64bit.
  6. Users would be frustrated when something doesn’t work right because an underlying piece of infrastructure is gone like an API server for logging into games.
  7. Developers who are still around would be stuck attempting to support versions of apps that they haven’t worked on in years. Is every app in this Classic App Store going to have a “probably unsupported” label on it?
  8. Apple still needs to encourage developers to transition to 64bit. All of that infrastructure would have to exist while they’re also encouraging developers to update old apps to 64bit.
  9. Apple has offered emulators for desktop software to run during transitions in the past, but desktops have the power and the room to explain what is going on with older software running in emulation or virtualization.
  10. I’m not sure if a modern iPhone even has the potential performance to virtualize an older device without destroying the performance of modern software running on the same device at the same time.
  11. How do you explain what is going on in any way that makes sense to a regular person?

Honestly, I think the only people in a position to solve this are software pirates, and they are still going to have a difficult time trying to do that for all of the games and other software out there on mobile platforms. Any other group would have hurdles that are too large to jump over in keeping this old software alive. It’s a problem for every modern online software distribution mechanism. Sony’s and Microsoft’s online stores for the PlayStation and Xbox both have this issue. Nintendo’s laughable online shop couldn’t even give you access to software you purchased previously without calling their support line and begging for it until they released the Switch and you’d still lose your game saves if your Switch is lost, broken, or stolen.

Software preservation is well and truly fucked.

Apple’s WWDC 2017 Press Event Notes

Every year Apple has at least two big events, their iPhone event in the second half of the year and their developers conference event in the first half of the year. Each event has it’s own press briefing, here are my notes from today’s WWDC 2017 press briefing. It is long as hell.

The show opened with a comedy skit about a new Apple engineer sitting down at a new desk and accidentally unplugging the iOS app store servers, disabling every app. Soon, the world descends into chaos as people can’t get directions and we see a marketplace of people trying to substitute for the missing apps:

Tim Cook was the first live human on stage with numbers and then to announce that Amazon’s Prime Video service is finally going to the Apple TV.

WatchOS 4:
Kevin Lynch announced new Siri functionality that would automatically present information you might want on the new Siri watch face. Rotating the crown scrolls through the available information. For example you might want to see your upcoming appointments and weather changes. They’re displayed as tiny cards as seen here:

There’s also a new kaleidoscope face along with several animated Toy Story character faces.

WatchOS 4 also has more customized notifications for activity encouragement and better reward animations when you’re done exercising.

The exercise tracking is more advanced and detects when you’re paused while swimming to determine your intervals, for example. It’s also easier now to move from one workout type into another without stopping.

New gym equipment will be Apple Watch enabled and will help you record your fitness activities on that equipment when you swipe your watch near a certain area on the equipment.

Apple Music is improved on the watch with a new app.

The new WatchOS will be out this Fall with a pre-release version available to developers today.


Mac Updates:
Tim Cook introduced Craig Federighi to talk about macOS updates. macOS 10.13 will be called High Sierra, another stability update along the lines of Snow Leopard and Mountain Lion. Craig joked about the name’s obvious weed reference.

Craig promises these new features in 10.13:

Faster Safari web browsing with faster Javascript.

Safari will also block autoplaying video and audio on web pages. That’s great.

Safari will have “Intelligent Tracking Prevention” to prevent sites from identifying you across domains.

Search is better somehow. Apple didn’t really explain how

Mail gets a split view in full-screen mode to edit new messages alongside reading others and is also more optimized for storage.

Photos gets all kinds of improvements such as improvements in detecting faces.

Finally, Photos’ machine learning stuff gets synced across devices.

Photos also has way more advanced editing features on the Mac.

Apple’s Apple File System came to iOS recently to replace HFS+, APFS will be on High Sierra as well.

MacOS will also support h.265 for video compression, which will also be supported with dedicated hardware on Macs.

Craig boasted about their graphics API Metal for a while before announcing Metal 2 that is supposed to give a 10x draw call throughput improvement. It’ll also have better debugging and optimization tools.

Interestingly, Craig talked about how there is a new Thunderbolt 3 enclosure with an AMD Radeon RX 580 and USB-C hub available for developers who want to offload Metal GPU computation tasks to an external device from a Mac. It isn’t clear if this can offload desktop graphics rendering, as well, but you would assume so.

Craig announced VR support coming to the Mac from Valve with SteamVR and engines from Epic (Unreal) and Unity.

All systems that support Sierra will support High Sierra. It’ll be out this Fall with a public beta later in June.


Mac Hardware Updates
New iMacs are coming with 50% brighter displays and Kaby Lake chipsets from Intel. The iMacs are getting support for more RAM, 32 gigs on the 21.5 inch and 64 gigs on the 27. The 27 also gets fusion drives by default. The 21.5 iMac gets improved integrated graphics. The 4k 21.5 inch iMac gets the Radeon Pro 555 and 560 with up to 4 gigs of RAM.

The 27inch 5K iMac gets the Radeon Pro 570, 575, and 580 with up to 6 gigs of RAM.

MacBooks and MacBook Pros are also getting updated with new chipsets.

Industrial Light & Magic’s John Knoll & Epic’s Lauren Ridge showed up to give a VR demo running on the new iMacs. The demo featured live edits to a Star Wars scene in the Unreal Engine. It was incredibly awkward and didn’t demonstrate anything new over what Windows desktop machines could do last year.


iMac Pro

 

There’s a new 27 inch iMac Pro in Space Gray to offer an option for anyone missing the Mac Pro that hasn’t been updated in years. With copious options in CPU (up to 18 cores) and RAM (up to 128GB).

According to Apple it’ll be much cheaper than a comparable workstation that would cost over $7000, this one will start at $5000. It’ll be available this December.


iOS 11
Tim Cook boasted about iOS 10’s lack of fragmentation compared to Android and then reintroduced Craig Federighi to talk about iOS 11.

iMessage apps get an updated app drawer.

iMessages is now in the cloud to sync your message history to all of your devices, and it is still encrypted end-to-end, which should mean that Apple can’t access your messages in any situation. Although practically they could do so and you might not know about it.

Apple Pay is getting person-to-person payments, and it’ll have integration with Messages. Although it’ll store by default to an Apple Pay Wallet you can transfer money to your bank account. Sounds like the introduction of an Apple Bank.

Siri is getting an improved voice that sounds more natural and is still the only option with both male and female voices. Siri will also support translation in beta with some languages. SiriKit for developers will do more and support more applications. The last update only worked with a few custom actions like sending messages in third-party apps.

iPhone photos and videos are getting new compression formats that are supposedly much more efficient. The portrait camera is getting more features for a higher quality end-result.

Memories are enhanced. Video editing is supposed to be much improved as well.

Control Center is now modular and a single page. If you 3D touch a control module you get more controls and more information for that module as it expands to take up more space on the display.

The lock screen notification area will now expand with a swipe to display all of your notifications, not just the most recent ones.

Live Photos are improved, you can pick the key photo and loop the short video in live photos or choose other effects.

Maps gets improved maps for inside buildings like malls and airports. Navigation gets speed limits and lane-guidance. There’s a new feature for drivers called Do Not Disturb while Driving, it’ll be automatically suggested while driving and hide notifications. It can also notify anyone texting you that you’re driving and that you won’t see their messages unless they’re on a VIP list or they can break through by responding with “urgent.”

AirPlay 2 supports multi-room speaker systems from all iOS devices as well as the 4th generation Apple TV.

Apple Music can optionally share your playlists and music library with your friends, developers can also access this information through new APIs.

Phil Schiller appeared on stage to announce some updates coming for developers and users. The iOS App Store gets a complete redesign that looks more like Apple Music. A new Today screen shows off new cards with different stories about new apps. There are new dedicated screens for showing off games and a different screen for non-game apps. Category editors can display videos demonstrating tips and tricks for different apps.

Craig came back to talk about other technology improvements, including ARKit for augmented reality. It’ll work with Unity, Unreal Engine, and Apple’s SceneKit. Craig demonstrated 3D objects like coffee cups and lamps displayed on surfaces like tables that are detected instead of just having the 3D objects overlaid on what the camera sees without any interaction with real-world objects. An updated version of Pokemon Go will support this new ARKit and display Pokemon on the ground instead of just floating in the air.

Peter Jackson’s WingnutAR team appeared on stage to demonstrate an AR application running on an iPad with a science fiction action scene. Using AR to display a pre-configured action sequence is pretty boring, honestly.


The 10.5 inch iPad Pro

Apple introduced a new iPad Pro with a 10.5 inch display and an A10x processor. Joz told us that this supports a full-size software keyboard along with a better keyboard case. It’ll have a better refresh rate that looks smoother when motion is displayed on the screen, so Apple called this ProMotion. It’ll also dynamically adjust the display refresh rate for whatever the content is you’re working on, for better performance or battery life at a slower refresh rate.

The new 10.5 inch iPad Pro also has the same camera as the iPhone 7 and the base model has 64 gigs of storage. They’re available for order today and will ship next week.

Craig came back to talk about iOS improvements for the iPad. The dock can now have a large amount of apps. It can be pulled up from the bottom of the screen in any app. You can pull apps out of the dock into a slide-over multi-tasking split-view. iOS 11 will also support drag and drop. There’s a new Files app that supports iCloud drive and third-party cloud file management services like Dropbox.

The Notes app now supports handwritten notes that get interpreted using OCR as text by the system for searching and can scan a document for editing.

The 12.9 (let’s be honest, 13) inch iPad Pro also gets the new display, processor, increased base storage (64GB) and other features. All of these iPads 10.5 and 12.9 also have the 2nd generation TouchID sensor with quicker fingerprint recognition, but still appear to have physical home buttons instead of the haptic-driven virtual home buttons of the iPhones 7 and 7 Plus.


HomePod

Phil Schiller came back on stage to talk about “Reinventing Home Music” with a new speaker from Apple called HomePod. The first non-Beats branded speaker from Apple since the iPod Hi-Fi. HomePod is a rotund  7-inch tall device with a little bit of an urn shape and has its own A8 chip inside. Supposedly this speaker is very advanced and detects the kind of environment it is in to adjust for different types of rooms and adjust the audio for clarity. It’ll also work with another HomePod automatically for better stereo separation.

Of course, this speaker will support Siri and be Apple’s competitor to Amazon’s Echo and Google’s Home device as was rumored for months. Apple says that this speaker will have advanced Siri functionality that is tailored specifically to it.

The HomePod will be $349, in white and space gray, this December.


Overall, this feels like Apple trying to catch up, in VR and AR, with a speaker for Siri and finally getting updates to the iPad Pro and an iMac Pro to help reassure Mac Pro users like video editors that Apple hasn’t completely abandoned them when the Mac Pro still doesn’t have a release scheduled beyond “not this year.”

It’s good to see VR on the Mac, but nothing today addressed the desire for a desktop Mac for gamers to play VR applications unless you want a machine with a built-in display and a GPU you can’t upgrade. As terrible as Nvidia can be as we’ve seen with the GTX 970 memory debacle, I don’t ever want to buy a desktop with an AMD graphics chipset.

I would probably want a HomePod as Apple is the only company offering an AI assistant speaker that gives a damn about privacy and isn’t just trying to sell you shit through their store like Amazon with their Echo.

Apple Releases Clips App

It’s the Ken Burns effect but for millennials. Joanna Stern has a guide to Clips on the WSJ:

Hang-gliding over the Swiss Alps, or just cleaning your dryer’s lint tray? Either way, the titans of tech want you to record a snippet of it. Then they want you to slap on some emojis and text. Then they want you to share it with the world.

Snapchat started it. Instagram copied it. Facebook really copied it. Even Microsoft’s doing it. And now… Apple. Available Thursday for the iPhone and iPad, Apple’s Clips app lets you shoot, edit and share mobile-friendly mini-movies. If you have iOS 10.3 (the latest version), you can download it from the App Store.

iOS & macOS Updates

Today’s software updates from Apple are notable for a lot of reasons. macOS 10.12.4 gets Night Shift from iOS, I can now find my AirPods in the Find My iPhone app, there are the beginnings of a standard to control apps that nag you for reviews on iOS 10.3, Siri enhancements on watchOS 3.2, but the most important change is he first switch from HFS+ to APFS. APFS is Apple’s new file system for iOS, macOS, tvOS, and watchOS. It’ll be installed with the latest iOS update, and will come to macOS in the future.

File systems keep track of where your files are on a storage medium like a hard drive or SD card, and Apple has been using some variation of the same file system, HFS and HFS+, for the past thirty years.

APFS should be an improvement in performance and reliability because it is targeted towards modern computing devices that use flash memory instead of spinning disks, but it is going to be a little bit like pulling a table cloth out from under a plated dinner and table setting in order for the operating system to replace the file system on the devices during the update.

I made sure my iCloud backups were all up to date before starting the upgrade, you should too.

Workflow Acquired by Apple

Matthew Panzarino has the scoop:

Workflow the app is being acquired, along with the team of Weinstein, Conrad Kramer, Ayaka Nonaka and Nick Frey. In a somewhat uncommon move for Apple, the app will continue to be made available on the App Store and will be made free later today.

This is terrific news. Workflow is an automation app that helps you get real shit done on iOS. I’ve used it to do things like clipping headlines out of articles and preparing a link in the format I use to link to posts like this one. Or you could use it for a simpler task like tapping one button to send a text message to your significant other and let them know how long it’ll take you to get home.

Workflow can do all kinds of things, but it is the kind of thing that it’s great to see Apple embrace because it means they’re serious about improving the functionality of iOS for more than reading, watching, and playing. If folks who grow up with iOS ever want to hope of being productive on those devices instead of learning desktop computing, this is going to be a tremendous start.

Apple’s WWDC 2016 Keynote Notes

Here’s what Apple announced at their WWDC 2016 Keynote, or you can just watch the video.

WatchOS 3

Kevin Lynch spoke about the changes coming in WatchOS 3.

Apps, that you pick, update in the background and launch faster as a result.

The long button below the crown will no-longer launch your VIP contact list. It’ll now launch the dock app switcher. The dock displays a live view of the applications so that you can see updated information even without entering one.

Glances are gone, replaced by control center when you swipe up on the watch face.

Messages immediately give you options to reply, instead of having to choose that you want to reply first.

When a message comes in you can scroll down to get a list of suggested replies.

WatchOS 3 will have a new input interface called Scribble. You draw out letters one at a time with the whole word appearing above the input interface. This supports both English and Chinese characters.

In addition to the Micky Mouse watch face you can now choose Minnie Mouse with different outfit color options.

There is a new watch face called Activity that more prominently features the activity rings which monitor your steps and other exercises throughout your day. They’ll be larger and displayed behind an analog style of watch hands if you choose this face, or in chronograph and digital variations.

Kevin Lynch isn’t done with new watch faces yet, Numerals is another. This one displays only analog watch hands and a prominent digit or digits in many different fonts for the current hour.

Another change to the watch face. You can now swipe to switch between them and choose which you would like available.

More complications are available on more watch faces.

Stacy Lysik gave a demo of WatchOS 3. She shows the audience how Apple’s watch apps have been updated for quicker interactions. The timer gives you a few preset timing options to launch, for example. This should allow my son to more quickly set timers that confuse me when they go off because he loves to grab my watch and mess with it while I am holding him.

Kevin Lynch returned to tell us about the SOS feature that can call your regional version of 911 and alert your emergency contacts with your location after the call. It’ll be activated in WatchOS 3 by holding down the side button and it’ll display a countdown before calling emergency services. Interestingly, Lynch mentioned that it’ll work either via bluetooth to your phone or over wifi if your Apple Watch is on a known network.

I wonder about the usefulness of an emergency call where your watch is broadcasting that call publicly on a speaker instead of through your phone’s earpiece.

Jay Blahnik appeared on stage to introduce a new fitness related improvements, starting with activity sharing as a competitive feature. With it you’ll be able to see your friends and family member’s activity rings and data like steps and calories burned. If your friends use third party apps and devices, it sounds like those will work with this feature as well as long as they use Apple’s HealthKit as a data intermediary.

When you view a friend’s activity information you’ll be able to send them suggested encouraging or competitive messages about their progress.

Jay Blahnik continued on to introduce new technology to recognize activity information for wheelchair users. He talked about how they have adjusted different notifications and other information if the wheelchair setting is enabled. For example, the Apple Watch can optionally remind you to stand for a minute once an hour. For wheelchair users the watch will remind you to take a break and push around a little. Some of the workouts will be specifically for wheelchair users.

Finally, Blahnik introduced an app called Breathe to remind people to take stress reducing breaks. It’ll have different kinds of optional reminders and will be able to guide you through a session with visuals or haptic feedback.

Lynch returned to the stage to discuss how WatchOS 3 will be improved for developers to enable integration with those new features and improvements.

Apps on the watch will be able to use Apple Pay. Fitness apps will be able to run in the background during workouts and have access to more data that will enable new kinds of workout apps on the watch according to Lynch.

There are many more improvements and new APIs to the SDK for the watch including SpriteKit and more. Games should be much better though it’ll still be a tough sell to keep your wrist raised up for a while. I’d still expect it to be really only for quicker interactions.

Eddie Cue came on stage to talk about tvOS for the 4th generation Apple TV. He spoke about new apps like Sling, Fox Sports Go, the French TV service Molotov and a few games like NBA 2k, Minecraft Story Mode, and Sketch Party.

Cue then introduced improvements to the Remote app for iPhone which has all of the features of the physical 4th generation Apple TV Remote.

Siri for the Apple TV will be able to find shows and movies by topic. Cue searched for high school comedy movies from the 80’s and got Ferris Bueller’s day off and other options.

Siri will be able to load into live channels. MLB was notably absent.

Logins with cable providers will be reduced to a single sign-on. The App Store will let you know what apps/channels you have access to once you sign in.

The Apple TV will have a dark mode, and will automatically download available apps if you download them to another device.

Cue finished the tvOS talk by briefly mentioning some of the improvements for the developer kit and mentioned that the new version will be released publicly this Fall.

Craig Federighi was introduced to discuss the improvements to OS X, now renamed to macOS. The new version will be macOS Sierra.

Sierra will introduce new features cribbed from third parties such as unlocking your Mac when you have your Apple Watch close to it.

Another new feature borrowed from a third party, Tapbot’s Pastebot in this case, is Universal Clipboard. You’ll be able to copy and paste text, images, video and more between Macs and iOS devices.

iCloud Drive will now include your files in the desktop folder and sync those between Macs as well as making them available via the iCloud Drive app on iOS.

If you are running out of space on your Mac’s hard drive, Sierra can attempt to offload older files to iCloud and remove other kinds of files that users don’t typically need like old application caches. There will be a new GUI specifically for doing all of these functions.

Apple Pay will now work on macOS Sierra through websites that support it. You’ll authenticate it on your iPhone with the Touch ID finger print reader or Apple Watch.

Craig announced another Sierra improvement, tabbing application windows like web browser tabs. Developers won’t need to do anything to support this functionality.

Videos will be able to go into a picture-in-picture mode from websites. You can drag to move or resize this always-on-top window which also persists across full screen application virtual desktops.

Finally, Siri will be available in macOS Sierra. She made a few jokes with Craig during a demo. Siri will be able to understand new questions that are more appropriate for a Mac. For example, Craig asked about files from a specific time period and location, and then did a follow up question to further refine the search. It is unusual for Siri to understand context.

Results from Siri can be pinned to the macOS Notification Center. Image results can be dragged directly into applications from Siri, or copied and pasted from an iOS device.

Sierra will be supported on these models of Mac:
Late 2009 & later
MacBook
iMac

2010 & later
MacBook Air
MacBook Pro
Mac mini
Mac Pro

Craig moved on to iOS 10 and discussed 10 new features.

First off the bat, user experience improvements.

The lock screen has been redesigned. You’ll be able to raise your phone and it’ll wake up, bypassing the issue with newer Touch ID sensors that are so fast you never see the lock screen if you press a finger to the home button.

Notifications on the lock screen are more interactive and designed for 3D Touch to display more contextual information. Craig demoed 3D touching a calendar invite and seeing more details about it, before accepting it.

The next example that Craig demonstrated was a more interactive iMessage conversation where he could see more of the context of the conversation and also get images in it without leaving the lock screen notification.

If you use the illegal taxi service, Uber, their updated notifications showed you the location of your incoming car with an unvetted driver that is probably making less than minimum wage after all is said and done.

Notification Center now lets you clear all of your notification with a 3D Touch gesture that reveals a clear all button.

Control Center was simplified in its initial display, but it also has another page you can swipe to with more bigger buttons for music control and an album art display.

It’ll be easier to get to the camera from the lock screen, you can now slide from right to left to open the camera application.

Sliding from left to right displays a new widget view. The demo broke when Craig tapped on his calendar widget to “show more” of his day. An engineer was immediately brought on stage to answer for this failure.

There are more 3D Touch improvements to apps on the home screen. The email application can now display a widgetized list of contacts and a count of unread messages from them. The activity application on the iPhone can also display your activity rings in a widget that appears when you 3D Touch on that from the home screen.

These widgets can also display live video through third party applications like ESPN.

Craig moved on to iOS 10’s improvements in Siri.

Developers will have access to Siri in IOS 10. Though it appears to be limited to certain categories of apps like messaging apps. Craig mentioned Slack, Whatsapp, and WeChat.

Siri will hail illegal taxi cabs through third party apps Uber, Lyft, and Didi. Search photos in Pinterest, IM and Shutterfly. Start and stop workouts with MapMyRun, Runtastic, and Runkeeper. Send payments with Number 26, Square Cash, and Alipay. Set up VoIP calls with Cisco Spark, Vonage, and Skype.

CarPlay will also work with third-party apps for messaging and VoIP.

The keyboard suggestions, QuickType, will be improved and understand the context of the conversation. Craig’s example, is that it’ll understand the difference between playing in the park versus the Orioles playing in the playoffs. If you’re asked “where are you?” in iMessage, QuickType suggestions will give you a big suggestion to give your current location on a map. If you’re asked for contact information, QuickType will suggest sending the contact. The calendar event suggestions that you’d see if you tapped on text like “Sunday at 2PM” before will now be more context-aware and understand that you’ve been talking about a certain type of food and a street address that get included in the calendar event suggestion.

The keyboard will now support multilingual typing, I won’t have to switch between German and English anymore!

Photos will now display your photos pinned on a map, and will run facial recognition to understand who is in what pictures. I hope it works better than iPhoto did at that. Photos will also now detect objects and scenes in pictures so that you can search for these better. Craig displayed an example picture of someone riding a horse by a lake with a mountain, and said it would understand the mountain and horse and the scenery if you search for those. I wonder if we’ll see progress bars scanning all of our photos the first time we open the new Photos app in iOS 10 as this occurs. I suspect that this will be done offline due to privacy concerns.

Photos will also group together different trips, people, groups, and topics like “on the water” or “at the beach” into a new interface. Craig demonstrated the new “Memories” tab in Photos on iOS 10 that shows the product of this computer vision work. It’ll even create a short video of different events with Ken Burns style zooms of different photos and videos as well as music layered on top. Craig assures us this will pick the right music, but he demonstrates overriding the music choice and how Photos will re-edit the “memory movie” to match the music change. These changes will also come to macOS Sierra.

Craig takes a break as Eddie Cue returns to show us updates to maps. iOS 10 Maps will offer you some suggestions based on calendars and where you commonly go at different times of day. If you search for restaurants it’ll display a horizontal list of cuisines and restaurant styles you can pick from, and adjust its suggestions based on which you select. Navigtaion is also improved. Eddie shows us how the view is more dynamic, zooming in and out intelligently based on where your next turn is and other information. Maps will give you suggestions for things along the route, and tell you how long it’ll take to reach your destination if you stop at these suggestions. There’s also a Maps extensions API, Eddie’s example is booking a table at a restaurant with an OpenTable restaurant, hailing an illegal taxi cab with Uber, and paying for it all inside Maps. Cool.

Apple Music is also getting updated with an “all-new redesign.” I don’t see the connect tab anymore. Bozoma Saint John went on stage to give a demo. The Music app looks to be streamlined without that connect tab, and pretty different. Lyrics are displayed right below the controls for the music if you scroll down. Very nice. Bozoma tried to get the audience to rap along with Rapper’s Delight, the audience cam showed us various Apple folks attempting to do so, it was extremely embarrassing. The new Apple Music interface looks great, though.

Eddie Cue comes back to discuss Apple News improvements. You can already read this site on there so they’ve done a bit of extra cleanup to improve the rest of the app. For example, they’ve added subscriptions and breaking news notifications.

Craig is back to discuss Homekit improvements. New categories of supported IoT devices have been added. A dedicated Home app is going to be available to control all of your homes. I’m sure monocles are popping out everywhere. Preset scenes will be available to adjust several devices at once. Siri can control these scenes and devices. If you swipe over on Control Center you’ll be able to control devices from there. Notifications will be able to interact with HomeKit devices and display live video from a doorbell device, for example. Apple TV will act as the hub for your HomeKit devices so that you can access them remotely when you’re away from home.

The Phone app will be updated to transcribe voicemails, though this functionality is in beta. An extension API will let other applications interact with the Phone app, in an example slide an extension from Tencent identifies an incoming caller that isn’t in your contacts as a possible spam caller.

When VoIP calls ring through, they’ll integrate into the lockscreen and the phone app just like regular calls. Contacts will be updated to display the options you have to call someone through multiple services. They’ve worked with Cisco to let you get your work calls through your iPhone.

The Messages app has been updated to provide previews of links inline. Videos and photos will display right in Messages. The camera will display a minified camera app inside of messages when you tap that button. Emoji are now three times larger. If you tap the emoji button after typing up a message, but before sending it, it will highlight words that can be replaced with emoji. Message bubbles can now have effects like shrinking or growing to emphasize emotion behind words. Messages can be hidden for a recipient to reveal with a swipe over the text or an image. They’ve also added a short series of icons to send a thumbs up, or a laugh to someone. Apparently we don’t have enough time to select the appropriate emoji now. This demo slide broke and displayed the thumbs-up above the message that it was replying to. You can also send “handwritten messages,” it isn’t clear if these are keyboard input being turned into a generated font with some ink physics or brushed on with a touch, but you can also send “digital touch” messages like drawing a smiley face on the Apple Watch. You can also draw on video or photos with digital touch. There are also fullscreen effects that appear behind the messaging transcript. A slide shows us some fireworks behind the chat bubbles after you send “Happy New Year!”

Bethany and Emron are introduced for a demo. They’re engineering and human interface design team leads. Bethany demonstrated sending and receiving some diffferent message types, Emron received a link to some music that you could play right inside Messages.

iMessage will now support developer-created apps that are launched from an “app drawer” for things like stickers. Craig tells us that some types of apps won’t even require code. Artists can make them a sticker app without coding, but you could also integrate other iOS functionality like the camera. Square Cash will work inside of Messages as well.

Craig started a demo. He demonstrates sending a sticker, and then applying some animated stickers to a photo message. The next demonstration is a group ordering food together through DoorDash, right inside of Messages. Each person selects the food they want from the restaurant (or food truck in this demonstration).

The Mac and Apple Watch will be able to receive these conversations, so it sounds like they won’t be able to work with messaging apps to create messages using them, which makes sense.

Here’s a video they showed to feature the new features in Messages:

Craig finishes up by blitzing through some other improvements to iOS 10. Live editing collaboration in the Notes app. Conversation view in Mail displays messages in a better threaded format. Live photos can be edited and they have stabilization now. Safari on iPad finally lets you use split view to display two websites at the same time. Previously you had to use third-party apps to do that. He also reminds us that many of these new features work for third-party developers.

Finally he re-emphasizes how all of this information that involves computer learning, the memories functionality in Photos for example, will be run on the device. The information that is sent to Apple’s servers will also be kept private by them not building profiles of users. Obviously this is a dig at Google who mines all of your data with no regard for your privacy if it’ll enable their advertisers to have better targeting.

Tim Cook comes back on stage to close things out and emphasize Apple’s commitment to developers. He discusses how great Swift is. He’s talking about how it’s the #1 language project on Github since the source was released. Cook also reiterates how important Swift is as a first language by releasing a new app called Swift Playgrounds for iPad.

Tim invites Cheryl Thomas on stage to demonstrate Swift Playgrounds. The app’s front door screen has lessons and challenges. QuickType suggestions are offered inside of lessons with code suggestions. There’s a simple turtle-style demo of moving a character on the screen with code like moveForward() and collectGem(). Cheryl demonstrates wrapping a for loop around some pre-existing code. This looks like it could give Codeacademy a run for its money, though obviously it’ll be limited to Swift many of the lessons could apply to other languages. We need XCode for iPad.

Cheryl opens a more advanced playground with a new coding keyboard to add new code to the playground.

Tim returns again to announce that Swift Playgrounds will be free, of course. Another video:

That could have been incredibly hokey, but I think they did a great job.

Tim Cook goes back over everything we’ve seen, and closes out the keynote.

Make it Rain Review

20140519-230122-82882247.jpg Make it Rain is difficult to categorize. It’s a game for iOS, and it stretches the definition of a game. Though it isn’t the first to do so.

Progress Quest was the first in this genre. It was simpler, though. Think of starting solitaire and watching it play itself except with high scores on an internet leaderboard more accurately measured in minutes hours and days the program has been running than in a score.

Cookie Clicker perfected it last year by adding upgrades and removing the leaderboards to redefine the goal. Not that there is a victory condition or any well-defined end game but you’re nolonger competing against other people who have just been running the game longer. Now you’re just having fun and experiencing the ridiculousness of the game as it changes over time through the upgrades you choose and your score or cookie count increases.

In Make it Rain they’ve simplified the game but I think they’ve lost something in doing so.

The upgrades names and icons in Make it Rain are funny and the game does have some of the same fun in upgrading your production. In this case it is a fountain of dollars instead of cookies. Much like Cookie Clicker theres fun to be had in trying to pick the optimal path through the upgrade tree. There’s even a great new mechanic in that you can metaphorically “make it rain” by swiping increasingly large bills from your virtual money clip.

But a lot of the charm is gone in the unavoidable comparison to Cookie Clicker. It seems like a slide back towards the gameplay of Progress Quest. When I was playing Make it Rain I kept hoping for some point where a meta game or a new addition to the gameplay.

The breaking point for me turned out to be when, after setting my iPhone to remain unlocked on the floor all night, I woke up to a piece of the new gameplay smacking me across the face. The FBI investigation is cute, but when the wheel of justice spins and lands on a penalty a few too many times you may be ready to toss your phone out the window.

The penalties are just too harsh and put your production into a hole it won’t be fun to dig out of.

It is a free-to-play game, and that almost makes the experience worse. It seems like the gameplay is designed to put you into a situation hat you can’t get out of without spending real money.

Maybe the developers will patch Make it Rain to be a little bit more player friendly, or at least weight that FBI investigation wheel towards fun, but right now I wouldn’t recommend the experience.

2 out of 5 robot pimps.