Categories
tech

Imagining the future of AI photography

Portrait mode and corollary features (e.g. portrait lighting) are halting first steps towards a true AI-augmented camera. Here’s a fanciful look at our smartphone future:


It’s April 4, 2027, and Julie is making good progress. For the seventh time that day, she clambers up the squeaky attic ladder and crouch-steps her way to a tall pile of cardboard boxes. She squints at the next box in the stack, just making out her mother’s scrawl: “FAMILY PHOTOS.” It slides easily across the dusty floorboards, and Julie descends the ladder, flopping the box from step to step above her.

With a heavy sigh, she sets the box down on a dining room chair. Her kitchen scissors make quick work of the duct tape; Julie glances inside—and winces. No neat stacks, no carefully-curated photo albums. Instead, the box is full to the brim with loose snapshots, unlabeled and unsorted. Just a few years back, organizing these photos would have taken an entire weekend.

Fortunately for Julie, times have changed. She works quickly, plucking photos from the box and tossing them into a messy grid on the table. Within a few minutes, she has strewn hundreds of memories across the oak panels. They’re arranged in no particular order; Julie spots a baby photo of her grandmother from the 40s, adjacent to a faded Kodak print of Aunt Susie driving in the mid–70s. The very next snapshot in the row is a Polaroid from Christmas 1991; her little brother triumphantly lifts a brand-new video game console package above his head.

With a nostalgic smile, Julie whips out her smartphone and opens the photo enhancement app that makes this spring cleaning possible. The real work begins; she waves the device over the table, lazily panning its viewfinder across the rows and columns of snapshots.

As she does, the camera does its magic. Each individual photograph is extracted, cropped, and saved to its own file. The process is nearly instant; after just a minute or two of haphazard scanning, the app beeps and confirms that it’s captured all the data it needs. Julie sweeps the impromptu collage into a waiting trash cash.

It’s almost hard to believe how much she trusts the phone to capture these photos. Once, she would have been horrified to throw away such precious memories. Now, in a single day, she has filled a half-dozen garbage bags with old snapshots.

As she breaks down the empty cardboard box, the phone app (and the cloud service that powers it) does its own tidying up. First, it leverages machine learning to automatically recognize every object in every photo: that’s a baby beneath a maple tree in late fall. That’s a 1976 AMC Hornet. That’s a Sega Genesis.

With that context in hand, the service can clean up the photos. First, the easy stuff: wiping away physical scratches. Removing decades’ worth of discoloration and fade. Filling in missing, damaged details using robust healing algorithms. The AMC logo on the Hornet’s hood, obliterated by a coffee stain on the photo, is now recreated from a library of vintage car logos. A gouge in the leaves gets repaired too; maple leaves have a distinctive shape, and the app generates a few more to fill the hole. The Sega Genesis, motion-blurred in the boy’s excitement, is sharpened using actual product photography.

The restoration isn’t limited to inanimate objects, though. The app knows that it’s Aunt Susie who’s sitting behind the wheel of the Hornet, even though she’s obscured by glare on the windshield and some lens flare. Using Susie’s existing visual profile, the tool sharpens her face and glasses and fills in the blown-out highlights with data from other images.

The service automatically assigns metadata to each image, too. Every calendar, clock, Christmas tree, or party hat in the background helps the service narrow down the image date. Less obvious visual clues can help, too; the app might recognize that ’76 Hornet from previous scans and assign the photo of Susie to the same era. Even the girl’s tight perm could help to date the photo; given enough data, the app might know exactly when she adopted—and abandoned—that distinctive look. In the same way, visual cues could help pin down each photo’s location, too.

As Julie sets the last of the trash bags by the curb, she feels a twinge of bittersweet guilt. The garbage truck is on its way; soon, the original photos will be gone for good.

But based on experience, she’s confident enough in the restoral to let them go. The digitized shots are far more portable, more flexible, and more interesting than the paper copies ever were. She can make edits that would otherwise have been impossible—like tweaking the exposure level or even the location and brightness of the original scene’s light sources. Or altering the camera’s focal length, decades after the shot was taken; the app’s AI has used the source image to extrapolate exactly where each object sits in three-dimensional space.

Finally, Julie can even perform that “Zoom… enhance” magic promised by science fiction movies for decades. As she steps back into the kitchen, she grabs her tablet and plops down at the counter. Time to take a closer look at Aunt Susie’s unbelievable 70s curls. ■


Categories
apple tech

Fixing Portrait mode’s grossness

A few weeks ago, I bought an iPhone X. I love its face-unlock authentication and its gorgeously tall screen, but its dual-lens camera is easily my favorite feature. The iPhone X is the best camera I’ve every owned, and it’s not even really a competition. I’ve never had an SLR and my last point-and-shoot was from the early days of digital photography.

In fact, the iPhone X camera is so good (or, rather, “good enough”) that it’s hard to imagine I’ll ever consider buying a standalone camera again.

That’s not to say there isn’t plenty of room for improvement. In particular, I find “portrait mode” (the fake bokeh blur effect) alternately intoxicating and maddening. In many cases, it does a great job isolating my photo’s foreground subject. But when it fails, it fails hard. As many others have pointed out, hair poses a serious challenge to its algorithm, as do non-contiguous background areas (e.g. a piece of the background visible through the crook of your subject’s arm) and ambiguous edges.

Could Apple fix these sorts of hiccups in software? This is my first dance with Portrait mode, so I can’t say whether the feature has improved since its first release last year. But I have at least some hope that edge detection will improve and fill in some of the gaps (pun intended).

Even if the algorithms improve, I’d like to see some way to touch up these problematic Portrait mode depth maps. There are already several interesting apps that let me see the underlying depth channel. Slør paints it as a black-and-white alpha channel; Focos lets me spin the depth layers around like I’m in some sort of sci-fi flick (“Rotate. Zoom. Enhance.”).

But neither of those apps—nor any others that I’ve heard of—let you actually fix the depth sensing errors that creep into Portrait mode photos. Take those non-continugous background areas I mentioned earlier. Given a decent channel-editing interface, it would be relatively simple to paint a foreground area back into the background, where it belongs.

It’s possible that Apple’s developer-facing APIs won’t allow depth changes to be written back into a fully editable Portrait mode photo. If not, that’s a shame and ought to be corrected. In the meantime, though, I’d love to see an app handle depth edits “in-house”, then export a flattened photo (with the proper blur baked in) back to the camera roll.

Hopefully that sort of functionality arrives soon. Portrait mode is a blast, but it’s hard to feel too enthusiastic when it produces such glaring flaws. ■

Categories
apple

Memories vs. math: how to justify paying $1,000 for the iPhone X

I have a dilemma. I can’t decide whether to buy the iPhone X or hang onto my iPhone 7 for another year. Day to day—and sometimes hour to hour—I waver:

Calculating the cost

On the one hand, I dig the X’s edge-to-edge display, its high-res OLED screen, and (especially) its dual lens camera system. And by my math, the costs of upgrading my phone every year are surprisingly comparable to upgrading every two—when I figure in the cash return of reselling the old phone.

But then I remember the X’s thousand-dollar price tag, and my determination falters. That’s a major investment, no matter the potential resale value. I hesitate to spend that much when my current phone works perfectly well.

Memories > math?

Of course, there’s more to this decision than just dollars and cents. If I do buy an iPhone X, it will be to get one marquee feature in particular: its best-in-class camera.[1]

I imagine myself forty years from now: a septuagenarian looking back on the past. From that vantage point, it’s likely that my current stage of life—young parenthood—will be the time I would be most grateful I had used a decent camera.

Kid in grass

Our adorable daughter is two years old—and growing like mad. It’s almost physically painful to see time flying by so fast, and we’re desperate to capture her quirks and discoveries with photos and videos. We snap hundreds of pictures every week—exponentially more than we ever did before she was born.

I want a phone camerathat takes amazing shots, even when my kid runs wild through the yard in the autumn twilight.[2] If I upgrade to the X, I will have a better record of my daughter’s second and third years of life.

To buy, or not to buy?

If I’m awake at 3 AM on October 27, preordering the iPhone X, here’s what I’ll be telling myself: you’re not buying a $1,000 phone. You’re buying a $1,000 camera with some amazing bonus features. Somehow, that seems easier to swallow. A thousand-dollar phone? That’s extravagant. But a thousand-dollar camera that helps me better remember my daughter’s early childhood? That makes some sense.[3] ■


  1. Technically, the Pixel 2 is the current champion, at least as judged by DxOMark. But the iPhone 8 Plus is close behind; assuming the X bests the 8 Plus (likely), it may approach the Pixel 2 in overall quality.  ↩

  2. Fortunately, even older iPhone models take snapshots that compare favorably to low-range DSLR photos. The iPhone is already good enough; it has completely usurped the place that standalone cameras once had in my life.  ↩

  3. Even hobbyists spend that much on photography gear without blinking an eye.  ↩

  4. Scale and camera vector artwork courtesy of Freepik.