Augmented Reality: Is It Real? Should We Care?
A talk on the state of augmented reality given at Interaction ’10 in Savannah, Georgia on February 6, 2010.
Update: The video for this talk is now online thanks to IxDA and I’ve included it here. If you’re in San Francisco, we’re also doing a redUX on March 6th, 2010 for free at Adaptive Path. There will be a number of local speakers that spoke at Interaction ‘10.
It all started around July or August 2009 when I saw this video:
It was the first time I’d seen the iPhone 3GS’s new compass feature utilized in such a way. I immediately thought, “this would be great as a Ghostbusters-like game.” So my friend Arshad, my wife Coley, and I with some help from many of our friends and family, spent a little bit of our spare time over the next few months working on a basic concept which eventually became ARGH: Augmented Reality Ghost Hunter.
How many people have taken the Savannah ghost tours? A few of you? Well, in case you didn’t know, Savannah is known for being haunted. So let’s look for some ghosts.
[showing ARGH game and finding a ghost in the presentation room]
This game isn’t really all that complex. In fact, it’s just a small part of what we originally envisioned. But as we were building and designing this game, I learned a lot about augmented reality and it’s current shape. So that’s what I’m going to talk about today. Augmented Reality: Is It Real? Should We Care?
When I asked on Twitter what people at this conference wanted to learn from this talk, Dan Willis helpfully replied to me:
So, I decided to oblige and have renamed the talk OMGWTFAR?!1!: A Hefty Heap.
What is Augmented Reality?
Well, augmented reality is a term that was actually coined 18 years ago:
“In 1992, Tom Caudell coined the term augmented reality when he was working at Boeing on a project to make it easier to assemble large bundles of electric wire for aircraft on the factory floor.”
But what does it mean exactly? According to Wikipedia (with my own emphasis):
“Augmented reality (AR) is a term for a live direct or indirect view of a physical real-world environment whose elements are merged with (or augmented by) virtual computer-generated imagery — creating a mixed reality.”
The live and mixed reality parts are very significant because it means taking a photograph and processing it doesn’t count and it means virtual reality doesn’t count.
However, it doesn’t sound as fanciful as you might think. We’ve been seeing it in Hollywood for ages. Minority Report features scenes where Tom Cruise is greeted at stores with virtual, personalized ads overlaid on top of his reality.
And even earlier than that, in 1984, The Terminator was showing an augmented reality view in the eyes of the Terminator.
It’s not just science fiction. Your reality has already been augmented. This being Superbowl weekend, it’s appropriate to note that the first-down line we see in football games is overlaid, or augmented, and not painted on. What, nobody told you?
Types of AR
There are different ways to create an augmented reality. What you ee with football games is one of them: show live video feeds and, while watching the feed, additional information is added. This experience is probably the most disconnected AR experience as you’re not actually interacting with the “reality”.
A very common type of AR you’ll see these days are the marker-based AR. The technology depends on the person holding up an easily recognizable and predefined marker to the camera. You can try it yourself with Boffswana’s Papervision prototype.
Another method, which is becoming increasingly popular, is location-based augmented reality. These systems depend on the knowledge of where you are, which direction you’re facing, and what the system already knows to be around your location. The most well known example of this is the Yelp Monocle which shows you restaurants and other businesses in a first-person view.
Another form of augmented reality is based on augmenting your world through audio instead. RjDj have created a suite of music apps which react to the environment and sounds around you as well as the speed you’re moving. Their latest collaboration with British pop singer Little Boots is probably their most polished demonstration of how delightful this experience can be.
Watching the people in the Little Boots video really serves to illustrate just how much of an experience AR can provide. While the term should and will remain the same now that it’s hit the mainstream, I do feel it’s more accurate to refer to it as an enhanced reality and adding information or interaction based on what is happening around you is in fact enhanced context and enhanced meaning which leads to enhanced experiences or even delightful experiences.
Why Now?
When you look at Google Trends for searches on the term augmented reality, you’ll find it’s skyrocketed in the last year. In particular, around halfway through the year, the term gained a lot more prominence.
This sudden increase came about because of a number of factors:
- Almost every phone now has a camera which means everyone is carrying a device which can capture what’s around them into a processor.
- Smartphones, which are also becoming increasingly common, now have reasonable internet connectivity to retrieve useful and contextual information (insert AT&T joke here).
- Phones are now equipped with GPS enabling phones to recognize where the user is.
- Most recently, and perhaps the tipping point, both Android phones and iPhones added digital compasses to their devices thus allowing software to detect not only where the user is but also which way they are facing.
- And finally, to a lesser extent, the accelerometers on these phones enable designs based on the user’s movement.
So What’s It For?
Augmented reality applications are being used in a lot of different ways. It’s used as art in installations like the N Building:
“Our proposed vision of the future is one where the facade of the building disappears, showing those inside who want to be seen. As you press on the characters their comments made on online appear in speech bubbles. You can also browse shop information, make reservations and download coupons.”
But it’s also starting to see some practical uses. For example, instead of just finding out the price of a product, LEGO has a prototype where you can preview the actual product from different angles.
The majority of mobile AR applications we’re seeing right now relate to location-based searches or visualizations. Urbanspoon, Yelp, and Wikitude are all examples of this. Layar, one of the leaders in the augmented reality space, allows you to overlay different data sets such as Wikipedia entries or tweets in the area.
Unfortunately, there’s a glut of rather useless augmented reality gimmicks that offer little value. You’ve probably seen some of these like the Robert Downey Jr. Esquire magazine cover, the action figures from the movie Avatar, or this one for Topps baseball cards.
Much like the pitching “game” included, most AR games are little more than poorly implemented retro games which seem hi-tech because they’re overlaid on top of a camera. Again, there are exceptions and innovators. This demo from Georgia Tech and SCAD-Atlanta creates a narrative around the player (flying in a helicopter) but also adds an additional dimension that involves … Skittles!
It’s actually been quite surprising how fast this technology is being adopted or experimented with. The United States Postal Service isn’t exactly the place one would expect digital experiments to come from. However, they have one of the first useful demos of how AR might be useful. By printing their custom marker, you can see a translucent shipping box in front of which you can then use to determine if the object you are shipping will fit.
Challenges
The future is so close we can virtually touch it but being as new as it is, developers and designers in this space have many challenges ahead. One clear problem — and opportunity — is the lack of any interaction design patterns emerging from common use cases such as AR browsers. The AR focused blog Augmented Planet has a nice summary of some of the ways applications are tackling one particular design problem but this overview also serves to point out just how disparate the interface currently is.
Augmented Planet also makes a case against AR which, among other points, expresses concern over privacy issues.
For example your mother comes over to your house and tweets about your priceless collection of Ming dynasty vases. Your home location is geotagged and out there for all to see along with details of your most valued possession. An enterprising thief using the latest version of BurglAR would be able to see high value items worth stealing in the local area.
As the mashup Please Rob Me demonstrates, these issues aren’t unique to AR but are certainly relevant.
When I first discuss AR with someone who hasn’t been paying attention to the space, their eyes often light up and their imagination starts to take off — much as mine did when I saw the Nearest Tube application. Often, ideas would revolve around turning a corner, or reacting to a step forwards or backwards.
Unfortunately, what we quickly found was that the accuracy of consumer-grade GPS is simply insufficient for many of these ideas. In particular, indoor GPS is even more inaccurate. Look at the disclaimer that Car Finder feels is necessary to show every time you launch the application.
If you were standing indoors, your compass may also be unreliable. On the iPhone, it asks that you wave the phone in a figure-8 fashion to get a better fix. I sometimes wonder if that motion cures hiccups as well …
Other technologies are also in their infancy. A lot of augmented reality’s promising comes in recognizing what your camera is pointed at but consumer engines are barely able to identify when something is a face, much less whose face it is. Much of the image recognition processing is also done after the fact rather than in real-time.
Even if the technology was there, the processing power for mobile devices would not be able to handle the requirements without burning a hole through your pocket.
It Can Be Useful
A lot of the products we see right now use augmented reality in a rather gimmicky fashion — adding little to the application’s utility. In most cases, the AR views are better served with maps or, in the case of games, are little more than overlaying a mediocre game over a live video feed.
However, we’re starting to see some real uses both on the market and in prototype form. As technology catches up and the ubiquity of AR-capable devices increases, we should be seeing more and more applications like these.
One way to offer better AR experiences now is to do so with strategically placed kiosks. Instead of depending on mobile technology, kiosks can be customized to the environment. This Japanese company tried a kiosk where you can try on different makeup in an augmented reality application. Once you’ve picked out some product you like, the kiosk then prints out an order form.
Some applications are particularly useful in mobile environments. Assuming sufficiently accurate GPS, the iPhone Car Finder application I mentioned earlier is a great use of augmented reality. You mark where you car is as you get out of the car and use that marker to find your car later. Such an app wouldn’t be useful to necessarily remember what street corner you parked in so much as what spot in a garage you might’ve parked in. The former is much better served with a marker on a map whilst the latter doesn’t have any map to mark!
In the near future, we may see even more utility from mobile applications. Nokia has announced that they are working on a mobile language translation application that utilizes Optical Character Recognition (OCR). We may be closer to a true Babel fish than we realize.
Once real-time image processing becomes more powerful, we can extend Nokia’s concept to recognizing faces and finding out more about people (or for starters, helping me remember names of people I’ve met).
And recognition can go beyond just faces. Recall that the term was coined at Boeing in reference to technology that might help people assemble aircrafts. BMW has the same idea and released a concept video of what an augmented reality training tool might be like. Hopefully, by the time you bring your 3-series in, the mechanic already knows what they’re doing.
Looking Ahead
Between the abundance of gimmicky AR “applications” out there and the technological hurdles we’ve yet to overcome, it may seem like it’s still too early to look at this form of interaction as something we should examine. But some companies, like Juniper Research, might claim otherwise:
“The market for mobile augmented reality (AR) services is expected to reach $732 million by 2014.
The annual number of mobile downloads featuring augmented reality (AR) content is expected to rise from less than 1 million in 2009 to more than 400 million by 2014.”
I’m dubious about those estimates, especially in relation to revenue. However, I do think AR may become an element of many applications in the near future. We will see many more applications that use AR as an alternate interface but hopefully, we will also see more applications that are specifically designed for AR’s strengths.
The rapid increase in smartphone market penetration will also mean a larger audience for AR applications. Based on this Changewave Research survey, we’re quickly headed towards technological ubiquity.
Smartphones may not even be the limiter when it comes to augmented reality. In addition to having location specific kiosks, we might be seeing some solutions that seem right out of science fiction. For example, instead of the awkward interface of holding up a phone in front of you, you could potentially be wearing augmented reality contact lenses.
Google Goggles hints at the future of augmented reality and they come the closest right now to making something that’s actual useful to the mainstream consumer. Watch their video and see what they have to say about their limitations, but also some truly interesting opportunities in the future.
Sadly, it seems like North America isn’t terribly interested in AR. Looking again at Google Trends and the searches for “augmented reality” in 2009, we don’t show anywhere in the top 10. Even English speakers as a whole are a distant third in AR interest.
Much of this might be attributed to the fact that mobile technology and adoption is much slower in the U.S. than in Asia but with iPhone and Android phones coming from right here in Silicon Valley, I would expect a much higher interest in the field than we’re seeing.
The following video shows some of our friends playing our ARGH iPhone game. Our game is simple and doesn’t even come close to the potential of what AR can offer in entertainment and utility but when you look at the expressions and engagement of the people that play the game, it’s clearly an immersive and delightful experience.
I think we may be missing an opportunity here to create incredibly interesting applications which actually take your existing environment and context into consideration. If design’s mantra is “it depends”, augmented reality offers a way to create interfaces that depend on what people are doing, looking at, and listening to.
