The future of Android: An up-close look at Google Now on Tap

Google's latest innovation is poised to change the way we interact with Android -- and there's far more to it than first meets the eye.

Computerworld | Jun 4, 2015 9:28 AM PT

When it comes to the evolution of Android, change usually seems to fall into one of three main areas.

First and most immediately noticeable are the broad and sweeping visual overhauls -- like what we've seen with Android Lollipop and its accompanying Material Design standard.

On a less obvious but no less significant level are the under-the-hood improvements, such as the efforts to boost battery life and performance in recent Android releases.

Then there's the third category -- the one that's trickiest to pinpoint when it first appears. That category is the addition of a seemingly simple service or feature that has the potential to dramatically change the nature of the operating system and how we use it. Items in this category are rare, but they can end up being the most transformative of all.

Google Now on Tap is one of those items.

Google Now on Tap, introduced in the middle ofa giant string of announcements at Google's I/O developers' conference last week, is an expanded version of the Google Now system present on Android since 2012. Back when Now first launched alongside the Android 4.1 Jelly Bean release, I referred to it as the platform's "new secret weapon."

Make no mistake about it: Now on Tap is no less pivotal. In fact, it makes the original Now launch look like little more than a stepping stone to the fleshed-out product we're about to experience.

Here's a detailed look at what Now on Tap is all about, how exactly it works, and how it'll change the way you interact with your Android devices -- including some new nuggets of info we hadn't previously heard.

What is Google Now on Tap?

In essence, Google Now on Tap takes the power of Google Now and makes it a core part of the Android operating system. Instead of just being a standalone app that you open sometimes -- and maybe see in your notifications once in a while -- Now on Tap turns Google Now into a connective tissue that ties everything on your device together with potent Google intelligence.

How does Now on Tap work?

Starting with this fall's Android "M" release, Now on Tap will pull up anytime you tap and hold your device's Home button. Instead of being a full-screen app-like experience, it'll appear in a card-like interface that sits atop whatever app you're using -- and it'll provide contextual info related to what's on your screen at any given moment.

Okay -- so what exactly does Now on Tapdo, then?

As Google explains it, tapping and holding the Home key tells Google Now to use the context on your screen to give you more information about whatever you're seeing. That allows you to get extra knowledge about something without having to stop what you're doing, go to Google or your Web browser, and manually search for what you want.

Real-world examples are really the best way to understand it. Say, for instance, you're looking at a text message from a friend suggesting a particular restaurant for lunch. You could tap and hold your Home key, and Now on Tap would instantly bring up a card showing you info about the restaurant your buddy mentioned -- ratings and reviews, a quick link for directions, and links to get even more info by jumping over to a relevant app like Yelp or OpenTable.

Google

Similarly, if you're reading an email where a friend mentions a movie title, calling up Now on Tap would give you a card with reviews of the movie along with links to watch the trailer and get more info from apps like IMDB or Flixster.

Google

The system can also use contextual info on your screen to answer a question. If you're listening to a song, for example, you could simply say: "Okay, Google, who's the lead singer?" -- and Now on Tap would pop up a card with the answer. Or if you're looking at a place in Yelp, you could say: "Okay, Google, how far is it from here?" and Now on Tap would figure out what you mean and show you a map with the info.

Google

What Readers Like

Oh dears! Does this mean Google is going to be watching everything I do?!

Nope -- not at all. First of all, Google has confirmed to me that Now on Tap scans your screen only when you press and hold the Home button to activate the feature -- never before, and never in any ongoing or continuous manner. It sees what you're doing only if and when you tell it to look.

Beyond that, Now on Tap will be a fully opt-in feature. You'll have to opt in to use Google Now, just like you do already, and you'll then have to opt in a second time to use the system-wide Now on Tap system. If you don't opt in, Now on Tap will never run.

Got it. But how exactly does Now on Tapdo all this stuff, anyway?

According to Google, Now on Tap uses a system known as Android's View Hierarchy. For the non-geek-speakers among us, it basically scans the words you're seeing on your screen, identifies the most relevant terms, pastes them into a Google search, and then comes back with the info it thinks you need in a nicely structured card format.

And -- theoretically, at least -- it does all of that in a fraction of a second.

So Google Now on Tap can only "see" and interpret plain text?

For the moment, it'll focus on text -- but Google has hinted that the system could eventually scan and recognize images as well and provide you with contextual info about pictures you're seeing. Think Google Goggles kind of magic -- identifying famous works of art, landmarks, bottles of wine, and so forth. But that's all speculative, and I'm told recognition beyond basic text is likely not going to be present right away.

Will Now on Tap work from every app, or will developers have to actively support it in order for it to function?

Now on Tap will work out of the box with any app -- no special coding or compatibility required.

That being said, Google tells me developers willbe able to take extra steps to mark up their apps and make them even more effective with the system. A game developer, for instance, might code in information that'd let Now on Tap know what level a player is in at any given moment. Then, if the player were to press and hold her device's Home key, Now on Tap could pull up content and links relevant to that specific level in the partial-screen card.

In order for an app to appear as a link within a Now on Tap card, meanwhile, -- like IMDB and Flixster did in our example a minute ago -- its developer would need to have implemented app indexing so the system can find the relevant data and point users to the right place. App indexing is nothing new, though, and most apps that should support it already do.

All right, wise guy: If Now on Tap works everywhere, what's to stop it from seeing sensitive info -- like something from a banking app?

It's a good question, and one I posed to Google as well. First of all, remember that Now on Tap sees what's on your screen only when you activate it -- so it's not going to be suddenly snooping at your financial info and transmitting it via the local tinfoil network. 

But beyond that, a Google spokesperson tells me the company is looking into ways in which it could allow apps to mark certain types of info as sensitive and thus not compatible with or accessible to Now on Tap at all, under anycircumstances -- even if you were to go out of your way to activate the system while such info is on-screen. The feature is still being developed and the specifics aren't yet concrete, but I'm told the issue is absolutely something engineers are considering.

What about data storage? Does Google maintain any type of history of the info it scans from my Now on Tap searches? 

Nope. Google assures me the data is processed and immediately discarded and that no sort of history is ever stored.

What happens if Now on Tap detects competing info on the screen -- say, if I'm looking at an email that mentions a movie title and a restaurant name?

It'll give you two cards -- one for each relevant bit of info. Easy enough, eh?

What happens if I activate Now on Tap and it can't find anything on my screen with an obvious card to accompany it?

I wondered the same thing (my, you ask excellent questions!). The answer, for now, is that Google is still thinking this through -- and it's still up in the air. The ladies and gents working on the feature figure folks will typically tap and hold the Home key when there's an obvious path and area of info to be obtained. But we all know someone is gonna test the system in all sorts of weird scenarios just to see what happens (hey -- why's everyone looking at me all of a sudden?!).

My first thought was that Now on Tap could just show you plain search results in such instances, much like what happens when you ask Google Now or even Google itself a question for which it has no simple answer. But the Now architects may have a better solution: Right now, the team is thinking about whether there might be basic helpful things the system could show on a card when there's no specific info connected to what you're doing.

If all else fails, a photo of a tuba-playing puffin in a top hat might also be quite enchanting. (Just my two cents.)

Crikey! That's a beautiful bird. But one last question: Is Google Now on Tap going to replace Google Now as we know it -- or will the full-screen app-like interface also still exist?

This one has another big fat "TBD" stamp attached to it, so things may change as we get closer to the launch -- but as of now, I'm told that tapping and holding the Home key is being positioned as the universal gesture to get to Now across the entire Android platform. And where you perform that gesture will determine what exactly transpires.

If you tap and hold the Home key from an app, you'll get the new Now on Tap cards. If you do it from the home screen, you'll likely get the full Now UI (similar to how you do today).

Consistency is the overarching goal, but different implementations may make sense for different scenarios.

The future of Android: An up-close look at Google Now on Tap

10:52 AM Android M, Google Now, smart phones

Google I/O 2015 The 12 most important announcements

At its 2015 I/O keynote, Google focused mostly on optimizing Android — but that's actually a pretty big project. A lot of the updates involved reducing friction: there were updates to Android designed to make moving between apps and the web smoother, updates to Android Wear designed to make it easier to glean information from watch faces at a glance, and a new photo app that will organize all your pictures and back them up for free.

There are also some new, potentially interesting features for the existing apps. Google Now will start to anticipate your needs wherever you are on your phone. If you’re in a region with poor connectivity, Chrome and Maps will be capable of more, including giving directions while offline. As for hardware, there’s a new Google Cardboard — and an educational field trip initiative — and a VR camera system that will be sold by GoPro, but the plans for which will be made available this summer for anyone to make their own.

Android M

Android M looks a lot like its predecessor Lollipop. It’s more like a reorganization of Android, with power optimization, new services, and changes to how your apps interact. If you’re a developer, you can check out the preview today. Otherwise, you’ll be getting it later this year.

App permissions get less intimidating

Android has traditionally given you a wall of permission requests — accessing your location, using your microphone — whenever you install a new app. That’s all supposed to be changing with Android M. There are fewer update categories, and you’ll no longer be asked to look at them on installation. Instead, Google will ask you to approve individual permissions when the developer calls them up. You can also look up everything that the app is accessing, or which ones are using, say, your calendar or camera.

Chrome in all your apps

Right now, when you click on a web link from within an app, you either have to load the browser or use a stripped-down web view in the app. Google announced something called Chrome Custom Tabs, which looks like an in-app Chrome browser and is supposed to make loading pages faster by preloading certain elements, including passwords and autofill. Google also announced measures to make it easier for apps to link to other apps, for example, loading the Twitter app automatically when you click on a link to a tweet.

Better battery life and USB-C

In an effort to save battery life, Google introduced a new feature called Doze. Android M uses motion detection to go into deeper sleep if inactive for longer periods of time. Google said it trades "app freshness" for battery life. While dozing, devices can still respond to high-priority messages and use alarms. In a test against a Nexus 9 running Lollipop, Google, said the M Nexus lasted up to two times longer in standby. USB Type-C will also be supported on Android.

Google Now on Tap

Google Now is getting smarter, and it’s going to be incorporated throughout the phone, through a program called Now on Tap. With Now on Tap, you can hold the home button and bring up Now cards with relevant information, whether you’re in an app, email, or web browser. For example, if someone emails you about going to a movie, summoning Now displays information about whatever was mentioned, with links to YouTube trailers, ratings, and other info. If you’re messaging about laundry and dinner, it can prompt you to set up a to-do item, and give you restaurant listings, along with buttons for Yelp, Maps, and OpenTable. It’s a big expansion for Now, and Google emphasized improvements in contextual understanding. In the demo, asking "What is his real name?" while listening to Skrillex in Spotify brought up the answer without needing to specify "Skrillex."

A new Google Photos app with free online storage

Google already has a photo tool for your Android phone, but Google Photos is a revamped appthat will back up an unlimited number of photos and videos for free (photos up to 16MP, video up to 1080p), organize them as a timeline, and group them together by locations, things like "beaches" or "boats," and people — using what purports to be some pretty advanced auto-tagging. And there are new interface tools for sharing your photos or making collages and movies out of them. Yes, it’s competing with some similar iOS tools, but it could stack up pretty well; it’s available starting today for Android, iOS, and web.

Offline Maps and Chrome for developing countries

Google announced a bunch of updates designed to make its products work better in parts of the world with poor connectivity. A new streamlined search results page will load faster with a spotty connection and optimize pages to load fewer images. Google already lets people save YouTube videos offline in some countries, but now, Chrome will also be able to save pages for offline use, and Google Maps will be able to run a bunch of features offline. In an onstage demo, it could autocomplete searches, show reviews and information, and give turn-by-turn directions offline.

Android everywhere

There's more to Android than phones, obviously. If Google has anything to say about it, it'll be on your wrist, in your house, and in your wallet... sort of.

Some more polishing for Android Wear

Google added some powerful new apps to its Android Wear smartwatch OS, including an easy option for calling Uber cars. It also reiterated some of the ways it's smoothed out the platform. The screen in Android Wear is always-on, Google noted several times, and now apps will be always-on as well, displaying information in a low-power black-and-white mode. If you’re navigating with Google Maps, for instance, directions will stay on the screen, making them glanceable as you move. There’s also a new app launcher designed to make loading apps faster and easier. There are a bunch of other small updates as well, including the ability to recognize drawings and turn them into emoji.

Google wants to manage your smart home

Project Brillo is an operating system built on the "lower layers of Android," Weave is a communications system that will let smart devices talk to each other, and Google hopes you’ll install both of them on your door locks, light bulbs, and thermostat. There’s built-in support for Wi-Fi and Bluetooth, and Google will have a certification platform for Weave, which companies can use with or without Brillo. A lot of people have tried to create a standard for home automation, including Google — it announced Android@Home back in 2011 — but Google will officially be throwing its hat in the ring again by the end of this year.

Android Pay gets closer

We’re getting some more details about Android Pay — which is both Google’s answer to Apple Pay and a second chance for the largely failed Google Wallet service. Android Pay will let you make purchases in apps or tap an NFC sensor to pay for physical goods. Android Pay will work with phones running KitKat and higher, and it’s supposedly being pre-installed by AT&T, Verizon, and T-Mobile; Google promises 700,000 stores will accept it, including Chipotle and Dunkin Donuts. And if your phone has a fingerprint sensor, you can use it to authenticate payment.

HBO Now is coming to Android

We thought we might hear more about TVs from Google this year than we did, but though there wasn’t much news about the Chromecast or Android TV, we did get one new development. HBO Now, the streaming-only service for watching everything from Game of Thrones toTrue Detective, is coming to Android. There’s no firm release date, but this marks the end of Apple’s exclusive access.

Virtual reality

Yes, this gets its own section now. It might still be a fledgling field, but Google put some decent effort into promoting VR during the keynote.

New, iPhone-compatible Google Cardboard

Phones have gotten a lot bigger in the year since Google Cardboard launched, so Google is releasing a new version which can fit phones as large as six inches. The new unit, which goes on sale from partners today, is ditching the headset’s original magnet clicker for a cardboard button that will work with any phone. Google is now making its SDK available for iOS as well as Android, and a dedicated Cardboard app launched on Apple’s App Store today. In order to help kick-start VR use, it’s also announced Expeditions, a virtual reality education tool that lets students take VR field trips while teachers control their experience with a tablet.

A VR Camera rig for everyone

In an effort to make it easier for people to film VR-compatible video, Google partnered with GoPro to build a circular 16-camera rig. The geometry of the rig, called Jump, will be made available this summer for anyone to build their own. Once you’ve recorded something with these cameras, Google’s assembler will figure out depth data from the different images and stitch them together into a stereoscopic VR video. Google said the assembler takes "thousands of computers" and would be available this summer to a "select few" people.

What's next

There wasn’t much talk of Google's new Project Fi wireless carrier at the keynote, or of Android Auto, though the introduction mentioned that 35 car brands are participating. Instead, the focus was on improvements in Google’s machine learning, deployed in Google Photo and Now on Tap. Expanding connectivity was also a theme. Sundar Pichai concluded by saying that Google is about solving problems for everyone: "it's about putting technology to work on important problems that users face and do it at scale for everyone in the world." In the last few minutes, he touched on Google’s self-driving car, the latest version of which will soon begin driving around Mountain View, and balloon-based internet service Project Loon, which he says will expand connectivity to the next billion users.

Google I/O 2015 The 12 most important announcements

8:34 AM Android M, featured, GOOGLE I/O, Google Now, smart phones

Native Fingerprint Authentication In Android M

Fingerprint Authentication

Google's yearly I/O conference is taking place at the end of next week, and if a recent report is worth taking into consideration, then we might witness the new fingerprint authentication feature being released with Android M

This year’s Google I/O conference will take place at the end of next week, on May 28-29, and it’s said to bring a lot of news regarding the upcoming Android M version. Google is also expected to discouss about aew Google Glass device, the updates on Android One, a new Chromecast and much more.

Now, a fresh rumor from BuzzFeed suggests that the upcoming Android M software version will feature native support for fingerprint authentication. Of course, there are already plenty of Android device which feature fingerprint readers but they are using software built specifically by the OEM. With this new native feature, Google wants to make it easier for phone-making companies to include a fingerprint scanner in their devices.

ArsTechnica too says that there have been hints in AOSP (Android Open Source Project, the Google leaded project for Android) that fingerprint readers were a planned feature. The publication says Google was working on the:

setup and enrollment process, a fingerprint lock screen, and a “service to manage multiple clients that want to access the fingerprint HAL API

Ron Amadeo with the publication further adds:

a Google-standardized API would encourage the entire Android ecosystem to finally jump onboard the fingerprint bandwagon. Widely used Google apps like the Play Store would support it, and it would even open up the possibility for Chrome to fingerprint website logins. A Google API would be stable enough and universally supported enough that third-party apps would begin to support it.

If Google does decide to announce fingerprint reader support at I/O, then chance are they will release preview build of Android M just like it happened with Android L back in 2014. But it also needs pointing out that Google currently has no device with a fingerprint sensor, even if the Nexus 6 was very close to getting one.

So, taking this into account, it makes sense for Google to announce this feature when it has a device with a fingerprint reader, as well, so that it could take the chance to promote it. But as Google usually releases its Nexus devices around fall, then we could wait for a little longer. Or, who knows, maybe they will surprise us and announce a new device at the event from next week. Whatever happens, we’re going to be here to report it.

Native Fingerprint Authentication In Android M

5:59 PM Android M, Fingerprint, GOOGLE I/O, NEXUS, smart phones