Comparing Siri and Google

Posted on November 18, 2012
Filed Under iPhone/iPad, Siri | 4 Comments

SiriDuring the time I was writing this post I had my worst experience with Siri ever. So honestly, I get all the complaints about Siri. I really do. I was driving home and trying to get iMessage to type via voice the simple phrase, “on my way home.” It took nearly ten tries to get it. And most of the time I just got the blinking three dots indicating Siri itself wouldn’t work rather than just bad recognition. Now to be fair to Apple this probably was partially been due to AT&T installing LTE in my area.1 However when I got home and tried to use Siri on WiFi I was still getting huge delays and confused recognition. I assume this meant Apple was working on the server during a low usage period.2 How much was Apple installing modifications to Siri or AT&T screwing up their service I don’t know. It was frustrating enough to me that I nearly took back every kind thing I’d ever said about Siri. And it’s hardly an excuse for Apple to say they are working on their servers. Still after a few hours it calmed down again and starting behaving much better.3 By evening though Siri was operating normally again only this time with significantly improved performance.

All that said the past few weeks, especially since the release of Google Voice Search, Siri has been savaged by bloggers and various tech sites. It’s just not been my experience. I think it does a very good job — and I thought this even before today’s updates.

Let’s cast off what Siri was like before now and especially before this summer. Siri’s been getting better. It’ll continue to get better. Those of us who first encountered using Siri with our new iPhone 5′s just didn’t have the experience 4s users have had over the past year. To make the comparison fair, I’ve broken the comparisons down between WiFi (which primarily compares the servers) and Cellular (which tests how well the service does under bad network conditions). Most importantly I’ve tried to test what I consider to be realistic types of queries. 

Let me address that last point a bit more. I’ve seen lots of reviews using Siri in frankly silly ways. First off are the reviews, best exemplified by Ars Technica’s review, that are basically bar bets or the sorts of queries you’d do watching TV. Those are silly because if you are in a bar or at a party it’s going to be too noisy to use Siri. Further you’re probably easily able to just do a regular Google or Wolfram Alpha search using the keyboard. What’s the value of Siri there? Also doing extra clicking within Siri is trivial. TV or Movie queries are silly because if you are watching TV the TV noise is loud enough to make voice recognition horrible.4 Why on earth would you do that rather than firing up Google or IMDB to do the query? It makes no sense. Honestly I don’t think either Google Voice Search or Siri should be judged as a search engine. But that’s how most people are testing them. I’ll lay great odds that in their day to day life though they never use Siri or Google Voice Search in that way. They are theoretical rather than work flow oriented reviews.

The place Siri or Google Voice Search make such a huge difference is where you can’t use two hands to type or can’t really look at the phone screen. For me that’s either when doing home improvement tasks or more prominently when driving. Driving in a car is the killer feature of Siri. 99% of my use occurs in the car. So if you are going to test Siri, test it with the types of things you do in a car.5

IMG 0184Here’s the problem. Both services still tend to be oriented around sight too much. To be fair Google often reads of the results of its query. It’s voice synthesis is vastly better than Apple’s as well. Apple far too often pops up a skeuomorphic continuous form hole punched printer paper with text that is too tiny.6 It also gives a lot of extra information but doesn’t distinguish clearly the main answer from extra information. That’s not a big deal when you can look at the screen. But if you could leisurely look at the screen would you really be using Siri? I have noticed that as of today Siri appears to be reading more results.7 I’ll make a guess that Apple (and Jony Ive in particular) is very aware of Google’s interface and Siri will start speaking even more in the future.

However as I said Google’s not great here either. It speaks some things more than Siri, but often it relies on giving a result that looks very much like a standard Google search. It’s just as an algorithm where it displays a summary of each page with what it thinks you are looking for. Sometimes it works. Sometimes it doesn’t. The problem is that, like Siri, it assumes you’re going to do a lot of reading. I will say this for Google though. When it is a simple fact it speaks it. So when I query, “who is the Prime Minister of Canada” it says, “Stephen Harper,” It displays a picture of him and then has some relevant Google results. Siri gets the right answer but doesn’t read it and displays a frankly pretty ugly bit of facts in the Wolfram Alpha results.

So if you are just doing Google like searches you should always use Google Voice Search. You’d be dumb to use Siri to be frank. I’d say you could launch Google from Siri but if you try you just get a list of all of Google’s applications. You then have to click on “Google” as opposed to “Google+” or “Google Earth.” Too bad Google didn’t rename it “Google Search.” I’ll give two caveats. If you are doing technical searches then Wolfram Alpha is vastly superior to Google. Just ask Siri, “ask Wolfram what the integral of x^2 is.” Try that with Google. Then try sports.

For sports scores Siri is just considerably superior to Google. The place where Siri is a little brain dead is in pulling up stats on previous games. It seems like Siri is tuned to the upcoming or current football games but can’t give me an overview of the season thus far. That info is easily available from their partner ESPN in a nice little table. But Siri can’t seem to get it. On the other hand Google does a regular Google search and nothing remotely like I want is anywhere in any of the results. This is pretty surprising given that Google undoubtedly can find that information. Further while I rarely do general queries while driving I frequently do want current sport scores. I think Apple’s been wise to optimize here. I just wish they’d do more to display scores from previous weeks. (If I specify the game, it’ll give me the score from a few weeks ago – I just want more)

One other type of query I expected either Siri or Google to get was TV listings. If I ask, “when does Mythbusters come on,” I surprisingly just get a regular Google search from both Google and Siri.8 The first hit is the Discovery Channel website but the summary doesn’t give me a clue when to watch my show. I tried other shows and neither service did a good job with those sorts of questions. For movies though Google does great. When I ask when a movie comes out it speaks the release date and has the IMDB page as the first hit. Siri seems to not know at all about future releases.9

Where Siri is most useful is in dictating Tweets and iMessage messages. That’s where I use it 95% of the time. There’s no reason that Google couldn’t do that with its voice services. There are plenty of apps that send SMS. Heck, even Google Voice 10 can send instant messages. And sending tweets is probably the sort of thing even Twitter would allow. Maybe this is an update coming soon. But right now it’s very disappointing that Google not only can’t do useful text sending or note taking with its recognition but it can’t make appointments in Google Calendar or call contacts out of Google contacts or the Apple equivalents. (Contra some comments there’s absolutely nothing keeping Google from accessing the iOS contacts or calendar services) 

It’s weird that everyone is gushing over how great a job Google does on a feature that I bet almost no one uses in practice. They then merely mention in passing the key features Siri does spectacularly that Google doesn’t do at all. I get it. Google pushed the UI for voice search in a way Apple really should have done a year ago. Let’s hope that with Ive handling UI design that he focuses in on workflow because it’s clear Forstall didn’t think along those lines with Siri.

The final place where I actually think Siri and Google Voice are useful is in directions. Now it’s a little early to give the total win to Siri here for a couple of reasons. First, if anything, Google Voice Search has actually gotten faster and better from the last time I used it. Secondly rumor is that Google Maps may be out for iOS 6 in early January. If it is out it should be trivial for Google to use URLs to have Google Voice Search launch Google Maps.11 While Google is faster than Siri, hands down12 I honestly have a hard time saying Google’s better because the total query took 10 seconds instead of 4 from the time I started speaking. It certainly makes Google’s service look better. However Apple’s results look much better and includes some animation that gives one a feel for where you are going. Also Siri has Maps pull up full turn by turn directions. Google gives you a traditional map with some written directions below. It doesn’t remotely do as much.

Some directions still are better in Google. For instance if I say, “find me the nearest hotels,” Google gives a small map showing the hotels and then lists them and their distance away. Siri merely gives me a list. It would be much, much better if Siri brought up a map as well. The best I can say about Siri is that it gives Yelp scores for the hotel and the list takes up much less space than Google, allowing me to see all of them at a glance. With Google there’s too much white space and I can’t see all the listings. Google is superior in providing a button I can click on to call the hotel. (Something I’m rather apt to want to do if I’m looking for them) Apple has this information but I have to first click on the hotel in Siri which brings up the location in Maps. Then I have to click on the detail triangle which brings up an information window. Admittedly it’s a very attractive information window with pictures, reviews, and more. But it’s just too many steps. I don’t have a button I can press to call the hotel’s phone until that screen.

While Google definitely does some location searches better it often is confused. So if I ask, “where is Sundance” Siri correctly shows me where the Sundance resort is located.13 Google does a regular Google search rather than a location search. The first hit is the film festival14

The Tests

I ran a slew of tests to come to all these conclusions. I had a list of 40 queries I ran with each service three times. Originally I was going to list them all but I decided that probably wasn’t too interesting. The results were pretty similar within categories of queries.

For direction queries Siri varied quite a lot. The typical result came about 8 seconds after I finished speaking. For Google the result came nearly as soon as I finished speaking or at worst 1 second after. As I said above the times aren’t exactly commensurate as Siri launches a separate program and then does some animation for displaying the result. While it’s great that Google’s results are so fast, in practice they aren’t as useful as Siri’s if you’re looking for directions. Where Google is better is in speaking the result and making it trivial to make a phone call to the location. Some might like Google’s results but in terms of practical workflow I think Siri is vastly superior here.

One must note Apple’s Map program is still somewhat limited. That said, it has improved dramatically throughout the past month. Two weeks ago when I first compared Siri and Google I found that Siri/Maps couldn’t find many of the locations I asked about. There appears to have been a major update today, which probably explains the weird problems I had earlier. Thus far in this test I’ve not found a point of interest, store, or address it couldn’t find. Even when looking up categories of stores it finds all of them correctly.15 That’s a much, much better situation. I can’t promise that’s true of everywhere but in terms of searching  in my state Maps is basically fixed for me. I can probably delete those other map programs I got from the app store. I can’t see needing them anymore. 

There are still a few things one might wish Apple would do to maps. I’d love better building names the way Google maps has. My local college campus has every building named in Google but no buildings are named or even displayed in Apple maps. I’m sure Apple will add that soon. I’m just glad they fixed the horrible state of point of interest data. That had made maps and Siri nearly unusable for searches back in October.16

Weather queries were pretty similar. There’s a slight delay with Siri as it decides you’ve finished speaking. Google, as I’ll discuss in a moment, starts parsing your voice as you speak. So before you’ve even finishes speaking it’s already deciphered much of your query and probably has already started accessing search data. Surprisingly both Siri and Google were equally fast in finding the time of sunset or sunrise. Google got the forecast for today or tomorrow as soon as you stopped speaking. There was a delay of an extra 3 – 5 seconds with Siri. I preferred Siri’s visual results. Siri does speak the results but they are more subjective than Google. Siri will say things like “it’s kind of cold and wet tomorrow” whereas Google speaks the high and low temperatures and then if there’s a chance of rain or snow. (It sounds remarkably what you’d hear spoken on the radio) For converting temperatures Google’s much better coming in with the spoken conversion from Fahrenheit to Celsius. Siri doesn’t speak the conversion but gives a standard Wolfram result with too much data that’s too hard to read quickly. Hopefully Apple will optimize for temperature conversion in the future. One weather query neither did well was asking when the moon would be out tonight. Google gave query results completely unrelated to the question (a song on YouTube and an ad) while Siri didn’t understand the question either and provided no results.

Asking about dates was more interesting. If I ask, “when’s Father’s day” Google just does a general query and doesn’t speak anything. That’s a fairly common type of query I’d potentially make when driving. It does find it but the result is in small text mixed in a paragraph summary I have to read. Here it’s Siri that helpfully reads the results. It found all the holidays I came up with. It also is good about telling me how many days until a holiday, although once again it’s unfortunately in that standard Wolfram printer page result. Google found the number of days until Christmas in its second result. But it was once again in the middle of a paragraph summary. It didn’t display the days until any other holiday. 

Things have definitely changed since I first considered both Google and Siri two weeks ago. Google started out ridiculously fast but for me frequently misunderstood my voice. Back then I only had about ⅓ of my queries understood properly by Google. Today neither Google nor Siri misunderstood a single one of my queries. Not only did their accuracy improve dramatically but their already fast speed got faster. It is no exaggeration to say Google frequently displays results as soon as I stop speaking. And that’s remarkable. Google still has a lot of work to do in order to make their service useful. As I said I’m not convinced people want to use speech to do regular queries. But that’s what Google is optimizing for. Right now Google just doesn’t offer voice on iOS in order to actually do things.

Siri started out strong on doing things like making appointments, calling contacts, or sending messages. It has a nice interface where you can interact adding more data by further speech. So you might say, “send a message.” Siri will say, “to whom shall I send it?” “Nicole” “What would you like it to say?” I then give the message. It uses this conversation model in several other places as well. It’s also smart enough to let you say it all in a single sentence. So I can say, “message Nicole ‘I love you’” and it sends the right message. It’s this sort of work flow that makes Siri useful. Not conducting bar bets. The places Siri does do useful queries seem to be the very type of queries it handles best. Undoubtedly that’s not an accident.

That said Siri still has places it needs to improve. It really should speak far more results. For certain classes of queries Siri should display a large simple result and speak it rather than the stylized Wolfram result it now provides. Given that Siri already has started speaking more results, I suspect we’ll see that added over the next month. Siri also has a problem of not letting you speak over it. I’d like it to be able to let me answer before I have to listen to every part of the question she’s asking. Finally I think there are several types of queries Siri needs to be optimized for. Temperature conversions, forthcoming movie releases, television schedules, and time series sporting event statistics really are all things Siri needs to do better.

Siri’s recognition with background noise seems much improved from a few weeks ago, but it still could be much better. I ran several tests with a football game playing on my laptop a few inches away from my phone. At moderate volume Siri surprisingly still recognized every query. (Google did as well) One thing Apple could learn from Google is displaying what you said. Google’s great here because it displays what you say as you are saying it. I suspect Google is deciphering your voice on the phone itself. That allows this sort of immediate feedback. Apple waits until it thinks you’re finished and then sends the entire voice to its servers. Hopefully Apple will move more of the processing onto the phone and potentially start interpreting it as you are still speaking as Google does. For a long time with Siri I had to use a slow, exaggerated annunciation.17 Now I find I can speak much more normally and Siri still recognizes me.

Once Apple gets Siri working the way it should have been last year it can start adding more features. I mentioned a few improvements this week. Certainly a general API so Siri can interact with other applications is in order. I’d be shocked if that isn’t already in development. Hopefully it’ll be a part of iOS7. I’d also love to be able to change settings like bluetooth with Siri. Those things probably are all coming.

In conclusion my basic stance is that most articles about Siri have really judged it unfairly. It’s being compared to doing general Google queries while trying to get the results with as few clicks as possible. However the only reason multiple clicks are bad is the presumption you need Siri when you can’t click. However if you can’t click honestly you probably can’t read either. The real use of Siri should be situations when you can’t really do a lot of reading. While Siri does a lot there, I can’t help but confess it could be better. That said for the types of things one would typically use Siri for it is more than adequate whereas Google just isn’t a contender at all. This is a rapidly changing area though. Strong competition from Google will undoubtedly spur Apple on to improve even more. Just looking at how much both services have improved in two short weeks I imagine things will look quite different by spring.

  1. When I ran speed test I was getting 1.25 mbps down and 0.8 up and with a horrible ping of 250 ms. That was far worse than the 4G they were replacing. I’m hoping this was a temporary problem.
  2. To be fair, I did not run any query tests during this time. Which was smart. When Siri started working right again it was dramatically improved.
  3. I swear it started talking back more as well. Unfortunately I didn’t write down when it did speak back ala Google Voice Search. I was only part way through the tests at the time so I restarted them.
  4. This is why I found inexplicable all the rumors that Apple was revolutionizing television with a Siri based television. That’s the absolute worst technology to use while watching television.
  5. To be fair in my initial review of Google Voice Search I did several silly queries too. I can’t think of any reason why I’d want to know the first 21 prime numbers while driving down the freeway, for example. I was more curious as to the limits of Google’s search.
  6. Some of Apple’s skeuomorphic choices are very weird to me. Exactly why would you want the output from Siri look like it was printed on an Imagewriter II from 1985? Maybe that accounts for the few seconds extra Siri takes over Google? Trying to keep that experience of a dot matrix printer?
  7. This might be my imagination. Unfortunately I didn’t have something to compare it to in order to be sure. But I’m pretty sure it didn’t speak as many results last week.
  8. I must confess I got rid of my TV a year and a half ago. I only miss it during football games. Everything gets watched on my AppleTV or my iPad. It helps that TV isn’t exactly strong right now.
  9. That may change now that Apple’s announced Fandango as a partner for Siri. I expect we’ll soon be able to buy tickets, watch trailers, and so forth all with Siri.
  10. The answering service not the voice search. Yes I find it confusing too.
  11. It’s possible they don’t do this. But if they don’t they are honestly being idiots.
  12. The tests I ran consistently had Google pulling up a map basically as soon as I stopped talking. That is Google had a result about the time Siri had figured out you’d actually stopped talking and started sending your voice to Apple for processing. That’s frankly amazing and is why everyone is raving about Google’s speed.
  13. A ski resort and convention area run by Robert Redford a few miles away.
  14. Oddly not where the festival is located despite my asking where. The festival is primarily held up in Park City about 40 miles away.
  15. So a list of nearby gas stations or grocery stores actually finds all the stores of that class. Even two weeks ago that wasn’t remotely true.
  16. I was a bit of an Eddy Cue skeptic. Color me converted if he fixed Siri and Maps so fast. There’s still a lot to do, but at least the big problems are gone now.
  17. I frequently joked that I had to speak like a villain from a bad 50′s B movie. It worked but wasn’t ideal.

Comments