Getting Ready for Self-Driving Cars
Our test of semi-autonomous technology that follows was created before the public disclosure of the fatal accident involving a Tesla Model S driver using its Autopilot feature in May. It was the first known fatality in an autonomous driving scenario, a moment we’ve all been bracing for and one that brings a new gravity to the conversation about this fast-moving technology.
I’d like to mention a few things, then. The juxtaposition of Joshua Brown’s crash and the image you see above of an editor with his hands in the air as a Tesla steers itself (with oncoming traffic) is unfortunate. Any news organization will attempt to pair a story’s words with a provocative image, just as this one is. However, in no way whatsoever do we mean to suggest that having your hands off the wheel like this is anything but irresponsible.
In the accident’s aftermath, just about every carmaker with even the slightest self-steer technology is loudly insisting that you must keep your hands on the wheel at all times. That’s what the lawyers are telling them to say, and for the vast majority of drivers, I’d agree that it’s seriously good advice. Yet as these systems improve, all of us will be increasingly letting go of the wheel (if only out of curiosity). For three of the four cars in this test, my hands were at the ready, an inch away from re-grabbing control, the wheel having been released purely to see how the car responds. In the case of Autopilot, its ability to significantly self-steer for lengthy periods of time (in the right conditions) meant my hands were on my thighs, palms toward the wheel rim for quick re-control. However, at all times we were hyper-attentive to the driving task at hand. What all this strongly suggests is a rethink of the driver environment, including comfortable arm supports on the door panel and center console.
Brown’s accident should be seen against a backdrop of the accidents this technology will increasingly help avoid. Moreover, it underlines the importance of third-party testers like us to better evaluate this technology, add our insights to the boarder conversation, and also not to artificially inflate the capabilities of these systems in the minds of drivers. – Kim Reynolds
I unfolded the business section of the “Los Angeles Times,” read the headline on page 3, and shook my head. The story was about the technological hurdles facing autonomous cars, and it included a picture of a bleak, snow-blanketed road—the kind of whitewash scenario that’ll be a real conundrum for self-driving cars. I put down the paper and took another drink of coffee. Strange coincidence.
The day before had been a rough 14 hours in the desert testing four cars with early evidence of autonomous features. On the way to the rough-and-ready outpost of Mojave, a fluke winter storm had come whistling through, building to 35-mph crosswinds as the temperature trapdoored toward freezing. Then, just past Willow Springs International Raceway, the snow started. I left the Tesla Model S’ Autopilot on, cautiously folded my arms, retracted my feet from the pedals, and sat amazed at what was happening.
At first the flakes streaked horizontally, a stutter of headlight shooting stars, the landed ones gradually windswept into a pile along the road’s sagebrush edge. I could still see the left lane markings, but as the ink night clouded to a white smear, it struck me. I’m doing 70 mph against a hard, quartering wind into snow flurries, the steering wheel is making robotic corrections in front of me, and I think this car is driving better than I could right now. OK, maybe I’m just dumb. But maybe this is it.
It? The moment—for me, at least—when autonomous technology is flickering to life. In 1997, I rode in a nose-to-tail platoon of experimental Buicks that driverlessly sniffed a trail of magnets buried along Interstate 15 in San Diego. In 2007, I watched empty cars and SUVs careen around a shuttered military neighborhood during the DARPA Urban Challenge. In 2014, Audi’s autonomous RS 7 (nicknamed Bobby) hot-lapped me around the twisting Ascari circuit in Spain. And later, its autonomous A7 (Jack) chauffeured five of us journos on what was then the technology’s longest real-world demonstration, from Palo Alto, California, to Las Vegas. Every instance, a step closer.
To, well, now, as I whistle through this snow-streaked night with no magnets to follow, no DARPA handlers with kill switches, no 20-car entourage of support vehicles following us. Just me, this car, these sensors, this software, and my faith that I can trust it all on this slippery, lonely road. At 100 feet per second with my arms folded. And so the march of the robot car begins.
Learn about the four cars we tested here.
Along with its shrill soundtrack, a Charlton Heston-like war cry that they’ll have to pry my steering wheel from my cold, dead hands. History is no romantic. It inevitably taps the delete button on the quaint (ocean liners, steam locomotives) when an industry’s financial spreadsheet tabulates a better bottom line. Nothing that’s ever been cheaper, more effective, and safer has failed to replace the familiar and fond. Ask horse owners. Or manual transmission holdouts.
Recently, I had an interesting dinner with a voluble car-company chief engineer. Over the main course, he explained his reasoning why attaining full autonomy will take longer than popularly thought. Whereas computers are great for instant decisions (we humans dither), their interpretation of objects and subtle situational cues is almost infantile. “A plastic bag blows onto the freeway in front of you” he began, putting his fork down and using his hands. “You know it’s just a bag and it’s better to drive through it than slam on the brakes and risk being rear-ended. But software, not knowing if it’s a bag or a boulder, might hit the brakes. Moreover, it can take 15 seconds for a driver to re-engage amid an emergency. For now, these things must be restricted to long, boring highway commutes with an attentive driver ready to take over.” Which of course is autonomous driving purgatory, wherein you sit there, arms slumped, half paying attention as you daydream a little. However, the guy gave an excellent boilerplate presentation of the industry’s corporate story line.
History is no romantic. Ask horse owners. Or manual transmission holdouts.
But it completely misunderstands human nature. Despite campaigns to hang up phone-distracted drivers, NHTSA says about 660,000 of us are messing with electronic devices behind the wheel every instant of the day (the fear of cops simply lowering the phone futzing below window height, making matters worse). Dumb, but it’s a human compulsion. Initially, lots of drivers will be wary of early autonomous systems; a just-published University of Michigan study found that 44 percent of drivers wouldn’t buy them, just 15 percent would, and 95 percent wanted a steering wheel and pedals available regardless of who (or what) is driving. They’re wary. But as a guy who’s spent an unusual amount of time in these things—among the first few people to get an autonomous driving license—my news from the front is that they’re phone-app addictive. Wouldn’t you rather punch out on that tedious drive to work and get online? It’s easy money that that scrawny 15 percent will rocket.
Putting pressure on car companies to strike a balance between fail-safe confidence in autonomous tech’s robustness and the market pull of less discerning, tech-thirsty driver-texters. Must a system’s reliability be 93 percent? Or 97 percent? Or 99.99 percent with concomitantly restricted capabilities while you watch sales leak away to more autonomy-aggressive companies (such as Tesla)? This is a business, after all.
It’s hard to predict when Earth’s zombie car invasion will reach D-Day levels, but “The Wall Street Journal” reported earlier this year that Mobileye (supplier of many of those sophisticated windshield-mounted video cameras) has agreed to support two carmakers for fully autonomous capability by 2019; Mobileye chairman and CTO Amnon Shashua declined to name the manufacturers. Next year, Volvo will commence its consumer test of 100 autonomous cars in Gothenburg, Sweden. Google and Chrysler have thrown in to create 100 autonomous Pacificas and build a new 53,000-square-foot test facility in Novi, Michigan. Chevrolet Bolts encrusted with autonomous gear from GM-purchased Cruise Automation have been seen in San Francisco, and similarly festooned Ford Fusions have been spotted at Uber’s Advanced Technologies Center in Pittsburgh (near Carnegie Mellon University, an autonomy hotbed). And dare I mention it, Apple just invested a billion bucks in Didi Chuxing (China’s Uber rival), raising renewed speculation that an Apple Car (or van) could become an autonomous ridesharing shuttle.
Meanwhile, associate road test editor Benson Kong and I recently attended Nvidia’s 2016 GPU Technology Conference in San Jose, California, where several presentations gave a glimpse at how fast autonomy’s knottiest problems have loosened. For instance, a critical puzzle piece for around-car perception is going to be affordable solid-state lidar, which laser-paints a 3-D perception of the world and massively complements video. Currently, a complex suite of sensors (including unreliable mechanical lidar) can cost $20,000. A simpler system based on Quanergy’s solid-state unit collapses this to $1,000. How good is a single $250 lidar? One of Quanergy’s slides showed a curious scene: two parallel visual disruptions on a lidar-imaged road ahead. The unit can’t see color or sense reflectivity, but the unit actually perceived the thickness of the lane-marking paint on the road. A Ford presentation explored navigation on snow-blanketed roads by reckoning from landmarks such as buildings and fences using high-def 3-D maps. And our host, Nvidia, showed a video of its new and remarkable Drive PX 2 (a dense supercomputer fashioned into an autonomous starter kit for researchers and car companies); without traditional software coding, an Nvidia system observed a human pilot for a while and gradually learned (“deep learned”) how to drive. Just by watching.
All this has been a big windup to my pitch that autonomous cars are no longer a speculation, and “testing” will necessarily follow. But how will we do it? In total honesty, I’m not quite sure. This is our first attempt to find out. We’ve been tinkering, and I’d like to invite you behind the curtain of our usual professional sheen to see what we’ve been up to. What we have here is sort of a half-finished play. Some of the scenes need rewrites, and for sure there’s been some post-rehearsal forehead slaps. But it’s a conversation starter.
That’s probably best begun by considering how we’ve classically tested cars. Way back when I began, the primary tool of the trade was a “fifth wheel”—a free-rolling, carefully calibrated bicycle wheel with a rotary encoder. The first I used was the original electronic one with a printer that made an eeeeer eeeeer eeeeer sound as it slowly transferred numbers onto a tiny roll of paper. (The subsequent plots were pencil-drawn on graph paper with French curves and a squinting right eye.) Now we’re using GPS loggers running at up to 100-Hz sampling rates; we’ve even rendered it for 3-D virtual reality goggles. But regardless of whether it’s a rolling bike tire or the Doppler shift of an antenna relative to a constellation of satellites, what’s happening is exactly the same. You drive as hard as possible, the car performs, and everything’s recorded.
We’re dreading this. We love driving. Isn’t this about taking the keys away?
Testing autonomous characteristics will be primally different, though. Here, you expose the car to actual-use demands at everyday g levels—threading between lane markings or tracking traffic ahead—and then judge its skillfulness. The driver is a passive experimenter, observing the test tube.
But we won’t be in the business of rolling basketballs in front of cars (or tossing plastic bags). Our target is more narrowly determining how expertly and efficiently they “drive”—and the confidence they inspire from making reasonable traffic decisions. If you’ve sampled today’s radar-based adaptive cruise control in stop-and-go traffic, you know the dead-in-the-water pause as the car ahead drives away, leaving a gap filled by three happy guys next to you. Recently, a Tesla Model S I was driving was self-steering via both lane markings and a visual lock on the car ahead. We passed through an intersection where the markings disappeared, and then the guy ahead immediately turned right into a gas station. Which the Tesla started to track into, too. A mistake on many levels.
We’ve split our testing Pangaea into two continents: adaptive cruise control (following another car around the 6-mile oval at Hyundai/Kia’s desert proving ground) and steering (road tests at two real-world venues: a well-marked and little-trafficked 65-mph toll road in Orange County and a curvier 55-mph two-laner with inconsistent markings, irregular pavement, and occasional tree-canopy shadows).
For this autonomous comparison rehearsal, we enlisted four cars as guinea pig proxies for the robo-brigade to come. It’s a mixed cast of an apple and three oranges. The Tesla Model S, with its ever-evolving, always-controversial Autopilot and Autosteer, is nearly a Level 3 car, meaning under predictable circumstances it can genuinely drive itself for lengthy periods (with supervision). The Mercedes S-Class (an S65 AMG hosting the last hurrah of the current Distronic Plus with Steering Assist) is sometimes Level 2 when it’s video-stalking another car, but it’s otherwise Level 1. We included the pre-Super Cruise Cadillac CT6 mainly as the “before” part of an envisioned before-and-after comparison as we await GM’s Autopilot-like system. The fourth car was the Hyundai Genesis 3.8, picked for the positive buzz around its highway drive and traffic jam assist systems. (Both the CT6 and Genesis 3.8 are Level 1 cars. These levels are based on SAE standards, from Level 1, driver assistance, to Level 5, full autonomy.) We kept our hands off the wheel as much as possible, but not to make them look bad—we know, of course, that lane keeping is meant to be just a warning nudge, not actual self-steering. It’s a rehearsal with stand-ins. The results of these early tests, which you can see in the graphs sprinkled throughout, aren’t ready for a full-blown comparison, just as the cars aren’t ready for full-blown autonomy, but they do give you an indication of the sort of thing we’ll look at.
Zooming out to the big picture, nearly every conversation surrounding this subject triggers a tinge of concern in my colleagues’ eyes. We’re dreading this because we love driving. Isn’t this about taking the keys away?
The snow never stopped on that drive to Mojave. I only touched the brakes at the first stoplight into town, and it was so bone-cold at the Supercharger station down the street that I hesitated to get out and plug it in. In the mid-’80s, I drove out here two or three days a week for six months, arriving at Zero Dark Thirty to help with acceleration tests of Burt Rutan’s Voyager airplane. That was in those fifth-wheel days, and at the time, the drama of that boundary-pushing plane and its nine-day flight—through Pacific typhoons and a cutting-out engine—was riveting stuff.
Recently, I’ve noticed an echo of that fascination with SpaceX’s Falcon 9. Here and there around our office, computer monitors blink from their usual work to a flaming 45,000-pound rocket slowing from 5,000 mph to a delicate, robotic landing on a bobbing robotic drone ship. I watch, too, and still can’t believe what I’m seeing. As the Tesla supercharged in the bleak parking lot for the next day’s tests—it was about midnight with snowflakes swirling—I stared at the steering wheel. This Autopilot has a pixie sprinkle of the same robotic wonder, but one that you and I can strap ourselves into and witness firsthand. And when more and better and smarter ones like this arrive, we’ll be ready for them.
It happens to all of us. You’re at highway speed, and suddenly a vehicle cuts in front of you. Closely in front of you. Dangerously in front of you. Here are the speed traces of each car’s reaction, each of them quite different. The extremes? The Hyundai Genesis decelerated languidly, its slowing building gradually. The Tesla Model S, Mercedes-Benz S65, and Cadillac CT6 slowed more abruptly, but the Model S re-accelerated more cautiously.
The S65 reacted dramatically.
How fast do they autonomously accelerate? Here’s each car’s response to the lead car abruptly pulling out of the way (the dotted lines) compared to the track-tested best runs (all without the usual 1-foot rollout). The Tesla Model S was the quickest, both at the track (no surprise) and in autonomous driving. The Cadillac was painfully laggy off the line, making it a target for aggressive drivers angling to cut in front of you.
How deftly does each car’s adaptive cruise control track the car ahead in stop-and-go traffic? To find out, we repeatedly accelerated to 35 mph and then slowed the lead car (at real-world rates) with a pause in between. Here, the Mercedes-Benz S65 responded the soonest and stopped the closest, too. The Hyundai Genesis left the greatest gap—4 feet longer (about three-quarters of a car length total). When the lead car pulled away, the Tesla Model S and the Cadillac CT6 responded soonest. All of these delays would probably be frustrating in the real world.
Follow the Leader
With their radar-based cruise control set to its smallest gap, we measured each car’s following distance. All four fell well short of the recommended safe-driving 2.0-second interval; a UC Berkeley study found that 70 percent of drivers followed at 1.6 seconds or less, so our autonomous cars match unsafe humans in that regard. However, even the Tesla Model S’ unusually shortening time gaps as it approaches highway speed leave it vulnerable to aggressive lane changers.
Autonomous Steering on two road types
We employed two venues to analyze self-steering behavior: a 35-mile stretch of an immaculately maintained 65-mph toll road and a windy 17-mile, 55-mph two-laner. What you see here is the popularity of certain steering angles needed to traverse each road. (The Cadillac was unable to complete these tests due to its weak lane keeping.) The red line shows us driving as perfectly as possible. The other depicts the car’s best autonomous attempts. There is a lot of data on-center (the graph is tall) because of the predominance of straights; those lumps from either side are also tall because they’re the most common steering angles. The low points indicate steering angles rarely used. If each car’s autonomous steering duplicated Kim’s, which would be ideal, then the two lines would overlap.
Human Intervention When The Robot Driver Fails
Whenever the car demanded that we hold the steering wheel—it began to cross the lane markings, or we became nervous; the Mercedes and Hyundai were constant interrupers—the data logger recorded when our hands touched the wheel and the duration of the steering corrections. To do this, we made the steering wheel a big switch with wires taped to a hand and aluminum foil wrapped around the rim (completing the circuit).
Number of times touching the wheel ……………..141
Proportion of time touching the wheel …………11%
Number of times touching the wheel …………..140
Proportion of time touching the wheel ……….10%
Number of times touching the wheel ……………..22
Proportion of time touching the wheel …………2%
Number of times touching the wheel……………148
Proportion of time touching the wheel……….28%
Number of times touching the wheel…………….113
Proportion of time touching the wheel………..29%
Number of times touching the wheel……………..12
Proportion of time touching the wheel………0.5%