What happens when you take away the quality assurance team in a software development operation? Fewer, not more errors, along with a vastly quicker development cycle.
That, at least, has been the experience at Yahoo, according to Amotz Maimon, the company’s chief architect, and Jay Rossiter, senior vice president of science and technology. After some small changes in development processes in 2013, and a larger push from mid-2014 to the first quarter of 2015, software engineering at Yahoo underwent a sea change. The effort was part of a program Yahoo calls Warp Drive: a shift from batch releases of code to a system of continuous delivery. Software engineers at Yahoo are no longer permitted to hand off their completed code to another team for cross checking. Instead, the code goes live as-is; if it has problems, it will fail and shut down systems, directly affecting Yahoo’s customers.
“Doing that,” Rossiter told me, “caused a paradigm shift in how engineers thought about problems.”
It has also, he said, forced engineers to develop tools to automate the kinds of checks previously handled by teams of humans. An engineer might go through an arduous process of checking code once—but then would start building tools to automate that process.
I met with Maimon and Rossiter at Yahoo’s annual TechPulse conference on Tuesday in Santa Clara. This private get together gives some 850 of Yahoo’s researchers and engineers an opportunity to publicize their projects by presenting papers and participating in poster sessions.
It was an odd time to be surrounded by Yahoo’s tech staff—all of whom were focused on software developments—because in that day’s newspapers and in news reports I heard on the car radio as I drove to the meeting, rumors swirled about Yahoo’s imminent restructuring. The researchers believe that any change will take some time to affect their operations, so they continue on, business as usual. (There may have been more of a buzz about the company’s future the following day, when Yahoo announced that it had decided to go forward with a reverse spinoff: that is, transferring all its businesses and liabilities except for its stake in China’s Alibaba group to a new company.)
Those structural and financial maneuvers notwithstanding, Yahoo’s decision to take away the safety net the company’s software engineers had come to rely upon was big news. The shift wasn’t easy, Rossiter recalled. It required some tough parenting, with no exceptions, he says. “People would come in and say I’m special, I’m working in UI, I’m on the back end, I’m this, I’m that.” But by consistently refusing to give any concessions, it forced a rethink. “We said ‘No more training wheels,’ and it made a huge difference. We forced excellence into the process.”
“It was not without pain,” Maimon says—though the problems were not as big as he feared. “We expected that things would break, and we would have to fix them. But the error that had been introduced by humans in the loop was larger than what was exposed by the new system.”
“It turns out,” Rossiter chimed in, “that when you have humans everywhere, checking this, checking that, they add so much human error into the chain that, when you take them out, even if you fail sometimes, overall you are doing better.”
Of course, taking away the quality assurance jobs meant, well, taking away jobs. “Some of the engineers really cared about system performance types of things,” Maimon explained, “so they joined related teams. Some started working on automation [for testing], and they thought that was great—that they didn’t have to do the same thing over and over. And others left.”
Now, a year after the change, “It’s 100 percent working,” Maimon says. “It’s amazing. Even people who didn’t think it could ever work now think it’s great, and we are applying it to everything we do in the company.”