The December 2022 issue of IEEE Spectrum is here!

Close bar

Hedge Funds Look to Machine Learning, Crowdsourcing for Competitive Advantage

Hedge funds are testing new quantitative strategies that could supplant traditional fund managers

4 min read
Two employees of Quantopian sketch algorithms onto a glass surface in the company's headquarters.
At Boston-based financial startup Quantopian, a team of analysts and engineers create tools that allow anyone to write investment algorithms based on a personal hypothesis about markets. The company then invests in the most promising algorithms.
Photo: Quantopian

Every day, financial markets and global economies produce a flood of data. As a result, stock traders now have more information about more industries and sectors than ever before. That deluge, combined with the rise of cloud technology, has inspired hedge funds to develop new quantitative strategies that they hope can generate greater returns than the experience and judgement of their own staff.

At the Future of Fintech conference hosted by research company CB Insights in New York City, three hedge fund insiders discussed the latest developments in quantitative trading. A session on Tuesday featured Christina Qi, the co-founder of a high-frequency trading firm called Domeyard LP; Jonathan Larkin, an executive from Quantopian, a hedge fund taking a data-driven systematic approach; and Andy Weissman of Union Square Ventures, a venture capital firm that has invested in an autonomous hedge fund.

Many of the world’s largest hedge funds already rely on powerful computing infrastructure and quantitative methods—whether that’s high-frequency trading, incorporating machine learning, or applying data science—to make trades. After all, human traders are full of biases, emotions, memories, and errors of judgment. Machines and data, on the other hand, can coolly examine the facts and decide the best course of action.

Deciding which technologies and quantitative methods to trust, though, is still a job for humans. There are many ways that hedge funds can use technology to create an advantage for investors. Just a few years ago, high-frequency trading was all the rage: Some firms built secret networks of microwave towers and reserved space on trans-Atlantic fiber optic cables to eke out competitors by a few milliseconds.

Now, speed alone isn’t enough. Qi co-founded Domeyard in 2013 to execute high-frequency trades through a suite of proprietary technologies. The firm built their own feed handlers, which are systems that retrieve and organize market data from exchanges such as Nasdaq. They also developed their own order management system, which are instructions that determine how proprietary algorithms make trades.

Qi says Domeyard’s system might gather 343 million data points in the opening hour of the New York Stock Exchange on any given day. The company can execute trades in just a few microseconds, and process data in mere nanoseconds.

Hedge funds have begun to incorporate machine learning into their systems, hand over key management decisions to troves of data scientists, and even crowdsource investment strategies. If they work, these experiments could give rise to a new breed of hedge funds that rely more on code and less on humans to make decisions than ever before.

But thanks to advances in the trading software and systems available for purchase, most any firm can now carry out the high-speed trades that once set Domeyard apart. “It’s not about the speed anymore,” Qi said. Hedge funds must find new ways to compete.

Over the past few years, hedge funds have started to get even more creative. Some have begun to incorporate machine learning into their systems, hand over key management decisions to troves of data scientists, and even crowdsource investment strategies. If they work, these experiments could give rise to a new breed of hedge funds that rely more on code and less on humans to make decisions than ever before.  

One hedge fund called Numerai pays data scientists in cryptocurrency to tweak its machine learning algorithms and improve its strategy. “The theory there is can you achieve consistent returns over time by removing human bias, and making it a math problem,” said Andy Weissman of Union Square Ventures, which has invested US $3 million in Numerai.

Not all funds will find it easy to compete on these new terms. Domeyard can’t incorporate machine learning, Qi says, because machine learning programs are generally optimized for throughput, rather than latency. “I can’t use standard machine learning techniques to trade because they’re too slow,” she said.  

The third fund represented on the panel, Quantopian, provides free resources to anyone who wants to write investment algorithms based on a personal hypothesis about markets. Quantopian takes the most promising algorithms, puts money behind them, and adds them to one big fund.

“We’re tapping into this global mindshare to make something valuable for our investors,” said Larkin, chief investment officer at Quantopian.

To help the process along, the firm provides educational materials, over 50 datasets on U.S. equities and futures, a library of ready-made modules that authors can borrow to code in the Python programming language, a virtual sandbox to test their hypotheses, and support from a team of 40 in-house developers. If authors wish to incorporate machine learning into their algorithms, they can do that with Python modules such as sci-kit learn.

One project, or strategy, consists of multiple algorithms written across several layers. An author’s first step it to generate a hypothesis. Then, they choose which data, instruments, and modules they will apply to test that hypothesis.

Next, the author must build what Larkin calls an “alpha,” or an expression based on the author’s hypothesis that has been tested and proven to have some degree of predictive value about market performance. “The best quantitative strategies will have a number of these,” Larkin said.

Each alpha should generate a vector, or a set of numbers, which can then be used to make trades that will align with that hypothesis. The next step, then, is to combine the alphas and add a risk management layer with safeguards to prevent the algorithms from getting carried away.

Finally, the author fills in the final details of the system which include the timing of trades. Quantopian’s approach is admittedly much slower than Domeyard—the fund has a minimum trading interval of one minute.

To date, 140,000 people from 180 countries have written investment algorithms for Quantopian, and the company has put money into 25 of those projects. Its largest allocation to a single project was $10 million.

Once they’ve built a strategy, the original author retains the intellectual property for the underlying algorithms. If their approach is funded, the author receives a cut (generally 10 percent) of any profits that their strategy generates. Larkin estimates it takes at least 50 hours of work to develop a successful strategy.

Larkin wouldn’t share any information about the fund’s performance so far. But he said the idea is to blend the best data-based hypotheses from many people. “We at Quantopian believe the strongest investment vehicle is a combination of strategies, not any one individual strategy,” he said.

Larkin refers to Quantopian’s methods as data-driven systematic investing, a separate category from high-frequency trading or discretionary investing based on data science. Still, he classifies all three of these quantitative methods as distinct from the longtime approach of simply relying on a fund manager’s judgement, without any formal way to organize and filter data.

Depending on how Numerai, Quantopian, and similar experiments fare, investors could be entering a new era of finance in which they entrust their money to machines, not managers.  

The Conversation (0)

Why Functional Programming Should Be the Future of Software Development

It’s hard to learn, but your code will produce fewer nasty surprises

11 min read
A plate of spaghetti made from code
Shira Inbar

You’d expectthe longest and most costly phase in the lifecycle of a software product to be the initial development of the system, when all those great features are first imagined and then created. In fact, the hardest part comes later, during the maintenance phase. That’s when programmers pay the price for the shortcuts they took during development.

So why did they take shortcuts? Maybe they didn’t realize that they were cutting any corners. Only when their code was deployed and exercised by a lot of users did its hidden flaws come to light. And maybe the developers were rushed. Time-to-market pressures would almost guarantee that their software will contain more bugs than it would otherwise.

Keep Reading ↓Show less