Baidu’s PaddlePaddle Spins AI up to Industrial Applications

TensorFlow, PyTorch, and Keras: Those three deep-learning frameworks have dominated AI for years even as newer entrants gain steam. But one framework you don’t hear much about in the West is China’s PaddlePaddle, the most popular Chinese framework in the world’s most populous country.

It is an easy-to-use, efficient, flexible, and scalable deep-learning platform, originally developed by Baidu, the Chinese AI giant, to apply deep learning to many of its own products. Today, it is being used by more than 4.77 million developers and 180,000 enterprises globally. While comparable numbers are hard to come by for other frameworks, suffice to say, that’s big.

Baidu recently announced new updates to PaddlePaddle, along with 10 large deep-learning models that span natural-language processing, vision, and computational biology. Among the models is a hundred-billion-parameter natural language processing (NLP) model called ERNIE 3.0 Zeus, a geography-and-language pretrained model called ERNIE-GeoL, and a pretrained model for compound representation learning called HELIX-GEM.

The company has also created three new industry-focused large models—one for the electric power industry, one for banking, and another one for aerospace—by fine-tuning the company’s ERNIE 3.0 Titan model with industry data and expert knowledge in unsupervised learning tasks.

Software frameworks are packages of associated support programs, compilers, code libraries, tool sets, and application programming interfaces (APIs) to enable development of a project or system. Deep-learning frameworks bring together everything needed to design, train, and validate deep neural networks through a high-level programming interface. Without these tools, implementing deep-learning algorithms would take a lot of time because otherwise reusable pieces of code would have to be written from scratch.

Baidu started to develop such tools as early as 2012 within months of Geoffrey Hinton’s deep-learning breakthrough at the ImageNet competition.

In 2013, a doctoral student at the University of California, Berkeley, created a framework called Caffe, that supported convolutional neural networks used in computer-vision research. Baidu built on Caffe to develop PaddlePaddle, which supported recurrent neural networks in addition to convolutional neural networks, giving it an advantage in the field of NLP.

The name PaddlePaddle is derived from PArallel Distributed Deep Learning, a reference to the framework’s ability to train models on multiple GPUs.

Google’s open-sourced TensorFlow in 2015 and Baidu open-sourced PaddlePaddle the next year. When Eric Schmidt introduced TensorFlow to China in 2017, it turns out China was ahead of him.

While TensorFlow and Meta’s PyTorch, open-sourced in 2017, remain popular in China, PaddlePaddle is more oriented toward industrial users.

“We dedicated a lot of effort to reducing the barriers to entry for individuals and companies,” said Ma Yanjun, general manager of the AI Technology Ecosystem at Baidu.

PyTorch and TensorFlow require greater deep-learning expertise on the part of users compared to PaddlePaddle, whose toolkits are designed for nonexperts in production environments.

“In China, many of the developers are trying to use AI in their work, but they do not have much AI background,” explained Ma. “So, to increase the use of AI in different industry sectors, we’ve provided PaddlePaddle with a lot of low-threshold toolkits that are easier to use so it can be used by a wider community.”

AI engineers normally don’t know much about industry sectors and industry-sector experts don’t know much about AI. But PaddlePaddle’s easy-to-understand code comes with a wealth of learning materials and tools to help users. It scales easily and has a comprehensive set of APIs to address various needs.

These developers used PaddlePaddle for a desert robot to automate the process of tree planting.Baidu

It supports large-scale data training and can train hundreds of machines in parallel. It provides a neural-machine translation system, recommender systems, image classification, sentiment analysis, and semantic role labeling.

Toolkits and libraries are the strong side of PaddlePaddle, Ma said. For example, PaddleSeg can be used for segmentation of images. PaddleDetection can be used for object detection. “We cover the whole pipeline of AI development from data processing to training, to model compression, to the adaptation to different hardware,” said Ma, “and then how to deploy them in different systems, for example, in Windows or in the Linux operating system or on an Intel chip or on an Nvidia chip.”

The platform also hosts toolkits for cutting-edge research purposes, like Paddle Quantum for quantum-computing models and Paddle Graph Learning for graph-learning models.

“That’s why PaddlePaddle is quite popular in China right now,” he said. “Developers are using such toolkits and not just the tool itself.”

Since it was open-sourced, PaddlePaddle has evolved quickly to have better performance and user experience in different industry sectors outside Baidu as well as countries outside China thanks to extensive English-language documentation. Currently, PaddlePaddle offers over 500 algorithms and pretrained models to facilitate the rapid development of industrial applications. Baidu has worked to reduce model size so they can be deployed in real-world applications. Some of the models are very small and fast and can be deployed on a camera or cellphone.

PaddlePaddle has parameter server technology to train sparse models that can be used in real-time recommender systems and search. But it has also merged models into even larger systems that are used for scenarios that don’t require real-time results, like text generation or image generation.

Baidu sees large, dense models as another way of reducing the barrier to AI adoption because so-called foundation models can be adapted to specific scenarios. Without the foundation model, you need to develop everything from scratch.

Ma said research areas are converging with cross-model learning of different modalities, like speech and vision. He said Baidu is also using knowledge graphs in the deep-learning process. “Previously a deep-learning system dealt with raw texts or raw images without any knowledge input and the system used self-supervised learning to gather rules outside the data,” Ma said. “But now we are seeing knowledge graphs as an input.”