THE INSTITUTEWhen I was a first-year Ph.D. student in computer science at Cambridge in the early 1990s, I met the inventor of subroutines in microprogramming, professor David John Wheeler, who made a comment that has stuck with me since. He said: “Look at the proceedings on the library shelves. They are collecting dust.” I understood that to mean that one should not publish for the sake of publishing but to make an impact on society. It was a wake-up call for me.
Many universities, especially in the United States, have a culture of “publish or perish,” in which researchers must continue to make their work public in order to further their career. This has resulted in many professors involved in “paper-cranking” to jack up their number of publications.
Worse is the possibility of researchers citing each other’s work to increase their citation counts. In a world where there are millions of professors, researchers, engineers, and Ph.D. students submitting articles, the number of papers being published in various proceedings and journals seems endless. Many of these works fade away with little or no significance or application to society today, while a small few do make a difference.
The topic of impact continues to surface time and again. So I’d like to propose the question here: How should we measure the impact of research?
COUNTING CITATIONS
I have given a lecture in several countries on “Perspectives of Good Quality, High Impact, and World Class Research,” which does not include the current ways of measuring impact.
For example, Google Scholar provides citation profiles of researchers. These include H index and total citation count. The i10-index indicates the number of publications that have been cited at least 10 times by others. The metrics are based on what’s called productivity, or the amount of papers an author has published.
One problem with measuring productivity is that industry researchers do not publish as much as university students and professors—which means the work of those in industry is not considered as impactful in their fields based on the metrics. Industries are focused on research outcomes that can drastically improve a product or create new ones. Those like Steve Jobs and Elon Musk arguably have a more significant impact on people’s lives than, say, a university researcher with several thousand citations.
Many top researchers will argue that a work that creates new knowledge or introduces a breakthrough is undeniably significant in impact—which is all the more reason why counting an index number alone is insufficient.
Those with a high number of citations are said to have “influence,” meaning others have read your paper, cited your work, and possibly used some of the research in their own work. The ability to influence other researchers certainly has its merits, because they can use that work to help solve a problem or add new insights into an existing one. Which is why several of the giants in the engineering field have had a handful of research papers with an extraordinary number of citations, while the rest of their works didn’t make a notable dent or they just didn’t publish often.
Take, for example, Tim Berners-Lee, considered the father of the World Wide Web. His total citation number is not available. Yet his book Weaving the Web: The Original Design and Ultimate Destiny of the World Wide Web has been cited more than 4,000 times. Alan Turing has been cited 38,197 times. Two of his research papers alone account for nearly 20,000 of those citations! Also, most such giants are the sole author of their high-impact papers, including Claude Shannon and Isaac Newton. The list goes on.
A better way
The way I believe we should define impact is based on criteria far beyond citations. The criteria should include: Did the researcher write the paper on his or her own, or with the help of others? Did the research uncover new knowledge? Did it help start a new discipline? Did it invent a new industry and, as a result, create new types of jobs? Does the research help improve the national economy in any way? Is it changing the lives of millions of people?
Those are examples of the type of impact our engineering giants have made.
I will continue to study them and their work to better understand how to measure impact that really makes a difference in our society. I hope this will help enlighten researchers to focus on what really matters, and discourage paper-cranking and an emphasis on citation count.
IEEE Fellow Chai K. Toh is honor chair professor of electrical engineering and computer science at National Tsing Hua University, in Hsinchu, Taiwan.
This article has been updated.
IEEE Fellow Chai K. Toh is an honor chair professor of electrical engineering and computer science at National Tsing Hua University, in Hsinchu, Taiwan.
