“The Party is like God,” one university professor in Beijing told Richard McGregor, a journalist, recently. “He is everywhere. You just can’t see him.” This was a remarkable statement considering that just over two decades ago, the Chinese Communist Party had a near-death experience when faced with protesters — in Tiananmen Square, in the heart of Beijing — who nearly brought China’s authoritarian political system to its knees. Yet today, the CCP is the world’s biggest political party, with over 76 million members. Nor is joining it easy: after all, party membership remains the key to advancement in every aspect of Chinese life, from government to business to academic life, and the gatekeepers do not give membership away lightly.

The paradox is that the CCP has become more powerful than ever in Chinese life at the same time that it seems to have disappeared from public view. Major Chinese cities are a mixture of the bland concrete and glass that marks most international metropolises, decorated with brand names that are both Chinese (Haier) and Western (McDonald’s, Louis Vuitton). There are few, if any, posters that chronicle the activities of the political elite. But in Chairman Mao’s China, more than four decades ago, politics was “in command,” to use a phrase of the era. During the Cultural Revolution, angry slogans denouncing the “running dogs of capitalism” were festooned over every boulevard. However, in some ways, the party of Mao’s era had an easier task in keeping hold on power. It was very hard for the ordinary Chinese to leave the country, and contact with the Western world or the major non-aligned countries such as India was rare, giving the CCP a monopoly on the information available to China’s citizens. But in today’s globalized world, Chinese students and visitors stream across the world, a large number of senior CCP members among them. Yet for the most part, they do not defect or disappear: they return to China assured, having seen the outside world, that China’s system is superior.

This trend seems to run against global currents. Every continent has been seized by the desire for a multi-party democracy, and Latin America, Asia and Africa are all dominated by political pluralism, however flawed. The CCP seems like an outlier: it is secretive, unpredictable, and entirely convinced that any opposition is subversive and must be eliminated. Only a few countries in the world explicitly reject pluralism in the way China does, and they are not company in which most leaders would care to be seen. Yet this same system has generated economic growth that has broken all records in history, and seems to be attempting political reforms that may tackle some of the country’s huge social problems (for instance, lack of a national pension scheme as well as huge income inequality). The one thing that does not seem to drive the party is communism, at least not in the classic sense of the collective ownership of property. So what exactly is the Chinese Communist Party that has unleashed some of the most carnivorous capitalism that the world has ever seen, and in a way that brings to mind the excesses of factories in Victorian England?

One excellent place to start answering this question is Richard McGregor’s book, The Party: The Secret World of China's Communist Rulers. McGregor spent many years in Beijing as correspondent for the Financial Times, and managed to use contacts and inside knowledge to compile an incisive view of the reality behind the shining new face of Chinese modernity. McGregor makes clear that the techniques of the Leninist party, where one small leadership group directs the rest of society, are still very much at the centre of the CCP’s view of how society should operate. This leads to an almost paranoid style of management, something illustrated by the details on the many news stories that the ‘Department of Information’ (formerly the ‘Department of Propaganda’, before an Orwellian genius decided to change its name but keep the function the same) has banned from the Chinese media. Stories about contaminated food, corrupt top-level officials, and arbitrary seizure of property are regularly censored to prevent what the party fears most — social dissent that might lead to the destabilization of one-party rule.

One of the longest, and most riveting chapters, in this book describes the ups and downs of China’s “Shanghai gang”. This group of top leaders, who were groomed in China’s most entrepreneurial city, stood at the very top of Chinese politics for more than a decade, pushing forward a neo-liberal economic agenda at fierce odds with anything that Mao could ever have associated with communism. The poster boy for this grouping was Chen Liangyu, party chief of Shanghai, whose period in control was marked by grand gestures such as the building of the world’s most advanced motor-racing track. Yet he fell foul of political enemies and, in 2008, was tried and sentenced to jail for corruption. The ups and downs of life in the party still have more to do with who you know than what laws you may or may not have broken. Another area where the CCP continues to exercise its veto is the honest investigation of the party’s own history. McGregor tells the fascinating story of Yang Jisheng, a reporter for the official Xinhua news agency who gathered damning data on the Great Leap Forward of 1958-62, an economic plan championed by Mao that led to a famine that killed 30 million people or more. The embarrassment that this data would cause to the CCP’s most iconic figure, even decades after his death, meant that Yang’s book could only be published in Hong Kong.

McGregor’s characterization of the CCP is subtle. He has avoided the trap of describing it as a Cold-War era monolith with totalitarian abilities to control the whole of society. Instead, he shows that it is engaged in a difficult — and perhaps unique — balancing act. The party wants China to develop aspects of what seem to mark a genuinely modern society, including internationally-competitive financial markets, a commercial media, and a globally relevant scientific and research capability. But it also wants to control and have the final say on all those aspects of society as well. The problem is that, ultimately, there is a fundamental incompatibility between these goals. It is hard to argue that there can be meaningful rule of law or academic freedom in a country where the party’s ultimate role cannot be questioned without fear of arrest.

McGregor does not venture too far into the realms of prediction, although he does suggest that the “protean” nature of the party and its ability to change pragmatically will keep it going for a long while, particularly while it continues to stamp on any opposition. Another equally sharp analyst who is willing to be much more explicit on the future direction of the party is Kerry Brown in Friends and Enemies: The Past, Present and Future of the Communist Party of China. Brown argues that the party will be “progressively more open and daring in its efforts to reform” not “because it wants to but because it has to”, and that this reform may well lead to democratization. McGregor’s book is more careful about predicting democratization, but the idea that China might find some way to open up would fit in well with his view about the party’s pragmatism.

Certainly, the party will do anything it can to survive, and it may well be that deciding to open up the media and allowing a certain amount of freedom to an opposition may well be its best hope of staying in power. After all, many of the party’s defenders point out that its record on social and economic progress is one that they can be pleased about. If they decide that they could put that record before their own people and might get freely elected in consequence, they would solve their legitimacy problem in a way that arresting hundreds of dissidents or censoring thousands of websites could never achieve. However, the party is also determined that its own form of democracy will look nothing like the Western form of that system. This sounds like an impossible square to circle. But then thirty years ago, few would have predicted that the world’s largest revolutionary communist power would become the second biggest capitalist economy on the planet.