Will Dominic Cummings’ reign bring the shadowy world of spads into the light?

“You can be a successful poisoner, or you can be a famous poisoner. But you can’t be both.” That was Frank Dobson’s verdict on Alastair Campbell, whose tenure as a special adviser to Tony Blair still shapes how “spads” are thought of today. Not since the New Labour nomenklatura has a spad attracted as much attention as Dominic Cummings, whose dilettante grab-bag of Silicon Valley management techniques, huckster science and penchant for “weirdos and misfits” provides ample material for those wondering how Boris Johnson will govern.

Cummings rode to power on invective against unelected elites, but he now occupies a position without a clear democratic mandate, effective limits on its power or obvious means of scrutiny. His mania for civil service reform and control of government prompted Sajid Javid’s resignation as chancellor, after Cummings insisted on replacing his advisers with a team run from No 10. He has similarly asserted his supremacy over spads working in other departments, expecting them to report on ministerial deviations or departmental slackness.

Conventional political wisdom agrees with Dobson’s verdict in two respects: a spad’s role is highly politicised – poisoning opposition, internal and external, through spin – and can only be conducted in the shadows. Believe that, and you have cause for optimism: any aide who attracts more attention than his master is soon destined for the exit. A more sanguine judgment might be that Dobson’s axiom held true in the Blair years, where powerful spads did dirty work behind closed doors to ensure the appearance of a smooth governmental machine, but that Cummings’s propensity for disruption better serves an era with a greater appetite for political bloodsports. His victories in the referendum and election provide armour against doubters. There is an advantage, too, to an aide briefing vivid attacks from which the PM can then resile, as happened with the BBC last weekend: the institution is panicked, perhaps cowed, and Johnson keeps his hands clean, or even looks moderate by comparison – with Cummings as his lightning rod.

Former spads bristle at their portrayal as shadowy manipulators. Most argue they fill a constitutional gap between Whitehall and Westminster, juggling the need for political commitment, media savvy and policy development in ways permanent officials cannot. Their status reflects that awkward gap: though formally temporary civil servants, they are exempted from the service’s requirement of political impartiality, and the rigours of its hiring process, but barred from direct decisions over public money and, at least nominally, from giving orders to permanent officials. Instead, they are personal appointees of their minister, with only the prime minister’s veto over their employment.

The modern spad emerged under Harold Wilson, who hoped that politically committed socialist advisers could provide a counterweight to a lumbering and hidebound civil service, widely distrusted among Labour politicians after long exile in opposition. Early spads – largely economists – were few in number, and eminent in their fields, though hardly immune from internecine gossip and intrigue. It was under Blair that their numbers ballooned and tilted more obsessively towards media management.

That shift was legitimate: the 24-hour news cycle made politics a media game that required sharp tactics. Blair’s presidential style extended the role further: a 1997 order empowered up to three of his advisers to manage permanent officials, a power not yet accorded to Cummings. It was hard to look at these figures – powerful, unelected, partisan and barely scrutinised – and not wonder about their democratic legitimacy, or how far they might influence elected politicians. Blair’s memoir describes agonised consultation with his advisers – Campbell, Jonathan Powell, David Miliband – over intervention in Kosovo; the foreign secretary, Robin Cook, who one might think ought to have had some input, goes unmentioned.

Scandal tarnished spads during the Blair period: Jo Moore’s attempt to “bury bad news” on 9/11, Damian McBride’s routine attempts to smear the rivals of Gordon Brown, Campbell’s dossier-tampering enthusiasm for the Iraq War. But select committee inquiries and election pledges to reduce their number produced little change. The role of the spad is too useful. Its power is attractive: the major spad scandal of the Cameron years occurred when Adam Werritty posed as a spad without ever having been appointed. Tenure in spadland remains a technocrat’s easiest path to a safe seat in the Commons – leapfrogging the drudgery of local government, or a thankless battle in unwinnable constituencies – and a golden ticket to preferment in ministry, should an ex-spad choose that route.

Spads are treated oddly. In evidence to one inquiry, a former Whitehall permanent secretary described them as having a “very significant influence” on government, with real autonomy and, if backed by their minister, real power. Even outside of Downing Street, many spads shape policy profoundly. But they are often regarded only as appendages of the minister, accountable just to them. They are too often hired in opaque processes favouring political insiders, with sketchy criteria for qualifications and minimal oversight, though paid from the public purse (£9.6m in the last report.). Checks on suitability for public office are absent, which explains how David Cameron employed former phone-hacker Andy Coulson as a media hitman.

Related: Cummings, Epstein, Sabisky – what fertile ground for ‘offence archaeologists’ | Catherine Bennett

Their defenders argue rightly that most spads are essential to their department. Their mix of policy and media competence is hard to advertise for, and less tangible qualities that are deemed essential – loyalty and political commitment – are difficult to put in a checkbox form. Other defences are less convincing: that ministers will always sack egregious freelancers, or that ministers will carefully police their advisers for fear of their own job. Still less convincing is the argument that grubby men are needed for the grubby work of politics, and we should simply look the other way; if it gets really bad, they can be booted out at the ballot box in a few years’ time – effectively an argument for temporary electoral dictatorship. The job’s mutability makes it difficult to regulate: raise the abuses by the most powerful spads and you will be met with the inarguable truth that those with policy specialisations are invaluable to their department – as if the one justifies the other. Proposals have been made for pre-appointment hearings, perhaps before a select committee, where MPs could raise concerns – but this was rejected as too cumbersome. Ministers are not required to hire spads purely on merit, a requirement that applies to the remainder of the civil service. The sheer number of spads, the personal nature of their appointments and the immediate need for them as soon as the government takes office would make such scrutiny unworkable.

The spad role evolved out of the notorious adaptability of the British constitution yet, like all its haziest parts, it depends unwisely on the prudence, virtue and self-denial of its occupants – not traits easily discerned in Cummings. The recent hiring scandal surrounding Andrew Sabisky suggests that Cummings has strongly advocated for his own agenda, with at least the tacit approval of Johnson. But his replacement of Treasury aides and the reputed culture of fear among spads across Westminster suggests he hopes to remake the spad network as a channel of influence and control for Downing Street, eliminating internal governmental checks. It is a move that will likely fail and cause dysfunction: Whitehall is not easily bent to individual wills, and spads are only one element in a machine with many powerful parts. Clare Short once spoke of prime ministerial spads as “people who live in the dark”. It is time they were dragged into the light.

• James Butler is co-founder of Novara Media and a regular contributor to the London Review of Books