The Serious Fraud Office (SFO) must pay up to £3m in legal costs over its failed investigation into businessmen Robert and Vincent Tchenguiz, a court has ordered.

Lord Justice Thomas, the President of the Queen's Bench, and Mr Justice Silber, ruled at the High Court that the SFO's handling of the investigation into the brothers during a probe into the collapse of Kaupthing Bank was unlawful.

The SFO must now make indemnity payments to Rawlinson & Hunter, the trustees of the Tchenguiz Family Trust, as well as to Vincent Tchenguiz and others, with Robert Tchenguiz also set to claim back his legal costs.

The SFO's budget in 2011-12 was £36.8m.

The judges ruled that the SFO's former director Richard Alderman "wholly failed to discharge this duty in circumstances in which the claimants' reputations were bound to be seriously damaged by the issue and the execution of the warrants, given the very public manner in which this was done".

Alderman, the judges said, took an "unacceptable approach… to his vitally important duties of disclosure on the applications for search warrants in this case and then to his subsequent failure to appreciate these very significant failures until a very late stage of the proceedings".

Last month, the SFO's new director David Green announced it had dropped its investigation into Robert, having abandoned its probe into Vincent in June.

"This is only the first financial manifestation of the folly of the former director of the SFO. His reckless, publicity driven, pursuit of me has caused very substantial damage to my life, family, reputation and commercial interests", said Vincent Tchenguiz in a statement.

"We are delighted that this judgment recognises the impact of the former SFO regime's unlawful actions," he added, noting the SFO's actions had exposed "the taxpayer to significant contingent liabilities".

The brothers have both indicated they will sue the agency for damages, with a claim set to be heard early next year.

Rawlinson & Hunter was advised by Stephenson Harwood, which instructed Hugo Keith QC of Three Raymond Buildings and Jonathan Glasson of Matrix Chambers.

Vincent Tchenguiz was advised by US firm Wilmer Cutler Pickering Hale and Dorr, which instructed Lord Goldsmith QC of Debevoise & Plimpton, Matrix Chambers' Ben Emmerson QC and Jonathan Barnard of Cloth Fair Chambers, while claimants Robert and R20 were advised by a BCL Burton Copeland team that used Lord Macdonald QC and Alex Bailin QC of Matrix and Cloth Fair's Clare Sibson.

Blackstone Chambers' James Eadie QC, QEB's Mark Ellison QC, 18 Red Lion Court's Allison Clare and Three Raymond Buildings' Ben Watson acted for the SFO.

Commenting on the ruling, a source close to the case said: "It's a huge disaster for the SFO and the Government. The fact that the judge has ruled for three lots of indemnity costs to be paid is unprecedented; I can't think of a case where that has happened.

"Some of what the judge said in the ruling regarding the former director was shocking. If someone said that about my professional conduct I would be devastated."