Artificial intelligence is becoming a common tool in legal work. Lawyers can use it to search for cases, draft submissions and go through large amounts of legal material much faster. Courts and government authorities are also increasingly coming across AI-generated material in legal proceedings.

But there is a serious problem: AI can sometimes make things up.

It can produce the name of a judgment that sounds real but doesn’t exist. It can provide a convincing-looking citation that is wrong. It can even cite a real judgment but incorrectly describe what the court actually said.

These AI “hallucinations” are no longer just a theoretical risk. Indian courts have recently encountered fake or non-existent judgments and citations in court proceedings and official orders.

In July, the Supreme Court set aside orders passed by the National Company Law Tribunal (NCLT) and National Company Law Appellate Tribunal (NCLAT) after finding that six citations relied upon in the proceedings either didn’t exist or didn’t support the legal arguments for which they were cited.

That raises a bigger question than whether lawyers should use AI: who checks what AI produces before it is used by a lawyer, government authority, tribunal or court?

One possible answer is to create a verification layer between AI and the legal system.

Raman Aggarwal, Founder and CEO of Jupitice Justice Technologies, calls this “trust infrastructure” — essentially, a system for checking whether an AI-generated legal reference is real, accurate and relevant before anyone relies on it.

Why are AI hallucinations particularly risky in law?

AI systems generate answers by predicting what information is likely to fit a user’s question. That means an answer can sound extremely convincing even when some of the information is wrong.

In many situations, an AI mistake may simply be inconvenient. In law, the consequences can be much more serious.

Imagine a lawyer asking an AI tool to find judgments supporting an argument. The AI provides a case that doesn’t exist. The lawyer includes it in a petition without checking it.

A court could then rely on that citation while writing its order.

The same thing could happen when a government officer uses AI to prepare a tax notice or when a tribunal uses AI-generated material while deciding a case.

At that point, something invented by an AI system has entered an official legal process.

“AI today has moved from AI intelligence to AI action,” Aggarwal told CNBC-TV18, pointing to the greater risks that arise when AI-generated information begins influencing real-world decisions.

The Supreme Court’s July ruling showed how serious the problem can become. The court found that six citations relied upon by the NCLT and NCLAT were fake, non-existent or incorrectly attributed and set aside the orders.

It also asked the Bar Council of India to examine the broader problem of lawyers relying on fake or hallucinated AI-generated material.

So the danger isn’t simply that AI can make mistakes. It is that those mistakes can look convincing enough to enter a system where judgments and legal precedents carry real authority.

The problem isn’t simply ‘hallucination’

It is easy to look at this as simply a problem of making AI more accurate. But there is another issue.

We already know generative AI can sometimes produce incorrect information. The more important question for the legal system is what happens **after** the AI produces that information.

Suppose an AI tool gives a lawyer the name of a case and a citation. Before using it, someone needs to check a few basic things.

Does the case actually exist? Is the citation correct? Does the paragraph being quoted actually appear in the judgment? Does it really support the argument being made? Has another court subsequently overruled or changed the legal position?

Making an AI model sound more convincing doesn’t answer those questions.

They require the AI’s output to be checked against reliable and authoritative legal sources.

That is why the debate is increasingly moving from AI accuracy to AI verification.

What could ‘trust infrastructure’ look like?

The idea behind trust infrastructure is fairly simple.

An AI system can suggest a potentially useful judgment. But before that judgment is relied upon, another system or process should verify that it is genuine and that the AI has represented it correctly.

Such a system could check whether the judgment exists, whether the citation is authentic, whether the quoted paragraph actually appears in it and whether the judgment supports the legal argument being made.

It could also check whether the judgment has subsequently been overruled, distinguished or otherwise affected by later decisions.

Simply asking the AI to check its own answer may not be enough. If the same system that made the mistake is asked to verify it using the same underlying process, it could simply repeat the error.

Verification therefore needs to be tied to authoritative legal sources.

In practice, an AI-powered legal research tool could do more than simply provide a list of cases. It could link each case to a reliable source, show the exact passage being relied upon and warn the user if the legal status of that judgment has subsequently changed.

The aim isn’t to replace lawyers or judges.

It is to give them a reliable way to check AI-generated information before using it.

The Supreme Court has made a similar distinction. In its July ruling, it said AI can assist adjudication, but humans must remain in control of the final decision.

Trust infrastructure, therefore, isn’t a replacement for legal judgement. It is a way to make AI-assisted legal judgement safer.

The warning signs go beyond court judgments

The problem isn’t limited to lawyers filing petitions.

In July, the Punjab and Haryana High Court quashed a GST show-cause notice after finding that it had been prepared primarily using an AI tool without the required independent application of mind by the competent authority.

The Delhi High Court also set aside a tax order after finding six non-existent judicial precedents in it. The court raised concerns about the growing use of hallucinated citations.

There had been warnings even earlier.

In January, the Bombay High Court imposed costs of ₹50,000 on a litigant after finding an unverified AI-generated submission that contained a non-existent judgment. The court stressed that the person using an AI tool remains responsible for checking the material it produces.

Together, these cases show that AI-generated errors can enter the legal system through many routes — lawyers, litigants, government authorities, tribunals and even courts.

That means verification isn’t just the responsibility of individual lawyers. It is becoming an issue for the legal system as a whole.

Should AI used for legal work be regulated?

If AI-generated legal material needs to be verified, the next question is who should be responsible for ensuring that happens.

Aggarwal believes some of that responsibility should lie with the companies building AI tools.

His argument is that simply telling lawyers and judges to be careful isn’t enough. AI applications specifically designed for legal work should have safeguards built into them.

“There has to be a regulation where they should introduce the trust infrastructure when they are building the solution around AI,” Aggarwal said.

That would put some responsibility on technology companies rather than leaving everything to the lawyer or judge using the product.

For example, minimum standards could be introduced for AI systems that generate or retrieve case law or help prepare legal submissions, tax notices and judicial material.

But that creates another difficult question: how much responsibility should an AI company bear when a human ultimately makes the legal decision?

An AI system might provide a verified judgment, but the lawyer still has to decide whether that judgment is relevant to the case.

Similarly, a court might use AI to help with research, but the judge remains responsible for the reasoning and final order.

Trust infrastructure, therefore, isn’t simply a technological issue. It is also about deciding how AI should be governed when it is used for legal work.

Who is responsible when AI gets it wrong?

Consider a simple example.

A lawyer uses AI to prepare a submission. The AI produces three judgments that don’t exist. The lawyer doesn’t check them and submits them to court. The court then relies on those judgments while writing its order.

Who is responsible?

The AI company? The lawyer? The judge?

There isn’t an easy answer because something went wrong at several stages.

The AI developer may not have provided adequate safeguards. The lawyer failed to check the citations. And the court relied on the material without independently verifying it.

“That is a fault on both sides,” Aggarwal said, referring to situations where fake AI-generated precedents submitted by lawyers are subsequently relied upon by courts.

What is clearer is that responsibility can’t simply be passed on to AI.

Lawyers remain responsible for what they submit to courts. Judges and tribunals remain responsible for the decisions they make.

The Supreme Court’s July ruling similarly called for verification by both the Bar and the Bench and adopted a zero-tolerance approach towards fake or hallucinated AI-generated precedents.

What does this mean for the future of AI in law?

None of this necessarily means AI has no place in India’s legal system.

AI can make legal research faster. It can find potentially relevant cases, summarise long documents and help lawyers prepare drafts.

The emerging approach instead appears to be AI-assisted law rather than AI-led law.

The important distinction is between an AI tool finding a case that *might* be useful and establishing that the case is real, current and actually supports the argument being made.

That distinction could become increasingly important as AI becomes more widely used in legal work.

For legal technology companies, the challenge isn’t simply to build AI that produces better answers. It is also to build systems that can demonstrate why those answers should be trusted.

For lawyers, judges and government authorities, the lesson is simpler: an AI-generated citation should be the beginning of the verification process, not the end of it.

As AI becomes more deeply embedded in the justice system, the question may no longer be whether lawyers and judges should use it. The bigger question will be whether there are enough safeguards to stop something invented by AI from becoming part of the legal record simply because nobody checked it.

Watch accompanying video for full conversation.