WASHINGTON — A federal judge secretly ruled last year that the F.B.I.’s procedures for searching for Americans’ emails within a repository of intercepted messages that were gathered without a warrant violated Fourth Amendment privacy rights, newly declassified files showed.
The files show that the F.B.I. resisted a new congressional mandate that required it to keep closer track of when it searched for Americans’ information gathered by the government’s warrantless surveillance program. The F.B.I.’s defiance set off a secret court fight that ultimately prompted the bureau to relent, the files showed.
Moreover, they show, the F.B.I. improperly searched the repository for information involving large numbers of Americans who fit within general categories but against whom there was no individualized basis for suspicion. In a twist, one March 2017 search used more than 70,000 identifiers, like email addresses, linked to the F.B.I.’s own work force.
The revelations were contained in a set of Foreign Intelligence Surveillance Act rulings that the Trump administration declassified on Tuesday. The rulings revealed a fight that has unfolded behind closed doors over the past year, adding a new chapter to the story of the warrantless surveillance program tracing back to the Sept. 11 attacks.
The latest chapter began in early 2018, when Congress extended the law that authorizes the warrantless surveillance program — known as Section 702 of the FISA Amendments Act — while adding a new requirement: Agencies that use the surveillance program’s repository of intercepted messages must win court approval for rules about how they may search it for Americans’ data.
In October 2018, the FISA court judge, James E. Boasberg, approved the procedures submitted by other agencies, like the C.I.A. and the National Security Agency. But he rejected the F.B.I.’s, saying the bureau’s rules were “not consistent with the requirements of the Fourth Amendment,” which protects Americans’ privacy from unreasonable searches.
Rather than comply with his ruling and adjust the F.B.I. rules, the Justice Department made a rare appeal to a three-judge panel that can review FISA court decisions. In July, the review court affirmed a central part of Judge Boasberg’s original ruling. After that, the F.B.I. acquiesced and changed its procedures, resolving the fight, the files showed.
The Section 702 program permits the government to collect communications from American companies like Google or AT&T without a warrant, so long as the target of the surveillance is a noncitizen abroad. The government can collect the data even when that person is talking to an American. A particular controversy has arisen over the F.B.I.’s ability to use it to find information about Americans.
Complicating that debate, the F.B.I. for years has not kept records of how often its agents searched the repository for information about Americans. So when the program was up for renewal last year, Congress required the bureau to track such queries.
But the procedures that the F.B.I. initially proposed simply called for it to keep track of all queries of the repository, without distinguishing which sought information about Americans and which were focused on foreigners. Judge Boasberg rejected that idea as inadequate, and the appeals court upheld his ruling.
The judge also wanted F.B.I. agents, before reviewing any emails of Americans that turned up in a search, to document in writing how that search term — like an email address or a phone number — met the standard of being likely to return foreign intelligence information or criminal evidence. The F.B.I. initially resisted that idea, too, but after the appeals court said it had to rewrite its procedures to track queries better, it accepted the other new mandate, too.
The idea of requiring agents to document their rationales for searching for an American’s information emerged from several recent episodes in which the Justice Department reported to the court that the F.B.I. had conducted improperly sweeping searches of the repository. Those events formed the backdrop to the fight over the new rules for querying the surveillance repository.
Specifically, F.B.I. agents had carried out several large-scale searches for Americans who generically fit into broad categories — like they were F.B.I. employees or contractors — so long as agents had a reason to believe that someone within that category might have relevant information. But there has to be an individualized reason to search for any particular American’s information.
The documents’ description of episodes involving large-batch queries that violated the rules were heavily redacted, so it was difficult to discern what the agents had been trying to learn. The discussion of the F.B.I.’s search for tens of thousands of identifiers involving its own work force also said that the bureau’s general counsel had raised objections beforehand. (The results were never scrutinized, the ruling said.)
Judge Boasberg wrote that the searches involving large numbers of queries using Americans’ identifiers raised a “serious concern” that agents either did not understand the standard or were indifferent to it.
The ruling also mentioned two searches in December 2017 that were flagged as problematic — one involving a query for messages containing any of 6,800 Social Security numbers, and the other involving 1,600 queries whose details were censored. Several other episodes in February 2018 apparently involved F.B.I. efforts to identify Americans whom it could try to develop into potential confidential sources of information, the rulings showed.
Several of the large-batch searches involved agents proactively trying to identify threats within the information the government had collected without waiting for a tip or criminal referral, said a senior F.B.I. official, speaking on the condition of anonymity to discuss the documents with reporters. But the details were classified, the official added.
The F.B.I. did not want to have to document a rationale every time it used an American’s identifying information to search the warrantless surveillance database because its systems allow agents to search all its databases at once — not just one at a time — and most of the time, the warrantless surveillance database, centered on foreigners abroad, yields no results anyway.
But Amy Jeffress, a former senior Justice Department official in the Obama administration whom Judge Boasberg had appointed as a friend of the court to critique the government’s position, proposed the idea of having agents document their rationales only if something came back, before the agents could look at it.
The presence of Ms. Jeffress and Judge Boasberg in the FISA court review process followed the uproar over surveillance after the 2013 leaks of National Security Agency files by the former intelligence contractor Edward J. Snowden.
Among other things, the resulting debate put a spotlight on the enigmatic FISA court, which, after hearing only from the government in secret court proceedings, had blessed an idiosyncratic interpretation of the law to approve a separate N.S.A. bulk phone records program that Mr. Snowden exposed. A federal appeals court later rejected that interpretation.
Those events brought to light that Chief Justice John G. Roberts Jr. had used his authority to decide who sits on the 11-member FISA court to assemble a panel that consisted of 11 federal judges either appointed by Republican presidents or selected by Republican senators in deals with a Democratic president.
In early 2014, when one of the seats became vacant and Chief Justice Roberts had an opportunity to fill it, he broke that partisan pattern by designating Judge Boasberg, whom President Barack Obama had appointed.
And in 2015, when Congress enacted the USA Freedom Act, which ended and replaced the bulk phone records program, lawmakers also enacted a provision that encouraged the FISA court to appoint friends of the court with security clearances to critique the Justice Department’s arguments in novel and significant cases. That set up Ms. Jeffress’s role in arguing that the F.B.I. should be required to obey more limits than it wanted.