Writing in The Observer, its boss Anthony Browne also says smaller banks could move operations overseas by 2017.
"Their hands are quivering over the relocate button," he wrote. Most banks had backed the UK remaining in the EU.
Mr Browne also said the current "public and political debate at the moment is taking us in the wrong direction."
His comments build upon those he made at the BBA annual conference last week, when he said banks had already "set up project teams to work out what operations they need to move by when, and how best to do it".
"Banking is probably more affected by Brexit than any other sector of the economy, both in the degree of impact and the scale of the implications," he told the newspaper.
"It is the UK's biggest export industry by far and is more internationally mobile than most. But it also gets its rules and legal rights to serve its customers cross-border from the EU."
One of the perks of Europe's Single Market - which also currently includes the UK, Norway and EU countries such as the Netherlands - is "passporting". Passporting allows banks and insurance companies to sell their services anywhere in the single market without having to establish a base in every country in Europe.
But single market membership comes with conditions: freedom of movement of goods, services, capital and (crucially) people. Theresa May has already said she intends to restrict the free movement of people from the EU after Brexit, while EU leaders have meanwhile said the four freedoms are indivisible i.e. non-negotiable.
That means the UK may have to quit the single market and lose passporting. What Anthony Browne from the BBA is doing is upping the ante on the government, by saying some banks will start to relocate in the coming months without passporting.
What he also says though is that erecting any barriers to cross-border banking will be just as bad for Europe as it would be for the UK.
Mr Browne added: "For banks, Brexit does not simply mean additional tariffs being imposed on trade - as is likely to be the case with other sectors. It is about whether banks have the legal right to provide services."
Banks want to see the continuation of the EU's "passporting" system, allowing UK-based financial services to operate across Europe without needing separate authorisation.
Banks have called for transition arrangements to be put in place after the UK leaves the EU.
But Mr Browne warned that in Europe and among UK eurosceptics the mood was "hardening".
"The problem comes - as seems increasingly likely, judging by the rhetoric - when national governments try to use the EU exit negotiations to build walls across the Channel to split Europe's integrated financial market in two, in order to force jobs from London," Mr Browne said.
"From a European perspective, this would be cutting off its nose to spite its face. It might lead to a few jobs moving to Paris or Frankfurt but it will make it more expensive for companies in France and Germany to raise money for investment, slowing the wider economy."
Meanwhile, Liberal Democrat foreign affairs spokesperson Tom Brake said that if the UK pursued a "hard Brexit" then it would "threaten the £65bn the UK financial services industry pays in taxes each year".
He added: "The Conservative government must explain how it will make up this funding shortfall if the UK leaves the single market."
Source: BBC Business News