Business and political circles in Switzerland had supported the plan, which was designed to prevent taxes rising sharply for foreign investors.
However 59% of voters opposed the plan in Sunday's referendum vote.
The government will now need to find an alternative, which may involve higher tax rates for multinational companies.
"It will not be possible to find a solution overnight," said Ueli Maurer, Switzerland's finance minister. He told a press conference it could take a year to come up with a new plan, and that Switzerland risked losing foreign investment as a result.
Currently Switzerland grants special status to foreign firms which allows cantons, or districts, to offer them lower rates of tax than domestic firms, making Switzerland an attractive destination for foreign investors.
But international organisations such as the OECD have deemed the system unacceptable and Switzerland has made a commitment to reform it by 2019.
To avoid raising taxes for multinational companies too sharply the government had agreed to abolish the special status for multinationals but instead offer new tax breaks for research and development and other activities.
As part of the proposal, Switzerland's cantons planned to lower their tax rates for businesses across the board, including domestic business, helping to avoid steep rises in tax bills when foreign firms' status was brought into line.
Some of the resulting budget shortfall in the cantons would have been plugged by federal funds.
But the plan was opposed by Social Democrats, Greens, trade unions and churches who argued that the reduced tax revenue would still lead to cuts in public services or higher personal taxes.
"The conservative parties wanted to push through tax reform with arrogance and haughtiness against the interests of the people. The Greens demand a new proposal with a sense of proportion," the opposition leftist party said of the vote.
Source: BBC Business News