A Long Island resident who once broke Cold War codes may have cracked the tax code for hedge funds.

James H. Simons, who became a billionaire when he turned his extraordinary mathematical ability from defense work to investing, has deployed an unusual strategy at Renaissance Technologies LLC to skirt hundreds of millions of dollars in taxes for himself and other investors, said people with knowledge of the matter.

The Internal Revenue Service is challenging the technique, which it called "particularly aggressive," without identifying the hedge fund in the dispute.

It is demanding more tax payments from investors in Renaissance's $10 billion Medallion fund, the people said.

Renaissance sought to convert profit from Medallion's rapid trading into long-term capital gains, said the people, who spoke on condition of anonymity because the dispute hasn't been made public. The top federal rate on long-term gains is about half that on short-term.

Versions of the strategy, which involved shifting ownership of Medallion's portfolio to banks including Barclays Plc, were used for most of the past decade, one person with knowledge of the matter said.

If East Setauket-based Renaissance prevails in its legal dispute with the IRS, dozens of other funds would probably take steps to mimic the firm's strategy, according to tax advisers.

advertisement | advertise on newsday

"If they win, that will signal to the rest of the hedge-fund community that aggressive strategies can work," said Steven Rosenthal, a fellow at the Urban Institute in Washington, D.C., and a former tax lawyer at Ropes & Gray LLP.

Jonathan Gasthalter, a spokesman for Renaissance, declined to comment, saying "the dispute is ongoing and being handled in the appropriate forum." The IRS also declined to comment.

Simons, 75, retired from Renaissance in 2010 as one of the best-performing managers of all time and now serves as nonexecutive chairman.

In the 1960s, Simons was a code breaker for the National Security Agency. Simons is a philanthropist and former math professor at Stony Brook University who has championed better math and science education. Simons has a fortune estimated by the Bloomberg Billionaires Index at $12 billion. He didn't respond to a written request for comment.