Forbes, Family Swindled By Madoff Fights $61 Million IRS Bill, by William P. Barrett:

First the family of Norman F. Levy, the late New York City real estate tycoon, was swindled out of hundreds of millions of dollars by close friend Bernard L. Madoff, forcing the closing of two Levy charities. Then Levy heirs coughed up $220 million to the Madoff bankruptcy trustee to repay personal withdrawals made before Madoff's Ponzi fraud collapsed in 2008.

Now, adding insult to injury, the unlucky clan is fighting a $61 million estate tax bill from the IRS.

In a previously unreported U.S. Tax Court lawsuit against the IRS, Levy's estate and his two grown children, Francis N. Levy and Jeanne Levy-Church, contend they don't owe the $61 million. Instead, they claim the feds actually owe the estate a $19 million refund for losses from Madoff's thievery of certain assets after their father's death in 2005 at age 93. …

According to Tax Court and probate filings, Levy's gross estate was valued at $1.09 billion in 2005. … Levy's will put $905 million of the estate into a charitable lead annuity trust. …  The Levy estate CLAT was set up to make periodic payments for a specified time period to a charity–the Betty and Norman F. Levy Foundation, named after Levy and his deceased wife–followed by the distribution of what remained in the trust to his children. As a result of the CLAT, the Levy family valued his taxable estate (largely meaning the portion not going to charity) at just $98 million and paid $46 million in federal taxes.

The IRS paperwork, which is part of the public court record, valued his taxable estate at more than twice as much–$226 million–and put the total tax at $105 million. The feds tacked on another $2 million in accuracy-related penalties, which the family also is disputing. …

[T]he IRS also disallowed as a deduction from the taxable estate $44 million listed as a commission to Levy's children for acting as co-executors of the estate. Such fees, calculated as a percentage of the estate, are set by state law and were figured based on the initial $1.09 billion gross estate–which, it now seems clear, wasn't all there. Justifying its denial, the IRS wrote there was no evidence the fees were "actually and necessarily incurred." The Levy lawsuit does not state explicitly that the $44 million actually was paid, saying only that the children are "entitled" to the sum.

The $19 million Levy refund claim stems from what the family calls a theft loss: After Levy died, two of his real estate interests were sold and $57 million in proceeds were invested with Madoff–where they went poof. The IRS rejected the refund request, saying there was no proof the loss was incurred while the assets were still within the estate itself. …

Fred T. Goldberg, a former IRS commissioner who is one of the Levy family lawyers in the Tax Court case, declined comment.