The billionaire's original proposal for a 72,000-square-foot home - bigger than Hearst Castle and double the size of the Taj Mahal - spawned an outcry of opposition when it was first introduced in 2005.
Now most opposing voices have been quelled, though the reasons why are controversial themselves. But the environmental group Save Mount Diablo is still fighting the project and hopes to have a hearing before the San Ramon Valley Regional Planning Commission in a few weeks.
Duffield is the founder of the Pleasanton-based software company PeopleSoft, which Oracle bought for $10 billion. He and his wife have purchased 21 acres of land in the quaint and quiet Bryan Ranch.
The location - an endangered species habitat that borders Mt. Diablo State Park and is surrounded by public trails - set off concerns that building the house would set a dangerous precedent for developing in delicate environments.
"Those are important issues to us and that's why we're fighting these things," said Seth Adams, the group's director of land programs.
The Bryan Ranch Home Owners Association and a group of distressed would-be neighbors, who dubbed their effort Save Bryan Ranch, also roared that the huge structure would disrupt their quiet community.
Upon hearing concerns, Duffield scaled back the mansion by 75 percent, to 17,000 square feet. Adams suggested the actual number would be closer to 21,000.
Since then the future Duffield home has gone through the hoops of bureaucratic review, the first of which was getting approval from the homeowners association's Architectural Review Commission.
Realizing it had no cap on house size, the commission added an amendment limiting homes to 10,000 square feet. It denied Duffield's proposal based on the new limit.
Duffield claimed the amendment targeted him specifically, was unfair, and had not even been properly adopted. He sued the association's board members and the amendment was eventually overturned.
From there the project went before the county, and the opposition's concerns followed.
But Adams said Duffield's lawsuit had taken its toll, intimidating Bryan Ranch residents into shutting their mouths.
"Many of the people who are opposed have been bullied into silence," he said. "We're not willing to be intimidated by a bully."
Steve Hill, a project manager for the Duffield home, had a different take, saying the association, Save Bryan Ranch, most neighbors, and the Alamo Improvement Association are now fully behind the project.
"We have worked very hard to win their support," he said.
The president of the homeowners association declined to comment.
Duffield spokesman Jim Dugdale said there's no controversy left; the issue at hand now is simply a Tree Permit application.
A request to cut down 20-plus trees was approved by the county, subject to public appeal. Save Mount Diablo appealed, and now the project is open to a hearing before the San Ramon Valley Regional Planning Commission, likely in April.
"This is a very controversial, political project. (County) staff knew that it would be decided in the political arena," Adams said. "Our hope is that the commission will do the reasonable thing."
The "reasonable thing," he said, is to put a scenic easement over the property stipulating that Duffield can't build structures on it, in order to protect the natural habitat.
Adams said the sticking point is that Duffield wouldn't agree to protect a large enough area, proposing a small scenic easement in legal language full of loopholes.
Hill called the nonprofit's insistence on having a larger easement - especially after the extensive review process Duffield has already undergone - "is just not a fair proposition from our perspective."
If the parties don't settle their differences, the debate will be heard by the planning commission. But Hill said "our hope is we can reconcile our differences with Save Mount Diablo in the next month."
That may, however, be tricky.
"We went into this thing with an open mind but Duffield hasn't moved one inch from what he was willing to do," said Adams. "He showed no willingness to compromise."