Developers are challenging a city decision to stop construction of a restaurant on Caldwell Road after a contractor tore down a building known as the “Little White House” without a demolition permit.
White House property owner Fritz Rybert is set to appear before the city’s Zoning Board of Appeals on Oct. 19 after the city told him last month he would need to reapply for zoning for his project. By tearing down the house without a demolition permit, the current zoning and plans for the project were void, according to the Community Development Department.

Located just off the thriving Dresden Drive corridor, the one-story White House at 2536 Caldwell Road was rezoned by the city last year to allow a restaurant by Chef Scott Serpas. The restaurant is called Dixie Moon. Serpas, a Brookhaven resident, is a renowned chef known for Serpas True Food in Atlanta’s Old Fourth Ward.
Rybert said the floor, walls and beams of the house were infested with termites and there was no way the contractor could build using the rotten wood. He said a city building inspector was called to the site to look at the walls and agreed the wood was rotten.
“We all agreed we couldn’t build on rotten wood,” Rybert said in an interview. “So we tore the walls down in preparation for new wood.”
A July 7 email exchange between city building official Paul Ivey and Community Development Director Ben Song, however, indicates only a single beam had termite damage.
“The contractor asked David [Smith, city building inspector] if he could reuse a particular beam, a double 2×10 adjacent to and parallel to the back side of the front porch. The answer was basically ‘No,’ as David had quickly determined that the beam was so heavily damaged by termites that it had no structural integrity,” Ivey wrote in an email to Song that was obtained through a request under the Open Records Act.
“In fact, David said that he was surprised to get such a question from a licensed contractor who should have had enough experience to already know the answer to his question. No termite damage was observed by David in the remaining floor joists, the walls, or the roof structure, which the plans showed were to remain and be reused,” Ivey’s email states.
Rybert denied that only one beam was infested. “It was the walls as well as the floor joints and beams. It was the entire framing,” he said. “We left the original foundation in place … we’re going to build the same size and design, but with good wood versus rotten wood.”

Song noted in an Aug. 11 email to City Manager Christian Sigman that the house and property were rezoned in 2015 and a land disturbance permit was issued on May 23, 2016. On July 1, the city learned the house was demolished. The house was considered a nonconforming structure and city ordinance prohibits enlarging, expanding, moving or altering in any manner that increases the degree of nonconformity.
Once the existing structure was demolished, the nonconforming or “grandfathering” designation no longer applied because the structure no longer existed, Song said. Any new structure must meet the requirements of the Brookhaven-Peachtree Overlay District, he said, and new, single-story buildings are not allowed to be built in the Overlay District.
City Attorney Chris Balch issued a memo to Sigman and the City Council saying that if a new single-story building is allowed to be constructed on the site, that “places the Overlay District, and perhaps the entire Zoning Code, in jeopardy of being found invalid because the city would have ignored its own code and state law.”
The White House served in 2012 as a kind of City Hall for Brookhaven’s new city government.
“I remember, I let them use it for free,” Rybert said of the history of the house. He also said the house is not being changed in any way other than the use of new wood.
“It’s going to look exactly the same,” Rybert said. “We left the foundation … it still has the same bones.”
Serpas said he and Rybert have been trying to build the restaurant at the location for nearly four years.
“And it’s been nothing but trials and tribulations” from the city, he said.
Serpas said he has reached out “time and time again” to the city for help, but to no avail.
“They’re forcing our hand,” he said. “There has been nothing but roadblocks, there is no leadership. It’s ridiculous and very frustrating. And I live in Brookhaven, I’m part of the fabric of the community. I’m just dumbfounded. I don’t know what to say anymore.”