▶ Watch Video: United Nations meets for emergency special session

Mystery and intrigue have always surrounded the Russian Diplomatic Compound in New York. Tall steel fencing and cameras, too.

The 20-story white building is unmissable. It towers over the tree-lined Henry Hudson Parkway and is far taller than any nearby structure in the Bronx’s Riverdale neighborhood. It is home to the families of diplomats serving in the Russian Mission to the United Nations — and maybe also a few spies.

The Russian Diplomatic Compound in the Bronx, New York.

Graham Kates / CBS News

On Monday, Russia’s U.N. ambassador, Vassily Nebenzia, interrupted a press conference to take a phone call. He then said he had just been informed that the U.S. Mission to the U.N. will be expelling 12 diplomats from Russia’s Mission to the U.N. 

Olivia Dalton, a spokesperson for the U.S. Mission, confirmed that the U.S. has begun “the process of expelling 12 intelligence operatives from the Russian Mission who have abused their privileges of residency in the U.S. by engaging in espionage activities that are adverse to our national security.”

She said the action “has been in the works for several months.”

The Russian Mission confirmed to CBS News that most of its New York-based diplomats and their family members live in the Bronx building, but did not say if that includes the 12 expelled people. 

The building has long been an enigma. It was built in 1974 using a unique top-down construction method that involved hoisting up prefabricated floors onto a frame. Floor 20 hovered high over the ground before the first floor was slid into place. Locals have long theorized this method was used to prevent American intelligence agencies from getting a foothold inside.

Jeffrey Dinowitz, the New York State Assembly Member who represents Riverdale, said other theories have been bandied about.

“I don’t know if it’s true or just some urban legend, but the story is that it was built there because it’s one of the highest points in the Bronx in terms of elevation and, you know, that it’s to spy from in terms of electronic monitoring and stuff like that,” Dinowitz said. “I don’t know, but I think a lot of people believe that’s the case.”

Steel fencing and security cameras surround the Russian Diplomatic Compound in the Bronx, New York.

Graham Kates / CBS News

During times of tension between the U.S. and Russia, even police and firefighters can be barred from entering. A small fire in the building in 2011 was handled internally, while the New York City fire department watched from outside the gate.

In 2014, local concerns about espionage seemed to be confirmed when Evgeny Buryakov, a Russian banker who lived in a small house abutting the compound, was arrested and charged with one count of conspiring to act in the United States as an agent of the Russian Federation. He later entered a guilty plea and was deported.

On Sunday, dozens of protesters gathered outside the building, honking car horns, waving Ukraine’s flag and singing its national anthem, according to The Riverdale Press. They chose that spot to demonstrate against the violent invasion of Ukraine by Russia, an unprovoked attack ordered by Russian President Vladimir Putin that in its first week has already claimed scores of lives and forced hundreds of thousands of people to flee their homes.

Dinowitz, who called the building an “eyesore,” said it’s “sort of like an island separated from the rest of the community.”

But it wasn’t always quite that way. Dinowitz recalls going inside the building once. It was the late 1980s, the era of glasnost, when the soon-to-fall Soviet Union was attempting to increase openness and transparency within its government. And there was a concert in an auditorium.

“I don’t remember it being a particularly beautiful place, I’ll tell you that. And it certainly isn’t beautiful from the outside, that’s for sure,” Dinowitz said of the building, which has its own  grade school and playground. “It’s a residence. I guess a lot of the Russian diplomats and their families may live there.”

Asked on Tuesday if any of the residents were among those expelled, a Russian Embassy spokesperson said, “I do not have this information, and for what reason would I get it? Just to share it with you?”

-Pamela Falk contributed reporting from the United Nations.