As the story goes, Caroline Schermerhorn Astor built a magnificent ballroom, which could hold about 400 guests, giving rise to the most coveted invitation in all of New York City’s high society. Her nephew and Fifth Avenue neighbor, William Waldorf Astor, felt that if anyone was going to be NYC’s leading lady, it ought to be his wife, not his aunt, so he built the Waldorf, a 13-story hotel that towered his aunt’s mansion.