As Congress Hall’s reputation grew, so did Cape May's. By the middle of the 19th Century Cape May had become a booming holiday destination, rivaling Saratoga and Newport for popularity. Congress Hall had doubled in size and was welcoming guests from around the region, but in 1878 the building was destroyed when a huge fire swept through 38 acres of Cape May’s seafront.

Within a year, the owners rebuilt the hotel, this time in brick rather than wood, and business blossomed once again. The hotel and Cape May proved so popular that they gained renown as a summer retreat for the nation's presidents. Ulysses S. Grant, Franklin Pierce, and James Buchanan all chose to vacation here. President Benjamin Harrison made Congress Hall his "summer White House" and conducted the affairs of state from the hotel.

The famous 19th Century composer John Philip Sousa also loved Congress Hall. After conducting concerts on Congress Hall's lawn with his Marine Corps band throughout the season in 1882 he composed a march in honor of the hotel, the "Congress Hall March."

The hotel fell into disrepair and remained closed for more than a dozen years, sometime between 1905 and the early 20s, after a long and bitter dispute between owner Annie Knight and the city council. Finally, an agreement to repave the roads around Congress Hall was reached and the hotel reopened in the early 20s, with a stunning renovation. Congress Hall went on to open Cape May's first post-Prohibition cocktail bar (where the Brown Room sits now) in 1934.

From 1968 until 1995 Congress Hall operated as part of the Cape May Bible Conference led by the Reverend Carl McIntire of Collingswood, New Jersey. By providing an alternate use for the building Dr. McIntire in effect helped preserve it during a time when many of Cape May's landmarks were being demolished to make way for modern motels. The present owners purchased the building in 1995 with the goal of undertaking a complete renovation to return Congress Hall to its former glory.

The hotel underwent an extensive renovation in 2001, during which a stash of original china, creamers and sugar bowls were unearthed. Both the Buffalo Pottery Company and D.W. Haber and Sons, which produced the items, are still in business and were able to reproduce them for the hotel’s 2002 re-opening.