Joseph Rogers built the inn in 1786. The original log-and-stone structure is now under clapboard siding. It is located at 205 South Rogers Street.
Legend tells about an early visitor to Rogers Tavern. One day, a stranger came to the inn and asked for a room. The landlady showed him a room that he could share with two other gentlemen. The stranger proceeded to make disparaging remarks about a town that could not offer a gentleman a private room. Another guest at the inn overheard the remarks and seized the stranger by the arm ushering the stranger to the corncrib out back and threw him in. Once locked inside, the stranger was asked if he was willing to apologize and accept the first-offered room. He agreed. The guest who requested the apology was none other than Andrew Jackson.