Peter Schermerhorn, father of Abraham Schermerhorn, built these counting houses in 1811–12 to serve the growing New York seaport. No 2 & No 4 Fulton Street were occupied from 1847 to the 1990s by Sweet's Seafood House, for over a century New York City's oldest fish restaurant. The building at the corner of Fulton and South Street (#2) was once a hotel; at that time it was altered – in 1868 – to add a mansard roof.[2]
The buildings were purchased in 1974 by the State of New York.[3] During the 2003 redevelopment, these buildings were linked to the A.A. Low Building, which faces John Street, to create gallery space. The permanent exhibits include paintings by the maritime artist James E. Buttersworth.[4]
Nearly contemporaneous to the building of Schermerhorn Row, other counting houses and warehouses were built in the immediate area, at 180-195 Front Street, 159-171 John Street, and 91-92 South Street, many of them in the Greek Revival style. These buildings were all restored in the 1980s under the supervision of the architect Jan Hird Pokorny, and are now part of the South Street Seaport.[2]