New York City was never the first capital of the United States.
The first capital under the Constitution was Philadelphia, Pennsylvania (1790–1800), where Congress met in Congress Hall. Before that, under the Articles of Confederation, the capital moved among several cities including Philadelphia, Princeton, Annapolis, Trenton, and New York City (1785–1790).
In 1790, the Residence Act designated a permanent capital on the Potomac—Washington, D.C.—which has served since 1800. New York City was the last temporary capital before D.C., not the first.