According to the New York City Administrative Code, Sec. 19-163- "it shall be unlawful for any person to keep, stable, house or shelter at any location in the city any animal, including, but not limited to, dogs, cats, horses and wild and exotic animals, which is not indigenous to the city, without first obtaining a permit therefor from the department."
Therefore, it is unlikely that obtaining a permit for horses in New York City would be granted, making horse-riding in the streets of the city effectively prohibited.