Do you capitalize street with the name of in front?

Yes, when Street is part of a formal street name, it should be capitalized: On Main Street, we took a left onto Elm Street.

However, when street is used in a general sense, it should not be capitalized: We live on a quiet street.