If you send your passport in with your application for a visa, in most cases it will be stamped with a date and returned to you once your visa has been denied. If you applied in person at an embassy or consulate, they will return it to you before you leave. If you mailed it in along with your application, they will mail it back to you.
Your passport could be held if your visa has been denied. This usually occurs if you travel to a country and request a visa at the port of entry. If the visa is denied, they will hold your passport and take you into custody to prepare for your deportation. Your passport will be returned to you once you have departed the country.
Once your application for a visa has been denied, you may wish to re-apply. A lack of documentation is one of the main reasons visas are denied. If you can include more information regarding why you need to return to your country of origin, this may strengthen, but not guarantee, your case.