You cannot take a bus directly to the Alcázar. Buses from Madrid (e.g., from Moncloa station) terminate at the Segovia bus station, about 100 km (62 miles) away. The journey takes ~1 hour 15 minutes.
From the Segovia bus station, the Alcázar is a 20-30 minute walk (uphill) or a short taxi/local bus ride to the historic center where the castle sits.