Was Gaelic ever the dominant language in south-east of Scotland e.g. Edinburgh area?

No, Gaelic was never the dominant language in the south-east of Scotland, including the Edinburgh area. The region has historically been predominantly English-speaking, with Scots as a secondary language.