Smith was born in Kingston in May 1945 and raised in St. Ann and May Pen. His father played guitar and bought him an instrument when he was twelve. He played guitar in the band The Vandals in Claremont, St. Ann, after leaving school, and was nicknamed ‘Ernie’ after Ernest Ranglin. He initially (unsuccessfully) sought a career as a radio announcer, but then approached Federal Studios looking for work as a songwriter, ending up recording his songs himself. His first recordings were not a success and Smith spent time working in life insurance before returning to the studio.