Rivera was born in Santurce, Puerto Rico, a sector of San Juan, Puerto Rico. He was the first of five children born to Luis and Margarita Rivera. His father, Luis, was a carpenter and his mother a housewife. As a child, Rivera was always singing and banging on cans with sticks. He received his primary education at the Pedro G. Goyco Elementary School and then went on to learn carpentry at a vocational school. He also shined shoes to help his family financially and, when he was 16 years old, he worked as a carpenter. During his free time he would hang around the corner with his best friend Rafael Cortijo and sing songs. In 1948, Rivera and Cortijo joined El Conjunto Monterrey, where Rivera played the conga and Cortijo the bongos. Rivera was unable to work full-time as a musician because of his work as a carpenter.