Born in Wolverhampton, England, Holland taught himself how to play stringed instruments, beginning at four on the ukulele, then graduating to guitar and later bass guitar. He quit school at the age of 15 to pursue his profession in a top 40 band, but soon gravitated to jazz. After seeing an issue of Down Beat where Ray Brown had won the critics' poll for best bass player, Holland went to a record store, and bought a couple of LPs featuring Brown backing pianist Oscar Peterson. He also bought two Leroy Vinnegar albums (Leroy Walks! and Leroy Walks Again) because the bassist was posed with his instrument on the cover. Within a week, Holland traded in his bass guitar for an acoustic bass and began practicing with the records. In addition to Brown and Vinnegar, Holland was drawn to the bassists Charles Mingus and Jimmy Garrison.

After moving to London in 1964, Holland played acoustic bass in small venues and studied with James Edward Merrett, principal bassist of the Philharmonia Orchestra and, later, the BBC Symphony Orchestra. Merrett trained him to sight read and then recommended he apply to the Guildhall School of Music and Drama. Holland received a full-time scholarship for the three-year programme. At 20, Holland was keeping a busy schedule in school, studios and Ronnie Scott's Jazz Club, London's premier jazz club, where he often played in bands that supported such touring American jazz saxophonists as Coleman Hawkins, Ben Webster and Joe Henderson. He also linked up with other British jazz musicians, including guitarist John McLaughlin, saxophonist Evan Parker, reedsman John Surman, South Africa-born London-based pianist Chris McGregor, and drummer John Stevens, and performed on the Spontaneous Music Ensemble's 1968 album Karyobin. He also began a working relationship with Canada-born, England-based trumpeter Kenny Wheeler that continued until Wheeler's death in 2014