Gray was born in Walsall, but his family soon moved to London, where Gray studied medicine. He assisted his father in writing The Natural Arrangement of British Plants (1821). After being blackballed by the Linnean Society he turned his interest from botany to zoology.[citation needed] He began his zoological career by volunteering to collect insects for the British Museum at age 15. He officially joined the Zoological Department in 1824 to help John George Children catalog the reptile collection. In 1840 he took over from Children as Keeper of Zoology, which he continued for 35 years, publishing well over 1000 papers. He named many cetacean species, genera, subfamilies, and families.

During this period he collaborated with Benjamin Waterhouse Hawkins, the noted natural history artist, in producing "Gleanings from the Menagerie at Knowsley". Knowsley Park, near Liverpool, had been founded by Edward Stanley, 13th Earl of Derby and was one of the largest private menageries in Victorian England.

In 1833, he was a founder of what became the Royal Entomological Society of London.

During his fifty years employed at the British Museum Gray wrote nearly 500 papers, including many descriptions of species new to science. These had been presented to the Museum by collectors from around the world, and included all branches of zoology, although Gray usually left the descriptions of new birds to his younger brother and colleague George.

He was also active in malacology, the study of mollusks. He described and named numerous marine snails include:

On his death he was buried at St Mary's church, Lewisham.