Born the son of William Cavendish, 3rd Duke of Devonshire and Catherine Cavendish, Cavendish was commissioned as an ensign in the 1st Regiment of Foot Guards on 29 April 1749. He entered politics as Member of Parliament for Derbyshire in 1751. He was promoted to lieutenant in his regiment and captain in the Army on 17 March 1752. In 1754, Cavendish gave up the Derbyshire seat for his brother George and was returned to Parliament as Member for Derby instead. He was seconded to the 29th Regiment of Foot as lieutenant-colonel and went to Ireland with his brother William Cavendish, Marquess of Hartington, newly made Lord Lieutenant of Ireland, in 1755. Cavendish was promoted to captain in the 1st Regiment of Foot Guards and lieutenant-colonel in the Army on 1 June 1756 and served as an aide-de-camp to the Duke of Cumberland in Germany in Summer 1757 during the early stages of the Seven Years' War. Promoted to colonel on 7 May 1758 and appointed an aide-de-camp to the King on 9 May 1758, he served under Charles Spencer, 3rd Duke of Marlborough during the raid on St Malo in June 1758 and then took part in the raid on Cherbourg in August 1758. He commanded the rear-guard during the re-embarkation following the disastrous battle of Saint Cast in September 1758 and, having been taken prisoner, gallantly offered to remain in captivity on the basis that he was a Member of Parliament. He was nevertheless released by the Duke of Aiguillon in an exchange for a French officer of equal rank in October 1758. Cavendish became colonel of the 67th Regiment of Foot in October 1759 and colonel of the 34th Regiment of Foot in October 1760. Promoted to major-general on 7 March 1761, he sailed for Germany where Prince Ferdinand of Brunswick gave him command of a brigade of chasseurs which he led to victory at the Battle of Wilhelmsthal in June 1762. Part of his brigade was ambushed during the Siege of Kassel in October 1762. Cavendish was promoted to lieutenant-general on 30 April 1770, but owing to his sympathies, took no part in the American Revolution. In 1780, he retired from Parliament and his seat was taken by his nephew Lord George Cavendish. He was promoted to full general on 20 November 1782 and to field marshal on 30 July 1796. He died at his home, Twickenham Park, on 21 October 1803 and was buried in the family vault at Derby Cathedral. He left most of his property to his nephew, Lord George Cavendish, later 1st Earl of Burlington.