On the formation of the Ministry of All the Talents in 1806, Lord Albemarle was appointed Master of the Buckhounds by Lord Grenville. Thereby he became an officer in the Master of the Horse's department in the Royal Household and also the equivalent of today's Representative of Her Majesty at Ascot. The Mastership of the Buckhounds being a political office, the holder changed with every government and because the Earl's patrons fell in March 1807 he lost his position after only one year. He remained out of office until 1830 when he was sworn of the Privy Council and made Master of the Horse by Lord Grey, which was the third ranking officer at court. He continued in this office until November 1834, the last few months under the premiership of Lord Melbourne, and held the same post under Melbourne between 1835 and 1841. Consequently, he was responsible for managing all matters equine at the changeover from one reign to the next and, in particular, at Queen Victoria's Coronation. The Earl was accorded the honour of travelling to Westminster Abbey inside the Gold State Coach with the nineteen-year-old, and as yet unmarried Victoria, who recorded in her diary: "At 10 I got into the State Coach with the Duchess of Sutherland and Lord Albemarle...It was a fine day, and the crowds of people exceeded what I have ever seen; their good humour and excessive loyalty was beyond everything, and I really cannot say how proud I feel to be the Queen of such a nation".
As Horseman and Racehorse Owner
In addition to managing the bloodstock of two successive heads of state, when the horse was still a main mode of transport, Lord Albemarle was also a leading racehorse owner of his day. As an owner, William Charles won two Classics, and the Ascot Gold Cup three times in 1843, 1844, and 1845. The second Gold Cup win, in 1844, was by a colt which the Earl had not yet named. One of the witnesses of this triumph, Tsar Nicholas I of Russia, let William Charles know how excited he had been by the race, and the Earl promptly named his horse "The Emperor" in honour of the distinguished Russian visitor. In 1845, when "The Emperor" won the Gold Cup again the Earl received a massive silver centrepiece paid for by the Tsar as the race prize based on Falconet's well known sculpture of Peter the Great in St Petersburg, the base flanked by Russian equestrian troops. William Charles's horses were also victorious in the 1840s in the Cesarevitch and Cambridgeshire major handicaps run at Newmarket. In 1833 he was made a Knight Grand Cross of the Hanoverian Order.
After his first wife's death in November 1817, aged 41, Lord Albemarle married, secondly, Charlotte Susannah, daughter of Sir Henry Hunloke, 4th Baronet, on 11 February 1822. This marriage was childless. He died at Quidenham, Norfolk, in October 1849, aged 77, and was succeeded in the earldom by his second but eldest surviving son, Augustus. The Dowager Countess, Charlotte Susannah, was nicknamed the "Rowdy Dow" by her stepchildren, who accused her of squandering the family's fortune. In the words of one biographer: " managed to disperse Keppel heirlooms with extravagant eccentricity." The Dowager Countess of Albemarle died at Twickenham, London, in October 1862, aged 88.