Pang Xi was from Henan County, Henan Yin, which is in present-day Luoyang, Henan. He started his career as a subordinate of Liu Yan, the Governor of Yi Province, and held the position of a Consultant. In 194, Liu Yan's sons Liu Fan and Liu Dan allied with the warlordMa Teng, who launched a military campaign against Li Jue and Guo Si, the warlords controlling the Hancentral government and the figureheadEmperor Xian. However, Ma Teng lost the battle and both Liu Fan and Liu Dan were killed. At the time, as Pang Xi was a family friend of Liu Yan, he turned to his network of friends for help in helping Liu Fan and Liu Dan's families escape from Chang'an and bringing them safely to Yi Province, thus preserving Liu Yan's posterity. Following Liu Yan's death later in 194, his youngest son Liu Zhang succeeded him as the Governor of Yi Province. During this time, Liu Zhang appointed Pang Xi as the Administrator of Baxi Commandery and ordered him to lead troops to attack his rival, Zhang Lu, in Hanzhong Commandery. Although Pang Xi never managed to make any territorial gains in Hanzhong Commandery, Liu Zhang still left him in charge of Baxi Commandery to guard against possible advances by Zhang Lu. Liu Zhang also arranged for his eldest son, Liu Xun, to marry Pang Xi's daughter. As time passed, Pang Xi became increasingly arrogant because he believed that Liu Zhang owed him a huge favour for saving his brothers' families in Chang'an and also because he thought that Liu Zhang deeply trusted him. It turned out that Liu Zhang gradually distrusted Pang Xi and began to alienate him. Around 211, when Liu Zhang's adviser Zhang Song urged his lord to invite another warlord Liu Bei into Yi Province to counter the threat posed by Zhang Lu, he warned Liu Zhang that Pang Xi might secretly harbour the intention of overthrowing him. In 214, after Liu Bei seized control of Yi Province from Liu Zhang, he appointed Pang Xi as a Major under the General of the Left. Pang Xi advised Liu Bei to allow Liu Zhang's eldest son, Liu Xun, to remain in Yi Province when Liu Bei relocated Liu Zhang and his other son to Jing Province.
