Zhao Xi was born in Wan, Nanyang Commandery. At a young age, he was known for his integrity. When Zhao Xi was 15, his elder cousin, who had no son, was murdered. Zhao Xi harboured the thought of avenging his cousin, so he brought a group of people with him to confront his cousin's killer. The murderer was ill, and Zhao Xi felt that it was inhumane to kill him at that time, so he spared him. When the killer had recovered, he bound himself and came to plead with Zhao Xi. However, Zhao Xi refused to meet him and had him killed.
In 9 CE, Wang Mang usurped the throne of the Western Han Dynasty and declared himself emperor of the Xin Dynasty. By 23 CE, several revolts had broken out throughout China due to Wang Mang's incompetent rule. Around the time, Liu Xuan, a descendant of the Han imperial clan, was proclaimed "Gengshi Emperor" by rebels aiming to restore the Han Dynasty. Gengshi's forces were attacking Wuyin, a city controlled by a certain Li family. Gengshi heard that Zhao Xi had some influence over the Li family, so he invited Zhao to help him persuade the Lis to surrender. When Gengshi met Zhao Xi, he saw that the latter was merely a youth, so he viewed Zhao Xi lightly. When Zhao Xi arrived in Wuyin, the Li family surrendered on their own accord, and Gengshi Emperor granted Zhao Xi the title of "Lieutenant General of Five Mights" for his achievement. Zhao Xi was injured when he fought in the Battle of Kunyang against the Xin Dynasty, and later received the title of "Marquis Yonggong". In 25 CE, Gengshi Emperor was defeated and killed by the Chimei, another rebel movement. Zhao Xi protected Gengshi and his subjects, along with their families, and led them through Wu Pass to safety. He instructed young women to smear dirt and mud on their faces to avoid being violated by the rebels if the latter caught up with them.
Gengshi Emperor was succeeded by Liu Xiu, who became known as Emperor Guangwu of the Eastern Han Dynasty. When Deng Feng, one of Guangwu's subjects, betrayed his lord, Zhao Xi wrote a letter to Deng to reprimand him. When the situation in China had become more stable, Emperor Guangwu appointed Zhao Xi as Marquis-Chancellor of Jianyang and Marquis-Chancellor of Pinglin. In 41 CE, when Zhao Xi was in office as Prefect of Huai County, he punished a wealthy landlord Li Zichun and his two grandsons for violating the law. Not long later, he was reassigned to be the Administrator of Pingyuan. In 50 CE, the women whom Zhao Xi saved at Wu Pass 25 years ago praised him, so Emperor Guangwu summoned him to the imperial court and commissioned him as Minister Coachman. A year later, Zhao Xi was promoted to Grand Commandant, received the title of a "Secondary Marquis", and was tasked with inspecting the border. In 56 CE, he accompanied Emperor Guangwu to attend a fengshan ceremony at Mount Tai.
Emperor Guangwu died in 57 CE. In his final edict, he ordered Zhao Xi to assist his successor, Emperor Ming. In 58 CE, Emperor Ming named Zhao Xi "Marquis of Jiexiang". In the spring of 60 CE, Zhao Xi was stripped of his post of Grand Commandant for making an erroneous report on Xue Xiu, the chancellor of Zhongshan. In winter that year he replaced Dou Rong as Minister of the Guards. In 65 CE, he took over Yu Yan and served as acting-Grand Commandant. When his mother died of illness, Zhao Xi returned home to mourn, and Emperor Ming attempted to persuade him to remain in court.
