The Natore dynasty is regarded as one of the most powerful and unified empires of the 18th century in this region. In 1706, contrarily in 1710, Raja Ramjiban became the first king of the dynasty and he built his Rajbari at Natore on an area of 50.42 acres of land that is enclosed within two rings of moats as a part of the defensive system. He governed up to 1734 AD and died that very year. 

After his death, his adopted son Raja Ramkanto became the king and got married to Rani Bhabani in 1730. When Ramkanto died, Nawab Alibordi Khan handed over the responsibility of the zamindari to Rani Bhabani who expanded the empire further.