Albury's old Saxon church of St Peter and St Paul is one of the most ancient in Surrey, with walls that may date from before the Norman Conquest. The building contains a remarkable 13th century wall painting of St Christopher and a brass commemorating John Weston, who held the manor in the 15th century. The church fell out of regular parish use when the village was effectively relocated in the 19th century.
In 1819, Henry Drummond, a wealthy banker and Member of Parliament, bought the Albury Park estate. Drummond was a follower of Edward Irving and the Catholic Apostolic Church, a millenarian movement that expected the imminent Second Coming. He commissioned Augustus Pugin, the architect most associated with the Gothic Revival and the Palace of Westminster, to build a new church for the congregation in 1840. Pugin also extensively remodelled Albury Park mansion, giving it sixty-three individually designed chimney stacks that form one of the most distinctive rooflines in Surrey. Each chimney is different, and they can be glimpsed above the trees from the road through the village.
The grounds of Albury Park were laid out in the 17th century by John Evelyn, the diarist, writer, and one of the founders of the Royal Society. Evelyn created a quarter-mile terrace walk and a tunnel modelled on a Roman grotto, both of which survive. The Drummond family held the estate for generations, and their influence shaped the village into the quiet, well-ordered place it remains. Silent Pool, a natural spring-fed lake with extraordinarily clear blue-green water, sits just off the road toward Shere. Legend connects it to King John, who is said to have caused the drowning of a local maiden, though the story is romantic fiction from a Victorian novel rather than documented history. The pool's clarity comes from chalk filtration, and it remains one of the most atmospheric spots in the county.