The historic core of the village is on the West Side of the settlement covering approximately a quarter of the current built up area. A conservation area covering the centre of the village was designated in 1981 to maintain its character.

Evidence of occupation in and around Potterne identified in archaeological investigations dates from pre-Roman times. Evidence of a late Bronze Age Settlement has been excavated to the North with the finding of a gold bracelet, fragments of Arreton Down Dagger and much pottery. The first church in the village is believed to have been a timber structure with baptistery, built in the 10th century in the gardens of Porch House just to the south of the present church. This was replaced in the middle of the 12th century with a stone church, itself replaced or incorporated into the present church in the middle of the 13th century. A south porch was added in the 14th century and the tower raised in the 15th, the church thus retains a rare architectural uniformity with much 13th century work still in evidence. Medieval Potterne was an important place with the quality of the church reflected by the fact that Potterne was a Manor of the Bishop of Salisbury. A Bishops Palace at Courthill overlooked the centre of the village from the 13th to mid 17th century when it was finally demolished. Early development took place in the High Street and around the church with scattered buildings being built at the same time along Whistley Road.

The Porch House, a grade 1 listed building of exceptional quality dates from 1460, whilst Church House, some of the houses in the High Street and parts of the George and Dragon Pub date back to the 1600. The Church of Saint Mary, being located at a high point in the landscape, forms a focal point and provides a vantage point of exceptional views of the village from the top of its tower. Down below on two sides of the churchyard is Rookes Lane, a somewhat tranquil place with its cobbled surface, established trees and shrubs edged by listed buildings. All the listed features and many others not mentioned contribute towards making the village of Potterne one of special character and of historical interest.