Until the late 19th century Swanmore was part of the large and important parish of Droxford, whose boundaries were first defined in a Saxon charter dated 826AD. The name Swanmore first appears in 1205 as Suanmere, assumed to mean "Swan's Pool" and referring to the Bishop of Winchester's swans. They were perhaps kept in the low-lying area of the present village known as The Lakes. The parish of Swanmore was based on two tithings: the one near the Rising Sun Inn and now known as Hill Pound, originally "Hill"; the other was Swanmore which was cottages in upper Swanmore and houses near what is now the centre of the village. The main route north-south through the village, at Hill Pound, carried the coach road from London to Gosport. Lanes from the village to Droxford arose from the routes followed by the villagers travelling to their parish church.
With the growth of Droxford population during the early nineteenth century facilities for worship were made available in outlying areas of the parish. In Swanmore the original school built in 1833 which stood on the site of the war memorial was licensed for evening services in 1836 and nine years later in December 1845 St Barnabas' church was opened and Swanmore became an ecclesiastical parish with its own resident priest, parsonage and so on. In 1863 the Primitive Methodists, who had worshipped in a private house since the 1830s, built their own chapel in the village.
The enclosure of Waltham Chase (or Horders Wood) in 1855 created the boundaries of the village as they are today. Ten acres were allocated "for exercise and recreation" which later became the recreation ground. In mid-century the population was 750. Apart from the usual rural employment there were brickworks in the village until just before the Second World War. The small school was rebuilt on the north side of the main road in 1864, where it remains, much enlarged and modernised in the 1960s and 1970s, to this day. Most children attended the school at least for a time, although education was neither compulsory nor free until 1891. A separate infant school was built in 1884 - it now forms part of the Paterson Centre.
Swanmore became a civil parish, following the passing of the Parish Councils Act in 1894. Amenities available in the towns gradually spread to the villages, but Swanmore was content for some time to use oil lamps, wood stoves and privies as well as water from wells or pumps. Pavements were not general until the 1960s. Mains water was available from 1903 but the parish was reluctant to incur the cost of installation until 1921. Gas and electricity both arrived in the 1930s, but mains sewerage was not installed until the 1960s. Dustmen did not arrive in the village until 1943.
With all amenities available, village life grew more attractive and in the 1970s Swanmore grew faster than in any previous decade. The rise of the motor car meant that shops closed and bus services declined or disappeared. In this period a new secondary school was built for the district, becoming comprehensive in 1973, and the much-loved corrugated iron Village Hall, formerly the British Legion Hall, was replaced by a new one, opened in 1980. Finally, the old Parish Room was sold, to be replaced in the late 1980s by the Paterson Centre, named to commemorate the ministry of the Rev Ron Paterson, Vicar from 1962-85 and three members of his family who died whilst he was vicar.