In the Addenda and Notes to " Abury Illustrated," the following is given as an explanation of the word "Sarsen": " The term Sarsen, or Saresyn, was applied by the Anglo-Saxons, simply in the sense of Pagan, to the stones which they found scattered about the Wiltshire Downs. As all the principal specimens of these mysterious blocks were perceived to be congregated into temples popu- larly attributed to heathen worship, it naturally came to pass that the entire formation acquired the distinctive appellation of Sarsen or Pagan stones. The same epithet of ' Saresyn ' the Saxons also applied to their invaders the Danes or Northmen, who, on their coming into this country, were universally pagan. Thus Robert Ricart (quoted in Roberts' History of Lyme) says, ' Duke Rollo Le Fort was a Saresyn come out of Denmark into France ; ' and a spot in Guernsey is still designated by the same term from having constituted the temporary stronghold of certain Norman freebooters." — Waylen's History of Marlborough, p. 529.
The following is from Mr. Henry Lawes Long's ' ' Survey of the Early Geography of Western Europe : " " In addition to the suggestion advanced that our word Sarsen, as applied to the Druid sandstone, is, in fact, a corruption of Saracen, I may add that Sarrasin is the name commonly given on the Continent to ancient objects whether of Celtic or Roman construction, thereby inferring a period anterior to any remains of Christian origin. Roman denarii, which in the north of France still occasionally are current as sous, bear the name of Sarrnsins. The Roman bridge near Aosta is called the Pont de Sarrasins." And I may add the following extract from the " Journal de I'Architecture," (of Brussels,) 4"". annee, p. 84 : " Les traditions locales attribuent la con- struction des chaussees romaines aux Sarrasins. Les mines, les tuiles antiques, les poteries, les medailles, etc., que Ton trouve chaque jour, ne sont connus, comme on salt, que sous les noms de Masures, de Vahes, de JiJonnaies, ou de puits des Sarrasins, Cette denomination remonte evidemment aux temps des Croisades, lorsque les esprits etaient remplis du nom des infideles. Du reste, les armees et les populations qui revenaient de Terre-Sainte, en suivant les chaussees romaines, n'auront pas peu contribue a repandre aux environs I'epithete injurieuse de Sarrasin et de payen, dans laquelle ils auront confondu les Romaines si, comme il est probable, un faible souvenir de ce grand peuple vivait encore a cette epoque dans le souvenir de nos peres." Mr. Long quotes the following from Col. Symonds's Diary, which his cousin, Mr. C. E. Long, edited for the Camden Society: " 12"' Nov. 1644, Tuesday, though a miserable wett windy day, the army moved over the playnes to Marlingsborough, where the King lay at the Lord Seymour's howse, the troopes to Fyfield, two myles distant, a so full of a grey pibble stone of great bigness as is not usually seene ; they breake them, and build their howses of them and walls, laying mosse betweene, the inhabaitants calling them Saracen's stones, and in this parish a myle and halfe in length, they lye so thick as you may goe upon them all the way. They call that place the Grey -weathers, because a far-off they looke like a flock of sheep”.
From Wiltshire Archaeological and Natural History Magazine Vol XVI 1876 pp.68-75
I will add the old farmworkers I grew up pronounced them Saracen Stones. I note also that some of the young of the Pewsey Vale still pronounce Saracen as Sarsen: My daughter for instance:
Update: It has been pointed out by Brian Edwards that "In an experiment in 1986 Kennet Valley locals that spoke with the Wiltshire dialect were recorded saying King's Hill which when spoken translated to "King Zil", Saracen which translated to "saracen" so the same, but Saxon translated to "sarsen". Subsequently, when asked to say sarsen, the tapes reveal what distinctly sounds like Saxon."
Dialect in Wiltshire: And Its Historical, Topographical and Natural Science Contexts by Malcolm Jones, Patrick Dillon Wiltshire County Council, Library & Museum Service, 1987 reproduces from M Wakelin's English Dialects: an introduction 1977 a map that shows the boundaries of the South Western Dialect as defined by the voicing of initial fricatives in the 1950s, ie saying Varmer or Farmer.
It is noticeable that the Kennet valley between Avebury and Marlborough is close to the boundary and lay just outside the South-western dialect area. I believe the dialect area had shrunk since Victorian times, and continues to do so.
Click to enlarge
And from 1903: