There is no doubt that Elfsen is an ancient settlement area, probably because of its favorable water conditions. Here was one of the upper courts of the Soest Börde, on which the noble family of Elfsen, in the old spelling "de Elfendehusen", sat. Elfsen is documented for the first time in 1179 as Eweldenehusen. The known interpretations of the place name are quite adventurous, but probably refer to a "swampy, boggy, wet area". Elfsen is a cluster village. The center of the village is the Elfser market. The layout of the village in the original cadastre already shows the present structure, but the individual buildings, apart from the Lange (Lütke) farm, no longer exist in their original form.