There was a time, before universal white male suffrage and the closing of the frontier, when the poor whites of the South were considered shiftless and without caste. If we were to look at the South as a hierarchical system, it could be argued that the poor whites were a kind of pariah. There’s a common misconception that all whites during the antebellum period had an equal standing economically and socially.
The Old South was a complex society, not just a place comprised merely of free whites and enslaved blacks. There were indentured servants, African slaves, Native Americans, poor white yeoman farmers, and a wealthier landowning aristocracy. Some of the first colonies were owned by wealthy English barons (like the Lords Proprietors in Carolina) and after the Revolution, most states had property requirements that kept the poor whites from voting.
More @ The Abbeville Institute