On 21 May 1829, he married Nancy Gray.  She was the daughter of Marcus Gray.

They had the following children:  Emiline, Eliza, Elihu, Levi, Josiah, Mary, Nancy, Marcus Spencis, Marshall, John Phifer Marshall, and Sylvania.

Joseph and his family lived near present-day Leighton on Barnes Mountain.  Due to county boundary changes, they lived in Owsley, Estill, Jackson and Madison County at various times.

Joseph and Nancy divorced in 1856.  Arrangements for the minor children aren't known, but they were living with their father on the 1860 Census.

Nancy died about 1859.