...I went to the Nelson house.
OR
...I went to the Nelsons' house.
Each of these is correct to say in English. Obviously, the second one has to deal with the possessive whereas the first one does not.
Remember that, when a surname has the word "the" in front of it, it is always plural.
It is the names that end in "s" or "z" that cause us the most problem. If you are unsure of what it should be and whether or not it should be plural, try a name that does not end in "s."
...He was driving the Millers' car.
...He was driving the Sanchezes' car.
Pronunciation comes into play in that many people do not know how to pronounce these plural names. My husband and I together are not the "Wells"; we are the "Wellses."
Happy punctuating!
Margie