The 4th Amendment!
Dan, don’t forget about the 4th Amendment against unreasonable searches and seizures.
It’s not just the 9th and 14th Penumbra stuff from the somewhat kooky Justice Douglas. The 4th Amendment guarantees privacy in a much more specific way and Justice Brandeis’s famous dissent (“the right to be left alone”) in 1928 cued up that (eventual) mainstream reading of the 4th.
I don’t think James Madison and gang were as dumb as you think, and they probably don’t need Ted Kennedy’s help.
Here are some good backgrounders on Brandeis and the 4th: