The former managing editor of the Oregonian has a really good essay about why political sex scandals deserve to be investigated by the press. It's not really possible to excerpt any single part of it, as it's sort of an explanation of the author's evolution from thinking that such stories were the equivalent of having reporters "peer[ing] in people's bedrooms" to thinking that they do matter. It's in the context of the David Wu mess, but touches upon other Oregonian political sex scandals of note.
To be sure, the incidents he discusses are all examples of unwanted sexual conduct -- in some cases, outright sexual assault. Consensual sexual affairs (even if adulterous) between adults may well be a different matter, assuming that no laws are broken (cf. Elliot Spitzer) or the politician is not lying under oath (cf. Bill Clinton).