Recently, a radio personality called out people justifiably for using the phrase, "To tell you the truth..." He said it made him wonder if everything the person said prior to that was a lie.
I think the same logic applies when people preface a sentence, "In all honesty..."