Open source doesn't imply trustworthiness and it's a very dangerous assumption to make.
Any open source system can be screwed with in a variety of ways. The simplest and most effective option is to publish both the source and the binaries, but built latter from the an altered source. This will work in a vast majority of cases, because a lot of people make this ridiculous assumption that publishing the source automatically implies that the guy is good, open and trustworthy all over. And won't bother verifying the binaries. Virtually everyone will assume that since it's open there will be someone who will do the verification. Guess what? That someone will assume the same thing.
That's your good old social engineering. It's the humans that are exploitable, not the tech.
But let's say, as unlikely as it is, this such person materialized. Easy enough to run an independent build and verify the binaries, right? Sure. In theory. In a lot of cases, due to dependencies, it's either hard or nearly impossible to do. In other cases it translates into an non-trivial amount of work, which needs to be justified. I am aware of just one project - PGPfone - that published not just the code, but the exact build instructions to produce matching binaries. Everything else is just the "open source, trust us" model. And so the bottom line is that in heck of a lot of cases you will not be able to produce matching binaries.
Now, even if the binary difference in just several bytes that is 100% enough to screw everyone over. This is done by messing with an initialization of an internal random number generator, which all crypto stacks have. All you need to do is make the PRNG (semi)predictable and the best crypto won't stand a chance as there'll be no secrets.
In the end, if you are using pre-made binaries (and who doesn't?) that are not built by a trusted entity from a specific peer-reviewed snapshot of the sources, you have the exact same chances of running a flawed version regardless of whether its source is open or not. Except that in a closed source case you are likely to be more on guard for the surprises.