In a refrigerator you pump in energy and it is used to separate cold (inside) from warm (outside). An RTG used to provide power on spacecrafts runs the same process backwards (I think) by finding a source of hot and a source of cold and extracting energy from the difference.
The RTG's source of hot is obvious - decaying radioactive material. But what is the source of cold? Is the near-vacuum around the spacecraft really sufficient?
Sure, it might be cold if we strictly look at temperature.
I wouldn't put my hand in water at 80 degrees C, but being in a sauna at that temperature is OK since the air has a lower particle density. Outer space certainly has considerably lower particle density than air on earth, so does its low temperature really "count"?