In physics, thought experiments are impressive heuristic tools. They are valuable instruments to help scientists find new results and to teach students the known ones. However, as we shall show, they should always be received with prudence, even when they are a shortcut to ‘prove’ well-established results. Here, we show that the most widely known thought experiments devised to derive the gravitational frequency shift from energy conservation are, in fact, problematic. When properly set and correctly read, those thought experiments reveal that the existence of the gravitational frequency shift is, in fact, at odds with energy conservation. We also propose two new simple thought experiments, one using energy conservation and the other the conservation of linear momentum, that corroborate that conclusion, showing that those conservation principles do not imply the gravitational frequency shift. Our results may be of some epistemological interest and could serve as a warning sign on how thought experiments should be received and trusted.