Condenser mics are not quite as fragile as you fear. But the obvious primary candidate for possible damage is the capsule with its micron-thin membranes.
And there, at least from my observation, it is fairly safe to assume damage to the performance if you can see wrinkles in the membrane (from the capsule whip-lashing against the inside of the protective basket, or, in the case of some pencil condensers, from the membrane making contact with the edge of the mic's capsule housing).
In the latter example, you will not be able to see the damage to the diaphragm platelet, but you may be able to see a dent in the housing, which would be indicative of a major hit having taken place.
If no diaphragm wrinkles, or deformation in the housing, can be detected, assume everything is ok.
The larger question (and answer) to whether more subtle damage occurred after a fall is indeed careful listening. And here, I would not rely on comparing current sound to past recordings, as exact mic positioning is always hard to recreate, but I would trust my ears which will tell me if the sound is grossly off from my expectation of stock performance.
What helps with even more subtle changes over time: compare your mic to a healthy specimen of the same model.