How do I do the above?
With smallish crafts, doing a soft re-entry on Droo is easy enough, speed is lost progressively while descending through the atmosphere that re-entry heating never becomes a problem. Also easy for gyros to keep the craft pointing the way it should while this is happening. It's a simple question of using parachutes or a small suicide burn a few km above impact to have a soft landing.
With larger crafts (such as trying to land a booster or a single-stage-to-orbit rocket) in the many hundreds of tons, I can't get this to work. Because of the square-cube law, I punch through the upper atmosphere without slowing down, and end up burning through ablative coating in seconds when I hit ~40km ASL still going at close to 3000m/s. Trying to keep the engines pointing forward and the nose-cone pointing backwards is also tough (because it's aerodynamically unstable) and even a ton of RCS and gyros fail at a certain point. How do I get around this?
The obvious solution is to have some deployable fins / airbrakes type of construct near the nose cone that increases resistance through the upper atmosphere, in order to both increase the rate at which I'm bleeding speed and keep the rocket aligned. However I've tried a few different ways and I can't get this to work:, either using structural panels or fins or wings: it doesn't seem to provide much extra stability or meaningful extra drag, and it burns up anyway. The other thing I tried is to do a starship-esque "belly flop" in the upper atmosphere, and then turn to have the engines prograde for the final controlled burn; but absolutely nothing I do seems to give me enough control authority to maintain attitude control during re-entry.
How do I solve this? What's the way of using fins / wings / airbrakes on a rocket to be able to control it during re-entry? Note that I while I consider myself proficient with rockets and engines in this game, I have only a very foggy idea how any of the aerodynamics works here (both in terms of the physics and the procedural parts).
Well, no. It's actually mass divided by cross sectional that matters, not just mass. Anyway my question wasn't about the underlying physics, but how I can engineer a solution.
I solved my problem by giving my rocket a flared bottom to increase drag. That slightly increases drag while launching too, but the speeds are so much lower on take off than re-entry that this doesn't matter in practice. In short, as you scale up rockets, you should scale up their radius more than their height if you want to keep aerodynamic properties the same.
As for stability, I found that the most reliable way is to add small stubby wings near the nose cone. No need for control surfaces, and by making each winglet symmetric (along a horizontal axis) they don't interfere much with launch either. In essence, they provide a restoring torque should the rocket stop pointing exactly retrograde. Although if you ever get more than a few degrees out of alignment nothing will stop you from entering a spin and ending up pointing prograde.
So, yeah, I solved my own problem, but hopefully this might be of use for someone else later.