In my 1946-54 service manual, it says "Caution, do not use knock-out type pullers or strike end of axle shaft to loosen hub. This may damage the rear wheel bearings, thrust block and opposite brake support plate"
Now, I've used a slide hammer to remove my rear axles, and I've also been guilty of hitting the hub puller center shaft to pop loose a drum. So far, no apparent damage to the other parts though. Maybe I was just lucky?
Edit: Easiest rear drums I ever pulled were on a buddy's P15. The castle nuts had been tightened in the manner you would tighten a front wheel bearing...snug it up, back off to the next available cotter key space. Those drums came off by hand! As a tribute to the great materials Plymouth used, there was no visible wear to the axle, drum, or key, and the car had been driven a lot, and on the freeways frequently. After some brake work, we tightened them to about 140 ft lbs with a torque wrench.