There have been no studies to suggest that any of the chemicals contained within any variety of psilocybin mushroom can cause any type of physical damage in the human's brain. what you are thinking of is not a hole in your brain but the use of indorvines to produce the high the psilocybe produces so when you come off of the high your brain is running a little low on some which produces the sight of a hole in your brain if you get a cat scan.
The answer to this question is not a simple yes or no, even for the average person with no emotional or psychological problems or any family history of mental illness the answer is surely an un-equivocal no.