I think it depends on how much you pissed off the tree. Really bad stuff may not stay with you for awhile, but sometimes that's all it takes to mess with you. Goofy annoying stuff may stick with you longer, but at that point you can still mostly go through life, it's just... humiliating.

Piss off the tree enough and you get a long-term bad-news done to you. Like dying, not coming back for awhile, and losing memories on top of that.