I found an insect inside of a plum... after I already ate three of them, should I be worried?

Nope, consider it extra protein if they were there and you didn't notice as you did with the fourth.