My server averages 20-40 players and it's run on a mid range VPS with only 3 plugins. My core plugin, world edit, and worlds. When I Iook at the timings my core plugin doesn't seem to be much of an issue except PlayerJoinEvent and PlayerDeathEvent are quite heavy, over 200%... Near the top there's lots of things I don't really understand but they don't look so good. I'm not sure how to fix this. https://timings.pmmp.io/?id=104890
From my experience, sometimes PlayerJoinEvent would go to ~150% with one player, and the overall plugin timing was around ~0.5%. Event timings are weird and I don't get them at all lol... Since your Core timings are overall fine, I would say there isn't much to worry about. (now unban me on Vasar!)