How to make item take damage

Let's say I have this pickaxe that shows how many blocks you mined in the lore.

Now in order to update the block mined count, I would have to increment the block mined count and then
$item->setItemInHand to update the lore.

The problem here is the item's durability gets reset every time I $item->setItemInHand to update the lore.

How would I go about making the item get damaged even when I $item->setItemInHand

I've tried doing $item->setDamage($item->getDamage() - 1) but that would make unbreaking enchant useless

 

Can’t you get the item then change the lore?

 

That’s what I’m doing right now but block break event gets the damage before breaking the block and not the damage after breaking the block so the durability remains the same