How can we prevent users from removing items from queues?

Dynamics CRM 2016 has a "Remove" button for queue items. I am looking at the security role and does not seem to be a security privilege for this? I am trying to prevent users on certain roles from being able to remove items from queues. 

However, removing the "Delete" permission on "Queue" and "Activity" does not achieve this?

Is anyone aware of a way?

I can't remove the button via ribbon workbench as queue items could be deleted via other ways. Like for instance the "Remove item from queue" tick box on the "Pick" button.