restrict access on some activity record

Hello 

I already brows the forum and people said it s impossible to restrict acces to activity but i Ask

for now people have read acces on activity at Organization level

i have activity created from outlook from different BU that are on the same level on tree Bu

but i Want people have access to all Activity for all BU unless One. do you have idea ? 

If I do it with owner , is there a risk of bad synchro with outlook agenda ?