You are free to set some rules, to decide where the boundaries are, and what game you are playing. Actually, it is your responsibility. You should do that.

And once that is done, the next step is for you to figure out who is in and who is out, and for others to figure whether they are in or out.

You can’t be everything to everybody.

Take ownership of the process.