Spending Time with Coworkers Outside of Work Hours

I have been spoken to several times regarding certain coworkers I spend my time with outside of normal working hours. I am told that I need to invite my whole group of 11 people every time I want to spend any time with anyone in my group. I believe that I should be free to spend my time with anyone I like outside of work and especially because I am spending my own money to do this (ie. I do not expense any of these outings). None of this affects my performance during work hours as I am considered a high performer as work. With year end bonsuses coming up, I am afraid that my employer will use this "issue" to reduce my year end figures and I believe that this unfair and borderline harrassment. Could anyone help me with some thoughts/advice? Thank you.