"It is not a matter of enjoying or not. My chambers are mine and therefore where I go to." He understands how that sounds. It's all his; he doesn't go anywhere but his separate chambers if he means to take in anything alone.