I would like to configure ssh to use one local authorized_keys file that is populated with the rsa public keys of all my users. I tried setting the parameter in sshd_config to a central key file in /etc/ssh/authorized_keys and added some public keys of some test users.

user1 can ssh user1@foo.com and not be challenged for a password.
user2 can ssh user2@foo.com and not be challenged for a password. Unfortunately user2 can also ssh as user1 to server foo.com and not be challenged for a password AND is logged in as user1. Thus user2 could login and do bad things and it looks like user1 did it.

Is there any way to have one authorized_keys file that users authenticate with but only allows users to login as themselves?
Thanx.