Can only a 'super user' log in as other users?
A second user is also in the administrator user group but cannot see the 'login as user' button. Is this correct behaviour, a bug, or a permission I can edit somewhere?
if you look in dashboard->system & settings->Permissions & Access->User Permissions
you should see a bunch of user permissions including 'Sign in as User', normally the administrators groups will be assigned to this but I always have advanced permissions turned on so I'm not sure if you must have Advanced permissions on or not.
This particular side only has basic permissions enabled so hadn't event thought to dig through the permissions settings, but the option was right there where you said it would be. Event without advanced permissions enabled.