As you have mentioned that for Admin, the slowness is not occuring, then mostly its access validation issue. For other users, it might be possible that you are ggetting 100's of unecessary records and then ACL or your custom security logic or both are running on each one of them to filter out. Its better, you try to find out a way in which you can reduce this initial load itself.