It's possible that your server is overloaded with far too many messages in queue and no optimization to handle it.
I think it's normal for a few messages to be delayed, even for this long, due to temporary server outages or simply undeliverables due to full accounts, etc. However, if a very large number of your messages are taking this long, even when sent to an account that is known to receive email very reliably, then there is likely something wrong with the server config.
Your best bet is to bring this to the attention of your host and see what they say. If you are running qMail (Plesk, etc) or Exim (cPanel, etc) then you might want to check out my optimizations, posted in the optimizations forum, and forward the information to your host.
Regards!