Answer by Abhishek Jain:
WhatsApp uses XMPP (eXtensible Messaging and Presence Protocol) to handle the message delivery system.
XMPP is mostly like HTTP where the client opens the socket with the XMPP server and keeps it open as long as the client is logged in. It's not like the regular REST API where the client opens the socket send/receive the data and close the socket. The socket is opened as long as you are signed in. In case of WhatsApp that's eternity (not really, WhatsApp reconnects automatically if the connection terminates)
XMPP protocol has been used in various chat applications such as Google Talk, Facebook messenger etc. I have been working on XMPP since last year and that's when I got to know what this protocol could bring. It's not limited to chat apps though. We're developing an application of our own which is not a chat app.
As far as actual technology goes, WhatsApp uses heavily customized version of Smack library on Android to build their client and uses customized eJabberd Server to handle the XMPP traffic. They might have different backend solution for handling the data though which might be on one of the cloud storage/computing network (I think it's heroku, no real idea though).
On iOS and other platforms, I suppose they might have developed their own libraries. Developing own libraries is not a lot of work, especially when you have customized needs and have a team of developers. I have used one of the libraries available for Windows Phone and heavily customized them to work for us. I made some improvement on the library, but due to time shortage I couldn't submit them to the original repo (Documentation is really tough).
Anyways, if you are interested in learning the tech, you can read Oriely's "XMPP: The Definitive Guide" and can visit.