My device is Atrix 2 with resolution of 960x540, but YouTube app only gives me options for 144p, 240p and 360p, which are not really "quality". How can I watch 720p videos on YouTube on my device?
Your screen resolution being 960*540 is not meant to play 720p videos because the resolution is lower. That's why the Youtube app detects the device resolution and does not show higher video qualities. The maximum video quality that the Youtube app will show for your device is 480p.
720p = 1280*720
However, you could use YouTube Vanced. It is a modded version that offers more features than the official Youtube app such as:
Override Maximal Resolution
Preferred Speed and Resolution
For rooted phones, L Tweaks, an Xposed module, allows to set video qualities up to 4320p.