If you are running a GPU job and you want to know how your application is using resources., you can make use of the tool nvtop.
This source of the tool that can be found here: https://github.com/Syllo/nvtop
- submit your GPU job
- login to the node where your job is running by using ssh
- start monitoring by enter command: nvtop at the prompt.
Below you see a example output generated by the tool.