IvLabs PC and GPUs Usage Guidelines
_____ _ _
|_ _| | | | |
| |_ _| | __ _| |__ ___
| \ \ / / | / _' | '_ \/ __|
_| |\ V /| |___| (_| | |_) \__ \
\___/\_/ \_____/\__,_|_.__/|___/
Welcome to the IvLabs PC. If you are on this page, chances are you have just been given access to the PC.
Follow the rules for using this PC at Lab New PC instructions.
Before using any GPU, you must book it in Lab new PC usage record.
If you don't understand how to use things, contact the admin on the Discord channel #pc.
GPUs and other resources
- Keep usage of resources under control. For RAM and cores usage :
htop and for GPU usage : nvidia-smi or gpustat
- After running your code (in case it is not stable), make sure that resources being used are under control (using above commands).
- To use a GPU, you must first book it at this google sheet provided, you will find some rules on how to do it in the spreadsheet. If you have any questions, please ask!
- In case you have to login using the monitor in lab, then remember to log out before leaving. This is necessary since logging in at the monitor will take up GPU memory which is not released even when the monitor is switched off.
- Use at most one GPU at a time. In case you need more, use the Discord #gpus channel to ask for permission and coordinate with the other users.
- By default, TensorFlow uses all GPUs available. Set up your experiments to use the correct GPU.
- In PyTorch, use
torch.cuda.set_device(id) to set the proper GPU. Replace id with 0 for GPU0 and so on.
- After completing your task make sure you release the GPU memory (if not done automatically). Use
nvidia-smi to look at running processes and kill them using kill <pid>.
- To determine the owner of process
<pid>, use ps -o user= -p <pid> to find the username of the owner. If GPU is not booked, use the #gpus channel on Discord to coordinate with the user or request the admin to kill the process. If unbooked GPU user does not respond quickly, then admin must kill the process.
How to setup your enviroment
- You must use conda to create an enviroment if you have custom dependencies for your project. DO NOT install conflicting packages in the base enviroment.
- Useful conda commands are listed below
Writing code over ssh
Using GUI applications over ssh with X11 forwarding
- The requirements for the server side are already setup by the admin.
- On your own Ubuntu machine, while logging in, use
ssh -X username@ip_address.
- To make X11 forwarding default in your machine, add
ForwardX11 yes in the ~/.ssh/config file in your local machine.
- Then, any GUI application can be started on the remote server (using the usual commands) and the display will be shown on your local machine.
Using ROS and ROS based applications
- ROS has already been installed system-wide. However, ROS environment variables will not be sourced by default in your bash terminals (since
source /opt/ros/kinetic/setup.bash hasn't been added to the .bashrc file of the root user).
- Thus, to use ROS, you must do
source /opt/ros/kinetic/setup.bash in each terminal that you wish to use for ROS related commands. To make it default to your user (NOT the root user), you may add this command to your user's .bashrc file (this file is present in your home folder. If you're unsure, you can contact the admin on #gpus channel on Discord).
- Note that there is no need to use the monitor in the lab for ROS based applications. You can do your work through ssh, and follow the X11 forwarding instructions above to get GUI applications working through ssh.
- WARNING: If you add the source command to your
.bashrc file, then you risk conflicts between your dependencies and dependencies in your conda or other environments (if any). So, if you face conflicts in (say) OpenCV (this is the most common conflict), then be sure to check your .bashrc file.
- WARNING: Your simulations may lag when any training is going on (since CPU is also used during training). So, check if training is going on using
watch nvidia-smi and coordinate with other users on the #gpus channel on Discord.
Instruction to current and future Admins: Never put the source command in the root user's .bashrc file. It will create conflicts for other users' environments who don't use ROS.
How to launch experiments
- Use
tmux to keep the session active even when you are disconnected from the server.
If you haven't used it before, read this tutorial to learn all the basics and this cheatsheet for some advanced functionalities.
- Run the program or start the Jupyter Notebook / Lab within the tmux session.
- Use the
watch nvidia-smi or gpustat --watch command to monitor GPU usage.
- Use
htop to keep resources under control.
- If other users are already using for training or running simulations or other tasks, then coordinate with them on the #pc channel on Discord.