瞧瞧我对服务器干了些什么！

Git与Github

  • 在Linux的Ubuntu发行版上一般都会默认安装了Git，所以不需要自己手动安装，拿来即用即可。
1
2
git config --global user.name "SSH keys Name"
git config --global user.email "SSH keys Email"
1
ssh-keygen -t rsa -C "Email of Github Account"
1
2
3
4
5
6
7
8
9
10
(base) houjinliang@3080server:~/userdoc/d2cv$ git config --global user.name 'hjl_3080server'
(base) houjinliang@3080server:~/userdoc/d2cv$ git config --global user.email 'cosmicdustycn@outlook.com'
(base) houjinliang@3080server:~/userdoc/d2cv$ ssh-keygen -t rsa -C "cosmicdustycn@outlook.com"
Generating public/private rsa key pair.
Enter file in which to save the key (/mnt/houjinliang/.ssh/id_rsa): 
Created directory '/mnt/houjinliang/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /mnt/houjinliang/.ssh/id_rsa.
Your public key has been saved in /mnt/houjinliang/.ssh/id_rsa.pub.
  • 不需要担心Git的用户配置会对本服务器上的其他用户会产生影响。ssh-keygen生产的的用户密钥会保存在个人账号的目录下。
1
2
3
4
5
6
7
8
9
(dlpy310pth113) houjinliang@3080server:~/.ssh$ pwd
/mnt/houjinliang/.ssh
(dlpy310pth113) houjinliang@3080server:~/.ssh$ ll
总用量 20
drwx------  2 houjinliang houjinliang 4096 11月  1 10:19 ./
drwxr-xr-x 12 houjinliang houjinliang 4096 11月  1 10:17 ../
-rw-------  1 houjinliang houjinliang 1675 11月  1 10:17 id_rsa
-rw-r--r--  1 houjinliang houjinliang  407 11月  1 10:17 id_rsa.pub
-rw-r--r--  1 houjinliang houjinliang  444 11月  1 10:19 known_hosts
  • 复制id_rsa.pub文件下的内容，到Github的Setting中设置SSH Keys。如下。

image-20231101105539214

image-20231102171332923

1
2
3
4
5
(base) houjinliang@3080server:~/userdoc$ git clone git@github.com:murphyhoucn/DeepLearningforCV.git
(base) houjinliang@3080server:~/userdoc/DeepLearningforCV$ git status
(base) houjinliang@3080server:~/userdoc/DeepLearningforCV$ git add .
(base) houjinliang@3080server:~/userdoc/DeepLearningforCV$ git commit -m "add new file"
(base) houjinliang@3080server:~/userdoc/DeepLearningforCV$ git push

查看GPU占用情况

nvidia-smi

image-20231101110320173

gpustat

GitHub - wookayin/gpustat: 📊 A simple command-line utility for querying and monitoring GPU status

1
2
(dlpy310pth113) houjinliang@3080server:~/userdoc$ pip install gpustat
(dlpy310pth113) houjinliang@3080server:~/userdoc$ gpustat

image-20231101110430041

nvitop

GitHub - XuehaiPan/nvitop: An interactive NVIDIA-GPU process viewer and beyond, the one-stop solution for GPU process management.

nvitop: 史上最强GPU性能实时监测工具 - 知乎 (zhihu.com)

1
2
3
4
5
6
(dlpy310pth113) houjinliang@3080server:~$ pip install nvitop
Requirement already satisfied: nvitop in ./miniconda3/envs/dlpy310pth113/lib/python3.10/site-packages (1.3.0)
Requirement already satisfied: nvidia-ml-py<12.536.0a0,>=11.450.51 in ./miniconda3/envs/dlpy310pth113/lib/python3.10/site-packages (from nvitop) (12.535.108)
Requirement already satisfied: psutil>=5.6.6 in ./miniconda3/envs/dlpy310pth113/lib/python3.10/site-packages (from nvitop) (5.9.5)
Requirement already satisfied: cachetools>=1.0.1 in ./miniconda3/envs/dlpy310pth113/lib/python3.10/site-packages (from nvitop) (5.3.1)
Requirement already satisfied: termcolor>=1.0.0 in ./miniconda3/envs/dlpy310pth113/lib/python3.10/site-packages (from nvitop) (2.3.0)

image-20231101110731169

Clash for Linux

Ubuntu配置 命令行Clash 教程 - 知乎 (zhihu.com)

终端使用代理加速的正确方式（Clash） | Ln’s Blog (weilining.github.io)

2024.01.10

1
2
3
4
gunzip clash-linux-amd64-v1.18.0.gz
mv clash-linux-amd64-v1.18.0 clash
chmod u+x clash
./clash
1
在 ~/.config/clash/config.yaml 写入订阅的内容

image-20240110153702123

1
2
3
4
5
6
7
8
9
10
11
`~/.bashrc`

function proxy() {
export http_proxy=http://127.0.0.1:7890
export https_proxy=$http_proxy
echo -e "proxy on!"
}
function unproxy(){
unset http_proxy https_proxy
echo -e "proxy off"
}
1
2
3
4
5
(base) houjinliang@3080server:~/userdoc$ source ~/.bashrc
(base) houjinliang@3080server:~/userdoc$ proxy
proxy on!
(base) houjinliang@3080server:~/userdoc$ unproxy
proxy off
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
(base) houjinliang@3080server:~/userdoc$ wget www.zhihu.com
URL transformed to HTTPS due to an HSTS policy
--2024-01-10 15:33:59--  https://www.zhihu.com/
正在连接 127.0.0.1:7890... 已连接。
已发出 Proxy 请求，正在等待回应... 302 Found
位置：//www.zhihu.com/signin?next=%2F [跟随至新的 URL]
URL transformed to HTTPS due to an HSTS policy
--2024-01-10 15:33:59--  https://www.zhihu.com/signin?next=%2F
再次使用存在的到 www.zhihu.com:443 的连接。
已发出 Proxy 请求，正在等待回应... 200 OK
长度： 39879 (39K) [text/html]
正在保存至: “index.html”

index.html                                           100%[===================================================================================================================>]  38.94K  --.-KB/s    用时 0.04s



2024-01-10 15:33:59 (944 KB/s) - 已保存 “index.html” [39879/39879])

(base) houjinliang@3080server:~/userdoc$ wget www.google.com
--2024-01-10 15:34:14--  http://www.google.com/
正在连接 127.0.0.1:7890... 已连接。
已发出 Proxy 请求，正在等待回应... 200 OK
长度： 未指定 [text/html]
正在保存至: “index.html.1”

index.html.1                                             [ <=>                                                                                                                ]  18.72K  --.-KB/s    用时 0.07s

2024-01-10 15:34:16 (257 KB/s) - “index.html.1” 已保存 [19169]

MMDetection

版本选择参考镜像：

open-mmlab/mmdetection3d/mmdetection3d-1.1: mmdetection3d-1.1版本 - CG (codewithgpu.com)

image-20240105160510106

CUDA 11.3.1 & CUDNN 8.9.5

  之前安装的是CUDA 11.6，后面感觉这个版本有点儿高了，在看到一些实例之后，决定退回到CUDA 11.3版本。首先第一步是要卸载掉CUDA 11.6，在搜索了之后，发现并没有找到能用的方法，于是决定直接rm -rf cuda-11.6，这样吧CUDA的文件删掉之后再重装。

CUDA Toolkit 11.3 Update 1 Downloads | NVIDIA Developer

1
2
wget https://developer.download.nvidia.com/compute/cuda/11.3.1/local_installers/cuda_11.3.1_465.19.01_linux.run
sudo sh cuda_11.3.1_465.19.01_linux.run

非root用户安装cuda与cudnn - 知乎 (zhihu.com)

image-20240105154958922
image-20240105154857088
image-20240105155054178
image-20240105154820598
image-20240105154838934
image-20240105154907733
image-20240105155142558
1
2
3
4
5
6
7
8
9
(base) houjinliang@3080server:~/userdoc/cuda_and_cudnn$ sh ./cuda_11.3.1_465.19.01_linux.run

= Summary =
Driver: Not Selected Toolkit: Installed in /mnt/houjinliang/cuda-11.3/ Samples: Not Selected
Please make sure that
PATH includes /mnt/houjinliang/cuda-11.3/bin
LD_LIBRARY_PATH includes /mnt/houjinliang/cuda-11.3/lib64, or, add /mnt/houjinliang/cuda-11.3/lib64 to /etc/ld.so.conf and run ldconfig as root
To uninstall the CUDA Toolkit, run cuda-uninstaller in /mnt/houjinliang/cuda-11.3/bin ***WARNING: Incomplete installation! This installation did not install the CUDA Driver. A driver of version at least 465.00 is required for CUDA 11.3 functionality to work. To install the driver using this installer, run the following command, replacing <CudaInstaller> with the name of this run file: sudo <CudaInstaller>.run --silent --driver
Logfile is /tmp/cuda-installer.log
1
2
3
4
5
6
7
8
9
vim ~/.bashrc

```
# cuda environment variables
# murpy insert
export CUDA_HOME=$CUDA_HOME:/mnt/houjinliang/cuda-11.3
export PATH=$PATH:/mnt/houjinliang/cuda-11.3/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/mnt/houjinliang/cuda-11.3/lib64
```
1
2
3
4
5
6
7
8
(base) houjinliang@3080server:~$ source ~/.bashrc

(base) houjinliang@3080server:~$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Mon_May__3_19:15:13_PDT_2021
Cuda compilation tools, release 11.3, V11.3.109
Build cuda_11.3.r11.3/compiler.29920130_0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
(base) houjinliang@3080server:~/userdoc/cuda_and_cudnn$ tar xvJf cudnn-linux-x86_64-8.9.5.29_cuda11-archive.tar.xz


(py38mmdetection) houjinliang@3080server:~/userdoc/cuda_and_cudnn/cudnn-linux-x86_64-8.9.5.29_cuda11-archive$ ll
总用量 48
drwxr-xr-x 4 houjinliang houjinliang  4096 8月   3  2022 ./
drwxrwxr-x 3 houjinliang houjinliang  4096 1月   5 16:32 ../
drwxr-xr-x 2 houjinliang houjinliang  4096 8月   3  2022 include/
drwxr-xr-x 2 houjinliang houjinliang  4096 8月   3  2022 lib/
-rw-r--r-- 1 houjinliang houjinliang 28994 8月   3  2022 LICENSE


(py38mmdetection) houjinliang@3080server:~/userdoc/cuda_and_cudnn/cudnn-linux-x86_64-8.9.5.29_cuda11-archive$ cp lib/* ~/cuda-11.3/lib64/
(py38mmdetection) houjinliang@3080server:~/userdoc/cuda_and_cudnn/cudnn-linux-x86_64-8.9.5.29_cuda11-archive$ cp include/* ~/cuda-11.3/include

chmod +x ~/cuda-11.3/include/cudnn.h
chmod +x ~/cuda-11.3/lib64/libcudnn*

(base) houjinliang@3080server:~$ cat ~/cuda-11.3/include/cudnn_version.h | grep CUDNN_MAJOR -A 2
#define CUDNN_MAJOR 8
#define CUDNN_MINOR 9
#define CUDNN_PATCHLEVEL 5
--
#define CUDNN_VERSION (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)

/* cannot use constexpr here since this is a C-only file */

PyTorch 1.11

1
2
3
4
5
6
7
8
9
10
11
(base) houjinliang@3080server:~$ conda create -n py38mmdetection python=3.8 -y
(base) houjinliang@3080server:~$ conda activate py38mmdetection
(py38mmdetection) houjinliang@3080server:~$ conda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3 -c pytorch

(py38mmdetection) houjinliang@3080server:~$ python
Python 3.8.18 (default, Sep 11 2023, 13:40:15)
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print(torch.cuda.is_available())
True

阿里云源

1
pip config set global.index-url https://mirrors.aliyun.com/pypi/simple

mmdet installation

开始你的第一步 — MMDetection 3.3.0 文档

MMYOLO

Overview — MMYOLO 0.6.0 documentation

1
2
3
4
5
6
7
8
9
10
11
(base) houjinliang@3080server:~$ conda create -n py38mmyolo python=3.8

(base) houjinliang@3080server:~$ conda activate py38mmyolo
(py38mmyolo) houjinliang@3080server:~$ pip config list
global.index-url='https://mirrors.aliyun.com/pypi/simple'

(py38mmyolo) houjinliang@3080server:~$ conda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3 -c pytorch

(py38mmyolo) houjinliang@3080server:~$ python -c "import torch; print(torch.__version__); print(torch.cuda.is_available())"
1.11.0
True
1
2
3
4
pip install -U openmim
mim install "mmengine>=0.6.0"
mim install "mmcv>=2.0.0rc4,<2.1.0"
mim install "mmdet>=3.0.0,<4.0.0"
1
mim install "mmyolo"
1
2
3
4
5
6
7
8
9
git clone https://github.com/open-mmlab/mmyolo.git
cd mmyolo
# Install albumentations
pip install -r requirements/albu.txt
# Install MMYOLO
mim install -v -e .
# "-v" means verbose, or more output
# "-e" means installing a project in editable mode,
# thus any local modifications made to the code will take effect without reinstallation.
1
2
3
4
5
6
7
8
9
10
(base) houjinliang@3080server:~/userdoc/offlinefile$ wget  http://images.cocodataset.org/zips/val2017.zip
--2024-01-10 16:17:46--  http://images.cocodataset.org/zips/val2017.zip
正在解析主机 images.cocodataset.org (images.cocodataset.org)... 3.5.7.141, 52.216.215.25, 52.216.185.83, ...
正在连接 images.cocodataset.org (images.cocodataset.org)|3.5.7.141|:80... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度： 815585330 (778M) [application/zip]
正在保存至: “val2017.zip”

val2017.zip                                          100%[===================================================================================================================>] 777.80M  3.89MB/s    用时 2m 22ss
2024-01-10 16:20:08 (5.48 MB/s) - 已保存 “val2017.zip” [815585330/815585330])

目录占用空间大小查询

  • 查看文件以及文件夹大小
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
(py38mmyolo) houjinliang@3080server:~/userdoc/offlinefile$ ll
总用量 26251480
drwxrwxr-x 6 houjinliang houjinliang        4096 1月  10 21:36 ./
drwxrwxr-x 9 houjinliang houjinliang        4096 1月  10 15:59 ../
-rw-rw-r-- 1 houjinliang houjinliang     3996930 1月  10 14:43 clash-linux-amd64-v1.18.0.gz
drwxr-xr-x 5 houjinliang houjinliang        4096 8月  26  2022 coco/
-rw-rw-r-- 1 houjinliang houjinliang     6983030 1月  10 17:00 coco128.zip
-rw-rw-r-- 1 houjinliang houjinliang    48639045 1月  10 16:21 coco2017labels.zip
-rw-rw-r-- 1 houjinliang houjinliang     4372979 1月  10 14:48 curl-8.5.0.tar.gz
-rw-rw-r-- 1 houjinliang houjinliang    12353723 1月   5 16:32 pandas-2.0.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
drwxrwxr-x 2 houjinliang houjinliang     1429504 8月  31  2017 test2017/
-rw-rw-r-- 1 houjinliang houjinliang  6646970404 1月  10 17:47 test2017.zip
drwxrwxr-x 2 houjinliang houjinliang     4112384 8月  31  2017 train2017/
-rw-rw-r-- 1 houjinliang houjinliang 19336861798 1月  10 21:35 train2017.zip
drwxrwxr-x 2 houjinliang houjinliang      167936 8月  31  2017 val2017/
-rw-rw-r-- 1 houjinliang houjinliang   815585330 7月  11  2018 val2017.zip

(py38mmyolo) houjinliang@3080server:~/userdoc/offlinefile$ ll -hl
总用量 26G
drwxrwxr-x 6 houjinliang houjinliang 4.0K 1月  10 21:36 ./
drwxrwxr-x 9 houjinliang houjinliang 4.0K 1月  10 15:59 ../
-rw-rw-r-- 1 houjinliang houjinliang 3.9M 1月  10 14:43 clash-linux-amd64-v1.18.0.gz
drwxr-xr-x 5 houjinliang houjinliang 4.0K 8月  26  2022 coco/
-rw-rw-r-- 1 houjinliang houjinliang 6.7M 1月  10 17:00 coco128.zip
-rw-rw-r-- 1 houjinliang houjinliang  47M 1月  10 16:21 coco2017labels.zip
-rw-rw-r-- 1 houjinliang houjinliang 4.2M 1月  10 14:48 curl-8.5.0.tar.gz
-rw-rw-r-- 1 houjinliang houjinliang  12M 1月   5 16:32 pandas-2.0.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
drwxrwxr-x 2 houjinliang houjinliang 1.4M 8月  31  2017 test2017/
-rw-rw-r-- 1 houjinliang houjinliang 6.2G 1月  10 17:47 test2017.zip
drwxrwxr-x 2 houjinliang houjinliang 4.0M 8月  31  2017 train2017/
-rw-rw-r-- 1 houjinliang houjinliang  19G 1月  10 21:35 train2017.zip
drwxrwxr-x 2 houjinliang houjinliang 164K 8月  31  2017 val2017/
-rw-rw-r-- 1 houjinliang houjinliang 778M 7月  11  2018 val2017.zip
  • 如要查看当前目录已经使用总大小及当前目录下一级文件或文件夹各自使用的总空间大小
1
2
3
4
5
6
7
8
9
10
11
12
(py38mmyolo) houjinliang@3080server:~$ du -h --max-depth=1
6.5M    ./.config
8.0K    ./.conda
1.1G    ./.vscode-server
12G     ./cuda-11.3
86G     ./userdoc
8.0K    ./.gnupg
16K     ./.ssh
8.0K    ./.nv
2.7G    ./.cache
24G     ./miniconda3
125G    .
高效工具
#Linux #Server #GPU #DeepLearning #CV
瞧瞧我对服务器干了些什么！
https://cosmicdusty.cc/post/Tools/WorkingWithGPUServer/
作者
Murphy
发布于
2023年11月1日
许可协议