Last active
July 24, 2025 22:49
-
-
Save zentavr/ff15ac51eda5bec1ae1b2f52fbf9c01a to your computer and use it in GitHub Desktop.
Revisions
-
zentavr revised this gist
Dec 28, 2021 . 1 changed file with 16 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -14,6 +14,9 @@ cat /sys/module/vhost/parameters/max_mem_regions ## Open files issue on PVE node Problem description is [here](https://forum.proxmox.com/threads/open-files-issue-on-pve-node.69783/). The situation is very critical when you use clustered file storage (Gluster/CEPH) and have dozens of attached disk drives inside the VM. Inside the VMs you can notice the strange locks of the block devices, high CPU load, etc. Check open file limits: ```shell @@ -107,3 +110,16 @@ Fix: ``` 7. By design, you are not allowed to restart **pve-guests.service** manually. So you might need to reboot your hypervisor machine. You can also change limits on already running process. ```shell #!/usr/bin/env bash for PID in $(ps aux | grep /usr/bin/kvm | grep -v grep | awk '{ print $2 }'); do SOFT_LIMIT="1048576" HARD_LIMIT="2097152" echo "Changing the limits for PID ${PID}" prlimit --nofile=${SOFT_LIMIT}:${HARD_LIMIT} --pid ${PID} done ``` -
Dec 28, 2021 .There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,109 @@ ## Cannot hotplug too much RAM The problem is described [here](https://forum.proxmox.com/threads/hotplug-memory-limits-total-memory-to-44gb.48781/). The quick fix: ```shell script echo "options vhost max_mem_regions=512" > /etc/modprobe.d/vhost.conf rmmod vhost_net rmmod vhost modprobe vhost_net cat /sys/module/vhost/parameters/max_mem_regions # 512 ``` ## Open files issue on PVE node Problem description is [here](https://forum.proxmox.com/threads/open-files-issue-on-pve-node.69783/). Check open file limits: ```shell # find open files limit per process root@hv05-htz-nbg1:~# ulimit -n 1024 # count all opened files by all processes: root@hv05-htz-nbg1:~# cat /proc/sys/fs/file-nr 12192 0 9223372036854775807 # get maximum allowed number of open files root@hv05-htz-nbg1:~# cat /proc/sys/fs/file-max 9223372036854775807 ``` How to check the limits per `kvm` process: ```shell #!/usr/bin/env bash for PID in $(ps aux | grep /usr/bin/kvm | grep -v grep | awk '{ print $2 }'); do SOFT_LIMIT=$(cat /proc/${PID}/limits 2>/dev/null | grep "Max open files" | awk '{ print $4 }') HARD_LIMIT=$(cat /proc/${PID}/limits 2>/dev/null | grep "Max open files" | awk '{ print $5 }') echo "PID ${PID} opened files: $(ls -1 /proc/${PID}/fd 2>/dev/null | wc -l)/${SOFT_LIMIT}/${HARD_LIMIT}" done ``` Fix: 1. **/etc/sysctl.d/90-rs-proxmox.conf**: ``` # Default: 1048576 fs.nr_open = 2097152 # Default: 8388608 fs.inotify.max_queued_events = 8388608 # Default: 65536 fs.inotify.max_user_instances = 1048576 # Default: 4194304 fs.inotify.max_user_watches = 4194304 # Default: 262144 vm.max_map_count = 262144 ``` Keep in mind that `fs.nr_open` should be greater-or-equal to the value of `nofile` you are going to set via limits.conf. If not - you'll have an error like: ``` Dec 23 03:10:01 hv04-htz-nbg1 CRON[94421]: pam_limits(cron:session): Could not set limit for 'nofile' to soft=1048576, hard=2097152: Operation not permitted; uid=0,euid=0 Dec 23 03:17:01 hv04-htz-nbg1 CRON[111402]: pam_limits(cron:session): Could not set limit for 'nofile' to soft=1048576, hard=2097152: Operation not permitted; uid=0,euid=0 Dec 23 03:18:35 hv04-htz-nbg1 sshd[115352]: pam_limits(sshd:session): Could not set limit for 'nofile' to soft=1048576, hard=2097152: Operation not permitted; uid=0,euid=0 Dec 23 03:18:35 hv04-htz-nbg1 sshd[115359]: pam_limits(sshd:session): Could not set limit for 'nofile' to soft=1048576, hard=2097152: Operation not permitted; uid=0,euid=0 ``` 2. **/etc/security/limits.d/90-rs-proxmox.conf**: ``` * soft nofile 1048576 * hard nofile 2097152 root soft nofile 1048576 root hard nofile 2097152 * soft memlock 1048576 * hard memlock 2097152 ``` 3. `mkdir -p /etc/systemd/system/pvedaemon.service.d && touch /etc/systemd/system/pvedaemon.service.d/limits.conf` Add the content of **/etc/systemd/system/pvedaemon.service.d/limits.conf**: ```ini [Service] LimitNOFILE=infinity LimitMEMLOCK=infinity LimitNPROC=infinity TasksMax=infinity ``` 4. `mkdir -p /etc/systemd/system/pve-guests.service.d && touch /etc/systemd/system/pve-guests.service.d/limits.conf` Add the content of **/etc/systemd/system/pve-guests.service.d/limits.conf**: ```ini [Service] LimitNOFILE=infinity LimitMEMLOCK=infinity LimitNPROC=infinity TasksMax=infinity ``` 5. `mkdir -p /etc/systemd/system/pve-ha-lrm.service.d && touch /etc/systemd/system/pve-ha-lrm.service.d/limits.conf` Add the content of **/etc/systemd/system/pve-ha-lrm.service.d/limits.conf**: ```ini [Service] LimitNOFILE=infinity LimitMEMLOCK=infinity LimitNPROC=infinity TasksMax=infinity ``` 6. Restart processes: ``` systemctl daemon-reload systemctl restart pvedaemon.service pve-ha-lrm.service ``` 7. By design, you are not allowed to restart **pve-guests.service** manually. So you might need to reboot your hypervisor machine.