SOMEONE WAS EATING MY RAM MEMORY
Today we received an alert on memory utilization alert for couple of VMs where our Kubernetes applications are deployed. We first suspected over utilization of memory by services deployed on K8S. We quickly ran OS commands like top, pidstat to find memory used by our applications. Overall memory usage was hardly crossing 30%. So, we suspected something other than applications like Kernel was consuming memory. Luckily, quick google search gave answer to our problem. The issue was caused by failure to unset Virtual Machine memory ballooning. We ran vmware-toolbox-cmd stat balloon to confirm it and issue was reported to concerned team. Below are some references which helped us in finding out problem: https://unix.stackexchange.com/questions/259659/high-memory-usage-but-no-process-is-using-it https://communities.vmware.com/t5/Virtual-Machine-Guest-OS-and-VM/Ballooning-memory-stuck-in-quot-is-resetting-quot/td-p/490441