Unexplained memory leak? Free RAM goes down over time

hopeful

Hi LowEndHelpDesk,

I have a VPS with 2GB of memory. I don't use it a lot, leaving it mostly idle except the IPv6 BGP session (over 6in4) I have with he.net and NetAssist. I don't load the routing table; I just route everything to he.net.

However, I noticed that the memory usage increases slowly over time. I need to hard-reset the VPS every few weeks to restore it. This behaviour started roughly since 2020 started but I haven't made any significant changes to the configuration

Thanks to this beautiful graph from hetrixtools hopefully it explains better:

Memory usage is quite high:

[me@lax2 ~]$ free -h
              total        used        free      shared  buff/cache   available
Mem:          2.1Gi       1.6Gi        95Mi       1.0Mi       366Mi       289Mi
Swap:         303Mi        20Mi       283Mi

You can see that there's actually no application actively consuming memory:

[me@lax2 ~]$ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.3 109744  8264 ?        Ss   May13   5:58 /sbin/init
root           2  0.0  0.0      0     0 ?        S    May13   0:00 [kthreadd]
root           3  0.0  0.0      0     0 ?        I<   May13   0:00 [rcu_gp]
root           4  0.0  0.0      0     0 ?        I<   May13   0:00 [rcu_par_gp]
root           6  0.0  0.0      0     0 ?        I<   May13   0:00 [kworker/0:0H-kblockd]
root           8  0.0  0.0      0     0 ?        I<   May13   0:00 [mm_percpu_wq]
root           9  0.0  0.0      0     0 ?        S    May13   1:39 [ksoftirqd/0]
root          10  0.0  0.0      0     0 ?        S    May13   0:00 [rcuc/0]
root          11  0.0  0.0      0     0 ?        I    May13  11:11 [rcu_preempt]
root          12  0.0  0.0      0     0 ?        S    May13   0:00 [rcub/0]
root          13  0.0  0.0      0     0 ?        S    May13   0:04 [migration/0]
root          14  0.0  0.0      0     0 ?        S    May13   0:00 [idle_inject/0]
root          16  0.0  0.0      0     0 ?        S    May13   0:00 [cpuhp/0]
root          17  0.0  0.0      0     0 ?        S    May13   0:00 [kdevtmpfs]
root          18  0.0  0.0      0     0 ?        I<   May13   0:00 [netns]
root          19  0.0  0.0      0     0 ?        S    May13   0:00 [rcu_tasks_kthre]
root          20  0.0  0.0      0     0 ?        S    May13   0:03 [kauditd]
root          21  0.0  0.0      0     0 ?        S    May13   0:00 [khungtaskd]
root          22  0.0  0.0      0     0 ?        S    May13   0:00 [oom_reaper]
root          23  0.0  0.0      0     0 ?        I<   May13   0:00 [writeback]
root          24  0.0  0.0      0     0 ?        S    May13   0:01 [kcompactd0]
root          25  0.0  0.0      0     0 ?        SN   May13   0:00 [ksmd]
root          26  0.0  0.0      0     0 ?        SN   May13   0:00 [khugepaged]
root         114  0.0  0.0      0     0 ?        I<   May13   0:00 [kintegrityd]
root         115  0.0  0.0      0     0 ?        I<   May13   0:00 [kblockd]
root         116  0.0  0.0      0     0 ?        I<   May13   0:00 [blkcg_punt_bio]
root         117  0.0  0.0      0     0 ?        I<   May13   0:00 [ata_sff]
root         118  0.0  0.0      0     0 ?        I<   May13   0:00 [edac-poller]
root         119  0.0  0.0      0     0 ?        I<   May13   0:00 [devfreq_wq]
root         120  0.0  0.0      0     0 ?        S    May13   0:00 [watchdogd]
root         121  0.0  0.0      0     0 ?        S    May13   4:59 [kswapd0]
root         124  0.0  0.0      0     0 ?        I<   May13   0:00 [kthrotld]
root         125  0.0  0.0      0     0 ?        I<   May13   0:00 [acpi_thermal_pm]
root         126  0.0  0.0      0     0 ?        I<   May13   0:00 [nvme-wq]
root         127  0.0  0.0      0     0 ?        I<   May13   0:00 [nvme-reset-wq]
root         128  0.0  0.0      0     0 ?        I<   May13   0:00 [nvme-delete-wq]
root         129  0.0  0.0      0     0 ?        I<   May13   0:00 [ipv6_addrconf]
root         140  0.0  0.0      0     0 ?        I<   May13   0:00 [kstrp]
root         146  0.0  0.0      0     0 ?        I<   May13   0:00 [zswap-shrink]
root         147  0.0  0.0      0     0 ?        I<   May13   0:00 [kworker/u3:0]
root         158  0.0  0.0      0     0 ?        I<   May13   0:00 [charger_manager]
root         188  0.0  0.0      0     0 ?        S    May13   0:00 [scsi_eh_0]
root         189  0.0  0.0      0     0 ?        I<   May13   0:00 [scsi_tmf_0]
root         190  0.0  0.0      0     0 ?        S    May13   0:00 [scsi_eh_1]
root         191  0.0  0.0      0     0 ?        I<   May13   0:00 [scsi_tmf_1]
root         195  0.0  0.0      0     0 ?        I<   May13   0:15 [kworker/0:1H-kblockd]
root         206  0.0  0.0      0     0 ?        S    May13   0:10 [jbd2/vda2-8]
root         207  0.0  0.0      0     0 ?        I<   May13   0:00 [ext4-rsv-conver]
root         234  0.0  2.2 172832 47940 ?        Ss   May13   3:28 /usr/lib/systemd/systemd-journald
root         242  0.0  0.0  78076   688 ?        Ss   May13   0:00 /usr/bin/lvmetad -f
root         247  0.0  0.2  30940  4732 ?        Ss   May13   0:02 /usr/lib/systemd/systemd-udevd
systemd+     250  0.0  0.2  26260  5208 ?        Ss   May13   0:05 /usr/lib/systemd/systemd-networkd
systemd+     276  0.0  0.1  91784  4276 ?        Ssl  May13   0:03 /usr/lib/systemd/systemd-timesyncd
root         289  0.0  0.0   6588  1968 ?        Ss   May13   0:07 /usr/bin/crond -n
dbus         290  0.0  0.1   6780  2772 ?        Ss   May13   3:35 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root         291  0.0  0.2  17504  5660 ?        Ss   May13   1:47 /usr/lib/systemd/systemd-logind
v2ray        294  0.1  0.6 1168500 13268 ?       Ssl  May13  14:52 /usr/bin/v2ray -config /etc/v2ray/0.json
bird         295  0.0  0.0   7156  1268 ?        Ss   May13   2:00 /usr/bin/bird -s /run/bird/bird.ctl
root         296  0.0  0.0   5424  1304 tty1     Ss+  May13   0:00 /sbin/agetty -o -p -- \u --noclear tty1 linux
http         315  0.0  0.5 1255500 12164 ?       Ssl  May13   0:54 /usr/bin/caddy -log stdout -agree -conf /etc/caddy/caddy.conf -root=/usr/share/caddy
root     3555765  0.0  0.0      0     0 ?        I    21:27   0:00 [kworker/0:0-rcu_gp]
root     3566553  0.0  0.0      0     0 ?        I    21:33   0:00 [kworker/u2:1-ext4-rsv-conversion]
root     3567234  0.0  0.0      0     0 ?        I    21:33   0:00 [kworker/0:1-events]
root     3569037  0.0  0.3  10704  7448 ?        Ss   21:34   0:00 sshd: me [priv]
me    3569138  0.0  0.4  18972  9776 ?        Ss   21:34   0:00 /usr/lib/systemd/systemd --user
me    3569140  0.0  0.0 113348  2116 ?        S    21:34   0:00 (sd-pam)
me    3569145  0.0  0.1  10704  4156 ?        S    21:34   0:00 sshd: me@pts/0
me    3569146  0.0  0.1   7488  4000 pts/0    Ss   21:34   0:00 -bash
root     3577610  0.0  0.0      0     0 ?        I    21:38   0:00 [kworker/u2:0-flush-254:0]
root     3584705  0.1  0.0      0     0 ?        I    21:42   0:00 [kworker/0:2-events]
root     3588276  0.0  0.0      0     0 ?        I    21:44   0:00 [kworker/u2:2-events_unbound]
root     3591768  0.0  0.1   9636  3600 ?        S    21:46   0:00 /usr/bin/CROND -n
hetrixt+ 3591769  0.0  0.1   7144  2860 ?        Ss   21:46   0:00 /bin/sh -c bash /etc/hetrixtools/hetrixtools_agent.sh >> /etc/hetrixtools/hetrixtools_cron.log 2>&1
hetrixt+ 3591770  0.3  0.1   7144  3392 ?        S    21:46   0:00 bash /etc/hetrixtools/hetrixtools_agent.sh
hetrixt+ 3593070  0.0  0.0   7144  1816 ?        S    21:46   0:00 bash /etc/hetrixtools/hetrixtools_agent.sh
hetrixt+ 3593071  0.0  0.0   7972  1184 ?        S    21:46   0:00 vmstat 3 2
hetrixt+ 3593072  0.0  0.0   5340   580 ?        S    21:46   0:00 tail -1
me    3593073  0.0  0.1   9500  3596 pts/0    R+   21:46   0:00 ps aux

[me@lax2 ~]$ cat  /proc/meminfo
MemTotal:        2163084 kB
MemFree:          111300 kB
MemAvailable:     302496 kB
Buffers:            5500 kB
Cached:            70908 kB
SwapCached:         1008 kB
Active:            56160 kB
Inactive:          54484 kB
Active(anon):      10912 kB
Inactive(anon):    22148 kB
Active(file):      45248 kB
Inactive(file):    32336 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:        311292 kB
SwapFree:         289952 kB
Dirty:                 8 kB
Writeback:             0 kB
AnonPages:         33544 kB
Mapped:            46160 kB
Shmem:              1624 kB
KReclaimable:     284608 kB
Slab:            1881652 kB
SReclaimable:     284608 kB
SUnreclaim:      1597044 kB
KernelStack:        1520 kB
PageTables:         1616 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1392832 kB
Committed_AS:     476668 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       10888 kB
VmallocChunk:          0 kB
Percpu:            37680 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:     1665012 kB
DirectMap2M:      563200 kB

Any ideas?

Harambe

Looks like it's all being eaten by Slab? Which iirc is kernel-related cache. Try slabtop and maybe that can give you some hints, or you can share the output here for someone much smarter than myself to help with. lol

hopeful

Thanks for the pointers! I never realised slab is a thing. Here's the output from slabtop and I will continue Googling tomorrow:

$slabtop
 Active / Total Objects (% used)    : 14075841 / 14088832 (99.9%)
 Active / Total Slabs (% used)      : 332194 / 332194 (100.0%)
 Active / Total Caches (% used)     : 94 / 129 (72.9%)
 Active / Total Size (% used)       : 1835557.66K / 1838777.06K (99.8%)
 Minimum / Average / Maximum Object : 0.01K / 0.13K / 8.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
4723200 4723200 100%    0.01K   9225      512     36900K kmalloc-8
2095170 2094988  99%    0.19K  99770       21    399080K cred_jar
1584128 1584128 100%    0.03K  12376      128     49504K kmalloc-32
1498624 1498324  99%    0.06K  23416       64     93664K anon_vma_chain
749248 749248 100%    0.09K  16288       46     65152K anon_vma
577376 577376 100%    0.12K  18043       32     72172K kernfs_node_cache
402400 402050  99%    0.25K  25150       16    100600K filp
296960 296960 100%    0.12K   9280       32     37120K pid
251538 251538 100%    0.09K   5989       42     23956K kmalloc-96
214336 214336 100%    0.06K   3349       64     13396K kmem_cache_node
214290 214290 100%    0.44K  23810        9     95240K kmem_cache
214284 211399  98%    0.19K  10204       21     40816K dentry
181760 181760 100%    0.02K    710      256      2840K kmalloc-16
177308 177308 100%    0.81K   9332       19    149312K sock_inode_cache
148304 148256  99%    0.25K   9269       16     37076K skbuff_head_cache
138630 138630 100%    1.06K   9242       15    147872K mm_struct
135808 135808 100%    0.06K   2122       64      8488K kmalloc-64
131911 131331  99%    0.60K  10147       13     81176K inode_cache
 74552  74552 100%    1.00K   9319        8     74552K PING
 71076  71050  99%    1.00K   8885        8     71080K kmalloc-1k
 46968  46968 100%    4.00K   5871        8    187872K kmalloc-4k
 32384  32384 100%    0.50K   4048        8     16192K kmalloc-512
 22197  22197 100%    0.19K   1057       21      4228K kmalloc-192
 19312  19312 100%    0.25K   1207       16      4828K kmalloc-256
 15639   8516  54%    0.10K    401       39      1604K buffer_head
 15246  15105  99%    0.57K   1089       14      8712K radix_tree_node
  9856   9856 100%    0.12K    308       32      1232K kmalloc-128
  9768   9752  99%    2.00K   1221        8     19536K kmalloc-2k
  9435   9435 100%    0.08K    185       51       740K Acpi-State
  4080   4080 100%    0.05K     48       85       192K ftrace_event_field
  2651   2174  82%    1.07K     95       29      3040K ext4_inode_cache
  2142   1836  85%    0.04K     21      102        84K ext4_extent_status
  1725   1556  90%    0.67K     75       23      1200K proc_inode_cache
  1568   1568 100%    0.07K     28       56       112K eventpoll_pwq
  1426   1426 100%    0.09K     31       46       124K trace_event_file
  1280   1280 100%    0.06K     20       64        80K vmap_area
  1120   1120 100%    0.07K     20       56        80K Acpi-Operand
  1116   1116 100%    0.11K     31       36       124K khugepaged_mm_slot
  1034   1016  98%    0.71K     94       11       752K shmem_inode_cache
   969    969 100%    0.08K     19       51        76K task_delay_info
   816    816 100%    0.04K      8      102        32K pde_opener
   714    546  76%    0.09K     17       42        68K kmalloc-rcl-96
   660    660 100%    0.31K     55       12       220K mnt_cache
   640    588  91%    0.06K     10       64        40K kmalloc-rcl-64
   510    510 100%    0.12K     15       34        60K jbd2_journal_head
   504    504 100%    0.19K     24       21        96K skbuff_ext_cache

[me@lax2 ~]$ sudo smem -tw
Area                           Used      Cache   Noncache 
firmware/hardware                 0          0          0 
kernel image                      0          0          0 
kernel dynamic memory       2018832     357292    1661540 
userspace memory              59244      33928      25316 
free memory                   85008      85008          0 
----------------------------------------------------------
                            2163084     476228    1686856 

[me@lax2 ~]$ df -i
Filesystem     Inodes  IUsed  IFree IUse% Mounted on
dev            268304    343 267961    1% /dev
run            270385    438 269947    1% /run
/dev/vda2      636480 462678 173802   73% /
tmpfs          270385      1 270384    1% /dev/shm
tmpfs          270385     18 270367    1% /sys/fs/cgroup
tmpfs          270385     17 270368    1% /tmp
tmpfs          270385     22 270363    1% /run/user/1000

mikho

What does cat /proc/slabinfo give?

seanho

V2ray and caddy in-memory cache?

hopeful

What is cred_jar under slabinfo? I have to reboot my box every week, otherwise the box will become unresponsive after 8 or 9 days.

[me@lax2 ~]$ sudo cat /proc/slabinfo && cat /proc/meminfo && free -h
slabinfo - version: 2.1
# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
ext4_groupinfo_4k     84     84    144   28    1 : tunables    0    0    0 : slabdata      3      3      0
ext4_inode_cache    4919  11687   1096   29    8 : tunables    0    0    0 : slabdata    403    403      0
ext4_system_zone     102    102     40  102    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_io_end           64     64     64   64    1 : tunables    0    0    0 : slabdata      1      1      0
ext4_extent_status   3278   3876     40  102    1 : tunables    0    0    0 : slabdata     38     38      0
jbd2_journal_handle    438    438     56   73    1 : tunables    0    0    0 : slabdata      6      6      0
jbd2_journal_head    748    918    120   34    1 : tunables    0    0    0 : slabdata     27     27      0
jbd2_revoke_table_s    256    256     16  256    1 : tunables    0    0    0 : slabdata      1      1      0
jbd2_revoke_record_s    128    128     32  128    1 : tunables    0    0    0 : slabdata      1      1      0
scsi_sense_cache      32     32    128   32    1 : tunables    0    0    0 : slabdata      1      1      0
fsverity_info          0      0    256   16    1 : tunables    0    0    0 : slabdata      0      0      0
fscrypt_info          32     32    128   32    1 : tunables    0    0    0 : slabdata      1      1      0
MPTCPv6                0      0   1664   19    8 : tunables    0    0    0 : slabdata      0      0      0
ip6-frags              0      0    184   22    1 : tunables    0    0    0 : slabdata      0      0      0
PINGv6                 0      0   1216   13    4 : tunables    0    0    0 : slabdata      0      0      0
RAWv6                 13     13   1216   13    4 : tunables    0    0    0 : slabdata      1      1      0
UDPv6                240    240   1344   12    4 : tunables    0    0    0 : slabdata     20     20      0
tw_sock_TCPv6         32     32    248   16    1 : tunables    0    0    0 : slabdata      2      2      0
request_sock_TCPv6     13     13    304   13    1 : tunables    0    0    0 : slabdata      1      1      0
TCPv6                247    247   2432   13    8 : tunables    0    0    0 : slabdata     19     19      0
bfq_io_cq              0      0    160   25    1 : tunables    0    0    0 : slabdata      0      0      0
mqueue_inode_cache      8      8    960    8    2 : tunables    0    0    0 : slabdata      1      1      0
dnotify_struct         0      0     32  128    1 : tunables    0    0    0 : slabdata      0      0      0
pid_namespace          0      0    208   19    1 : tunables    0    0    0 : slabdata      0      0      0
ip4-frags              0      0    200   20    1 : tunables    0    0    0 : slabdata      0      0      0
MPTCP                  0      0   1472   11    4 : tunables    0    0    0 : slabdata      0      0      0
request_sock_subflow      0      0    344   23    2 : tunables    0    0    0 : slabdata      0      0      0
xfrm_state             0      0    768   10    2 : tunables    0    0    0 : slabdata      0      0      0
PING               70000  70000   1024    8    2 : tunables    0    0    0 : slabdata   8750   8750      0
RAW                    8      8   1024    8    2 : tunables    0    0    0 : slabdata      1      1      0
UDP                   98     98   1152   14    4 : tunables    0    0    0 : slabdata      7      7      0
tw_sock_TCP           16     16    248   16    1 : tunables    0    0    0 : slabdata      1      1      0
request_sock_TCP      39     39    304   13    1 : tunables    0    0    0 : slabdata      3      3      0
TCP                   42     42   2304   14    8 : tunables    0    0    0 : slabdata      3      3      0
hugetlbfs_inode_cache     12     12    640   12    2 : tunables    0    0    0 : slabdata      1      1      0
dquot                 48     48    256   16    1 : tunables    0    0    0 : slabdata      3      3      0
eventpoll_pwq       1288   1288     72   56    1 : tunables    0    0    0 : slabdata     23     23      0
dax_cache             10     10    768   10    2 : tunables    0    0    0 : slabdata      1      1      0
request_queue         15     15   2104   15    8 : tunables    0    0    0 : slabdata      1      1      0
biovec-max            78    112   4096    8    8 : tunables    0    0    0 : slabdata     14     14      0
biovec-128             8      8   2048    8    4 : tunables    0    0    0 : slabdata      1      1      0
biovec-64              8      8   1024    8    2 : tunables    0    0    0 : slabdata      1      1      0
khugepaged_mm_slot   1044   1044    112   36    1 : tunables    0    0    0 : slabdata     29     29      0
user_namespace         0      0    536   15    2 : tunables    0    0    0 : slabdata      0      0      0
dmaengine-unmap-256     15     15   2112   15    8 : tunables    0    0    0 : slabdata      1      1      0
dmaengine-unmap-128     15     15   1088   15    4 : tunables    0    0    0 : slabdata      1      1      0
sock_inode_cache  166478 166478    832   19    4 : tunables    0    0    0 : slabdata   8762   8762      0
skbuff_ext_cache     420    420    192   21    1 : tunables    0    0    0 : slabdata     20     20      0
skbuff_fclone_cache     48     48    512    8    1 : tunables    0    0    0 : slabdata      6      6      0
skbuff_head_cache 139488 139488    256   16    1 : tunables    0    0    0 : slabdata   8718   8718      0
file_lock_cache       36     36    216   18    1 : tunables    0    0    0 : slabdata      2      2      0
file_lock_ctx        146    146     56   73    1 : tunables    0    0    0 : slabdata      2      2      0
fsnotify_mark_connector    128    128     32  128    1 : tunables    0    0    0 : slabdata      1      1      0
net_namespace          0      0   4928    6    8 : tunables    0    0    0 : slabdata      0      0      0
task_delay_info      765    765     80   51    1 : tunables    0    0    0 : slabdata     15     15      0
taskstats             11     11    352   11    1 : tunables    0    0    0 : slabdata      1      1      0
proc_dir_entry       357    357    192   21    1 : tunables    0    0    0 : slabdata     17     17      0
pde_opener           612    612     40  102    1 : tunables    0    0    0 : slabdata      6      6      0
proc_inode_cache    1610   1702    688   23    4 : tunables    0    0    0 : slabdata     74     74      0
bdev_cache            19     19    832   19    4 : tunables    0    0    0 : slabdata      1      1      0
shmem_inode_cache   1029   1034    728   11    2 : tunables    0    0    0 : slabdata     94     94      0
kernfs_node_cache 541632 541632    128   32    1 : tunables    0    0    0 : slabdata  16926  16926      0
mnt_cache            708    708    320   12    1 : tunables    0    0    0 : slabdata     59     59      0
filp              378420 378656    256   16    1 : tunables    0    0    0 : slabdata  23666  23666      0
inode_cache       122986 123487    616   13    2 : tunables    0    0    0 : slabdata   9499   9499      0
dentry            198930 205359    192   21    1 : tunables    0    0    0 : slabdata   9779   9779      0
names_cache           16     16   4096    8    8 : tunables    0    0    0 : slabdata      2      2      0
buffer_head        15458  20865    104   39    1 : tunables    0    0    0 : slabdata    535    535      0
uts_namespace          9      9    440    9    1 : tunables    0    0    0 : slabdata      1      1      0
mm_struct         129135 129135   1088   15    4 : tunables    0    0    0 : slabdata   8609   8609      0
files_cache          121    121    704   11    2 : tunables    0    0    0 : slabdata     11     11      0
signal_cache         255    255   1088   15    4 : tunables    0    0    0 : slabdata     17     17      0
sighand_cache        195    195   2112   15    8 : tunables    0    0    0 : slabdata     13     13      0
task_struct          138    164   7424    4    8 : tunables    0    0    0 : slabdata     41     41      0
cred_jar          2030693 2030931    192   21    1 : tunables    0    0    0 : slabdata  96711  96711      0
anon_vma_chain    1407381 1407872     64   64    1 : tunables    0    0    0 : slabdata  21998  21998      0
anon_vma          696072 696072     88   46    1 : tunables    0    0    0 : slabdata  15132  15132      0
pid               276544 276544    128   32    1 : tunables    0    0    0 : slabdata   8642   8642      0
Acpi-Operand        1120   1120     72   56    1 : tunables    0    0    0 : slabdata     20     20      0
Acpi-ParseExt         39     39    104   39    1 : tunables    0    0    0 : slabdata      1      1      0
Acpi-State          8823   8823     80   51    1 : tunables    0    0    0 : slabdata    173    173      0
numa_policy          170    170     24  170    1 : tunables    0    0    0 : slabdata      1      1      0
trace_event_file    1426   1426     88   46    1 : tunables    0    0    0 : slabdata     31     31      0
ftrace_event_field   4080   4080     48   85    1 : tunables    0    0    0 : slabdata     48     48      0
pool_workqueue       128    128    256   16    1 : tunables    0    0    0 : slabdata      8      8      0
radix_tree_node    15189  15624    584   14    2 : tunables    0    0    0 : slabdata   1116   1116      0
task_group            36     36    640   12    2 : tunables    0    0    0 : slabdata      3      3      0
vmap_area           1216   1216     64   64    1 : tunables    0    0    0 : slabdata     19     19      0
dma-kmalloc-8k         0      0   8192    4    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-4k         0      0   4096    8    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-2k         0      0   2048    8    4 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-1k         0      0   1024    8    2 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-512        8      8    512    8    1 : tunables    0    0    0 : slabdata      1      1      0
dma-kmalloc-256        0      0    256   16    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-128        0      0    128   32    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-64         0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-32         0      0     32  128    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-16         0      0     16  256    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-8          0      0      8  512    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-192        0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-96         0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-8k         0      0   8192    4    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-4k         0      0   4096    8    8 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-2k         0      0   2048    8    4 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-1k         0      0   1024    8    2 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-512        0      0    512    8    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-256        0      0    256   16    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-192        0      0    192   21    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-128       96     96    128   32    1 : tunables    0    0    0 : slabdata      3      3      0
kmalloc-rcl-96      1988   2184     96   42    1 : tunables    0    0    0 : slabdata     52     52      0
kmalloc-rcl-64      1145   2048     64   64    1 : tunables    0    0    0 : slabdata     32     32      0
kmalloc-rcl-32         0      0     32  128    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-16         0      0     16  256    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-rcl-8          0      0      8  512    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-8k            35     56   8192    4    8 : tunables    0    0    0 : slabdata     14     14      0
kmalloc-4k         43968  43968   4096    8    8 : tunables    0    0    0 : slabdata   5496   5496      0
kmalloc-2k          9165   9176   2048    8    4 : tunables    0    0    0 : slabdata   1147   1147      0
kmalloc-1k         64640  64640   1024    8    2 : tunables    0    0    0 : slabdata   8080   8080      0
kmalloc-512        32608  32608    512    8    1 : tunables    0    0    0 : slabdata   4076   4076      0
kmalloc-256        18192  18192    256   16    1 : tunables    0    0    0 : slabdata   1137   1137      0
kmalloc-192        20643  20643    192   21    1 : tunables    0    0    0 : slabdata    983    983      0
kmalloc-128         9280   9280    128   32    1 : tunables    0    0    0 : slabdata    290    290      0
kmalloc-96        236292 236292     96   42    1 : tunables    0    0    0 : slabdata   5626   5626      0
kmalloc-64        127570 127872     64   64    1 : tunables    0    0    0 : slabdata   1998   1998      0
kmalloc-32        1585408 1585408     32  128    1 : tunables    0    0    0 : slabdata  12386  12386      0
kmalloc-16        170752 170752     16  256    1 : tunables    0    0    0 : slabdata    667    667      0
kmalloc-8         4296704 4296704      8  512    1 : tunables    0    0    0 : slabdata   8392   8392      0
kmem_cache_node   201344 201344     64   64    1 : tunables    0    0    0 : slabdata   3146   3146      0
kmem_cache        201330 201330    448    9    1 : tunables    0    0    0 : slabdata  22370  22370      0
MemTotal:        2163104 kB
MemFree:          133316 kB
MemAvailable:     346344 kB
Buffers:           17868 kB
Cached:            99716 kB
SwapCached:          264 kB
Active:            84384 kB
Inactive:          90608 kB
Active(anon):      18548 kB
Inactive(anon):    36044 kB
Active(file):      65836 kB
Inactive(file):    54564 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:        311292 kB
SwapFree:         310768 kB
Dirty:               816 kB
Writeback:             0 kB
AnonPages:         57180 kB
Mapped:            69860 kB
Shmem:              1796 kB
KReclaimable:     281144 kB
Slab:            1796448 kB
SReclaimable:     281144 kB
SUnreclaim:      1515304 kB
KernelStack:        1568 kB
PageTables:         1912 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1392844 kB
Committed_AS:     462684 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       10832 kB
VmallocChunk:          0 kB
Percpu:            35412 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:     1638388 kB
DirectMap2M:      589824 kB
              total        used        free      shared  buff/cache   available
Mem:          2.1Gi       1.6Gi       130Mi       1.0Mi       389Mi       338Mi
Swap:         303Mi       0.0Ki       303Mi