19 June, 2008

Monitoring "free memory" on Linux

Linux's memory management allows the kernel to use as much RAM as possible for the filesystem cache. This is reflected in the 'cached' column of "top" or "free" listings.
If you or your manager tries to look for "memory utilisation", then the 'free' memory is NOT what you should be looking it. More so if you have a database running on FileSystems on the server.

Here is a simple demo on 64bit OELinux 5.1 (using "default" Kernel parameters, no changes to memory/paging/vm etc setups) :


Beginning with 1.8GB RAM on the server. I see 1.36GB free
in both "top" and "free" listings.
"top" :

Mem: 1863144k total, 503048k used, 1360096k free, 25652k buffers
Swap: 2031608k total, 0k used, 2031608k free, 309992k cached


"free -o" :

total used free shared buffers cached
Mem: 1863144 503964 1359180 0 25684 310152
Swap: 2031608 0 2031608


Startup of 1 database instance :

SQL> ORACLE instance started.

Total System Global Area 1325400064 bytes
Fixed Size 2020576 bytes
Variable Size 469764896 bytes
Database Buffers 838860800 bytes
Redo Buffers 14753792 bytes
Database mounted.
Database opened.
SQL>


Although the Database has an SGA if 1.3GB, it hasn't yet significantly reduced
the "free" memory (which is now 919MB)

"top" :

Mem: 1863144k total, 943656k used, 919488k free, 27008k buffers
Swap: 2031608k total, 0k used, 2031608k free, 619968k cached

"free -o" :

total used free shared buffers cached
Mem: 1863144 944424 918720 0 27096 620076
Swap: 2031608 0 2031608


Startup of second database instance :

SQL> startup
ORACLE instance started.

Total System Global Area 541065216 bytes
Fixed Size 2085320 bytes
Variable Size 150998584 bytes
Database Buffers 381681664 bytes
Redo Buffers 6299648 bytes
Database mounted.
Database opened.
SQL>

Itill have 608MB free memory, after the second database instance startup.

"top" :

Mem: 1863144k total, 1255084k used, 608060k free, 28380k buffers
Swap: 2031608k total, 0k used, 2031608k free, 851996k cached



How do I "bump up" the "free memory" without shutting down the databases ?
1. Create a large file
2. Delete the file

Here's the first step :
Create a 1GB file :

$>dd if=/dev/zero of=DUMMY.ZERO bs=1024K count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 38.5035 seconds, 27.2 MB/s
$>


The process of creating a 1GB file sudeenly brought "free" memory down
to 12MB. Notice how "cached" has increased. Neither of the two really
changed by 1GB ! "free" memory reduced by 596MB and "cached" increased
by 908MB !

"top" :

Mem: 1863144k total, 1851188k used, 11956k free, 14804k buffers
Swap: 2031608k total, 0k used, 2031608k free, 1462696k cached

Here's the second step :
Remove the 1GB file :

$>ls -l DUMMY.ZERO
-rw-r--r-- 1 oracle2 dba 1048576000 Jun 19 22:14 DUMMY.ZERO
$>rm DUMMY.ZERO


Now, "free" memory has suddenly jumped to 1GB. Remember that I still
have the two database instances with a sum of 1.8GB SGA size in a
1.8GB RAM machine. And "swap" isn't even used yet.

"top" :

Mem: 1863144k total, 823668k used, 1039476k free, 15140k buffers
Swap: 2031608k total, 0k used, 2031608k free, 465152k cached

Let's verify those two database instances :
shared memory segments :

"ipcs -m" :

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00000000 65536 root 600 393216 2 dest
0x00000000 98305 root 600 393216 2 dest
0x00000000 131074 root 600 393216 2 dest
0x00000000 163843 root 600 393216 2 dest
0x00000000 196612 root 600 393216 2 dest
0x00000000 229381 root 600 393216 2 dest
0x00000000 262150 root 600 393216 2 dest
0x00000000 294919 root 600 393216 2 dest
0x00000000 327688 root 600 393216 2 dest
0x00000000 360457 root 600 393216 2 dest
0x0133492c 393226 oracle1 640 1327497216 17
0xc4afc244 425995 oracle2 640 543162368 18


Now I run a query on the first database to see if "loading" the SYSTEM
datafile has an impact on "free" memory.
Query the "oracle1" database :

SQL> select sum(bytes)/1048576 from dba_data_files;

SUM(BYTES)/1048576
------------------
3561.25

SQL> select sum(value)/1048576 from v$sga;

SUM(VALUE)/1048576
------------------
1264

SQL>


My "free" memory has shrunk from 1,039MB to 973MB and "cached" usage increased
from 465MB to 521MB.. Evidently, not the entire SYSTEM datafile (which is
actually 492MB in size) did get loaded into memory.
"top" :

Mem: 1863144k total, 889992k used, 973152k free, 15848k buffers
Swap: 2031608k total, 0k used, 2031608k free, 521256k cached

"free -o" :

total used free shared buffers cached
Mem: 1863144 889612 973532 0 15864 521256
Swap: 2031608 0 2031608


Running a few large queries, DML, sort operations etc (ie accessing multiple datafiles)
and then check available memory :
My "free" memory has dropped from 973MB to 23MB and "cached" increased
from 521MB to 1,467MB.
"top" :

Mem: 1863144k total, 1839452k used, 23692k free, 8280k buffers
Swap: 2031608k total, 3576k used, 2028032k free, 1467392k cached

"free -o" :

total used free shared buffers cached
Mem: 1863144 1839404 23740 0 8248 1467436
Swap: 2031608 3576 2028032


Can I try that "trick" again to release memory ?
Create and remove a 1GB dummy file :

sh-3.1$ dd if=/dev/zero of=DUMMY.ZERO bs=1024K count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 44.776 seconds, 23.4 MB/s
sh-3.1$ rm DUMMY.ZERO

I see "free" memory up to 795MB and "cached" down to 680MB.
"top" :

Mem: 1863144k total, 1067748k used, 795396k free, 3368k buffers
Swap: 2031608k total, 96580k used, 1935028k free, 680900k cached


"free -o" :

total used free shared buffers cached
Mem: 1863144 1068244 794900 0 3392 680896
Swap: 2031608 96580 1935028





Evidently, Linux is dynamically allocating filesystem buffer space to/from
"cached" and "cached" can grow and shrink as required. The values in the
"free" column are quite misleading at all times.

1 comment:

Andres said...

Hi, you can compile this program in C to find oracle process memory usage

http://ejemplosprogramacionc.blogspot.com/2013/06/memoria-de-procesos-oracle-sobre.html