First of all, why do we want do change the memory settings for the ASM instance at all? There are many reasons why you might want to do it. The main reasons are that HugePages cannot be swapped, and the relief of TLB (Translation Lookaside Buffer) pressure, meaning that HugePages uses fewer pages to cover the physical address space, so the size of “book keeping” (mapping from the virtual to the physical address) decreases, so it requiring fewer entries in the TLB.
Another reason is, that you don’t want to mix different memory allocation techniques on your system. E.g. if your databases are already using HugePages, then ASM should do it also. In case you want to learn more about HugePages, check out the following MOS Note.
Oracle Linux: HugePages What It Is… and What It Is Not… (Doc ID 361323.1)
To implement HugePages for your ASM Instance, just do the following steps:
- Disable Transparent Hugepages first (if not already done)
- Implement Huge Pages on the OS Level (if not already done)
- Remove all the __shared_pool_size parameters and recreate the ASM spfile
- Set the parameters sga_max_size, sga_target and use_large_pages
- Restart HAS
On my system, the memory target defaulted to about 1GB. It might be different on your system. So let’s get started and change the Oracle 21c ASM instance from MEMORY_TARGET to SGA_TARGET.
Disable Transparent HugePages first
There is a quite good documentation of how to disable the Transparent HugePages features. The Transparent HugePages is something we do not want.
To check if the Transparent HugePages feature is really disabled, run the following command.
# cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never]
Implement HugePages on the OS Level
Now implement the HugePages if not already done.
To check if HugePages are configured, run the following command.
# sysctl -a | grep -i huge vm.hugetlb_shm_group = 54321 vm.nr_hugepages = 6144 vm.nr_hugepages_mempolicy = 6144 vm.nr_overcommit_hugepages = 0
Remove all the __shared_pool_size parameters and recreate the ASM spfile
This is an optional step, and it is only required if you have some “double underscore values” in your parameter file.
Check the current values and do a copy of your spfile settings.
SQL> create pfile='/u01/app/oracle/admin/+ASM/pfile/init+ASM.ora.20230524a' from spfile; File created. SQL> $ cat /u01/app/oracle/admin/+ASM/pfile/init+ASM.ora.20230524a +ASM.__oracle_base='/u01/app/oracle'#ORACLE_BASE set from in memory value +ASM.asm_diskgroups='FRA','ACFS'#Manual Mount *.asm_diskstring='/dev/sd*','/dev/nvme0*' *.asm_power_limit=1 *.large_pool_size=12M *.remote_login_passwordfile='EXCLUSIVE' SQL> select NAME,BYTES from v$sgainfo ; NAME BYTES -------------------------------- ---------- Fixed SGA Size 9456600 ASM Cache Size 25165824 Shared Pool Size 620756992 Large Pool Size 12582912 Granule Size 4194304 Maximum SGA Size 1137724376 Startup overhead in Shared Pool 190917512 Free SGA Memory Available 469762048 8 rows selected. SQL> show sga Total System Global Area 1137724376 bytes Fixed Size 9456600 bytes Variable Size 1103101952 bytes ASM Cache 25165824 bytes SQL> show parameter memory NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ memory_max_target big integer 1076M memory_target big integer 1076M SQL> show parameter sga NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ allow_group_access_to_sga boolean FALSE lock_sga boolean FALSE sga_max_size big integer 1088M sga_target big integer 0 SQL>
Change the parameters
SQL> alter system reset memory_max_target scope=spfile sid='*'; System altered. SQL> alter system reset memory_target scope=spfile sid='*'; System altered. SQL> alter system set memory_max_target=0 scope=spfile sid='*'; System altered. SQL> alter system set memory_target=0 scope=spfile sid='*'; System altered. SQL> alter system set sga_max_size=2048M scope=spfile sid='*'; System altered. SQL> alter system set sga_target=2048M scope=spfile sid='*'; System altered. SQL> alter system set use_large_pages='ONLY' scope=spfile sid='*'; System altered. SQL> SQL> create pfile='/u01/app/oracle/admin/+ASM/pfile/init+ASM.ora.20230524b' from spfile; File created. SQL> $ cat /u01/app/oracle/admin/+ASM/pfile/init+ASM.ora.20230524b +ASM.__oracle_base='/u01/app/oracle'#ORACLE_BASE set from in memory value +ASM.asm_diskgroups='FRA','ACFS'#Manual Mount *.asm_diskstring='/dev/sd*','/dev/nvme0*' *.asm_power_limit=1 *.large_pool_size=12M *.memory_max_target=0 *.memory_target=0 *.remote_login_passwordfile='EXCLUSIVE' *.sga_max_size=2147483648 *.sga_target=2147483648 *.use_large_pages='ONLY'
Restart HAS (High Availability Services)
$ crsctl stop has CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ocm199' CRS-2673: Attempting to stop 'ora.evmd' on 'ocm199' CRS-2673: Attempting to stop 'ora.ACFS.dg' on 'ocm199' CRS-2673: Attempting to stop 'ora.cdb.db' on 'ocm199' CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'ocm199' CRS-2677: Stop of 'ora.ACFS.dg' on 'ocm199' succeeded CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'ocm199' succeeded CRS-2677: Stop of 'ora.evmd' on 'ocm199' succeeded CRS-2677: Stop of 'ora.cdb.db' on 'ocm199' succeeded CRS-2673: Attempting to stop 'ora.DATA.dg' on 'ocm199' CRS-2673: Attempting to stop 'ora.FRA.dg' on 'ocm199' CRS-2677: Stop of 'ora.FRA.dg' on 'ocm199' succeeded CRS-2677: Stop of 'ora.DATA.dg' on 'ocm199' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'ocm199' CRS-2677: Stop of 'ora.asm' on 'ocm199' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'ocm199' CRS-2677: Stop of 'ora.cssd' on 'ocm199' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ocm199' has completed CRS-4133: Oracle High Availability Services has been stopped. $ crsctl start has CRS-4123: Oracle High Availability Services has been started.
Check the alert.log if everything is ok.
Starting ORACLE instance (normal) (OS id: 4283)
2023-05-24T08:40:22.477100+02:00
************************************************************
Instance SGA_TARGET = 2048 MB and SGA_MAX_SIZE = 2048 MB
************************************************************
2023-05-24T08:40:22.477209+02:00
****************************************************
Sys-V shared memory will be used for creating SGA
****************************************************
2023-05-24T08:40:22.477644+02:00
**********************************************************************
2023-05-24T08:40:22.477672+02:00
Dump of system resources acquired for SHARED GLOBAL AREA (SGA)
2023-05-24T08:40:22.477720+02:00
Domain name: system.slice/ohasd.service
2023-05-24T08:40:22.477746+02:00
Per process system memlock (soft) limit = 128G
2023-05-24T08:40:22.477772+02:00
Expected per process system memlock (soft) limit to lock
instance MAX SHARED GLOBAL AREA (SGA) into memory: 2044M
2023-05-24T08:40:22.477820+02:00
Available system pagesizes:
4K, 2048K
2023-05-24T08:40:22.477867+02:00
Supported system pagesize(s):
2023-05-24T08:40:22.477892+02:00
PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s)
2023-05-24T08:40:22.477942+02:00
2048K 6144 1022 1022 NONE
2023-05-24T08:40:22.477966+02:00
...
...
Using parameter settings in server-side spfile +DATA/ASM/ASMPARAMETERFILE/registry.253.1135337751
System parameters with non-default values:
sga_max_size = 2G
use_large_pages = "ONLY"
large_pool_size = 16M
sga_target = 2G
memory_target = 0
memory_max_target = 0
remote_login_passwordfile= "EXCLUSIVE"
asm_diskstring = "/dev/sd*"
asm_diskstring = "/dev/nvme0*"
asm_diskgroups = "FRA"
asm_diskgroups = "ACFS"
asm_power_limit = 1
...
...
Check the /proc/meminfo and ipcs output to see if the HugePages are really used.
$ cat /proc/meminfo | grep -i huge AnonHugePages: 0 kB ShmemHugePages: 0 kB FileHugePages: 0 kB HugePages_Total: 6144 HugePages_Free: 3024 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 12582912 kB $ ipcs --shmems ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x00000000 2 grid 600 10485760 31 0x00000000 3 grid 600 2130706432 31 0x73a656dc 4 grid 600 2097152 31
Ready. We have implemented HugePages for the ASM instance. 🙂
In case you want to learn more about ASM and Huge Pages, check out the following MOS Notes.
- Oracle Linux: HugePages What It Is… and What It Is Not… (Doc ID 361323.1)
- Oracle Linux: HugePages on Oracle Linux 64-bit (Doc ID 361468.1)
- HugePages and Oracle Database Automatic Memory Management (AMM) on Linux (Doc ID 749851.1)
- Oracle Linux: Shell Script to Calculate Values Recommended Linux HugePages / HugeTLB Configuration (Doc ID 401749.1)
- Default and Minimum MEMORY_TARGET & MEMORY_MAX_TARGET Value for ASM 11.2.0.4 and Onwards (Doc ID 1982132.1)
- ASM & Shared Pool (ORA-4031) (Doc ID 437924.1)
Conclusion
Implementing HugePages for you ASM instance is not so complicated, and it really makes sense if you already use it for your Oracle database instances anyway.