Oracle 11g RAC 数据库ASM扩容实战-详细过程记录
系统：CentOS7.9
数据库：oracle 11.2.0.4
环境：RAC 双节点
磁盘组：asmlib创建+multipath多路径
问题描述：磁盘组由5块20g的磁盘构成，当添加一块40g的磁盘进磁盘组时，出现无法重平衡的问题.
SQL> r
1* select path,total_mb,free_mb from v$asm_disk_stat where group_number=1 order by 1
PATH TOTAL_MB FREE_MB
—————————— ———- ———-
ORCL:DATA01 20479 20067
ORCL:DATA02 20479 20069
ORCL:DATA03 20479 20067
ORCL:DATA04 20479 20066
ORCL:DATA05 20479 20067
ORCL:DATA06 40959 40143
6 rows selected.
报错如下，ARB0进程会自动被停止.
SQL> alter diskgroup data rebalance power 8
NOTE: GroupBlock outside rolling migration privileged region
NOTE: requesting all-instance membership refresh for group=1
Wed May 11 22:24:08 2022
NOTE: membership refresh pending for group 1/0xbb195d83 (DATA)
Wed May 11 22:24:11 2022
GMON querying group 1 at 12 for pid 18, osid 4346
SUCCESS: refreshed membership for 1/0xbb195d83 (DATA)
SUCCESS: alter diskgroup data rebalance power 8
NOTE: starting rebalance of group 1/0xbb195d83 (DATA) at power 8
Starting background process ARB0
Wed May 11 22:24:11 2022
ARB0 started with pid=33, OS id=7966
NOTE: assigning ARB0 to group 1/0xbb195d83 (DATA) with 8 parallel I/Os
cellip.ora not found.
NOTE: stopping process ARB0
SUCCESS: rebalance completed for group 1/0xbb195d83 (DATA)
NOTE: Attempting voting file refresh on diskgroup DATA
NOTE: Refresh completed on diskgroup DATA. No voting file found.
此问题花了两天时间，经多次测试终于弄清楚，原来添加的磁盘要和之前构成磁盘组的磁盘大小要一致.
实验测试：
SQL> alter diskgroup data drop disk data06;
Diskgroup altered.
SQL> select path,total_mb,free_mb from v$asm_disk_stat where group_number=1 order by 1;
PATH TOTAL_MB FREE_MB
—————————— ———- ———-
ORCL:DATA01 20479 19905
ORCL:DATA02 20479 19906
ORCL:DATA03 20479 19904
ORCL:DATA04 20479 19903
ORCL:DATA05 20479 19904
建表快速消耗磁盘组+DATA
SQL> r
1* select file_name,autoextensible,bytes/1024/1024,tablespace_name from dba_data_files
FILE_NAME AUT BYTES/1024/1024 TABLESPACE_NAME
————————————————– — ————— ——————————
+DATA/orcl/datafile/system.259.1103587173 YES 760 SYSTEM
+DATA/orcl/datafile/sysaux.260.1103587175 YES 670 SYSAUX
+DATA/orcl/datafile/undotbs1.261.1103587179 YES 825 UNDOTBS1
+DATA/orcl/datafile/undotbs2.263.1103587185 YES 200 UNDOTBS2
+DATA/orcl/datafile/users.264.1103587185 YES 5 USERS
速度建表,指定users表空间
SQL> create table t1 tablespace users as select * from dba_objects;
然后不断insert数据.
SQL> insert into t1 select * from t1;
11051776 rows created.
Elapsed: 00:01:39.79
SQL> insert into t1 select * from t1;
22103552 rows created.
Elapsed: 00:29:11.08
SQL> insert into t1 select * from t1;
44207104 rows created.
Elapsed: 00:07:30.63
SQL> select bytes/1024/1024 from user_segments where segment_name=’T1′;
BYTES/1024/1024
—————
9855
Elapsed: 00:00:00.09
SQL> alter system set db_recovery_file_dest_size=35g;
System altered.
Elapsed: 00:00:00.01
SQL> insert into t1 select * from t1;
88414208 rows created.
Elapsed: 00:14:42.63
SQL> select bytes/1024/1024 from user_segments where segment_name=’T1′;
BYTES/1024/1024
—————
19711
Elapsed: 00:00:00.08
查看磁盘组使用情况
SQL> select path,total_mb,free_mb from v$asm_disk_stat where group_number=1 order by 1;
PATH TOTAL_MB FREE_MB
—————————— ———- ———-
ORCL:DATA01 20479 15766
ORCL:DATA02 20479 15768
ORCL:DATA03 20479 15765
ORCL:DATA04 20479 15764
ORCL:DATA05 20479 15765
添加1块20g共享磁盘/dev/sdo(添加方法可查看此前博客)
# lsblk -p
/dev/sdo 8:224 0 20G 0 disk
└─/dev/mapper/mpatha 253:5 0 20G 0 mpath
节点1分区
# fdisk /dev/sdo
[root@hisdb1 ~]# partprobe
[root@hisdb2 ~]# partprobe
# lsblk -p
/dev/sdo 8:224 0 20G 0 disk
├─/dev/sdo1 8:225 0 20G 0 part
└─/dev/mapper/mpathb 253:6 0 20G 0 mpath
└─/dev/mapper/mpathb1 253:31 0 20G 0 part
[root@hisdb1 ~]# /usr/lib/udev/scsi_id -g -u /dev/sdo
36000c29070ff67056a025a83380ed941
配置multipath，修改/etc/multipath.conf文件
双节点添加以下内容：
multipath {
wwid “36000c29070ff67056a025a83380ed941”
alias data07
}
双节点激活多路径
[root@hisdb1 ~]# multipath -F
[root@hisdb1 ~]# multipath -v2
[root@hisdb1 ~]# multipath -ll
…
create: data07 (36000c29070ff67056a025a83380ed941) undef VMware, ,VMware Virtual S
size=20G features=’0′ hwhandler=’0′ wp=undef
`-+- policy=’service-time 0′ prio=1 status=undef
`- 0:0:15:0 sdo 8:224 undef ready running
…
创建asm磁盘，节点1执行
[root@hisdb1 ~]# oracleasm createdisk data07 /dev/mapper/data07p1
Writing disk header: done
Instantiating disk: done
[root@hisdb1 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks…
Scanning system for ASM disks…
[root@hisdb1 ~]# oracleasm listdisks
DATA01
DATA02
DATA03
DATA04
DATA05
DATA06
DATA07
FRA01
FRA02
FRA03
OCR01
OCR02
OCR03
OCR04
OCR05
[root@hisdb2 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks…
Scanning system for ASM disks…
Instantiating disk “DATA07”
[root@hisdb2 ~]# oracleasm listdisks
DATA01
DATA02
DATA03
DATA04
DATA05
DATA06
DATA07
FRA01
FRA02
FRA03
OCR01
OCR02
OCR03
OCR04
OCR05
SQL> show parameter power
NAME TYPE VALUE
———————————— ——————– ——————————
asm_power_limit integer 8
SQL> r
1* select path,total_mb,free_mb from v$asm_disk_stat
PATH TOTAL_MB FREE_MB
—————————— ———- ———-
ORCL:DATA01 20479 15766
ORCL:DATA02 20479 15768
ORCL:DATA03 20479 15765
ORCL:DATA04 20479 15764
ORCL:DATA05 20479 15765
ORCL:FRA01 20479 12758
ORCL:FRA02 20479 12765
ORCL:FRA03 20479 12756
ORCL:OCR01 2047 1795
ORCL:OCR02 2047 1795
ORCL:OCR03 2047 1794
PATH TOTAL_MB FREE_MB
—————————— ———- ———-
ORCL:OCR04 2047 1795
ORCL:OCR05 2047 1800
13 rows selected.
SQL> alter diskgroup DATA add disk ‘ORCL:DATA07’;
Diskgroup altered.
SQL> select path,total_mb,free_mb from v$asm_disk_stat;
PATH TOTAL_MB FREE_MB
—————————— ———- ———-
ORCL:DATA01 20479 15973
ORCL:DATA02 20479 15976
ORCL:DATA03 20479 15972
ORCL:DATA04 20479 15972
ORCL:DATA05 20479 15972
ORCL:FRA01 20479 12758
ORCL:FRA02 20479 12765
ORCL:FRA03 20479 12756
ORCL:OCR01 2047 1795
ORCL:OCR02 2047 1795
ORCL:OCR03 2047 1794
PATH TOTAL_MB FREE_MB
—————————— ———- ———-
ORCL:OCR04 2047 1795
ORCL:OCR05 2047 1800
ORCL:DATA07 20479 19438
14 rows selected.
SQL> select path,total_mb,free_mb from v$asm_disk_stat;
PATH TOTAL_MB FREE_MB
—————————— ———- ———-
ORCL:DATA01 20479 16549
ORCL:DATA02 20479 16551
ORCL:DATA03 20479 16549
ORCL:DATA04 20479 16550
ORCL:DATA05 20479 16550
ORCL:FRA01 20479 12758
ORCL:FRA02 20479 12765
ORCL:FRA03 20479 12756
ORCL:OCR01 2047 1795
ORCL:OCR02 2047 1795
ORCL:OCR03 2047 1794
PATH TOTAL_MB FREE_MB
—————————— ———- ———-
ORCL:OCR04 2047 1795
ORCL:OCR05 2047 1800
ORCL:DATA07 20479 16554
14 rows selected.
在asmcmd下lsdg命令显示Rebal为N代表磁盘组扩容完毕，此处未保存到数据.
说明：当将ORCL:DATA07加入+DATA磁盘组时，asmcmd下lsdg命令Rebal会由N变为Y，结束后又会变为N.
