■ RAC 의 Storage 구성 (02.Shared Storage 구성)
여기서는 각 DB 노드에 Shared Volumn 을 공급하기 위해 앞서서 설치한 Shared Storage 솔루션의 구성 후속 작업이다.
[RAC 의 Storage 구성 (02.Shared Storage 구성)] 01. /etc/hosts 파일 수정.
[root@storage ~]# vi /etc/hosts <-- 아래 사항들을 추가한다.
127.0.0.1 localhost.localdomain localhost
# Storage
192.168.12.5 storage.test.com storage
# 11gR2 RAC Setting
# Public
192.168.12.11 mk01n01.test.com mk01n01
192.168.12.12 mk01n02.test.com mk01n02
# Private
192.168.13.11 mk01n01-priv.test.com mk01n01-priv
192.168.13.12 mk01n02-priv.test.com mk01n02-priv
# Vitural
192.168.12.13 mk01n01-vip.test.com mk01n01-vip
192.168.12.14 mk01n02-vip.test.com mk01n02-vip
# SCAN
192.168.12.15 scan.test.com mark01-scan
46. 콘솔에서 root user로 위와 같이 접속후 /etc/hosts 파일을 수정한다.. (위사항 추가.)
Storage 서버에서 위와 같이 각 DB 노드의 IP 를 정의 한다.
※ 통신을 위한 pulic IP와 private IP만 정의 되어있어도 된다.
[RAC 의 Storage 구성 (02.Openfiler 설치)] 02. vi /etc/initiators.deny 파일 수정
[root@storage ~]# vi /etc/initiators.deny --> 이파일 열어서 주석처리 아래와 같이 iqn 부분 주석처리.
# PLEASE DO NOT MODIFY THIS CONFIGURATION FILE!
# This configuration file was autogenerated
# by Openfiler. Any manual changes will be overwritten
# Generated at: Mon May 16 16:20:20 KST 2016
# iqn.2006-01.com.openfiler:tsn.2151a0d1ff7e ALL
# End of Openfiler configuration
47. /etc/initiators.deny 이 파일에 Iqn이 정의되어 있으면 설정된 IP 에서의 접근을 거부 하는 것으로 default 값은 ALL 이다. 이 파일의 iqn에 ALL 로 설정되어있으면 모든 접근을 거부한다는 의미 이므로 주석처리한다.
[RAC 의 Storage 구성 (02.Shared Storage 구성)] 03. 각 DB 서버의 iscsi 서비스 start 및 iSCSI discovery
-- 1번노드
# 1. iscsi 서비스 start 및 부팅시 동작하도록 chkconfig
[root@mk01n01 ~]# service iscsi start
iscsid (pid 1735) is running...
Setting up iSCSI targets: iscsiadm: No records found
[ OK ]
[root@mk01n01 ~]# chkconfig iscsid on
[root@mk01n01 ~]# chkconfig iscsi on
# 2. DB 서버 에서 storage 서버로 discovery
[root@mk01n01 ~]# iscsiadm -m discovery -t sendtargets -p storage-priv
192.168.13.5:3260,1 iqn.2006-01.com.openfiler:tsn.2151a0d1ff7e
192.168.12.5:3260,1 iqn.2006-01.com.openfiler:tsn.2151a0d1ff7e
-- 2번노드
# 1. iscsi 서비스 start 및 부팅시 동작하도록 chkconfig
[root@mk01n02 ~]# service iscsi start
iscsid (pid 1732) is running...
Setting up iSCSI targets: iscsiadm: No records found
[ OK ]
[root@mk01n02 ~]# chkconfig iscsid on
[root@mk01n02 ~]# chkconfig iscsi on
# 2. DB 서버 에서 storage 서버로 discovery
[root@mk01n02 ~]# iscsiadm -m discovery -t sendtargets -p storage-priv
192.168.13.5:3260,1 iqn.2006-01.com.openfiler:tsn.2151a0d1ff7e
192.168.12.5:3260,1 iqn.2006-01.com.openfiler:tsn.2151a0d1ff7e
48. Openfiler 에서 설정한 Shared storage 를 각 DB 노드에서 인식을 시키려면 위와 같이
# 1. iscsi 서비스를 start 및 부팅시 동작하도록 chkconfig 로 모든 runlevel 에서 동작하도록 설정 후
# 2. DB 서버에서 shared disk 를 인식하도록 discovery 한다.
(위 화면은 discovery 시 private IP 로만 discovery 시켰기 때문에 private IP 만 확인되어야 하나 public IP 도 같이 나타난다 아무래도 openfiler 버그로 생각된다.)
[RAC 의 Storage 구성 (02.Shared Storage 구성)] 04. 각 DB 서버에서 Shared volumn 확인 및 fdisk
# 1. 모든 노드 에서 아래와 같이 disk 확인
[root@mk01n01 ~]# fdisk -l |perl -ne 'print if /\/dev\/sd[b-z]/'
Disk /dev/sdj: 5368 MB, 5368709120 bytes
/dev/sdj1 1 7599 5242849 83 Linux
Disk /dev/sdk: 5368 MB, 5368709120 bytes
/dev/sdk1 1 7599 5242849 83 Linux
Disk /dev/sdg: 2147 MB, 2147483648 bytes
/dev/sdg1 1 21846 2097121 83 Linux
Disk /dev/sdf: 2147 MB, 2147483648 bytes
/dev/sdf1 1 21846 2097121 83 Linux
Disk /dev/sdh: 5368 MB, 5368709120 bytes
/dev/sdh1 1 7599 5242849 83 Linux
Disk /dev/sdb: 2147 MB, 2147483648 bytes
/dev/sdb1 1 21846 2097121 83 Linux
Disk /dev/sdd: 2147 MB, 2147483648 bytes
/dev/sdd1 1 21846 2097121 83 Linux
Disk /dev/sde: 2147 MB, 2147483648 bytes
/dev/sde1 1 21846 2097121 83 Linux
Disk /dev/sdc: 2147 MB, 2147483648 bytes
/dev/sdc1 1 21846 2097121 83 Linux
Disk /dev/sdi: 5368 MB, 5368709120 bytes
/dev/sdi1 1 7599 5242849 83 Linux
Disk /dev/sdl: 5368 MB, 5368709120 bytes
/dev/sdl1 1 7599 5242849 83 Linux
Disk /dev/sdo: 5368 MB, 5368709120 bytes
/dev/sdo1 1 7599 5242849 83 Linux
Disk /dev/sdm: 5368 MB, 5368709120 bytes
/dev/sdm1 1 7599 5242849 83 Linux
Disk /dev/sdr: 5368 MB, 5368709120 bytes
/dev/sdr1 1 7599 5242849 83 Linux
Disk /dev/sdn: 5368 MB, 5368709120 bytes
/dev/sdn1 1 7599 5242849 83 Linux
Disk /dev/sdp: 5368 MB, 5368709120 bytes
/dev/sdp1 1 7599 5242849 83 Linux
Disk /dev/sds: 5368 MB, 5368709120 bytes
/dev/sds1 1 7599 5242849 83 Linux
Disk /dev/sdq: 5368 MB, 5368709120 bytes
/dev/sdq1 1 7599 5242849 83 Linux
[root@mk01n01 ~]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): u
Changing display/entry units to sectors
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First sector (62-4194303, default 62):
Using default value 62
Last sector or +size or +sizeM or +sizeK (62-4194303, default 4194303):
Using default value 4194303
Command (m for help): p
Disk /dev/sdb: 2147 MB, 2147483648 bytes
67 heads, 62 sectors/track, 1009 cylinders, total 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 62 4194303 2097121 83 Linux
Partition 1 does not end on cylinder boundary.
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.
49. #1 과 같이 모든 node 에서 disk 를 확인한다.
(openfiler 의 버그로 2배정도 disk 가 좀더 많이 나타난다. 따라서 fdisk 시에 되는것과 안되는것을 체크해서 admgroup 생성시 잘 사용해야 한다.)
-- 확인한 실제 DISK명
Disk /dev/sdb: 2147 MB, 2147483648 bytes Disk /dev/sdc: 2147 MB, 2147483648 bytes Disk /dev/sdd: 2147 MB, 2147483648 bytes Disk /dev/sdh: 5368 MB, 5368709120 bytes Disk /dev/sdi: 5368 MB, 5368709120 bytes Disk /dev/sdk: 5368 MB, 5368709120 bytes Disk /dev/sdl: 5368 MB, 5368709120 bytes Disk /dev/sdm: 5368 MB, 5368709120 bytes |
[RAC 의 Storage 구성 (02.Shared Storage 구성)] 05. oracle asm 을 위한 환경 설정(모든노드)
-- 1번노드
[root@mk01n01 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: grid
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
-- 2번노드
[root@mk01n02 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: grid
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
05. 여기서는 위와 같이 oracleasm 설정을 위해 위와 같이 oracleasm library 를 호출하여 oracleasm 환경을 설정한다.
[RAC 의 Storage 구성 (02.Shared Storage 구성)] 06. ocr_vote 디스크 생성
[root@mk01n01 ~]# oracleasm createdisk OCR_VOTE01 /dev/sdb1
Writing disk header: done
Instantiating disk: done
[root@mk01n01 ~]# oracleasm createdisk OCR_VOTE02 /dev/sdd1
Writing disk header: done
Instantiating disk: done
[root@mk01n01 ~]# oracleasm createdisk OCR_VOTE03 /dev/sdf1
Writing disk header: done
Instantiating disk: done
06. 여러분은 위 [RAC 의 Storage 구성 (02.Shared Storage 구성)] 03 에서 discovery 시 생성한 volumn 보다 2배정도 많은 disk 가 보인것을 확인했을 것이다.
이는 openfiler 버그 때문인것으로 추정되며
위와 같이 asmdisk 생성시 중복되는 disk 가 없이 생성해야 한다.
(물론 이미 생성한 disk 라면 아래와 같이 이미 할당된 disk 라고 나타난다.)
-- 아래와 같이 error 를 만나도 당황하지 않아도 된다 이미 할당된것은 에러메시지와 함께 재할당되지 않는다.
[root@mk01n01 ~]# oracleasm createdisk OCR_VOTE01 /dev/sdb1 Writing disk header: done Instantiating disk: done [root@mk01n01 ~]# oracleasm createdisk OCR_VOTE02 /dev/sdc1 Device "/dev/sdc1" is already labeled for ASM disk "OCR_VOTE01" [root@mk01n01 ~]# oracleasm createdisk OCR_VOTE02 /dev/sdd1 Writing disk header: done Instantiating disk: done [root@mk01n01 ~]# oracleasm createdisk OCR_VOTE03 /dev/sde1 Device "/dev/sde1" is already labeled for ASM disk "OCR_VOTE02" [root@mk01n01 ~]# oracleasm createdisk OCR_VOTE03 /dev/sdf1 Writing disk header: done Instantiating disk: done |
[RAC 의 Storage 구성 (02.Shared Storage 구성)] 07. DATA 및 RECO 디스크 생성
[root@mk01n01 ~]# oracleasm createdisk OCR_VOTE01 /dev/sdb1
Writing disk header: done
Instantiating disk: done
[root@mk01n01 ~]# oracleasm createdisk OCR_VOTE02 /dev/sdc1
Device "/dev/sdc1" is already labeled for ASM disk "OCR_VOTE01"
[root@mk01n01 ~]# oracleasm createdisk OCR_VOTE02 /dev/sdd1
Writing disk header: done
Instantiating disk: done
[root@mk01n01 ~]# oracleasm createdisk OCR_VOTE03 /dev/sde1
Device "/dev/sde1" is already labeled for ASM disk "OCR_VOTE02"
[root@mk01n01 ~]# oracleasm createdisk OCR_VOTE03 /dev/sdf1
Writing disk header: done
Instantiating disk: done
[root@mk01n01 ~]#
[root@mk01n01 ~]# oracleasm createdisk DATA01 /dev/sdh1
Writing disk header: done
Instantiating disk: done
[root@mk01n01 ~]# oracleasm createdisk DATA02 /dev/sdi1
Device "/dev/sdi1" is already labeled for ASM disk "DATA01"
[root@mk01n01 ~]#
[root@mk01n01 ~]# oracleasm createdisk DATA02 /dev/sdj1
Writing disk header: done
Instantiating disk: done
[root@mk01n01 ~]# oracleasm createdisk DATA03 /dev/sdk1
Device "/dev/sdk1" is already labeled for ASM disk "DATA02"
[root@mk01n01 ~]# oracleasm createdisk DATA03 /dev/sdl1
Writing disk header: done
Instantiating disk: done
[root@mk01n01 ~]# oracleasm createdisk RECO01 /dev/sdm1
Device "/dev/sdm1" is already labeled for ASM disk "DATA03"
[root@mk01n01 ~]# oracleasm createdisk RECO01 /dev/sdn1
Writing disk header: done
Instantiating disk: done
[root@mk01n01 ~]# oracleasm createdisk RECO02 /dev/sdo1
Device "/dev/sdo1" is already labeled for ASM disk "RECO01"
[root@mk01n01 ~]# oracleasm createdisk RECO02 /dev/sdp1
Writing disk header: done
Instantiating disk: done
[root@mk01n01 ~]# oracleasm createdisk RECO03 /dev/sdq1
Device "/dev/sdq1" is already labeled for ASM disk "RECO02"
[root@mk01n01 ~]# oracleasm createdisk RECO03 /dev/sdr1
Writing disk header: done
Instantiating disk: done
07. 여기도 마찬가지로 중복으로 할당되었기 때문에 위와 같은 시행착오를 거쳐야 한다.
(버그가 고쳐지면 재 수정하도록 하겠다.)
[RAC 의 Storage 구성 (02.Shared Storage 구성)] 08. 제대로 생성된 asmdisk list
oracleasm createdisk OCR_VOTE01 /dev/sdb1 oracleasm createdisk OCR_VOTE02 /dev/sdd1 oracleasm createdisk OCR_VOTE03 /dev/sdf1
oracleasm createdisk DATA01 /dev/sdh1 oracleasm createdisk DATA02 /dev/sdj1 oracleasm createdisk DATA03 /dev/sdl1 oracleasm createdisk RECO01 /dev/sdn1 oracleasm createdisk RECO02 /dev/sdp1 oracleasm createdisk RECO03 /dev/sdr1 |
08. 위에서 adm disk 가 제개로 생성되었 다면 위 list 대로 생성되었을 것이다.
(high rependency 로 diskgroup을 생성하기 위해서는 최소 3개이상의 disk 가 필요하기 때문에 위와 같이 difk를 3개 씩 생성 했다.)
[RAC 의 Storage 구성 (02.Shared Storage 구성)] 09. ASM disk scandisks
-- mk01n01
[root@mk01n01 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@mk01n01 ~]# oracleasm listdisks
DATA01
DATA02
DATA03
OCR_VOTE01
OCR_VOTE02
OCR_VOTE03
RECO01
RECO02
RECO03
-- mk01n02
[root@mk01n02 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "OCR_VOTE02"
Instantiating disk "DATA02"
Instantiating disk "OCR_VOTE03"
Instantiating disk "DATA01"
Instantiating disk "OCR_VOTE01"
Instantiating disk "DATA03"
Instantiating disk "RECO01"
Instantiating disk "RECO02"
Instantiating disk "RECO03"
[root@mk01n02 ~]# oracleasm listdisks
DATA01
DATA02
DATA03
OCR_VOTE01
OCR_VOTE02
OCR_VOTE03
RECO01
RECO02
RECO03
09. 위 [RAC 의 Storage 구성 (02.Shared Storage 구성)] 07 에서 만든 asm disk 를 scandisks 한다.
그냥 oracleasm listdisks 를 하면 disk를 scan 하지 않았기 때문에 원격지 node 에서는 안보일수 있다.
따라서 모든 노드에서 한번은 scandisks 를 해줘야 한다.
이로써 모든 DB 서버에 shared disk 를 인식 시켰다.
