安装Veritas Cluster Server 4.0

硬件简述：
1. Node1: Netra 20 (2 X UltraSPARC-III+, 2048M RAM, 72G*2 HardDisk)
2. Node2: Netra 20 (2 X UltraSPARC-III+, 2048M RAM, 72G*2 HardDisk)
3. Shared Storage: D1000 (36G*3 HardDisk) 一． 安装 操作系统
安装2/04版本（2004年2月份）的Solaris 8。在安装过程中需要选择英文作为主要语言。

二． 安装EIS-CD
安装EIS-CD 2/04版本，EIS-CD用于设置使用cluster的环境变量。

三． 安装patch
为了避免CPU虚高的问题，需要安装117000-05补丁。该补丁可以从SUN公司官方网站下载。下载该补丁以后解压将生成117000-05目录。使用如下命令安装patch：
patchadd 117000-05

四． 安装共享磁盘
在本次环境中，我们使用SUND1000盘阵作为共享磁盘，是SCSI接口的盘阵，如果是光纤接口的盘阵需要另行设置。
1． 给盘阵加电
2． 将Node1和盘阵用SCSI连线连接
3． Node1加电
4． 使Node1进入ok模式（在console窗口监控Node1的启动，当出现启动信息时，迅速按下Ctrl+Break，即可进入ok模式）
5． {0} ok probe-scsi-all
6． {0} ok boot ?Cr
7． 重新启动以后进入操作系统，使用format命令确认盘阵已经被此系统加载
8． Node1断电，Node2加电，重复4-7步，确认盘阵也可以被此台机器加载
9． 为了使两台机器同时读取共享存储，需要修改其中一台机器的SCSI ID。给Node1加电，进入ok模式（此时的状态应该是Node1，Node2，存储均已加电，Node2目前使用format命令已经可以观察到盘阵被正常加载，Node1处于ok模式）。
10． 设置Node1的SCSI ID为5（默认为7）
{0} ok setenv scsi-initiator-id 5
11． 通过第5步中的屏幕信息我们可以知道该SCSI设备的系统表示符为/pci@8,700000/scsi@2,1，这个标识符将在下一步使用
12． 设置Node1的SCSI ID为5。(注意第2行的输入中" scsi-initiator-id"，双引号之后有一个空格！)
{0} ok nvedit
0: probe-all
1: cd /pci@8,700000/scsi@2,1
2: 5 " scsi-initiator-id" integer-property
3: device-end
4: install-console
5: banner
13． 保存第12步的配置
{0} ok nvstore
14． 设置环境变量
{0} ok setenv use-nvramrc? true
use-nvramrc? = true
{0} ok setenv auto-boot? true
auto-boot? = true
15． 重新启动Node1
{0} ok reset-all
16． 在Node1，Node2上均用format命令检查，确认都成功挂载了共享存储

五． 安装VCS
1． 设置rhosts文件，在两台机器上都要执行，以下以Node1为例。
在/目录下添加rhosts文件，使远程登陆生效，用以简便地在两台机器上同时安装VCS
root@uulab-s22 # echo “+” > /.rhosts
root@uulab-s22 # more /.rhosts
+

2． 设置多个网卡IP
由于Cluster系统中的心跳网卡最好是一个独立的网络设备，所以我们在两台机器上设置另外一个网卡和IP地址，专门用于心跳设备。在两台机器上都要执行，以下以Node1为例。
root@uulab-s22 # echo "uulab-p22" > /etc/hostname.qfe0
root@uulab-s22 # touch /etc/notrouter
root@uulab-s22 # vi /etc/hosts
添加如下行:
192.168.0.6 uulab-p22
192.168.0.8 uulab-p23
root@uulab-s22 # vi /etc/netmasks
添加如下行:
192.168.0.0 255.255.255.0
root@uulab-s22 # sync
root@uulab-s22 # reboot

3． 开始安装
VCS 4.0的安装已经可以简化到只使用一个命令，将同时安装Veritas Volume Manager 4.0, Veritas File System 4.0，Veritas Cluster Server 4.0以及其他一些相关 软件。以下命令只需要在一个节点中执行即可。
root@uulab-s22 # cd /opt/sf_ha.4.0.sol/storage_foundation
root@uulab-s22 # ./installsf
安装过程一路都是选择或者按照提示输入，总的来说比较简单，不再赘述。

六． 创建磁盘组
VCS安装完毕以后，将要求使用如下命令重新启动所有节点：
shutdown -y -i6 -g0

重启完毕，开始创建磁盘组。以下命令只需要在一个节点中执行即可。
使用format命令确认我们需要添加到磁盘组中的共享磁盘为c4t0d0，c4t8d0，c4t9d0
root@uulab-s22 # format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
0. c1t0d0
/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100000c50569190,0
1. c1t1d0
/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100000c5056c1a7,0
2. c4t0d0
/pci@8,700000/scsi@2,1/sd@0,0
3. c4t8d0
/pci@8,700000/scsi@2,1/sd@8,0
4. c4t9d0
/pci@8,700000/scsi@2,1/sd@9,0
Specify disk (enter its number):

root@uulab-s22 # vxdisksetup -i c4t0d0
如果此命令执行以后报错如下，那么参看附录中的解决方法1。
VxVM vxdisksetup ERROR V-5-2-3535 c4t0d0s2: Invalid dmpnodename for disk device c4t0d0.
如果此命令执行以后报错如下，那么参看附录中的解决方法2。
VxVM vxdisksetup ERROR V-5-2-1813 c4t0d0: Disk is part of ipasdg disk group, use -f option to force setup.

root@uulab-s22 # vxdisksetup -i c4t8d0
root@uulab-s22 # vxdisksetup -i c4t9d0

root@uulab-s22 # vxdg init hlrdg hlrdg-01=c4t0d0
root@uulab-s22 # vxdg -g hlrdg adddisk hlrdg-02=c4t8d0
root@uulab-s22 # vxdg -g hlrdg adddisk hlrdg-03=c4t9d0

七． 创建卷
root@uulab-s22 # vxassist -g hlrdg -b make oradata_vol 15g layout=nostripe,nolog nmirror=2 &
root@uulab-s22 # vxassist -g hlrdg -b make oraredo_vol 5g layout=nostripe,nolog nmirror=2 &
root@uulab-s22 # vxassist -g hlrdg -b make oraarch_vol 8g layout=nostripe,nolog nmirror=2 &
root@uulab-s22 # vxassist -g hlrdg -b make hlr_vol 4g layout=nostripe,nolog nmirror=2 &

八． 使用VxFS创建文件系统
root@uulab-s22 # mkfs -F vxfs -o bsize=8192,largefiles /dev/vx/rdsk/hlrdg/oradata_vol
root@uulab-s22 # mkfs -F vxfs -o bsize=8192,largefiles /dev/vx/rdsk/hlrdg/oraredo_vol
root@uulab-s22 # mkfs -F vxfs -o bsize=8192,largefiles /dev/vx/rdsk/hlrdg/oraarch_vol
root@uulab-s22 # mkfs -F vxfs -o bsize=8192,largefiles /dev/vx/rdsk/hlrdg/hlr_vol

九． 配置VCS
我们的环境是创建了4个文件系统，分别是oradata_vol, oraredo-vol, oraarch_vol, hlr_vol，文件系统可以被使用需要挂载（mount）到相应的目录中。先创建相应的目录，在两台机器上都要执行，以下以Node1为例。（假设系统中dba组和oracle用户都已经创建）
root@uulab-s22 # mkdir -p /opt/oracle/data
root@uulab-s22 # mkdir -p /opt/oracle/redo
root@uulab-s22 # mkdir -p /opt/oracle/arch
root@uulab-s22 # mkdir ?Cp /opt/hlr

root@uulab-s22 # chown oracle:dba /dev/vx/rdsk/hlrdg/oradata_vol
root@uulab-s22 # chown oracle:dba /dev/vx/rdsk/hlrdg/oraredo_vol
root@uulab-s22 # chown oracle:dba /dev/vx/rdsk/hlrdg/oraarch_vol

root@uulab-s22 # chown oracle:dba /opt/oracle/data
root@uulab-s22 # chown oracle:dba /opt/oracle/redo
root@uulab-s22 # chown oracle:dba /opt/oracle/arch

修改/etc/VRTSvcs/conf/config/main.cf文件，这是VCS的配置文件，该文件的修改可以使用命令行（比如hagrp,hares等命令）修改，也可以使用任何文本编辑器（比如vi）直接修改，此处我们选择使用vi进行修改。粗体字部分为需要新添加的行。

修改完毕以后该文件如下：
include "types.cf"

cluster vcs_hlr_cluster (
UserNames = { admin = hijBidIfjEjjHrjDig }
ClusterAddress = "10.7.1.7" --此处为Cluster环境的虚拟IP
Administrators = { admin }
CounterInterval = 5
)

system uulab-s22 (
)

system uulab-s23 (
)

group ClusterService (
SystemList = { uulab-s22 = 0, uulab-s23 = 1 }
UserStrGlobal = "LocalCluster@https://10.7.1.7:8443;LocalCluster@https://10.7.1.7:8443;"
AutoStartList = { uulab-s22, uulab-s23 }
OnlineRetryLimit = 3
OnlineRetryInterval = 120
)

DiskGroup hlrdg (
DiskGroup = hlrdg
MonitorReservation = 1
)

IP webip (
Device = eri0
Address = "10.7.1.7"
NetMask = "255.255.0.0"
)

Mount arch_mnt (
MountPoint = "/opt/oracle/arch"
BlockDevice = "/dev/vx/dsk/hlrdg/oraarch_vol"
FSType = vxfs
FsckOpt = "-y"
)

Mount data_mnt (
MountPoint = "/opt/oracle/data"
BlockDevice = "/dev/vx/dsk/hlrdg/oradata_vol"
FSType = vxfs
FsckOpt = "-y"
)

Mount hlr_mnt (
MountPoint = "/opt/hlr"
BlockDevice = "/dev/vx/dsk/hlrdg/hlr_vol"
FSType = vxfs
FsckOpt = "-y"
)

Mount redo_mnt (
MountPoint = "/opt/oracle/redo"
BlockDevice = "/dev/vx/dsk/hlrdg/oraredo_vol"
FSType = vxfs
FsckOpt = "-y"
)

NIC csgnic (
Device = eri0
)

VRTSWebApp VCSweb (
Critical = 0
AppName = vcs
InstallDir = "/opt/VRTSweb/VERITAS"
TimeForOnline = 5
RestartLimit = 3
)

VCSweb requires webip
arch_mnt requires hlr_mnt
data_mnt requires redo_mnt
hlr_mnt requires hlrdg
redo_mnt requires arch_mnt
webip requires csgnic

修改完毕，使用如下命令进行语法检查：
root@uulab-s22 # hacf -verify /etc/VRTSvcs/conf/config

十． 测试VCS
配置完成以后，同时重新启动两台机器，启动完毕，使用如下命令检查VCS是否运行正常。
root@uulab-s22 # hares -display -group ClusterService
正常情况应该如下显示：
#Resource Attribute System Value
VCSweb Group global ClusterService
VCSweb Type global VRTSWebApp
VCSweb AutoStart global 1
VCSweb Critical global 0
VCSweb Enabled global 1
VCSweb LastOnline global uulab-s22
VCSweb MonitorOnly global 0
VCSweb ResourceOwner global unknown
VCSweb TriggerEvent global 0
VCSweb ArgListValues uulab-s22 vcs /opt/VRTSweb/VERITAS 5
VCSweb ArgListValues uulab-s23 vcs /opt/VRTSweb/VERITAS 5
VCSweb ConfidenceLevel uulab-s22 100
VCSweb ConfidenceLevel uulab-s23 0
VCSweb Flags uulab-s22
VCSweb Flags uulab-s23
VCSweb IState uulab-s22 not waiting
VCSweb IState uulab-s23 not waiting
VCSweb Probed uulab-s22 1
VCSweb Probed uulab-s23 1
VCSweb Start uulab-s22 1
VCSweb Start uulab-s23 0
VCSweb State uulab-s22 ONLINE
VCSweb State uulab-s23 OFFLINE
VCSweb AppName global vcs
VCSweb ComputeStats global 0
VCSweb InstallDir global /opt/VRTSweb/VERITAS
VCSweb ResourceInfo global State Valid Msg TS
VCSweb RestartLimit global 3
VCSweb TimeForOnline global 5
VCSweb MonitorTimeStats uulab-s22 Avg 0 TS
VCSweb MonitorTimeStats uulab-s23 Avg 0 TS
#
…………

也可以使用df命令检查是否所有需要挂载的文件系统都已正确挂载在Node1上，而在Node2上则无法看到。

做切换测试：
root@uulab-s22 # hagrp -switch ClusterService -to uulab-s23
正常情况大概是5秒左右，所有的资源包括IP，文件系统加载点都会转移到Node2上。


十一． 附录
解决方法1：
root@uulab-s22 # vxdiskadm


Volume Manager Support Operations
Menu: VolumeManager/Disk

1 Add or initialize one or more disks
2 Encapsulate one or more disks
3 Remove a disk
4 Remove a disk for replacement
5 Replace a failed or removed disk
6 Mirror volumes on a disk
7 Move volumes from a disk
8 Enable access to (import) a disk group
9 Remove access to (deport) a disk group
10 Enable (online) a disk device
11 Disable (offline) a disk device
12 Mark a disk as a spare for a disk group
13 Turn off the spare flag on a disk
14 Unrelocate subdisks back to a disk
15 Exclude a disk from hot-relocation use
16 Make a disk available for hot-relocation use
17 Prevent multipathing/Suppress devices from VxVM's view
18 Allow multipathing/Unsuppress devices from VxVM's view
19 List currently suppressed/non-multipathed devices
20 Change the disk naming scheme
21 Get the newly connected/zoned disks in VxVM view
22 Change/Display the default disk layouts
23 Mark a disk as allocator-reserved for a disk group
24 Turn off the allocator-reserved flag on a disk
list List disk information


? Display help about menu
?? Display help about the menuing system
q Exit from menus

Select an operation to perform: 17


Exclude Devices
Menu: VolumeManager/Disk/ExcludeDevices
VxVM INFO V-5-2-1239
This operation might lead to some devices being suppressed from VxVM's view
or prevent them from being multipathed by vxdmp (This operation can be
reversed using the vxdiskadm command).

Do you want to continue ? [y,n,q,?] (default: y) y

Volume Manager Device Operations
Menu: VolumeManager/Disk/ExcludeDevices

1 Suppress all paths through a controller from VxVM's view
2 Suppress a path from VxVM's view
3 Suppress disks from VxVM's view by specifying a VID:PID combination
4 Suppress all but one paths to a disk
5 Prevent multipathing of all disks on a controller by VxVM
6 Prevent multipathing of a disk by VxVM
7 Prevent multipathing of disks by specifying a VID:PID combination
8 List currently suppressed/non-multipathed devices

? Display help about menu
?? Display help about the menuing system
q Exit from menus

Select an operation to perform: 5

Exclude controllers from DMP
Menu: VolumeManager/Disk/ExcludeDevices/CTLR-DMP
Use this operation to exclude all disks on a controller from being multipathed
by vxdmp.

As a result of this operation, all disks having a path through the specified
controller will be claimed in the OTHER_DISKS category and hence, not
multipathed by vxdmp. This operation can be reversed using the vxdiskadm
command.
VxVM INFO V-5-2-1263
You can specify a controller name at the prompt. A controller name is of
the form c#, example c3, c11 etc. Enter 'all' to exclude all paths on all
the controllers on the host. To see the list of controllers on the system,
type 'list'.

Enter a controller name [,all,list,list-exclude,q,?] c4
VxVM INFO V-5-2-1129
All disks on the following enclosures will be excluded from DMP ( ie
claimed in the OTHER_DISKS category and hence not multipathed by vxdmp) as a
result of this operation :


Disk OTHER_DISKS


Continue operation? [y,n,q,?] (default: y) y

Do you wish to exclude more controllers ? [y,n,q,?] (default: n) n

Volume Manager Device Operations
Menu: VolumeManager/Disk/ExcludeDevices

1 Suppress all paths through a controller from VxVM's view
2 Suppress a path from VxVM's view
3 Suppress disks from VxVM's view by specifying a VID:PID combination
4 Suppress all but one paths to a disk
5 Prevent multipathing of all disks on a controller by VxVM
6 Prevent multipathing of a disk by VxVM
7 Prevent multipathing of disks by specifying a VID:PID combination
8 List currently suppressed/non-multipathed devices

? Display help about menu
?? Display help about the menuing system
q Exit from menus

Select an operation to perform: q

VxVM vxdiskadm NOTICE V-5-2-1187 Please wait while the device suppression/unsuppression operations take effect.