HACMP concurrent Vg synchronizing

On a HACMP/PowerHA cluster, with concurrent VG, if a filesystem is created outside HACMP, the cluster is "out of sync"... if we do a lsvg -l on both nodes, some mount point will be N/A on the remote node.

The online way to resynchronize is :

on the passive node :
varyoffvg VGname

on the active node :
/usr/es/sin/cluster/sbin/cl_updatevg -cspoc -g ResourceGroupName VGname
or via Cspoc menu -> HACMP Logical Volume Management -> synchronize a Shared VG definition

On the passive node :
varyonvg -n -c -P VGname

trace réseau

# tcpdump -v -i ent1 -A -w /tmp/tcpdump.out
Peut etre lu par ethereal ou par un tcpdump -r
# tcpdump host machine1 port 53
filtre sur le nom de host et le port named

distant tar

# cd /destination
# ssh "cd /origine; tar -cBf - ." | tar -xvBf -

Verify maxdata value with dump command....

# dump -X64 -o /machine/oracle/ora10203/bin/oracle

List PowerHA resources status

Command is under : /usr/es/sbin/cluster/utilities

quicky : create / extend a fs on a redhat 5 with lvm

list volume groups :
#vgs
or
#vgdisplay

create a lv :
#lvcreate -L 10G -n lvname vgname

format the lv to ext3:
#mkfs.ext3 /dev/vgname/lvname

to create the fs, need to add the correct entry in /etc/fstab

list the lvm /fs :

#lvdisplay

extend lv :
#lvextend -L +54 /dev/vg01/lvol10 /dev/sdk3

extend fs :
#resize2fs /dev/vg01/lvol10

Verify the PowerHa POL (Priority Override Location)

/usr/es/sbin/cluster/utilities/clRGinfo -p

Primary instance POL : must be blank

Resetted via Cspoc, should not be disruptive.

useful hmc commands

lspartition -dlpar <= Active <1> for partition to accept commands.



if not 1, then, on the partition :

partition :root / # /usr/sbin/rsct/bin/rmcctrl -z

partition :root / # /usr/sbin/rsct/install/bin/recfgct

0513-071 The ctcas Subsystem has been added.

partition :root / # /usr/sbin/rsct/bin/rmcctrl -s

0513-071 The ctrmc Subsystem has been added.

0513-059 The ctrmc Subsystem has been started.

Subsystem PID is 401430.

partition :root / # /usr/sbin/rsct/bin/rmcctrl -p





Migrate lpar (live mobility) :

migrlpar -o m -m P570origine -t P520destination -p Partitionabasculer



Migrate lpar (from /to MSP):

migrlpar -o v -m srcSystem -t destSystem -p myLPAR –I source_msp_id=2,dest_msp_name=S2_VIOS2

List all the systems :

lssyscfg -r sys -F name

List the scsi devices defined for a partition :

lssyscfg -r prof -m P570 --filter "lpar_names=darecb1" -F lpar_id,virtual_scsi_adapters

List all the defined scsi devices for a physical server :

lshwres -r virtualio --rsubtype scsi -m P570 --level lpar -F lpar_name,lpar_id,slot_num,remote_lpar_id,remote_slot_num --header

Problem determination PowerHA / HACMP

Collect logs : clsnap -d '/tmp' -p2 -n 'node1,node2'

or

snap -e

If cspoc pb : /tmp/cspoc.log <= more details

If Vg configuration is inconstitent between nodes :

1) Validate that all disks in a VG are know, on both nodes

node1 # lspv grep vg1

hdisk3 005a2b2a4dc045f3 vg1 active

hdisk4 005a2b2ab58a59b2 vg1 active

node 2 # lspv grep vg1

hdisk3 005a2b2a4dc045f3 vg1

hdisk4 is missing ....

if HACMP is > 5.4, and VG are not ehanced concurrent (which is not the case, since the vg is not "concurrent" but "active" then :

node 2 # lspv grep 005a2b2ab58a59b2

hdisk4 005a2b2ab58a59b2 None

2) Integrate it in the vg correctly

node1 # lqueryvg -p hdisk3 -T > /usr/es/sbin/cluster/etc/vg/vg1 <= this is to save the good timestamp for the cluster

node1 # varyonvg -ub vg1 <= From now on, NO MORE manipulation on vg1, on node 1, must occur...

node2 # importvg -L vg1 hdisk3

vg1

node2 # lqueryvg -p hdisk3 -T > /usr/es/sbin/cluster/etc/vg/vg1 <= This way, the timestamp is correct on both nodes.

node1 #varyonvg vg1 <= Things are back to normal, now.

This is the simplest way to refefine correctly the vg on backup node... But, this is when things are going smooth.... its not always that way....

First, if the pvid is not known on node 2... First, is it's zoning correctly defined ? if yes, you MUST have a disk in "none None" on your backup node. If you want it to be correctly defined on you second node, you must do a "rmdev/cfgmgr" while vg1 is in mode 'unlocked' on node 1, via the varyonvg -ub command.

If it has been known, and now, it is no more, it means you have 'phantom' disks. Some disks must be "Defined" on node 2, as others are defined in place, with no definition (None = no pvid none = no vg defined). The good way to define them correctly, is to remove the "None none" disk, and to "mkdev" the Defined one, again.

For the timestamp definition, since HACMP 5.4, the timestamp is synchronised via the clveryfy command.

Beware of the storm...

While configuring two vio server the other day, i wanted to transform both vio server SEA cards into SEA failover mode. I fell into the following trap :

If one Vio server is configured as followed :

1 virtual ethernet in vlan / pvid 1 with external network access yes, and trunk pri 1

SEA created between the adapter and this card, and, an internal adress configured on the SEA.

While configuring the other VIO server, in the same way than the first one, in order to transform it in failover mode, later, when you create the SEA adapter (with the virtual adapter on the same Vlan / pvid than the first vio server), you generate a biiig arp storm / broadcast storm, that can put your vlan, and more, down.

So, the good way to do it is to make directly the failover mode, as you create the SEA, or, to transform it before creating the second SEA on the second VIO server.

NOT : mkvdev -sea ent1 -vadapter ent4 -default ent4 -defaultid 3

but directly

mkvdev -sea ent1 -vadapter ent4 -default ent4 -defaultid 3 ha_mode=auto ctl_chan=ent3

Before, you should have the virtual adapter ent3 created, on vlan 3.

OR : if you just need to modify your existing SEA into failover mode :

chdev -dev ent3 -attr ha_mode=auto ctl_chan=ent4

there seems also to be a bypass at the switch level, which could be helpful : its the BPDU guard setting, which disables the port if bridging loop or packet storm occurs.

This is what it looks like at the end (2 differents networks for every partition)