fun with probevue...

Today, we had a fun problem... Someone, or something is modifying /tmp on an AIX server, with chmod command (from 777 + sticky to 755 ).
Nothing in .sh_history, neither bash_history, so i decided to use a simple probevue script to catch the mouse :


darkstar:root:/home/moi# cat pvchmod

int chmod(char *buf, unsigned long size);
@@BEGIN
{
}

@@syscallx:*:chmod:entry
{
        __auto String arg[128];
        arg = get_userstring(__arg1, 128);
        if (arg == "/tmp")
        {
                printf("%s sur %s ,pid => %d, ppid => %d , uid => %d \n",__pname,arg,__pid,__ppid, __uid) ;
                exit();
        }
}

simple command to send a mail, when mouse is in the box :

darkstar:root:/home/moi# probevue pvchmod | mail -s moi@gmail.com

and from another screen, i tryed to test the trap.

darkstar:root:/home/moi# chmod 755 /tmp
darkstar:root:/home/moi#

it, then, triggered the following mail :

chmod sur /tmp ,pid => 54460624, ppid => 3080846 , uid => 0

et voila !

modify sharepool attributes on command line

to show resources of sharepools :

lshwres -r procpool -m Managed_System

to change max cpu on command line, for a given sharepool :

 chhwres -r procpool -m Managed_System -o s --poolname  Sharepool -a "max_pool_proc_units=1"

Remove tabs and spaces before # with sed

In order to remove leading spaces and tabs on Linux, i use following standard sed command :

cat aaa |sed 's/[ \t]*#/#/g'

but.... on AIX \t doesn't work ...

In order to run it on AIX, replace \t by Ctrl+v and tab (key tab)

cat aaaa |sed 's/^[ ]*#/#/g'

 

probevue to trace accesses to a file

Using probevue to trace who is accessing a file :

A lot of debugging can be done with truss, in order to see what system calls are called by programs.

While it can be a real time saver, it can only run on a pid or a specific command.

What if you want to see, who is doing that particular system call ?

The answer is probevue.

Probevue is a useful dynamic  tracing command, which can be used to trace specific system calls, and it allows to do some code, in order to get specific info.

I have used it for some time, now, and it helped a lot for debugging purposes.

In this example, I want to know, who is writing to a file, and even what it is writing :

darkstar:root:/home/moi# touch /tmp/myfile

darkstar:root:/home/moi#

this is the content of my probevue file

#!/bin/probevue

#

# PArametre : nom de fichier entre quotes : \"/tmp/tutu\"

int open(char *filename, int m, int p);

int kwrite(int fd, char *s, int size);

int write(int fd, char *s, int size);

@@BEGIN

{

        printf ("\n ==> Surveillance des ecritures sur %s <== \n",$1);

}

@@syscall:*:open:entry

{

        __auto String filename[256];

        filename = get_userstring(__arg1, -1);

        if (filename == $1)

        {

                printf("\n (%s) ouverture de %s par %s (pid: %d) \n",get_function(),filename,__pname,__pid);

                thread:open = 1;

        }

}

@@syscall:*:open:exit

when (thread:open == 1)

{

        thread:fd = __rv;

        //printf(" (%s) thread:fd = %d\n",get_function(),thread:fd);

}

@@syscall:*:write:entry

when (thread:open == 1)

{

                __auto String buffer[256];

                buffer = get_userstring(__arg2, 128);

                //printf("file descriptor : %ld \n",thread:fd);

                printf(" (%s) programme %s (%d) a ecrit ",get_function(),__pname, __pid);

                printf(" :  %s ",buffer);

                thread:write = 1;

                thread:open = 0;

}

@@syscall:*:write:exit

when (thread:write ==  1 )

{

                printf(" (%s) rc = %d \n",get_function(),__errno);

                thread:write = 0;

}

Execute it :

darkstar:root:/home/moi# ./pvwrite3 \"/tmp/myfile\"

 ==> Surveillance des ecritures sur /tmp/myfile <==

And try access in another window :

darkstar:root:/home/moi# echo AAAAAAAAAAAAAAAAAASSSSSDDDDDSQDaaa > /tmp/myfile

darkstar:root:/home/moi#

the result should be, in the first window :

darkstar:root:/home/moi# ./pvwrite3 \"/tmp/myfile\"

 ==> Surveillance des ecritures sur /tmp/myfile <==

 (kopen) ouverture de /tmp/myfile par ksh (pid: 53870698)

 (kwrite) programme ksh (53870698) a ecrit  :  AAAAAAAAAAAAAAAAAASSSSSDDDDDSQDaaa

  (kwrite) rc = 0

In order to work correctly, the system should be in AIX 6.1 TL7 minimum.  The script can work in lower AIX TL, but needs to be adapted.

I will post other useful scripts, asap.

script to open console on partition

I never remember on what frame is a partition... And i dont want to open an HMC to just launch a console...
I use this simple, small shell to open the console for me :
Before using it, you must have a valide ssh connexion (with key validation) to all hmc you need to access, and you must change the HMC variable to your HMCs...


It doesn't launch anything, it just gives you the right command to juste copy / paste. (as recommanded by the brilliant Brian Smith says, it is safer to "Write scripts that don't actually do anything").
I try to write all of my stuff like that, now.

It can be called with all or part of the server name in $1.

#!/bin/ksh
#
#       $1 : partition name
#
HMC="hmc1 hmc2 hmc3"
if [[  -z $1 ]]
then
        echo "Need a partition name"
        exit
fi
PART=$1
for i in $HMC
do
        for PSERIE in $(ssh $i -l hscroot lssyscfg -r sys -F name)
        do
                        for j in $(ssh $i -l hscroot  "lssyscfg -r lpar -m $PSERIE -Fname "| grep -i $PART)
                        do
                                ssh $i -l hscroot  "lssyscfg -r lpar -m $PSERIE -Fname "  | grep -i $PART  && echo ssh $i -l hscroot -t mkvterm -m $PSERIE  -p $j
                        done
        done
done
 

rsyslogd on aix

installing :

I installed rsyslogd based on http://www.rsyslog.com/ on several AIX servers (both AIX6.1 and 7.1). I wanted to install ibm's own compiled module for several reasons :

1. obvious security matters (rsyslogd is security related product, which will help us to have stronger means to keep traks on servers, so for that reason, we cannot install unstamped products)
2. it's an installp product, and it's integrated to srcmstr daemon.
3. we will have someone to yell on, if it doesn't work! (or maybe not!)

rsyslog is now on the expansion pack of the two main aix versions i work on. The minimum server level is :

6.1.8.0 : AIX 6.1, TL8, SP0

7.1.2.0 : AIX 7.1, TL2, SP0

After having installed rsyslogd, we have a new command, which is helpful to convert from the normal syslogd version :

syslog_ssw

syslog_ssw -c /etc/syslog.conf /etc/rsyslog.conf : will convert the normal syslog.conf into rsyslog.conf.

It will need some tuning to get into your own organisation  :
*.* @@ip_central_server:port
This will be the server that will receive all the syslog messages. you can add some parameters, according to rsyslog wiki. But a good starting poing is the following options (you have to create /var/rsyslog before launching rsyslog) :

$WorkDirectory /var/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1G # 1Gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList   # run asynchronously
$ActionResumeRetryCount -1    # infinite retries if host is down
*.* @@ip_central_server:port
Then, to replace syslog with rsyslog, you can use the command syslog_ssw with the -r option.

root@darkstar:/var/lib/rsyslog# syslog_ssw -r

0513-077 Subsystem has been changed.

Start daemon: syslogd

0513-059 The syslogd Subsystem has been started. Subsystem PID is 37027944.

As you keep the /etc/syslog.conf, you wan switch easily from rsyslog to syslog back and forth, via syslog_ssw -r and -s options.

It is started and stopped via startsrc -s syslogd, and via stopsrc -s syslogd command.

debugging :

To debug rsyslog is quite simple :

stop daemon via stopsrc command.
export the two RSYSLOG parameters :

export RSYSLOG_DEBUG="DebugOnDemand NoStdOut"
export RSYSLOG_DEBUGLOG=/tmp/gileb.out

then launch rsyslog on the command line as follow :

rsyslogd  -n -c5

then,  send the usr signal via the kill command on the running process

kill -USR1 syslogpid

you will then, have all the debug info into the indicated file  :

root/datestaix7 / #cat /tmp/gileb.out

7600.312486000:1:

7600.312524000:1:

7600.312537000:1: ********************************************************************************

7600.312550000:1: Switching debugging_on to true at 14:46:40

7600.312561000:1: ********************************************************************************

7635.540547000:203: Message from UNIX socket: #4

7635.540627000:203: MsgSetTAG in: len 3, pszBuf: ege

7635.540661000:203: MsgSetTAG exit: pMsg->iLenTAG 3, pMsg->TAG.szBuf: ege

7635.540709000:203: main Q: entry added, size now log 2, phys 3 entries

7635.540759000:203: main Q: EnqueueMsg advised worker start

7635.540821000:203: --------imuxsock calling select, active file descriptors (max 4): 4

7647.435687000:102: file netstrms.c released module 'lmnsd_ptcp', reference count now 1

7647.435722000:102: Action 20059608 transitioned to state: rtry

to stop, you can ctrl-c or kill the process.

 

apply command (in AIX only?)

Here is a fast way to do function loop : 

apply "ssh server%1_pers date" 2 3 5 6  7 8

Replaces %1 with following parameters 

same as : 

for i in 2 3 5 6 7 8 
do
     ssh server${i}_pers date
done

Exclude files or directories from commands

An easy way to remove elements from command line : 

root:/# rm !(*.c|*.h)  => remove every files except *.c or *.h) 

another syntax : 

root:/# rm !(*.@(c|h))  => same thing ...

Can be used with grep, for directories exlusion. :

root:/#  grep P770_P7_1_A30 !(www|logs)/* => grep P770_P7_1_A30 from every underlying directories, except www and logs

Be sure to be on different switchs !

It's good to have more than one fcs adapter... but it's better to be sure they are connected on different switchs !
Each id is different, if it is on another switch

in this case we have two  adapters (or 2 ports on the same card):

root:/# lsdev -c adapter | grep fcs
fcs0   Available 30-T1 Virtual Fibre Channel Client Adapter
fcs1   Available 31-T1 Virtual Fibre Channel Client Adapter

but on the same vio server ! ....

root:/# echo "vfcs" | kdb
           START              END
0000000000001000 0000000005770000 start+000FD8
F00000002FF47600 F00000002FFDF9C0 __ublock+000000
000000002FF22FF4 000000002FF22FF8 environ+000000
000000002FF22FF8 000000002FF22FFC errno+000000
F1000F0A00000000 F1000F0A10000000 pvproc+000000
F1000F0A10000000 F1000F0A18000000 pvthread+000000
read vscsi_scsi_ptrs OK, ptr = 0xF1000000C01A9380
(0)> vfcs
NAME      ADDRESS             STATE   HOST      HOST_ADAP  OPENED NUM_ACTIVE
fcs0      0xF10001019999A000  0x0008  vioprex1vfchost2  0x01    0x0000
fcs1      0xF10001019A3AC000  0x0008  vioprex1vfchost3  0x01    0x0000and on the same swicth !!!


root:/# lsattr -El fscsi0
attach       switch    How this adapter is CONNECTED         False
dyntrk       yes       Dynamic Tracking of FC Devices        True
fc_err_recov fast_fail FC Fabric Event Error RECOVERY Policy True
scsi_id      0x340206  Adapter SCSI ID                       False
sw_fc_class  3         FC Class for Fabric                   True
root:/# lsattr -El fscsi1
attach       switch    How this adapter is CONNECTED         False
dyntrk       yes       Dynamic Tracking of FC Devices        True
fc_err_recov fast_fail FC Fabric Event Error RECOVERY Policy True
scsi_id      0x340501  Adapter SCSI ID                       False
sw_fc_class  3         FC Class for Fabric                   True


this other partition is ok :

# lsattr -El fscsi0
attach       switch    How this adapter is CONNECTED         False
dyntrk       yes       Dynamic Tracking of FC Devices        True
fc_err_recov fast_fail FC Fabric Event Error RECOVERY Policy True
scsi_id      0x1db40   Adapter SCSI ID                       False
sw_fc_class  3         FC Class for Fabric                   True
# lsattr -El fscsi1
attach       switch    How this adapter is CONNECTED         False
dyntrk       yes       Dynamic Tracking of FC Devices        True
fc_err_recov fast_fail FC Fabric Event Error RECOVERY Policy True
scsi_id      0x20db40  Adapter SCSI ID                       False
sw_fc_class  3         FC Class for Fabric                   True

Merci Patrice, pour l'info !

patch alt_disk install image

create disk copy :

     alt_disk_copy -O -B -d "hdisk25 hdisk26"

wake up disk copy :

     alt_rootvg_op -W -d hdisk25

update alternate disk copy : 

     alt_rootvg_op -C -b update_all -l /updates/5300-12-04

close alternate disk copy, and bosboot it:

     alt_rootvg_op -St

how to find the underlying vio server

I need to know, from a partition, how to find which vio server it relies on :

We need to use kdb  : 

on Npiv servers

root @test #  echo "vfcs" | kdb
           START              END
0000000000001000 0000000004090000 start+000FD8
F00000002FF47600 F00000002FFDF9C0 __ublock+000000
000000002FF22FF4 000000002FF22FF8 environ+000000
000000002FF22FF8 000000002FF22FFC errno+000000
F1000F0A00000000 F1000F0A10000000 pvproc+000000
F1000F0A10000000 F1000F0A18000000 pvthread+000000
read vscsi_scsi_ptrs OK, ptr = 0x0
(0)> vfcs
NAME      ADDRESS             STATE   HOST      HOST_ADAP  OPENED NUM_ACTIVE
fcs0      0xF1000A0000156000  0x0008  vioserver1vfchost0  0x01    0x0000
root @test # 

on vscsi servers

root @test #  echo "cvai" | kdb | grep vscsi

Preserving 30612 bytes of symbol table [./usr/lib/drivers/vscsi_initdd]

read vscsi_scsi_ptrs OK, ptr = 0xF1000000A0138378

vscsi0     0x000007 0x0000000000 0x0                vio73->vhost7

vscsi1     0x000007 0x0000000000 0x0                vio74->vhost7

Some stuff....

SED : (merci nico)

to remove, with a sed command, every thing before the dot :

echo " aaa.bbb" | sed 's/^.*\.//g'
bbb

^ = from the line beginning
. = one caracter

.* (dot with *) every caracters


Python : 

I have the following file, which contains server names, and dates.

root@test # cat file
server_a
09192012
09292012
09302012
server_b
 04022012
04082012
04262012
10072012
...

The date is american format, so i need to convert it in europeen format, so
reverse chain 2 by 2  (convert date from mmddyyyy to ddmmyyyy)

the simplest python command to do that is :

>>> x=06242012 
>>>x=x[2:4:]+x[0:2:]+x[4:8::]
>>> x
'24062012'

As i need to use it on the command line, to process a verrryy large file, i built a simple python file :

root@test #cat conv.py

#!/bin/python
# -*-coding:utf-8 -*

import sys
for line in sys.stdin:
        line=line.strip("\r\n") # need to remove trailing \n because my var is not numbers if there is a \n
        if line.isdigit():          # we only rework lines that are numbers
                line=line[2:4:]+line[0:2:]+line[4:8:]
        print line

root@test #cat file | ./conv.py 

server_a
19092012
29092012
30092012
server_b
22022012
18042012
26042012
07102012
...
 job's done...

same stuff with perl (one liner!) (merci Guigui)

root@test #echo 09192012 | perl -ne "s/(\d\d)(\d\d)(\d\d\d\d)/\2\/\1\/\3/;print"
19/09/2012

How to find the devices in pre defined subclass

root @ test # lsdev -P -H

PCM            iscsiother      friend         SCSI/FCP Disk Path Control Module
PCM            sasother        friend         SCSI/FCP Disk Path Control Module
PCM            scsiscsd        friend         SCSI/FCP Disk Path Control Module
PCM            sisarray        friend         SCSI/FCP Disk Path Control Module
adapter        df1000fa        pci            FC Adapter
adapter        df1000fd        pci            FC Adapter
adapter        df1000f114108a0 pciex          8Gb PCI Express Dual Port FC Adapter (df1000f114108a03)
cdrom          oscd            fcp            Other FC SCSI CD-ROM Drive
 

TTL et cache DNS

Si on interroge un dns cache, et qu'aucun des primaires ne connait l'adresse demandée, il y aura un TTL avant de pouvoir vraiment redécouvrir l'adresse, une fois que celle ci sera connue dans un DNS. Ce TTL se déclenche à la premiere interrogation (ici il est de 3600 secondes):

darkstar:root:# dig +nocmd toto.titi.zzz.fr  +answer

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 1094

;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:

;toto.titi.zzz.fr.          IN      A

;; AUTHORITY SECTION:

Titi.zzz.fr.              3600    IN      SOA     toto.titi.zzz.fr. admin. 2913617 900 600 86400 3600

;; Query time: 14 msec

;; SERVER: 192.168.0.244#53(192.168.0.244)

;; WHEN: Tue Mar 20 07:21:54 2012

;; MSG SIZE  rcvd: 99

darkstar:root:# dig +nocmd toto.titi.zzz.fr  +answer

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 1094

;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:

;toto.titi.zzz.fr.          IN      A

;; AUTHORITY SECTION:

Titi.zzz.fr.              3448    IN      SOA     toto.titi.zzz.fr. admin. 2913617 900 600 86400 3600

;; Query time: 14 msec

;; SERVER: 192.168.0.244#53(192.168.0.244)

;; WHEN: Tue Mar 20 07:21:54 2012

;; MSG SIZE  rcvd: 99

Formatter la sortie en ksh

Pour avoir des chiffres sur 3 digits en shell, il suffit de formatter la variable via un typeset : 

let a=1

typeset -RZ3 variable

echo "a=$a"

a=001

tcpdump with headers and datas

tcpdump -s 0 -vX

sed -i doesn't work on AIX

To change a file on only one command  :

perl -p -i -e "s/iiii/uuu/" titi

[POWERHA + AIX ] info : clstat ne fonctionne pas

 

Si clstat ne fonctionne pas sur un cluster aix, et que snmpd est correctement configuré comme noté dans le readme,  il reste la commande cldump pour connaître l’état du cluster :

monserveur1(backup de monserveur1):root:/usr/es/sbin/cluster# /usr/es/sbin/cluster/clstat -o

Failed retrieving cluster information.

There are a number of possible causes:

clinfoES or snmpd subsystems are not active.

snmp is unresponsive.

snmp is not configured correctly.

Cluster services are not active on any nodes.

Refer to the HACMP Administration Guide for more information.

Additional information for verifying the SNMP configuration on AIX 6

can be found in /usr/es/sbin/cluster/README5.5.0.UPDATE

 

La commande suivante doit fonctionner si snmpd est bien configuré :

 

monserveur1(backup de monserveur1):root:/usr/es/sbin/cluster# snmpinfo -m dump -v -o /usr/es/sbin/cluster/hacmp.defs

ibm.2.1.1.1.0 = 51218

clusterId.0 = 1281957358

clusterName.0 = "MONCLUSTER"

clusterConfiguration.0 = ""

Si Clstat et cldump ne fonctionnent toujours pas, il faut faire les manipulations suivantes :

Editer le fichier /etc/snmpdv3.conf

Aller à la fin du fichier et modifier les lignes suivantes

smux 1.3.6.1.4.1.2.3.1.2.1.2 gated_password # gated

smux 1.3.6.1.4.1.2.3.1.2.1.5 clsmuxpd_password # HACMP/ES for AIX clsmuxpd

en

smux 1.3.6.1.4.1.2.3.1.2.1.2 gated_password

smux 1.3.6.1.4.1.2.3.1.2.1.5 clsmuxpd_password

en clair : retirer les commentaries

 

D’abord, valider que la version utilisée de snmp, est bien une version v3 non encryptée :

 

ls -ail /usr/sbin/snmpd

lrwxrwxrwx 1 root system 19 Aug 14 14:43 /usr/sbin/snmpd@ -> /usr/sbin/snmpdv3ne*

 

Si, non passer en ssl v3 non encrypté, par la commande suivante :

/usr/sbin/snmp3_ssw -n

Ensuite, il faut commenter la ligne (si vous n’utilisez pas ip v6 sur le serveur) la ligne suivante dans /etc/hosts :

::1                    loopback localhost      # IPv6 loopback (lo0) name/address

 En

#::1                    loopback localhost      # IPv6 loopback (lo0) name/address

Passer enfin toutes les commandes suivantes pour relancer tous les processus concernés (ça peut etre fait à chaud)

stopsrc -s aixmibd

stopsrc -s snmpmibd

stopsrc -s hostmibd

stopsrc -s snmpd

sleep 10

startsrc -s snmpd

startsrc -s hostmibd

startsrc -s snmpmibd

startsrc -s aixmibd

sleep 10

startsrc -s clinfoES

sleep 60

lssrc -ls clstrmgrES

refresh -s clstrmgrES

attendre quelques secondes, et tenter le cldump/clstat.