fun with probevue...

Today, we had a fun problem... Someone, or something is modifying /tmp on an AIX server, with chmod command (from 777 + sticky to 755 ).
Nothing in .sh_history, neither bash_history, so i decided to use a simple probevue script to catch the mouse :


darkstar:root:/home/moi# cat pvchmod

int chmod(char *buf, unsigned long size);
@@BEGIN
{
}

@@syscallx:*:chmod:entry
{
        __auto String arg[128];
        arg = get_userstring(__arg1, 128);
        if (arg == "/tmp")
        {
                printf("%s sur %s ,pid => %d, ppid => %d , uid => %d \n",__pname,arg,__pid,__ppid, __uid) ;
                exit();
        }
}

simple command to send a mail, when mouse is in the box :

darkstar:root:/home/moi# probevue pvchmod | mail -s moi@gmail.com

and from another screen, i tryed to test the trap.

darkstar:root:/home/moi# chmod 755 /tmp
darkstar:root:/home/moi#

it, then, triggered the following mail :

chmod sur /tmp ,pid => 54460624, ppid => 3080846 , uid => 0

et voila !

modify sharepool attributes on command line

to show resources of sharepools :

lshwres -r procpool -m Managed_System

to change max cpu on command line, for a given sharepool :

 chhwres -r procpool -m Managed_System -o s --poolname  Sharepool -a "max_pool_proc_units=1"

Remove tabs and spaces before # with sed

In order to remove leading spaces and tabs on Linux, i use following standard sed command :

cat aaa |sed 's/[ \t]*#/#/g'

but.... on AIX \t doesn't work ...

In order to run it on AIX, replace \t by Ctrl+v and tab (key tab)

cat aaaa |sed 's/^[ ]*#/#/g'

 

probevue to trace accesses to a file

Using probevue to trace who is accessing a file :

A lot of debugging can be done with truss, in order to see what system calls are called by programs.

While it can be a real time saver, it can only run on a pid or a specific command.

What if you want to see, who is doing that particular system call ?

The answer is probevue.

Probevue is a useful dynamic  tracing command, which can be used to trace specific system calls, and it allows to do some code, in order to get specific info.

I have used it for some time, now, and it helped a lot for debugging purposes.

In this example, I want to know, who is writing to a file, and even what it is writing :

darkstar:root:/home/moi# touch /tmp/myfile

darkstar:root:/home/moi#

this is the content of my probevue file

#!/bin/probevue

#

# PArametre : nom de fichier entre quotes : \"/tmp/tutu\"

int open(char *filename, int m, int p);

int kwrite(int fd, char *s, int size);

int write(int fd, char *s, int size);

@@BEGIN

{

        printf ("\n ==> Surveillance des ecritures sur %s <== \n",$1);

}

@@syscall:*:open:entry

{

        __auto String filename[256];

        filename = get_userstring(__arg1, -1);

        if (filename == $1)

        {

                printf("\n (%s) ouverture de %s par %s (pid: %d) \n",get_function(),filename,__pname,__pid);

                thread:open = 1;

        }

}

@@syscall:*:open:exit

when (thread:open == 1)

{

        thread:fd = __rv;

        //printf(" (%s) thread:fd = %d\n",get_function(),thread:fd);

}

@@syscall:*:write:entry

when (thread:open == 1)

{

                __auto String buffer[256];

                buffer = get_userstring(__arg2, 128);

                //printf("file descriptor : %ld \n",thread:fd);

                printf(" (%s) programme %s (%d) a ecrit ",get_function(),__pname, __pid);

                printf(" :  %s ",buffer);

                thread:write = 1;

                thread:open = 0;

}

@@syscall:*:write:exit

when (thread:write ==  1 )

{

                printf(" (%s) rc = %d \n",get_function(),__errno);

                thread:write = 0;

}

Execute it :

darkstar:root:/home/moi# ./pvwrite3 \"/tmp/myfile\"

 ==> Surveillance des ecritures sur /tmp/myfile <==

And try access in another window :

darkstar:root:/home/moi# echo AAAAAAAAAAAAAAAAAASSSSSDDDDDSQDaaa > /tmp/myfile

darkstar:root:/home/moi#

the result should be, in the first window :

darkstar:root:/home/moi# ./pvwrite3 \"/tmp/myfile\"

 ==> Surveillance des ecritures sur /tmp/myfile <==

 (kopen) ouverture de /tmp/myfile par ksh (pid: 53870698)

 (kwrite) programme ksh (53870698) a ecrit  :  AAAAAAAAAAAAAAAAAASSSSSDDDDDSQDaaa

  (kwrite) rc = 0

In order to work correctly, the system should be in AIX 6.1 TL7 minimum.  The script can work in lower AIX TL, but needs to be adapted.

I will post other useful scripts, asap.

script to open console on partition

I never remember on what frame is a partition... And i dont want to open an HMC to just launch a console...
I use this simple, small shell to open the console for me :
Before using it, you must have a valide ssh connexion (with key validation) to all hmc you need to access, and you must change the HMC variable to your HMCs...


It doesn't launch anything, it just gives you the right command to juste copy / paste. (as recommanded by the brilliant Brian Smith says, it is safer to "Write scripts that don't actually do anything").
I try to write all of my stuff like that, now.

It can be called with all or part of the server name in $1.

#!/bin/ksh
#
#       $1 : partition name
#
HMC="hmc1 hmc2 hmc3"
if [[  -z $1 ]]
then
        echo "Need a partition name"
        exit
fi
PART=$1
for i in $HMC
do
        for PSERIE in $(ssh $i -l hscroot lssyscfg -r sys -F name)
        do
                        for j in $(ssh $i -l hscroot  "lssyscfg -r lpar -m $PSERIE -Fname "| grep -i $PART)
                        do
                                ssh $i -l hscroot  "lssyscfg -r lpar -m $PSERIE -Fname "  | grep -i $PART  && echo ssh $i -l hscroot -t mkvterm -m $PSERIE  -p $j
                        done
        done
done
 

rsyslogd on aix

installing :

I installed rsyslogd based on http://www.rsyslog.com/ on several AIX servers (both AIX6.1 and 7.1). I wanted to install ibm's own compiled module for several reasons :

1. obvious security matters (rsyslogd is security related product, which will help us to have stronger means to keep traks on servers, so for that reason, we cannot install unstamped products)
2. it's an installp product, and it's integrated to srcmstr daemon.
3. we will have someone to yell on, if it doesn't work! (or maybe not!)

rsyslog is now on the expansion pack of the two main aix versions i work on. The minimum server level is :

6.1.8.0 : AIX 6.1, TL8, SP0

7.1.2.0 : AIX 7.1, TL2, SP0

After having installed rsyslogd, we have a new command, which is helpful to convert from the normal syslogd version :

syslog_ssw

syslog_ssw -c /etc/syslog.conf /etc/rsyslog.conf : will convert the normal syslog.conf into rsyslog.conf.

It will need some tuning to get into your own organisation  :
*.* @@ip_central_server:port
This will be the server that will receive all the syslog messages. you can add some parameters, according to rsyslog wiki. But a good starting poing is the following options (you have to create /var/rsyslog before launching rsyslog) :

$WorkDirectory /var/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1G # 1Gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList   # run asynchronously
$ActionResumeRetryCount -1    # infinite retries if host is down
*.* @@ip_central_server:port
Then, to replace syslog with rsyslog, you can use the command syslog_ssw with the -r option.

root@darkstar:/var/lib/rsyslog# syslog_ssw -r

0513-077 Subsystem has been changed.

Start daemon: syslogd

0513-059 The syslogd Subsystem has been started. Subsystem PID is 37027944.

As you keep the /etc/syslog.conf, you wan switch easily from rsyslog to syslog back and forth, via syslog_ssw -r and -s options.

It is started and stopped via startsrc -s syslogd, and via stopsrc -s syslogd command.

debugging :

To debug rsyslog is quite simple :

stop daemon via stopsrc command.
export the two RSYSLOG parameters :

export RSYSLOG_DEBUG="DebugOnDemand NoStdOut"
export RSYSLOG_DEBUGLOG=/tmp/gileb.out

then launch rsyslog on the command line as follow :

rsyslogd  -n -c5

then,  send the usr signal via the kill command on the running process

kill -USR1 syslogpid

you will then, have all the debug info into the indicated file  :

root/datestaix7 / #cat /tmp/gileb.out

7600.312486000:1:

7600.312524000:1:

7600.312537000:1: ********************************************************************************

7600.312550000:1: Switching debugging_on to true at 14:46:40

7600.312561000:1: ********************************************************************************

7635.540547000:203: Message from UNIX socket: #4

7635.540627000:203: MsgSetTAG in: len 3, pszBuf: ege

7635.540661000:203: MsgSetTAG exit: pMsg->iLenTAG 3, pMsg->TAG.szBuf: ege

7635.540709000:203: main Q: entry added, size now log 2, phys 3 entries

7635.540759000:203: main Q: EnqueueMsg advised worker start

7635.540821000:203: --------imuxsock calling select, active file descriptors (max 4): 4

7647.435687000:102: file netstrms.c released module 'lmnsd_ptcp', reference count now 1

7647.435722000:102: Action 20059608 transitioned to state: rtry

to stop, you can ctrl-c or kill the process.

 

apply command (in AIX only?)

Here is a fast way to do function loop : 

apply "ssh server%1_pers date" 2 3 5 6  7 8

Replaces %1 with following parameters 

same as : 

for i in 2 3 5 6 7 8 
do
     ssh server${i}_pers date
done

Exclude files or directories from commands

An easy way to remove elements from command line : 

root:/# rm !(*.c|*.h)  => remove every files except *.c or *.h) 

another syntax : 

root:/# rm !(*.@(c|h))  => same thing ...

Can be used with grep, for directories exlusion. :

root:/#  grep P770_P7_1_A30 !(www|logs)/* => grep P770_P7_1_A30 from every underlying directories, except www and logs

Be sure to be on different switchs !

It's good to have more than one fcs adapter... but it's better to be sure they are connected on different switchs !
Each id is different, if it is on another switch

in this case we have two  adapters (or 2 ports on the same card):

root:/# lsdev -c adapter | grep fcs
fcs0   Available 30-T1 Virtual Fibre Channel Client Adapter
fcs1   Available 31-T1 Virtual Fibre Channel Client Adapter

but on the same vio server ! ....

root:/# echo "vfcs" | kdb
           START              END
0000000000001000 0000000005770000 start+000FD8
F00000002FF47600 F00000002FFDF9C0 __ublock+000000
000000002FF22FF4 000000002FF22FF8 environ+000000
000000002FF22FF8 000000002FF22FFC errno+000000
F1000F0A00000000 F1000F0A10000000 pvproc+000000
F1000F0A10000000 F1000F0A18000000 pvthread+000000
read vscsi_scsi_ptrs OK, ptr = 0xF1000000C01A9380
(0)> vfcs
NAME      ADDRESS             STATE   HOST      HOST_ADAP  OPENED NUM_ACTIVE
fcs0      0xF10001019999A000  0x0008  vioprex1vfchost2  0x01    0x0000
fcs1      0xF10001019A3AC000  0x0008  vioprex1vfchost3  0x01    0x0000and on the same swicth !!!


root:/# lsattr -El fscsi0
attach       switch    How this adapter is CONNECTED         False
dyntrk       yes       Dynamic Tracking of FC Devices        True
fc_err_recov fast_fail FC Fabric Event Error RECOVERY Policy True
scsi_id      0x340206  Adapter SCSI ID                       False
sw_fc_class  3         FC Class for Fabric                   True
root:/# lsattr -El fscsi1
attach       switch    How this adapter is CONNECTED         False
dyntrk       yes       Dynamic Tracking of FC Devices        True
fc_err_recov fast_fail FC Fabric Event Error RECOVERY Policy True
scsi_id      0x340501  Adapter SCSI ID                       False
sw_fc_class  3         FC Class for Fabric                   True


this other partition is ok :

# lsattr -El fscsi0
attach       switch    How this adapter is CONNECTED         False
dyntrk       yes       Dynamic Tracking of FC Devices        True
fc_err_recov fast_fail FC Fabric Event Error RECOVERY Policy True
scsi_id      0x1db40   Adapter SCSI ID                       False
sw_fc_class  3         FC Class for Fabric                   True
# lsattr -El fscsi1
attach       switch    How this adapter is CONNECTED         False
dyntrk       yes       Dynamic Tracking of FC Devices        True
fc_err_recov fast_fail FC Fabric Event Error RECOVERY Policy True
scsi_id      0x20db40  Adapter SCSI ID                       False
sw_fc_class  3         FC Class for Fabric                   True

Merci Patrice, pour l'info !

patch alt_disk install image

create disk copy :

     alt_disk_copy -O -B -d "hdisk25 hdisk26"

wake up disk copy :

     alt_rootvg_op -W -d hdisk25

update alternate disk copy : 

     alt_rootvg_op -C -b update_all -l /updates/5300-12-04

close alternate disk copy, and bosboot it:

     alt_rootvg_op -St