Subha AIX Info Center

Saturday, March 9, 2013

Correct BLV by using NIM Maintenance Mode

This Post will provide a detail overview on how to fix corrupted boot logical volume in AIX server by using NIM server with maintenance mode.

My Environment details as below:

NIM Server Name: NIMMASTER

Client Server Name: subha

Consider by I have corrupted BLV on my server. Here in this case I have removed my BLV to work on this on my Lab environment.

subha:/ # rmlv hd5 ------------------------------> I have removed my BLV from client.

Warning, all data contained on logical volume hd5 will be destroyed.

rmlv: Do you wish to continue? y(es) n(o)?

0516-1246 rmlv: If hd5 is the boot logical volume, please run 'chpv -c '

as root user to clear the boot record and avoid a potential boot

off an old boot image that may reside on the disk from which this

logical volume is moved/removed.

rmlv: Logical volume hd5 is removed.

0516-1734 rmlv: Warning, savebase failed. Please manually run 'savebase' before rebooting.

subha:/ #

Still you can see your BLV in ODM. In real case you no need to remove hd5 from ODM but here I did on my LAB to watch how my server will react with BLV …

Subha:/ # chpv –c hdisk0 ------------------------------> Remove hd5 from disk and ODM.

Subha:/ # lsvg -l rootvg | grep -i hd5

Subha:/ #

After this step if you try to reboot your server, you will get HMC error code AA060011 (Firmware didn’t find operating system image).

Now consider this is real scenario in production environment and this need to be fix … to do this either you need base OS CD or NIM server … as I have NIM in my environment I took NIM maintenance mode.

It is better to have NIM instead of depending on CD (It required FE to present on DC it may took couple of hours to reach when you have issue on production server).

To fix this issue Your NIM Server OS level should be higher or equal to your client OS level.

You need to have a SPOT created same level of your client OS and make it maint_boot to identify spot.

NIMMASTER:/ # nim -o maint_boot -a spot=spot_aix530_12SP2

NIMMASTER:/ # lsnim -l spot_aix530_12SP2

spot_aix530_12SP2:

--------------------

Rstate = ready for use -------------------------> Spot state should be ready for use.

NIMMASTER:/ #

Bring Client in Maintenance Mode:

From HMC console reboot server while booting select opition 1 = SMS Menu

IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM

1 = SMS Menu 5 = Default Boot List

8 = Open Firmware Prompt 6 = Stored Boot List

Once server is in Maintenance mode we will get below display …. Here we Need to work on RIPL.

PowerPC Firmware

Version EM350_143

-------------------------------------------------------------------------------

Main Menu

1. Select Language

2. Setup Remote IPL (Initial Program Load)

3. Change SCSI Settings

4. Select Console

5. Select Boot Options

Type menu item number and press Enter or select Navigation key:2------------> Select Setup Remote IPL

##########################################################################

PowerPC Firmware

Version EM350_143

-------------------------------------------------------------------------------

NIC Adapters

Device Location Code Hardware

Address

1. Interpartition Logical LAN U8234.EMA.0609CC4-V10-C11-T1 62a63c90140b

2. Interpartition Logical LAN U8234.EMA.0609CC4-V10-C12-T1 62a63c90140c

Type menu item number and press Enter or select Navigation key:1----------------> Select Adapter

############################################################################

PowerPC Firmware

Version EM350_143

-------------------------------------------------------------------------------

Select Internet Protocol Version.

1. IPv4 - Address Format 123.231.111.222

2. IPv6 - Address Format 1234:5678:90ab:cdef:1234:5678:90ab:cdef

Type menu item number and press Enter or select Navigation key:1----------------> Select IPv4

###########################################################################

PowerPC Firmware

Version EM350_143

-------------------------------------------------------------------------------

Select Network Service.

1. BOOTP

2. ISCSI

Type menu item number and press Enter or select Navigation key:1----------------> Select BOOTP

##########################################################################

PowerPC Firmware

Version EM350_143

-------------------------------------------------------------------------------

Network Parameters

Interpartition Logical LAN: U8234.EMA.0609CC4-V10-C11-T1

1. IP Parameters

2. Adapter Configuration

3. Ping Test

4. Advanced Setup: BOOTP

Type menu item number and press Enter or select Navigation key:1----------------> Select IP Parameters

Enter Cleint IP/ NIM IP /Gateway IP/ Subanet Mask as below for security reason I have placed my IP as xxx.xxx.xxx.xxx

PowerPC Firmware

Version EM350_143

-------------------------------------------------------------------------------

IP Parameters

Interpartition Logical LAN: U8234.EMA.0609CC4-V10-C11-T1

1. Client IP Address [xxx.xxx.xxx.xxx]

2. Server IP Address [xxx.xxx.xxx.xxx]

3. Gateway IP Address [xxx.xxx.xxx.xxx]

4. Subnet Mask [xxx.xxx.xxx.xxx]

Type menu item number and press Enter or select Navigation key:----------------> To go back menu

######################################################################

PowerPC Firmware

Version EM350_143

-------------------------------------------------------------------------------

Network Parameters

Interpartition Logical LAN: U8234.EMA.0609CC4-V10-C11-T1

1. IP Parameters

2. Adapter Configuration

3. Ping Test

4. Advanced Setup: BOOTP

Type menu item number and press Enter or select Navigation key:2----------------> Set adapter settings

PowerPC Firmware

Version EM350_143

-------------------------------------------------------------------------------

Speed,Duplex

Interpartition Logical LAN: U8234.EMA.0609CC4-V10-C11-T1

1. auto,auto ( none )

Type menu item number and press Enter or select Navigation key:----------------> To go back menu

#####################################################################

PowerPC Firmware

Version EM350_143

-------------------------------------------------------------------------------

Adapter Configuration

Interpartition Logical LAN: U8234.EMA.0609CC4-V10-C11-T1

1. Speed,Duplex

2. Spanning Tree Enabled

3. Protocol

Type menu item number and press Enter or select Navigation key:2----------------> Disable spanning Tree

PowerPC Firmware

Version EM350_143

-------------------------------------------------------------------------------

Spanning Tree Enabled

Interpartition Logical LAN: U8234.EMA.0609CC4-V10-C11-T1

1. Yes

2. No <===

-------------------------------------------------------------------------------

Navigation keys:

M = return to Main Menu

ESC key = return to previous screen X = eXit System Management Services

-------------------------------------------------------------------------------

Type menu item number and press Enter or select Navigation key:----------------> Go to previous menu by esc to verify Ping test.

########################################################################

PowerPC Firmware

Version EM350_143

-------------------------------------------------------------------------------

Network Parameters

Interpartition Logical LAN: U8234.EMA.0609CC4-V10-C11-T1

1. IP Parameters

2. Adapter Configuration

3. Ping Test

4. Advanced Setup: BOOTP

-------------------------------------------------------------------------------

Navigation keys:

M = return to Main Menu

ESC key = return to previous screen X = eXit System Management Services

-------------------------------------------------------------------------------

Type menu item number and press Enter or select Navigation key:3----------------> Verify Ping test from Client to Master.

#########################################################################

PowerPC Firmware

Version EM350_143

-------------------------------------------------------------------------------

Ping Test

Interpartition Logical LAN: U8234.EMA.0609CC4-V10-C11-T1

Speed, Duplex: auto,auto

Client IP Address: xxx.xxx.xxx.xxx

Server IP Address: xxx.xxx.xxx.xxx

Gateway IP Address: xxx.xxx.xxx.xxx

Subnet Mask: xxx.xxx.xxx.xxx

Protocol: Standard

Spanning Tree Enabled: 0

Connector Type:

1. Execute Ping Test

-------------------------------------------------------------------------------

Navigation keys:

M = return to Main Menu

ESC key = return to previous screen X = eXit System Management Services

-------------------------------------------------------------------------------

Type menu item number and press Enter or select Navigation key:1

.---------------------.

| Attempting Ping... |

`---------------------'

xxx.xxx.xxx.xxx: 24 bytes from xxx.xxx.xxx.xxx: icmp_seq=1 ttl=? time=20 ms

.-----------------.

| Ping Success. |

`-----------------'

Press any key to continue..........

1. Execute Ping Test

-------------------------------------------------------------------------------

Navigation keys:

M = return to Main Menu

ESC key = return to previous screen X = eXit System Management Services

-------------------------------------------------------------------------------

Type menu item number and press Enter or select Navigation key: + + + +

#####################################################################################

PowerPC Firmware

Version EM350_143

-------------------------------------------------------------------------------

Main Menu

1. Select Language

2. Setup Remote IPL (Initial Program Load)

3. Change SCSI Settings

4. Select Console

5. Select Boot Options

-------------------------------------------------------------------------------

Navigation Keys:

X = eXit System Management Services

-------------------------------------------------------------------------------

Type menu item number and press Enter or select Navigation key:5 ----------------> To set boot options

#####################################################################################

PowerPC Firmware

Version EM350_143

-------------------------------------------------------------------------------

Multiboot

1. Select Install/Boot Device

2. Configure Boot Device Order

3. Multiboot Startup

4. SAN Zoning Support

-------------------------------------------------------------------------------

Navigation keys:

M = return to Main Menu

ESC key = return to previous screen X = eXit System Management Services

-------------------------------------------------------------------------------

Type menu item number and press Enter or select Navigation key:1----------------> To set boot device

#####################################################################

PowerPC Firmware

Version EM350_143

-------------------------------------------------------------------------------

Select Device Type

1. Diskette

2. Tape

3. CD/DVD

4. IDE

5. Hard Drive

6. Network

7. List all Devices

Type menu item number and press Enter or select Navigation key:6----------------> To set boot from network

######################################################################

PowerPC Firmware

Version EM350_143

-------------------------------------------------------------------------------

Select Network Service.

1. BOOTP

2. ISCSI

Type menu item number and press Enter or select Navigation key:1----------------> Select BOOTP

#######################################################################

PowerPC Firmware

Version EM350_143

-------------------------------------------------------------------------------

Select Device

Device Current Device

Number Position Name

1. - Interpartition Logical LAN

( loc=U8234.EMA.0609CC4-V10-C11-T1 )

2. - Interpartition Logical LAN

( loc=U8234.EMA.0609CC4-V10-C12-T1 )

Type menu item number and press Enter or select Navigation key:1 -----------------> Select LAN adapter.

############################################################################

PowerPC Firmware

Version EM350_143

-------------------------------------------------------------------------------

Select Task

Interpartition Logical LAN

( loc=U8234.EMA.0609CC4-V10-C11-T1 )

1. Information

2. Normal Mode Boot

3. Service Mode Boot

Navigation keys:

M = return to Main Menu

ESC key = return to previous screen X = eXit System Management Services

-------------------------------------------------------------------------------

Type menu item number and press Enter or select Navigation key:2 ----------------> Select NORMAL Boot

###########################################################################

PowerPC Firmware

Version EM350_143

-------------------------------------------------------------------------------

Are you sure you want to exit System Management Services?

1. Yes

2. No

Type menu item number and press Enter or select Navigation key:1 -------------------> Exit SMS menu

TFTP BOOT ---------------------------------------------------

Server IP.....................xxx.xxx.xxx.xxx

Client IP.....................xxx.xxx.xxx.xxx

Gateway IP....................xxx.xxx.xxx.xxx

Subnet Mask...................xxx.xxx.xxx.xxx

( 1 ) Filename................./tftpboot/subha

TFTP Retries..................5

Block Size....................512

FINAL PACKET COUNT = 29578

FINAL FILE SIZE = 15143424 BYTES

Elapsed time since release of system processors: 875750 mins 46 secs

Type a 1 and press Enter to use this terminal as the

system console.

>>> 1 Type 1 and press Enter to have English during maintenance.

>>> Choice [1]: 1

Maintenance

Type the number of your choice and press Enter.

>>> 1 Access a Root Volume Group

2 Copy a System Dump to Removable Media

3 Access Advanced Maintenance Functions

4 Erase Disks

5 Configure Network Disks (iSCSI)

6 Select Storage Adapters

99 Previous Menu

>>> Choice [1]: 1

Warning:

If you choose to access a root volume group, you will NOT be able to return

to the Base Operating System Installation menus without rebooting.

Type the number of your choice and press Enter.

0 Continue

>>> Choice [99]: 0

Access a Root Volume Group

Type the number for a volume group to display the logical volume information

and press Enter.

1) Volume Group 00c09cc400004c0000000133d92890b6 contains these disks:

hdisk0 32768 vscsi -----------> My rootvg disk

2) Volume Group 00c09cc400004c000000013d4da24a79 contains these disks:

hdisk1 32768 vscsi

3) Volume Group 00c09cc400004c0000000133d9902be2 contains these disks:

hdisk2 32768 vscsi

Choice: 1

Volume Group Information

------------------------------------------------------------------------------

Volume Group ID 00c09cc400004c0000000133d92890b6 includes the following

logical volumes:

hd6 hd8 hd4 hd2 hd9var hd3

hd1 hd10opt hd7 sysdump1 hd11admin

------------------------------------------------------------------------------

Type the number of your choice and press Enter.

1) Access this Volume Group and start a shell

2) Access this Volume Group and start a shell before mounting filesystems

99) Previous Menu

Choice [99]: 1

# bosboot -ad hdisk0

0301-168 bosboot: The current boot logical volume, /dev/hd5,

does not exist on /dev/hdisk0.

# lsvg -l rootvg | grep -i hd5

Now create hd5 on boot disk with 1PP on first edge.

# mklv -y hd5 -t boot -ae rootvg 1

hd5

# bosboot -ad hdisk0

bosboot: Boot image is 43303 512 byte blocks.

# lsvg -l rootvg | grep –I hd5

hd5 boot 1 1 1 closed/syncd N/A

# bootlist -m normal -o

hdisk0 blv=hd5

#reboot

# bootlist -m normal -o

hdisk0 blv=hd5

Saturday, February 23, 2013

How to get BASH shell behaviour on AIX server?

As you know BASH shell is having more advantages over any other Unix shells ... some of these features like command history,command editing by using up/down arrow keys, deleting/modifying typos ...

By default AIX uses ksh shell and we will not be able to get all of the above features what bash is having. This post will provide how to achive bash features on AIX machines.

Log in with your User name and you can see .profile under your home directory.

Take backup of your profile.
#cp -p /home//.profile /home//.profile.Date

Add the below lines to your .profile
#vi /home//.profile
set -o emacs
alias __A=`echo "\020"` # up arrow = ^p = back a command
alias __B=`echo "\016"` # down arrow = ^n = down a command
alias __C=`echo "\006"` # right arrow = ^f = forward a character
alias __D=`echo "\002"` # left arrow = ^b = back a character
alias __H=`echo "\001"` # home = ^a = start of line
export EXTENDED_HISTORY=ON

save the file.

Now restart your ssh session, you will be able to do bash operations in AIX machine.

Monday, February 4, 2013

Portmir in AIX and Unix

What is portmir?
Portmir is a very useful command to share your unix session with another remote user.
The portmir utility allows on TTY stream (monitor) to attach another TTY stream (Target).

Requirements:
1. Two Users should login on server.
Limitations:
1. At any point of time Only one mirror session will be active.

How To do/setup Portmir in AIX:
1. Log in to your Unix server from putty. Here my login ID is subha and host is aixtest.
# who
aixtest:/ # who
subha     pts/0       Feb 4 09:07     (X.X.X.X)

2. Issue command tty it will show your terminal details ..
#tty
aixtest:/ # tty
/dev/pts/0

3. Ask your remote user to login on server ... Here my remote user is reddy
#who
aixtest:/home/reddy $ who
reddy         pts/1       Feb 4 09:10     (X.X.X.X)   -------->Here remote terminal marked in RED
subha         pts/0       Feb 4 09:07     (X.X.X.Y)
aixtest:/home/reddy $

Here you can find your remote colleague terminal (my case remote user terminal /dev/pts1)

4. Run portmir command as Below to start sharing ..
#/usr/sbin/portmir -t pts/1
aixtest:/ # /usr/sbin/portmir -t pts/1
portmir: Remote user connected, mirroring active.
aixtest:/home/reddy $
portmir: Remote user connected, mirroring active.
hostname
aixtest
aixtest:/home/reddy $

5. Once your session is over terminate your portmirror with -o option ..

#/usr/sbin/portmir -o
aixtest:/home/reddy $ /usr/sbin/portmir -o
portmir: Mirroring is stopped.

If you're admin user and the other person temporarily requires root access to do
something (and you can't provide sudo's on emergency basis), then you can su -
in the portmir session, allowing the other person to have root access, while you
can see what he/she is doing.

Friday, December 21, 2012

AIX WPAR Configuration and Administration

IBM introduced work load partitions (WPAR) in AIX from v6.1. WAPR enables virtuvalization features in Logical partition i.e nothing but LPAR.

The main difference between LPAR and WPAR is, every LPAR required its own Operating system and certain amount hardware resources like CPU, memory ,disks ...

whereas WPAR is independent of all these.

Advantage of WPAR:

AIX admin can create a WPAR through a single command or SMIT menu within 30 Minutes (quick deployment).
You can create individual WPAR for each of your application.
WPAR is independent of your rootvg and you can maintain individual environment for each WPAR (like file systems, configurations, users, network, tuning settings..etc).
You can maintain WPAR image for each of your environment like Prod/Test/Dev/Non-prod/Lab.

Disadvantage of WPAR:

If any hardware/Network failures at LPAR level it will affect all WPAR’s on that box.
WPAR is not capable of high load / high I/O applications and data bases.
WPAR is not recommended for Production environment so far..
WPAR is not recommended for highly security environments.
No Support for physical devices on WAPR point (You can’t add storage..etc).

WPAR’s categorized as two types:

System Workload Partition: This type of WPAR’s is much closer to complete version of AIX. The system WPAR has its own dedicated, completely writable file systems along with its own inetd and cron.
Application Workload Partitions: Application WPARs are real, lightweight versions of virtualized OS environments. They are extremely limited and can only run application processes, not system daemons such as inetd or cron. One cannot even define remote access to this environment.

WPAR Administration:

Here I will be creating a System wpar named testwpar that is independent of my LPAR rootvg.

To accommodate this request I will create a separate VG named wparvg for all my WPAR's and under that VG I will create base file system (/wpar) for my testwpar.

1. Create wparvg on hdisk0

subha:/ # mkvg -B -s 128M -y wparvg hdisk0

wparvg

subha:/ #

2. Create a base File system for testwpar install.

subha:/ # crfs -v jfs2 -g wparvg -a size=20G -m /wpar

File system created successfully.

20970676 kilobytes total disk space.

New File System size is 41943040

subha:/ #

subha:/ # mount /wpar

3. Change FS permissions to 700 for security

subha:/ # chmod -R 700 /wpar

4. Create a system wpar named testwpar with IP 10.x.x.x under wparvg and base file system is /wpar/testwpar …

-A for to add WPAR info to /etc/rc.wpars and add to global /etc/inittab to start wpar at LPAR Boot.

-s to start WPAR after creation (Here I havn’t user –s, I will start WPAR with startwpar command).

subha:/ # mkwpar -A -n testwpar -N address=10.x.x.x -g wparvg -d /wpar/testwpar

mkwpar: Creating file systems...

/home

/opt

/proc

/tmp

/usr

/var

subha:/ # lswpar

Name State Type Hostname Directory RootVG WPAR

-------------------------------------------------------------

testwpar D S testwpar /wpar/testwpar no

subha:/ #

To Start WPAR with #startwpar

subha:/ # startwpar testwpar

Starting workload partition 'testwpar'.

Mounting all workload partition file systems.

Loading workload partition.

Exporting workload partition devices.

Starting workload partition subsystem 'cor_testwpar'.

0513-059 The cor_testwpar Subsystem has been started. Subsystem PID is 14024940.

Verifying workload partition startup.

To login to WPAR use command #clogin

subha:/ # clogin testwpar

*******************************************************************************

* Welcome to AIX Version 6.1! *

* *

* Please see the README file in /usr/lpp/bos for information pertinent to *

* this release of the AIX Operating System. * ********************************************************************************

# hostname

testwpar

To list all WPAR’s running on given LPAR #lswpar

Use –L option for WPAR detailed information including all configurations.

To get WPAR resource limit usage you can use below command

subha:/ # lswpar -a Name,State,Hostname,CPU,memory,Shares_CPU,Shares_memory

Name State Hostname CPU Limits Memory Limits CPU Shares Memory Shares

----------------------------------------------------------------------------------

testwpar A testwpar 0%-100%,100% 0%-100%,100% unlimited unlimited

subha:/ #

To change CPU & Memory allocations to WPAR use #chwpar

subha:/ # chwpar -R CPU=5%-40%,60% testwpar

subha:/ # chwpar -R memory=10%-30%,40% testwpar

subha:/ #

subha:/ # lswpar -a Name,State,Hostname,CPU,memory,Shares_CPU,Shares_memory

Name State Hostname CPU Limits Memory Limits CPU Shares Memory Shares

--------------------------------------------------------------------------------

testwpar A testwpar 5%-40%,60% 10%-30%,40% unlimited unlimited

subha:/ # lswpar

Name State Type Hostname Directory RootVG WPAR

----------------------------------------------------------

testwpar A S testwpar /wpar/wpars no

subha:/ #

Here State –A (Active), type –S (System WPAR), Hostname =WPAR Name, Directory=WPAR Base FS, Rootvg=no (Independent of LPAR rootvg).

To stop running WAPR #stopwpar

subha:/ # stopwpar testwpar

Stopping workload partition 'testwpar'.

Stopping workload partition subsystem 'cor_testwpar'.

0513-044 The cor_testwpar Subsystem was requested to stop.

stopwpar: 0960-261 Waiting up to 600 seconds for workload partition to halt.

Shutting down all workload partition processes.

wio0 Defined

Unmounting all workload partition file systems.

Here you can see State of WPAR is D – Deactivated.

subha:/ # lswpar

Name State Type Hostname Directory RootVG WPAR

----------------------------------------------------------

testwpar D S testwpar /wpar/wpars no

subha:/ #

To remove WPAR including its local file systems #rmwpar

If you don’t want remove FS’s use –p and –s options.

subha:/ # rmwpar testwpar

rmwpar: Removing file system /wpar/wpars/var.

rmlv: Logical volume fslv07 is removed.

rmwpar: Removing file system /wpar/wpars/usr.

rmwpar: Removing file system /wpar/wpars/tmp.

rmlv: Logical volume fslv06 is removed.

rmwpar: Removing file system /wpar/wpars/proc.

rmwpar: Removing file system /wpar/wpars/opt.

rmwpar: Removing file system /wpar/wpars/home.

rmlv: Logical volume fslv05 is removed.

rmwpar: Removing file system /wpar/wpars.

rmlv: Logical volume fslv04 is removed.

subha:/ #

subha:/ # lswpar --------------------------------------------No WPARS.

subha:/ #

Will Be adding more on WPAR Soon until then keep rocking with AIX :) (: