Thursday, May 29, 2014
AIX Mirror Pool
Mirror Pool
Starting with 6.1 TL2 so-called mirror pools were introduced that make it possible to divide the physical volumes of a scalable volume group into separate pools. Mirror Pools allow to group physical volumes in a scalable volume group so that a mirror copy of a logical volume can be restricted to only allocate partitions from physical volumes in a specified group.
A mirror pool is made up of one or more physical volumes. Each physical volume can only belong to one mirror pool at a time. When creating a logical volume, each copy of the lv being created can be assigned to a mirror pool.
Mirro Pool name can be up to 15 characters long and is unique to the volume group it belongs to. Therefore, two separate volume groups could use the same name for their mirror pools.
Any changes to mirror pool characteristics will not affect partitions allocated before the changes were made. The reorgvg command should be used after mirror pool changes are made to move the allocated partitions to conform to the mirror pool restrictions.
------------------------------------------------------
Strict Mirror Pool:
When strict mirror pools are enabled any logical volume created in the volume group must have mirror pools enabled for each copy of the logical volume. (If this is enabled all of the logical volumes in the volume group must use mirror pools.)
mkvg -M y -S <hdisk list> creating a vg with strict mirror pool
chvg -M y <vg name> turn on/off strict miror pool setting for a vg (chvg -M n... will turn off)
lsvg <vg name> shows mirro pool sctrictness (at the end of the output: MIRROR POOL STRICT: on)
------------------------------------------------------
Super strict Mirror Pool:
A super strict allocation policy can be set so that the partitions allocated for one mirror cannot share a physical volume with the partitions from another mirror. With this setting each mirror pool must contain at least one copy of each logical volume.
mkvg -M s -S <hdisk list> creating a vg with super strict setting
chvg -M s <vg name> turn on/off super strict setting for a vg (chvg -M n... will turn off)
lsvg <vg name> shows mirro pool sctrictness (at the end of the output: MIRROR POOL STRICT: super)
------------------------------------------------------
Creating/Removing/Renaming a Mirror Pool (adding disk to a Mirror Pool):
mkvg -S -p PoolA hdisk2 hdisk4 bbvg <--creating a new VG with mirror pool
extendvg -p PoolA bbvg hdisk6 <--extending a VG with a disk (while adding disks to mirror pools)
If we already have a vg:
root@bb_lpar: / # lsvg -P bbvg <--lists the mirror pool that each physical volume in the volume group belongs to
Physical Volume Mirror Pool
hdisk6 None
hdisk7 None
root@bb_lpar: / # chpv -p PoolA hdisk6 <--creating mirror pool with the given disks (disks should be part of a vg)
root@bb_lpar: / # chpv -p PoolB hdisk7 (or if the mirror pool already exists, it will add the specified disk to the pool)
root@bb_lpar: / # lsvg -P bbvg
Physical Volume Mirror Pool
hdisk6 PoolA
hdisk7 PoolB
root@bb_lpar: / # chpv -P hdisk7 <--removes the physical volume from the mirror pool
root@bb_lpar: / # lsvg -P bbvg
Physical Volume Mirror Pool
hdisk6 PoolA
hdisk7 None
root@bb_lpar: / # chpv -m PoolC hdisk6 <--changes the name of the mirror pool
root@bb_lpar: / # lsvg -P bbvg
Physical Volume Mirror Pool
hdisk6 PoolC
hdisk7 None
------------------------------------------------------
Creating/Mirroring lv to a Mirror Pool:
mklv -c 2 -p copy1=PoolA -p copy2=PoolB bbvg 10 <--creates an lv (with default name:lv00) in the given mirror pools with the given size
mklvcopy -p copy2=PoolB bblv 2 <--creates a 2nd copy of an lv to the given mirror pool
mirrorvg -p copy2=MPoolB -c 2 bbvg <--mirrors the whole vg to the given mirror pool
------------------------------------------------------
Adding/Removing an lv to/from a Mirror Pool:
root@bb_lpar: / # lsvg -m bbvg <--shows lvs of a vg with mirror pools
Logical Volume Copy 1 Copy 2 Copy 3
bblv None None None
root@bb_lpar: / # chlv -m copy1=PoolA bblv <--enables mirror pools to the given copy of an lv
root@bb_lpar: / # chlv -m copy2=PoolB bblv
root@bb_lpar: / # lsvg -m bbvg <--checking again the layout
Logical Volume Copy 1 Copy 2 Copy 3
bblv PoolA PoolB None
root@bb_lpar: / # chlv -M 1 bb1lv <--disables mirror poools of the given copy for the lv
root@bb_lpar: / # lsvg -m bbvg <--checking again the layout
Logical Volume Copy 1 Copy 2 Copy 3
bb1lv None PoolB None
------------------------------------------------------
Viewing Mirror Pools:
lsmp bbvg <--lists the mirror pools of the given vg
lspv hdisk6 <--shows PV characteristic (at the last line shows mirror pool the pv belongs to)
lspv -P <--shows all PVs in the system (with mirror pools)
lsvg -P bbvg <--shows the PVs of a VG (with mirror pools)
lsvg -m bbvg <--shows the LVs of a VG (with mirror pools)
lslv bblv <--shows LV characteristics (at the end shows lv copies and the mirror) pools)
------------------------------------------------------
Correct steps of creating and removing a mirror pool (totally):
Mirror pool is a separate entity from LVM. (I imagine it as a small database, which keeps rules and strictness, so the underlying LVM commands, based on those rules are successful or not.) It can happen that you remove the 2nd copy of an LV with rmlvcopy (not in LVM anymore), but mirror pool commands will still show it as an existent copy. So make sure LVM commands (mirrorvg, mklvcopy...) and Mirror Pool commands (chpv -p, chlv -m copy1=.., chvg -M....) are in synchron all the time!
Mirror pool informations are stored in 3 places: PV, LV and VG
If you need to create or remove a mirror pool, make sure mirror pool entry is taken care it all 3 places.
Creating mirror pool on a VG which is already mirrored at LVM level:
0. check if mirrors are OK (each copy in separate disk)
1. chpv -p <poolname> <diskname> <--add disks to the mirror pool
# lspv hdisk0 | grep MIRROR
MIRROR POOL: PoolA
2. chlv -m copy1=PoolA fslv00 <--add lv to the given pool (add all lvs to both pools: copy1 and copy2)
# lslv fslv00 | grep MIRROR
COPY 1 MIRROR POOL: PoolA
COPY 2 MIRROR POOL: PoolB
COPY 3 MIRROR POOL: None
3. chvg -M <strictness> <vgname> <--set strictness for the VG (usually chvg -M s ...)
# lsvg testvg | grep MIRROR
MIRROR POOL STRICT: super
------------------------------------------------------
Removing mirror pool from a system:
1. chvg -M n <vgname> <--turn off strictness
# lsvg testvg | grep MIRROR
MIRROR POOL STRICT: off
2. chlv -M 2 <lvname> <--remove 2nd copy of the LV from mirror pool (remove 1st copy as well: chlv -M 1...)
# lslv fslv00 | grep MIRROR
COPY 1 MIRROR POOL: PoolA
COPY 2 MIRROR POOL: None
COPY 3 MIRROR POOL: None
If every mirror pool is removed from LV level, only then!:
3. chpv -P <diskname> <--remove disk from mirror pool (do it with all disks)
# lspv hdiskpower0| grep MIRROR
MIRROR POOL: None
4. check with lsvg -m <vgname>
------------------------------------------------------
If you remove mirror pool from a disk, but it still exist on LV level (step 2 and 3 are not in correct order), you will get this:
# chpv -P hdiskpower0
0516-1010 chpv: Warning, the physical volume hdiskpower0 has open
logical volumes. Continuing with change.
0516-1812 lchangepv: Warning, existing allocation violates mirror pools.
Consider reorganizing the logical volume to bring it into compliance.
# lsvg -m testvg
Logical Volume Copy 1 Copy 2 Copy 3
loglv00 None None None
fslv00 None None <--it will show incorrect data (Pool was not deleted at LV level)
fslv01 None None None
# chlv -M 1 fslv00 <--remove pool from LV level (copy 1)
# lsvg -m testvg <--it will show correct info
Logical Volume Copy 1 Copy 2 Copy 3
loglv00 None None None
fslv00 None None None
fslv01 None None None
------------------------------------------------------
Changing from one Mirror Pool to another:
If you have a good working system with mirror pool (A and B) and requested to remove disks from pool A and assign new disks from Pool C:
My suggestion:
1. remove mirror pools totally from the system: from VG, LV and PV level
2. remove unnecessary mirror at LVM level (unmirrorvg from the disks of Pool A)
3. delete disks on the system (from Pool A) and assign new disks to the system (Pool C)
4. create LVM mirror to the new disks on Pool C (mirrorvg)
5. create new mirror pools, Pool A and C (PV, LV and VG level)
------------------------------------------------------
0516-622 extendlv: Warning, cannot write lv control block data.
0516-1812 lchangelv: Warning, existing allocation violates mirror pools.
Consider reorganizing the logical volume to bring it into compliance.
This can come up when you want to increase fs (or lv), but the lv layout on the disks is not following fully the mirror pool restrictions. (For example there is an lp which exists on a disk in one pool, but it should reside in the other pool.)
The reorgvg command can solve this (it can run for a long time):
reorgvg <vg name> <lv name>
Sometimes reorgvg can't solve it and you have to manually find where is the problem:
1. check lv - mirror pool distribution:
root@aixdb2: /root # lsvg -m P_NAVISvg
Logical Volume Copy 1 Copy 2 Copy 3
p_admlv VMAX_02 VMAX_03 None
p_datlv VMAX_02 VMAX_03 None
p_archlv VMAX_02 VMAX_03 None
...
As you see all of the 1st copy belongs to VMAX_02 and the 2nd copy to VMAX_03
2. check disk - mirror pool distribution
root@aixdb2: /root # lspv -P
Physical Volume Volume Group Mirror Pool
hdiskpower1 P_NAVISvg VMAX_03 <--it should contain only 2nd copy of lvs
hdiskpower2 P_NAVISvg VMAX_03 <--it should contain only 2nd copy of lvs
...
hdiskpower18 P_NAVISvg VMAX_02 <--it should contain only 1st copy of lvs
hdiskpower19 P_NAVISvg VMAX_02 <--it should contain only 1st copy of lvs
hdiskpower20 P_NAVISvg VMAX_02 <--it should contain only 1st copy of lvs
3. check lv - disk distribution
From the output of lsvg -M <vg name>, you can see the 1st and 2nd copy of an lv resides on which disk.
After that you can check if that disk belongs to the correct mirror pool or not.
this will sort the disks with lvs on it and show which copy (1st or 2nd) is there:
root@aixdbp2: /root # lsvg -M P_NAVISvg | awk -F: '{print $1,$2,$4}'| awk '{print $1,$3,$4}'| sort -u | sort -tr +1 -n
P_NAVISvg:
hdiskpower18 t_datlv 1
hdiskpower18 t_oralv 1
hdiskpower19 p_datlv 2 <--2nd copy of p_datlv resides on hdiskpower19, but hdiskpower19 should contain only 1st copy
hdiskpower19 p_oralv 1
hdiskpower19 t_archlv 1
(the above command: lsvg -M...sort -tr +1 -n, was written for hdiskpower disks (-tr:delimeter is 'r'))
(if you have only hdisk, you can change it to lsvg -M...sort -tk +1 -n, or if you omit this sort, the command should work as well)
4. migrating the wrong lps to a correct disk
checking lps of an lv:
root@aixdb2: /root # lspv -M hdiskpower19 | grep p_datlv
hdiskpower19:889 p_datlv:9968:2
hdiskpower19:890 p_datlv:9969:2
hdiskpower19:891 p_datlv:9970:2
After finding the correct disk with free pps (e.g. this will show you the freepps: lspv -M <disk>):
root@aixdb2: /root # migratelp p_datlv/9968/2 hdiskpower2/329
(Sometimes for migratelp not enough to give diskname only (e.g. hdiskpower2), pp number is needed as well (e.g. hdiskpower2/329))
AIX LVM
LVM (Logical Volume Manager):
LVM manages the storage to have a structured overview of it.
/var/adm/ras/lvmcfg.log lvm log file shows what lvm commands were used (alog -ot lvmcfg)
alog -ot lvmt shows lvm commands and libs
The LVM consists of:
-high level commands: can be used by users, e.g.: mklv (this can call an intermediate level command)
-intermediate level commands: these are used by high-level commands, e.g. lcreatelv (users should not use these)
-LVM subroutine interface library: it contains routines used by commands, e.g. lvm_createlv
-Logical Volume Device Driver (LVDD): manages and processes all I/O; it is called by jfs or lvm library routines
-Disk Device Driver: It is called by LVDD
-Adapter Device Driver: it provides an interface to the physical disk
This shows how the execution of a high level command goes through the different layers of LVM:
After you create a volume group, you can create logical volumes within that volume group. Logical partitions and logical volumes make up the logical view. Logical partitions map to and are identical in size to the physical partitions. A physical partition is the smallest unit of allocation of disk where the data is actually stored. A logical volume is a group of one or more logical partitions that can span multiple physical volumes. All the physical volumes it spans must be in the same volume group.
A logical volume consists of a sequence of one or more logical partititons. Each logical partition has at least one and a maximum of three corresponding physical partitions that can be located on different physical volumes.
When you first define a logical volume, the characteristics of its state (LV STATE) will be closed. It will become open when, for example, a file system has been created in the logical volume and mounted.
It is also possible that you might want to create a logical volume and put nothing on it. This is known as a raw logical volume. Databases frequently use raw devices
Logical Volume types:
- log logical volume: used by jfs/jfs2
- dump logical volume: used by system dump, to copy selected areas of kernel data when a unexpected syszem halt occurs
- boot logical volume: contains the initial information required to start the system
- paging logical volume: used by the virtual memory manager to swap out pages of memory
users and appl.-s will use these lvs:
- raw logical volumes: these will be controlled by the appl. (it will nit use jfs/jfs2)
- journaled filesystems:
Striped logical volumes:
Striping is a technique spreading the data in a logical volume across several physical volumes in such a way that the I/O capacity of the physical volumes can be used in parallel to access the data.
LVCB (Logical Volume Control Block)
First 512 byte of each logical volume in normal VGs (In big VGs it moved partially into the VGDA, and for scalable VGs completely.)(traditionally it was the fs boot block) The LVCB stores the attributes of the LV. Jfs does not access this area.
# getlvcb -AT <lvname> <--shows the LVCB of the lv
--------------------
LOGICAL VOLUME: hd2 VOLUME GROUP: rootvg
LV IDENTIFIER: 0051f2ba00004c00000000f91d51e08b.5 PERMISSION: read/write
VG STATE: active/complete LV STATE: opened/syncd
TYPE: jfs WRITE VERIFY: off
MAX LPs: 512 PP SIZE: 32 megabyte(s)
COPIES: 2 SCHED POLICY: parallel
LPs: 73 PPs: 146
STALE PPs: 0 BB POLICY: relocatable
INTER-POLICY: minimum RELOCATABLE: yes
INTRA-POLICY: center UPPER BOUND: 32
MOUNT POINT: /usr LABEL: /usr
MIRROR WRITE CONSISTENCY: on/ACTIVE
EACH LP COPY ON A SEPARATE PV ?: yes
Serialize IO ?: NO
inter-policy inter-physical volume allocation policy, can be minimum or maximum
minimum: to allocate pp's the minimum pv will be used (not spreading to all pv's tha data if possible)
maximum: to spread the physical partitions of this logical volume over as many physical volumes as possible.
This illustration shows 2 physical volumes. One contains partition 1 and a copy of partition 2. The other contains partition 2 with a copy of partition 1. The formula for allocation is Maximum Inter-Disk Policy (Range=maximum) with a Single Logical Volume Copy per Disk (Strict=y).
each lp copy on separate pv The strictness value. Current state of allocation, strict, nonstrict, or superstrict. A strict allocation states that no copies for a logical partition are allocated on the same physical volume. If the allocation does not follow the strict criteria, it is called nonstrict. A nonstrict allocation states that copies of a logical partition can share the same physical volume. A superstrict allocation states that no partition from one mirror copy may reside the same disk as another mirror copy. (mirror 2 and mirror 3 cannot be on the sam edisk)
(So inter-policy and strictness have effect together how many disks are used: spreading to maximum disks (1st lps) then mirroring them we need another bunch of disks; however spreading to minimum disks and mirroring, we need less disks.)
intra-policy intra-physical volume allocation policy, it specifies what startegy should be used for choosing pp's on a pv.
it can be: edge (outer edge), middle (outer middle), center, inner middle, inner edge
If you specify a region, but it gets full, further partitions are allocated from near as possible to far away.
The more i/o-s used, the pp's should be allocate to the outer edge.
mirror write consistency If turned on LVM keeps additional information to allow recovery of inconsistent mirrors.
Mirror write consistency recovery should be performed for most mirrored logical volumes
MWC is necessary to mirror lvs with parallel scheduling policies.
sched policy how reads and writes are handled to mirrorred logical volumes
parallel (default): read from least busy disk, write to all copies concurrently (at the same time)
sequential: read from primary copy only (if not available then next copy). write sequential (one after another)
(1 book suggests sequential because it works with MWC)
Write verify If turned on, all writes will be verified with a follow-up read. This will negatively impact performace but useful.
BB policy Bad block relocation policy. (bad blocks are relocatable or not)
Relocatable Indicates whether the partitions can be relocated if a reorganization of partition allocation takes place.
Upper Bound what is the maximum number of physical volumes a logical volume can use for allocation
------------------
# lslv -l pdwhdatlv
PV COPIES IN BAND DISTRIBUTION
hdiskpower5 125:000:000 3% 000:004:000:076:045
Copies shows information of each copies (separated by :) on the disks (125 first copy and no other mirrors are on the disk)
In Band the percentage of pps on the disk which were allocated within the region specified by Intra-physical allocation policy
Distribution how many pps are allocated in: outer edge, outer middle, center, inner middle, and inner edge (125=4+76+45)
------------------
lslv lvname displays information about the logical volume
lslv -m lvname displays the logical partitions (LP) and their corresponding physical partititons (PP)
lslv -l lvname displays on which physical volumes is the lv resides
lslv -p <hdisk> displays the logical volume allocation map for the disk (shows used, free, stale for each physical partition)
lslv -p <hdisk> <lv> displays the same as above, just the given lv's partitions will be showed by numbers
Open Indicates active if LV contains a file system
Closed Indicates inactive if LV contains a file system
Syncd Indicates that all copies are identical
Stale Indicates that copies are not identical
mklv -y newlv1 datavg 1 create logical volumes (mklv -y'testlv' -t'jfs' rootvg 100 <--creates jfs with 100 lp)
-y newlv1 name of the lv
datavg in which vg the lv will reside
1 how many logical partitions add to the lv
mklv -t jfs2log -y <lvname> <vgname> 1 <pvname> creates a jfs2log lv (after creation format it: logform -V jfs2 <loglvname>)
rmlv removes a logical volume
rmlv -f loglv removes without confirmation
mklvcopy bblv 2 hdisk2 make a 2nd copy (1LP=2PP) of bblv to hdisk2 (synchronization will be needed: syncvg -p hdisk2 hdisk3)
rmlvcopy bblv 1 hdisk3 leave 1 copy (1LP=1PP) only and remove those from hdisk3
getlvcb display the LVCB (Logical Volume Control Block) of a logical volume
extendlv increasing the size of a logical volume
cplv copying a logical volume
chlv changes the characteristic of a logical volume
migratelp testlv/1/2 hdisk5/123 migrates testlv's data from the 1st lp's second copy to hdisk5 on pp 123
(output of lspv -M hdiskx can be used:lvname:lpnumber:copy, this sequence is needed)
(if it is not mirrorred than easier this way: migratelp testlv/1 hdisk3)
(if it is mirrorres and we use the above commande, than 1st copy will be used: testlv/1/1...)
migratelp in for cycle:
for i in $(lslv -m p1db2lv | grep hdiskpower11 | tail -50 | cut -c 2-4); do migratelp p1db2lv/$i hdiskpower3; done
lresynclv resync a logical volume (???maybe if mirrorred???
------------------
Creating a new log logical volume:
1. mklv -t jfs2log -y lvname vgname 1 pvname <-- creates the log lv
2. logform -V jfs2 /dev/lvname
3. chfs -a log=/dev/lvname /fsname <--changes the log lv (it can be checked in /etc/filesystems)
------------------
Resynchronizing a logical volume:
1. root@aix16: / # lslv hd6 | grep IDENTIFIER
LV IDENTIFIER: 00c2a5b400004c0000000128f907d534.2
2. lresynclv -l 00c2a5b400004c0000000128f907d534.2
------------------
Striped lv extending problems:
extending is only possible by the stripe width (if it is 2, the extended lp should be 2,4,6...)
if lv can't be extended upper bound can cause this:
lslv P02ctmbackuplv | grep UPPER
UPPER BOUND: 2
It means that the lv can only be on 2 disks, but if on those 2 disks has no more space it can't be extebded to other disks.
upper bound should be changed: chlv -u 4 P02ctmbackuplv
After this extension should be possible
.
------------------
Unable to find lv in the define configuration database
1. synclvodm <vgname> <-- rebuild the volume group descriptors on the physical volume. Enter:
2. rmlv <lvname> <-- remove the unwanted logical volume.
------------------
Migrating PPs between disks:
checking the PPs of test1lv:
lslv -m test1lv
test1lv:/home/test1fs
LP PP1 PV1 PP2 PV2 PP3 PV3
0001 0001 hdisk6
0002 0002 hdisk6
0003 0003 hdisk6
...
0057 0057 hdisk6
0058 0058 hdisk6
0059 0059 hdisk6
the command: migratelp test1lv/59 hdisk7
(it wil migrate LP #59 to hdisk7)
in a for cycle:
for i in $(lslv -m shadowlv | grep hdisk1 | tail -10 | cut -c 2-4); do
migratelp shadowlv/${i} hdisk0
done
------------------
Once had a problem with an lv and its mirror copies:
root@bb_lpar: / # lsvg -l bbvg
bbvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
0516-1147 : Warning - logical volume bblv may be partially mirrored.
bblv jfs2 16 20 3 closed/syncd /bb
root@bb_lpar: / # mirrorvg bbvg
0516-1509 mklvcopy: VGDA corruption: physical partition info for this LV is invalid.
0516-842 mklvcopy: Unable to make logical partition copies for
logical volume.
0516-1199 mirrorvg: Failed to create logical partition copies
for logical volume bblv.
0516-1200 mirrorvg: Failed to mirror the volume group.
root@bb_lpar: / # lslv -l bblv
0516-1939 : PV identifier not found in VGDA.
root@bb_lpar: / # rmlvcopy bblv 1 hdisk2
0516-1939 lquerypv: PV identifier not found in VGDA.
0516-304 getlvodm: Unable to find device id 0000000000000000 in the Device
Configuration Database.
0516-848 rmlvcopy: Failure on physical volume 0000000000000000, it may be missing
or removed.
The partial mirrored lps caused a big mess in VGDA and LVM, so the solution was the removal of these lps with a low-level command: lreducelv
1. checking the problematic lps:
root@bb_lpar: / # lslv -m bblv
bblv:/bb
LP PP1 PV1 PP2 PV2 PP3 PV3
0001 0008 hdisk2
0002 0009 hdisk2
0003 0010 hdisk2
0004 0011 hdisk2
0005 0012 hdisk2
0006 0013 hdisk2
0007 0014 hdisk2
0008 0015 hdisk2
0009 0008 hdisk3 0016 hdisk2
0010 0009 hdisk3 0017 hdisk2
0011 0010 hdisk3 0018 hdisk2
0012 0012 hdisk3 0019 hdisk2
0013 0001 hdisk2
0014 0002 hdisk2
0015 0003 hdisk2
0016 0004 hdisk2
2. creating a text file with these wrong lps which will be used by lreducelv:
1st column: PVID of the disk with wrong lps (lspv hdisk2: 00080e82dfab25bc)
2nd column: PP# of the wrong lps (lslv -m bblv: PP2 column)
3rd column: LP# of the wrong lps (lslv -m bblv: LP column)
root@bb_lpar: / # vi partial_mir.txt
00080e82dfab25bc 0016 0009
00080e82dfab25bc 0017 0010
00080e82dfab25bc 0018 0011
00080e82dfab25bc 0019 0012
3. removing the partial mirror copies:
lreducelv -l <LV ID> -s <NUMBER of LPs> <TEXT FILE>
LV ID: 00080e820000d900000001334c11e0de.1 (lslv bblv)
NUMBER of LPs: 4 (wc -l partial_mir.txt)
TEXT FILE: partial_mir.txt
root@bb_lpar: / # lreducelv -l 00080e820000d900000001334c11e0de.1 -s 4 partial_mir.txt
Now the lvm deallocates all PP's of your partially mirror.
4. After these, lslv -m will show correct output, but LVCB or VGDA could still show we have 2 copies
root@bb_lpar: /tmp/bb # odmget -q name=bblv CuAt | grep -p copies
CuAt:
name = "bblv"
attribute = "copies"
value = "2"
type = "R"
generic = "DU"
(We can see this paragraph only if there is mirroring, otherwise there will be no output of odmget command)
root@bb_lpar: /tmp/bb # getlvcb -AT bblv
AIX LVCB
intrapolicy = m
copies = 1
(odmget shows we have 2 copies and getlvcb shows we have only 1 copy.)
Probably it is safer if we update both with the correct value:
putlvodm -c <COPYNUM> <LV ID>
putlvcb -c <COPYNUM> <LV NAME>
COPYNUM: 1
LV ID: 00080e820000d900000001334c11e0de.1 (lslv bblv)
root@bb_lpar: /tmp/bb # putlvodm -c 1 00080e820000d900000001334c11e0de.1
root@bb_lpar: /tmp/bb # putlvcb -c 1 bblv
source of this solution: http://archive.rootvg.net/cgi-bin/anyboard.cgi/aix?cmd=get&cG=73337333&zu=37333733&v=2&gV=0&p=
------------------
AIX-File Systems
FS-File system
A file system is a hierarchical tree structure of files and directories. Some tasks are performed more efficiently on a file system than on each directory within the file system. For example, you can back up, move, or secure an entire file system.
File systems are associated with devices (logical volumes) represented by special files in /dev. When a file system is mounted, the logical volume and its contents are connected to a directory in the hierarchical tree structure. You can access both local and remote file systems using the mount command.
AIX supports these file system types:
JFS Journaled File System which exists within a Logical Volume on disk
JFS2 Enhanced Journaled File System which exists within a Logical Volume on disk
CDRFS CD-ROM File System on a Compact Disc
NFS Network File System accessed across a network
UDF Universal Disk Format (DVD ROM media)
GPFS General Parallel Filesystem
SMBFS Server Message Block Filesystem (cifs_fs, samba share)
All of the information about the file system is centralized in the /etc/filesystems file. Most of the file system maintenance commands take their defaults from this file. The file is organized into stanza names that are file system names and contents that are attribute-value pairs specifying characteristics of the file system.
/tmp: <-- names the directory where the file system is normally mounted
dev = /dev/hd3 <-- for local mounts identifies the block special file where the file system reside
for remote mounts, it identifies the file or directory to be mounted
vfs = jfs2 <-- specifies the type of mount. For example, vfs=nfs
log = /dev/hd8 <-- full path name of the filesystem logging logical volume (only for jfs and jfs2)
mount = automatic <-- used by the mount command to determine whether this file system should be mounted by default
type = nas <-- several file systems can be mounted by giving the value as an argument to the -t flag (mount -t nas)
check = false <-- used by the fsck command to determine the default file systems to be checked
vol = /tmp <-- used by the mkfs command when initiating the label on a new file system
free = false <-- it is there because of unix traditions only (It is totally ignored by any and all AIX commands)
(df command in traditional UNIX would use it to determine which file systems to report)
For the option mount, these are valid entries: automatic, true, false, removable, and readonly:
automatic fs is to be mounted at boot; this is usually used for system-defined file systems.
true mount all is allowed to mount this file system.
false mount will only occur when the file system is specified as an argument to the mount command, or the type is used for mount.
The asterisk (*) is the comment character used in the /etc/filesystems file.
To remove a file system data from /etc/filesystems: imfs -x -l <lvname>
System-Created File Systems in AIX
The six standard file systems in AIX Versions 5 and higher are /, /home, /usr, /proc, /tmp, and /var. Each of these file systems is always associated with a logical volume name:
Logical Volume File System or Description
------------------------------------------------------
hd1 /home (users' home dir)
hd2 /usr (operating system commands, libraries and application programs)
hd3 /tmp (temporary space for all users)
hd4 / (critical files for system operations, programs that complete the boot process)
hd5 <boot logical volume>
hd6 <primary paging space>
hd8 <primary JFS or JFS2 log>
hd9var /var (variable spool and log files)
hd10opt /opt (freeware programs)
/proc /proc (pseudo fs kept in memory to support threads)
------------------------
Superblock
In a JFS, the superblock is the first addressable block (and a backup at the thirty-first addressable block) on a file system. It is 4096 bytes in size. The superblock is very important because a file system cannot be mounted if the superblock is corrupted. This is why there is a secondary or backup superblock at block 31. The superblock contains the following: size of the filesystem, number of datablocks in the fs, state of the fs...
# dd count=1 bs=4k skip=31 seek=1 if=/dev/hd4 of=/dev/hd4 <--this will restore the superblock from block #31
# fsck -p <fs> <--this will copy also the superblock from #31
# dumpfs /usr <--shows the superblock, i-node map, and disk map information
# od -x -N 64 /dev/hd1 +0xF000 <--display first superblock (JFS2)
# od -x -N 64 /dev/hd1 +0x8000 <--display second superblock (JFS2)
------------------------
i-node (index node)
A file system has a fixed number of i-nodes that are located following the superblock. i-nodes contain information about files, including the location of the data on the disk. They contain all of the identifying information about files (file type, size, permissions, user/group/owner, create/modification and last access dates) except for the file name, which is stored in the directory, and the contents of the file, which are stored in the data blocks. Each file or directory has an i-node associated with it. AIX reserves a number of i-nodes for files and directories every time a file system is created, and if all the available inodes are used, no more files can be created, even if the fs has free space.
------------------------
jfslog
AIX uses a journaled file system, meaning that certain i-node information is stored in a transaction log during writes. Its real value is in maintaining the integrity of the file system. Journaled file systems enable faster system reboots after system crashes. Each volume group has a jfslog file that is automatically created when the first file system is created in that volume group. The jfslog ensures the integrity of a file system by immediately writing all meta-data information to itself. Meta-data is information about the file system, such as changes to the i-nodes and the free lists. The jfslog keeps track of what is supposed to happen to the file system and whether it gets done. You are allowed to have a separate log for each filesystem.
(If a jfslog has been created manually, the logform command should be used to activate it as the jfslog for that vg.)
------------------------
Special or device files
A special file, sometimes called device file is associated with a particular hw device or other resource of the computer system. AIX uses them to provide file I/O access to character and block device drivers. Special files are distinguished from other files by having a "c" or "b" stored in the i-nodes, and they are located under the /dev directory. Character and block I/O requests are performed by issuing a read or write request on the device file:
- Character device file: Character devices (tape drives, tty devices) are capable of doing sequential I/O.
- Block device file: Block devices can only do random I/O, such as disk devices.
mknod: creates new special files (i-node and the file type (c or b) sould be set), major minor numbers will be written into the i-node
------------------------
Directories
The directory entry contains an index number associated with the file's i-node, the file name....
Every well formed directory contains the entries: . and ..
-.: points to the i-node for the directory itself
-..: points to the i-node for the parent directory
Because directory entries contain file names paired with i-nodes, every directory entry is a link.
------------------------
Links
Links are connection between a file name and an i-node. The i-node number actually identifies the file not the filename. By using links, any i-node or file can be known by many different names.
-hard link: Hard links can be created only between files that are in the same fs.
(when the last hard link is removed , the i-node and its data are deleted)
# ls -li: (bello is a hard link, and link count 2 shows it)
4 -rw-r--r-- 2 root system 0 Jul 8 23:27 bello
-symbolic link: Allow access to data in other filesystems from a new filename.
# ls -li: (bello is a sym. link, and the first character "l" shows this)
lrwxrwxrwx 1 root system 15 Jul 8 23:30 bello -> /test_fs1/hello
------------------------
NameFS (Name File System)
A NameFS is a pseudo-file system that lets you access a directory through two different path names. With a NameFS you can create a second path and you don't have to change permissions, copy, move, rename or even touch the original file system.
You could use a Name File System to set up a directory with:
-an alternate path (so it’s like a shortcut)
-different permissions (e.g. when some applications or users should have read-only access)
-other mount attributes such as Direct I/O (dio) or Concurrent I/O (cio)
How to setup NameFS:
1. mkdir -p /shortcut <--create a dir, which will be the mount point for the new Name file System
2.:
mount -v namefs /some/long/path /shortcut <--with this you got access to the files via 2 paths
mount -v namefs -o ro /data/report /data_reports <--with this you can make a read only mount of a dir
mount -v namefs -o cio /db2/W01/redo /deb2redo <--with this you can mount a dir with CIO to improve I/O
------------------------
crfs ... creates a file system (crfs creates lv as well; mkfs will create an fs over an already created lv)
crfs -v jfs2 -d W01origlogAlv -m /oracle/W01/origlogA -A yes -p rw -a options=cio -a agblksize=512
mount displays information about all the currently mounted file systems
mount dir_name mount the file system according to the entry in /etc/filesystems
mount lv_name dir_name mount the file system to another lv than in /etc/filesystems
umount dir_name umount the filesystem
mount -a or mount -all mounts all the file systems at one time
lsfs displays the characteristics of file systems
lsfs -q more detailed info about the fs (lv size...) (it queries the superblock)
(-v: list filesytems belonging to given fs type (jfs2, nfs); -u: lists filesystems in the given mount group)
rmfs /test removes a file system
rmfs -r /test removes the mount point also
chfs -a size=+5G /shadowtemp it will increase by 5G the fs (-5G can be used as well, or 5G will set tthe size of the fs to 5GB)
(if fs was reduced but space is not freed up defragfs could help)
chfs -a options='rw' /shadow shows with lsfs rw (I think rw is the deafault anyway)
imfs -x -l <lvname> remove a file system data from /etc/filesystems
fsck checks file system consistency (should not run on a mounted file system)
defragfs /home improves or reports the status of contiguous space within a file system
ls -ldi <dir> shows inode number in the first column
istat /etc/passwd display information regarding a particular inode (last updated, modified, accessed)
(update: change in the inode (file name, owner...); modified: change in the content of the file or dir)
df monitor file system growth
du dir_name (disk usage), to find which files are taking up the most space
du -sm * | sort -rn | head shows every dir size in MB (du -sk * the same in KB), the first 10 largest
skulker cleans up file systems by removing unwanted or obsolete files
fileplace <filename> displays the placement of file blocks within logical or physical volumes, it will show if a file fragmented
fuser /etc/passwd lists the process numbers of local processes using the /etc/passwd file
fuser -cux /var shows which processes are using the given filesystem
fuser -cuxk /var it will kill the above processes
fuser -dV /tmp shows deleted files (inode) with process ids which were open by a process (so its space could not be freed up)
(-V: verbose will show the size of the files as well)
if we rm a file, while it is opened by a process its space will not free up.
solution: kill the process, wait for the process to finish or reboot the system
---------------------------------------
HOW TO FIND FILES AFTER A SPECIFIC DATE:
touch mmddhhmm filename creates a file at a specific date
find /var -xdev -newer filename -ls
---------------------------------------
CREATING FS with commands:
1. mkvg -y oravg -s 128 hdiskpower62 <--creates vg with 128MB pp
2. mklv -y oraloglv -t jfs2log -a e -r n oravg 1 hdiskpower62 <--creates loglv (-a: allocation (e:edge), -r: relocatable (n:no))
3. mklv -y oralv -t jfs2 -a e oravg 500 hdiskpower62 <--creates lv (-a: allocation (e:edge))
4. crfs -v jfs2 -a logname=oraloglv -d oralv -m /ora <--creates fs with specified loglv (set auto mount if needed)
5. mount /ora <--mount fs
6. chown -R oracle.dba /ora_backup <--set owner/permission
---------------------------------------
EXTENDING FS with commands:
1. extendvg oravg hdiskpower63 <--extends vg with hdisk
2. chlv -x 1024 oralv <--set the maximum number of logical partitions if needed
3. extendlv oralv 20 hdiskpower63 <--extends lv to the specified hdisk
4. lslv -m oralv <--check allocations if needed
5. lsfs -q /ora <--shows new size of the lv (copy value of 'lv size')
6. chfs -a size=146800640 /ora <--use the 'lv size' value to enlarge fs
--------------------------------------------
HOW TO CORRECT CORRUPTED FS:
1. fsck /fs1 <--checks fs consistency (should not run on a mounted file system)
2. umount /fs1 <--umounts fs
If umount fails:
fuser -cux /fs1 <--shows processes running in the fs
fuser -kcux /fs1 <--kills the above processes (kill -9 works as well) (inittab/repawn processes will be there again)
umount /fs1
3. fsck -y /fs1 <--corrects errors
--------------------------------------------
CANNOT UNMOUNT FILE SYSTEMS:
-files are open or a user is using a directory in the fs:
fuser <--determines the PIDs for all processes that have open references within the fs
kill <--these processes can be killed
-loaded kernel extension:
genkex <--reports all loaded kernel extension
-file systems are still mounted within that file system:
umount <--umount first the embedded file systems!!!!!
-you can check processes using it with lsof:
lsof /home <--it will show the pids what should be terminated (kill <pid>)
--------------------------------------------
HOW TO COPY A FILESYSTEM TO A NEW VG:
command cplv is useful as it will copy at pp (lp) level not files (if there are many files cplv is better)
1. create vg if needed (loglv as well) <--we created bbvg, with log devce:bbloglv
2. umount /domo <--umount fs that you want to copy (/domo is the fs, domolv is its lv)
3. cplv -v bbvg domolv <--copy domolv to bbvg (it will create a new lv there like fslv01)
4. chfs -a dev=/dev/fslv01 -a log=/dev/bbloglv /domo <--changes the log device for fs to the one which we created in the new vg
(chfs -a dev=/dev/fslv01 -a log=INLINE /domo <--if we have inline log)
5. fsck -p /dev/fslv01 <--ensure fs integrity
6. mount /domo <--mount fs
--------------------------------------------
HOW TO COPY A FILESYSTEM (in the same vg):
1. create temporary fs
extendvg macavg hdiskX hdiskY <--add storage to vg
mkdir /maca_u10/macatmp <--create tmp mount point
chown oramaca.dbamaca /maca_u10/macatmp <--set rights
mklv -t jfs2 -y macatmplv macavg 1596 hdiskX hdiskY <--create temp lv (-y: new lv name, 1596: number of lps)
(if no jfslog, then mklv -t jfs2log..., and logform)
crfs -v jfs2 -d macatmplv -m /maca_u10/macatmp <--create temp fs (-d: device name, -m: mount point)
check if everything is identical <--df, lsfs, mount
2. copy data to temporary fs
umount /maca_u10/macaoradata
mount -r /maca_u10/macaoradata <--mount with read only
a.
cp -prh /maca_u10/oradata/* /maca_u10/macatmp/ <--this is good if small number of files need to be copied
or
b.
cd /maca_u10/macaoradata
tar cvf - . | (cd <tempnewfs> && tar xvf -) <--it copies everything from here to tempnewfs (file size limit? 6GB OK)
ls -lR | wc -l <--check if everything is identical, for both dir
3. rename lv and fs
umount /maca_u10/oradata
umount /maca_u10/macatmp
chlv -n macaoradatalv_o macaoradatalv <--rename oradatalv to old (chlv -n newlv oldlv)
chfs -m /maca_u10/oradata_o /maca_u10/oradata <--rename oradata fs to old fs (chfs -m newmnt oldmnt)
chlv -n macaoradatalv macatmplv <--rename tmplv to oradatalv
chfs -m /maca_u10/oradata /maca_u10/macatmp <--rename tmp fs to oradata fs
fsck /maca_u10/oradata <--fsck, before mount
mount /maca_u10/oradata
chown oramaca.dbamaca /maca_u10/oradata <--check (set) rights (if needed)
4. create mirror
rmfs /maca_u10/oradata_o <--remove old fs (if space needed)
rmdir /maca_u10/macatmp; rmdir /maca_u10/oradata_o <--remove dirs: macatmp, oradata_o
mklvcopy macaoradatalv 2 hdiskX hdiskY <--mirror oradatalv (2: 2 copies)
syncvg -l macaoradatalv <--synchronization
--------------------------------------------
BACKUP/RECREATE/RESTORE FILESYSTEM:
1. cd /filesystem
2. tar -cvf /tmp/filesystem.tar ./* <--it creates a backup of all the files in the fs
3. cd /
4. umount /filesystem
5. mkfs /filesystem
6. mount /filesystem
7. cd /filesystem
8. tar -xvf /tmp/filesystem.tar > /dev/null <--restores the data (output redirected, as displaying is time consuming)
--------------------------------------------
CHANGING THE LOG LOGICAL VOLUME:
If an fs is write extensive, the use of the log logical volume can cause io bottleneck if it is placed on the same disk.
(e.g. data-fs is located on hdisk2)
1. umount /data-fs <--umount the fs for which you want to create the new log logical volume
2. mlv -t jfs2log -y datafsloglv datavg 1 hdisk1 <--create a new log logical volume
3. logform /dev/datafsloglv <--format the log
4. chfs -a log=/dev/datafsloglv /data-fs <--it will modify /etc/filesystems to consist the new settings
5. getlvcb -ATdatalv <--just for checking if lvcb is updated
6. mount /data-fs <--mount back the changed filesystem
--------------------------------------------
REMOVING A FILE WITH SPECIAL CHARACTERS:
1. find inode number
root@bb_lpar: /tmp/bb # ls -i
49 <--the name of the file is empty
35 commands
47 lps.txt
2. remove file by inode number
root@bb_lpar: /tmp/bb # find . -inum 49 -exec rm '{}' \;
--------------------------------------------
FILESYSTEM CLEANUP HINTS:
find large files:
find . -xdev -size +4000000c -exec ls -l {} \; <--it will list files larger than 4MB in the fs
find . -type f | xargs ls -s | sort -rn | head <--10 largest file (if there is another fs under it, it will search there too)
find . -type f -size +10000 | while read X ; do du -sm "$X" ; done | sort -n | tail -n 15 <--15 largest file
/etc:
/etc/perf/daily/ <--xmdaily logs can be removed if not needed
(can be removed from inittab and xm processes can be killed)
who /etc/security/failedlogin <--lists failed logins
> /etc/security/failedlogin <--clears that file
/usr:
/usr/websm/pc_client <--windows, exe files can be removed
/var:
/var/adm/wtmp:
who /var/adm/wtmp <--shows the contents of that file
/usr/sbin/acct/fwtmp < /var/adm/wtmp | tail -5000 > /tmp/wtmp.asc <--converts wtmp to ascii, saves last 500 lines
/usr/sbin/acct/fwtmp -ic < /tmp/wtmp.asc > /var/adm/wtmp <--converts back to original format
rm /tmp/wtmp.asc <--delete the ascii file
/var/adm/cron/log:
> /var/adm/cron/log <--this can be cleared
/var/spool/lpd:
stopsrc -s qdaemon <--stops qdaemon
rm /var/spool/lpd/qdir/* <--clears dir
rm /var/spool/lpd/stat/*
rm /var/spool/qdaemon/*
startsrc -s qdaemon <--starts qdaemon
/var/spool/mail <--under this dir, not needed mails can be cleared as well
/var/adm/sulog <--this file can be reduced (cleared) as well
Subscribe to:
Posts (Atom)