VOLUME GROUP
When you install a system, the first volume group (VG) is created. It is
called the rootvg. Your rootvg volume group is a base set of logical
volumes required to start the system. It includes paging space, the
journal log, boot data, and dump storage, each on its own separate
logical volume.
A normal VG is limited to 32512 physical partitions. (32 physical volumes, each with 1016 partitions)
you can change it with:
chvg -t4 bbvg (the factor is 4, which means: maximum partitions:4064 (instead of 1016), max disks:8 (instead of 32))
How do I know if my volume group is normal, big, or scalable?
Run the lsvg command on the volume group and look at the value for MAX
PVs. The value is 32 for normal, 128 for big, and 1024 for scalable
volume group.
VG type Maximum PVs Maximum LVs Maximum PPs per VG Maximum PP size
Normal VG 32 256 32,512 (1016 * 32) 1 GB
Big VG 128 512 130,048 (1016 * 128) 1 GB
Scalable VG 1024 4096 2,097,152 128 GB
If a physical volume is part of a volume group, it contains 2 additional
reserved areas. One area contains both the VGSA and the VGDA, and this
area is started from the first 128 reserved sectors (blocks) on the
disk. The other area is at the end of the disk, and is reserved as a
relocation pool for bad blocks.
VGDA (Volume Group Descriptor Area)
It is an area on the hard disk (PV) that contains information about the
entire volume group. There is at least one VGDA per physical volume, one
or two copies per disk. It contains physical volume list (PVIDs),
logical volume list (LVIDs), physical partition map (maps lps to pps)
# lqueryvg -tAp hdisk0 <--look into the VGDA (-A:all info, -t: tagged, without it only numbers)
Max LVs: 256
PP Size: 27 <--exponent of 2:2 to 7=128MB
Free PPs: 698
LV count: 11
PV count: 2
Total VGDAs: 3
Conc Allowed: 0
MAX PPs per PV 2032
MAX PVs: 16
Quorum (disk): 0
Quorum (dd): 0
Auto Varyon ?: 1
Conc Autovaryo 0
Varied on Conc 0
Logical: 00cebffe00004c000000010363f50ac5.1 hd5 1 <--1: count of mirror copies (00cebff...c5 is the VGID)
00cebffe00004c000000010363f50ac5.2 hd6 1
00cebffe00004c000000010363f50ac5.3 hd8 1
...
Physical: 00cebffe63f500ee 2 0
<--2:VGDA count 0:code for its state (active, missing, removed)
00cebffe63f50314 1 0 (The sum of VGDA count should be the same as the Total VGDAs)
Total PPs: 1092
LTG size: 128
...
Max PPs: 32512
-----------------------
VGSA (Volume Group Status Area)
The VGSAs are always present, but used with mirroring only. Needed to
track the state of mirror copies, that means whether synchronized or
stale. Per-disk stucure, but twice on each disk.
Quorum
Non-rootvg volume groups can be taken offline and brought online by a
process called varying on and varying off a volume group. The system
checks the availability of all VGDAs for a particular volume group to
determine if a volume group is going to be varied on or off.
When attempting to vary on a volume group, the system checks for a
quorum of the VGDA to be available. A quorum is equal to 51 percent or
more of the VGDAs available. If it can find a quorum, the VG will be
varied on; if not, it will not make the volume group available.
Turning off the quorum does not allow a varyonvg without a quorum, it
just prevents the closing of an active vg when losing its quorum. (so
forced varyon may needed: varyonvg -f VGname)
After turning it off (
chvg -Qn VGname) it is in effect immediately.
LTG
LTG is the maximum transfer size of a logical volume (volume group?).
At 5.3 and 6.1 AIX dynamically sets LTG size (calculated at each volume group activation).
LTG size can be changed with:
varyonvg -M<LTG size>
(The chvg -L has no effect on volume groups created on 5.3 or later (it was used on 5.2)
To display the LTG size of a disk:
/usr/sbin/lquerypv -M <hdisk#>
lsvg lists the volume groups that are on the system
lsvg -o lists all volume groups that are varied on
lsvg -o | lsvg -ip lists pvs of online vgs
lsvg rootvg gives details about the vg (lsvg
-L <vgname>, will doe not wait for the lock release (useful
during mirroring))
lsvg -l rootvg info about all logical volumes that are part of a vg
lsvg -M rootvg lists all PV, LV, PP deatils of a vg (PVname:PPnum LVname: LPnum :Copynum)
lsvg -p rootvg display all physical volumes that are part of the volume group
lsvg -n <hdisk> shows vg infos, but it is read from the VGDA on the specified disk (it is useful to compare it with different disks)
mkvg -s 2 -y testvg hdisk13 creates a volume group
-s specify the physical partition size
-y indicate the name of the new vg
chvg changes the characteristics of a volume group
chvg -u <vgname> unlocks the volume group (if a command core dumping, or the system crashed and vg is left in locked state)
(Many LVM commands place a lock into the ODM to prevent other commands working on the same time.)
extendvg rootvg hdisk7 adds hdisk7 to rootvg (-f forces it: extendvg -f ...)
reducevg rootvg hdisk7 tries to delete hdisk7 (the vg must be varied on) (reducevg -f ... :force it)
(it will fail if the vg contains open logical volumes)
reducevg datavg <pvid> reducevg command can use pvid as well (it is useful, if disk already removed from ODM, but VGDA still exist)
syncvg synchronizes stale physical partitions (varyonvg better, becaue first it reestablis reservation then sync in backg.)
varyonvg rootvg makes the vg available (-f force a vg to be online if it does not have the quorum of available disks)
(varyonvg acts as a self-repair program for VGDAs, it does a syncvg as well)
varyoffvg rootvg deactivate a volume group
mirrorvg -S P01vg hdisk1 mirroring rootvg to hdisk1 (checking: lsvg P01vg | grep STALE) (-S: background sync)
(
mirrorvg -m rootvg hdisk1 <--m makes an exact copy, pp mapping will be identical, it is advised this way)
unmirrorvg testvg1 hdisk0 hdisk1 remove mirrors on the vg from the specified disks
exportvg avg removes the VGs definition out of the ODM and /etc/filesystems (for ODM problems after importvg will fix it)
importvg -y avg hdisk8 makes the previously exported vg known to the system (hdisk8, is any disk belongs to the vg)
reorgvg rearranges physical partitions within the vg to conform with the placement policy (outer edge...) for the lv.
(For this 1 free pp is needed, and the relocatable flag for lvs must be set to 'y': chlv -r...)
getlvodm -j <hdisk> get the vgid for the hdisk from the odm
getlvodm -t <vgid> get the vg name for the vgid from the odm
getlvodm -v <vgname> get the vgid for the vg name from the odm
getlvodm -p <hdisk> get the pvid for the hdisk from the odm
getlvodm -g <pvid> get the hdisk for the pvid from the odm
lqueryvg -tcAp <hdisk> get all the vgid and pvid information for the vg from the vgda (directly from the disk)
(you can compare the disk with odm: getlvodm <-> lqueryvg)
synclvodm <vgname> synchronizes or rebuilds the lvcb, the device configuration database, and the vgdas on the physical volumes
redefinevg it helps regain the basic ODM informations if those are corrupted (redefinevg -d hdisk0 rootvg)
readvgda hdisk40 shows details from the disk
Physical Volume states (and quorum):
lsvg -p VGName <--shows pv states (not devices states!)
active <--during varyonvg disk can be accessed
missing <--during varyonvg disk can not be accessed + quorum is available
(after disk repair varyonvg VGName will put in active state)
removed <--no disk access during varyonvg + quorum is not available --> you issue varyonvg -f VGName
(after force varyonvg in the above case, PV state will be removed, and it won't be used for quorum)
(to put back in active state, first we have to tell the system the failure is over:)
(chpv -va hdiskX, this defines the disk as active, and after that varyonvg will synchronize)
---------------------------------------
Mirroring rootvg (i.e after disk replacement):
1. disk replaced -> cfgmgr <--it will find the new disk (i.e. hdisk1)
2. extendvg rootvg hdisk1 <--sometimes extendvg -f rootvg...
(3. chvg -Qn rootvg) <--only if quorum setting has not yet been disabled, because this needs a restart
4. mirrorvg -s rootvg <--add mirror for rootvg (-s: synchronization will not be done)
5. syncvg -v rootvg <--synchronize the new copy (lsvg rootvg | grep STALE)
6. bosboot -a <--we changed the system so create boot image (-a: create complete boot image and device)
(hd5 is mirrorred, no need to do it for each disk. ie. bosboot -ad hdisk0)
7. bootlist -m normal hdisk0 hdisk1 <--set normal bootlist
8. bootlist -m service hdisk0 hdisk1 <--set bootlist when we want to boot into service mode
(9. shutdown -Fr) <--this is needed if quorum has been disabled
10.bootinfo -b <--shows the disk which was used for boot
---------------------------------------
Export/Import:
1. node A: umount all fs -> varyoffvg myvg
2. node A: exportvg myvg <--ODM cleared
3. node B: importvg -y myvg hdisk3 <-- -y: vgname, if omitted a new vg will be created (if needed varyonvg -> mount fs)
if fs already exists:
1. umount the old one and mount the imported one with this:
mount -o log=/dev/loglv01 -V jfs2 /dev/lv24 /home/michael
(these details have to added in the mount command, and these can be retreived from LVCB: getlvcb lv24 -At)
2.
vi /etc/filesystems, create a second stanza for the imported filesystems with a new mountpoint.
---------------------------------------
VG problems with ODM:
if varyoff possible:
1. write down VGs name, major number, a disk
2. exportvg VGname
3. importvg -V MajorNum -y VGname hdiskX
if varyoff not possible:
1. write down VGs name, major number, a disk
2. export the vg by the backdoor, using: odmdelete
3. re-import vg (may produce warnings, but works)
(it is not necessary to umount fs or stop processes)
---------------------------------------
Changing factor value (chvg -t) of a VG:
A normal or a big vg has the following limitations after creation:
MAX PPs per VG = MAX PVs * MAX PPS per PV)
Normal Big
MAX PPs per VG: 32512 130048
MAX PPs per PV: 1016 1016
MAX PVs: 32 128
If we want to extend the vg with a disk, which is so large that we would have more than 1016 PPs on that disk, we will receive:
root@bb_lpar: / #
extendvg bbvg hdisk4
0516-1162 extendvg: Warning, The Physical Partition Size of 4 requires the
creation of 1024 partitions for hdisk4. The limitation for volume group
bbvg is 1016 physical partitions per physical volume. Use chvg command
with -t option to attempt to change the maximum Physical Partitions per
Physical volume for this volume group.
If we change the factor value of the VG, then extendvg will be possible:
root@bb_lpar: / #
chvg -t 2 bbvg
0516-1164 chvg: Volume group bbvg changed. With given characteristics bbvg
can include up to 16 physical volumes with 2032 physical partitions each.
Calculation:
Normal VG: 32/factor = new value of MAX PVs
Big VG: 128/factor= new value of MAX PVs
-t PPs per PV MAX PV (Normal) MAX PV (Big)
1 1016 32 128
2 2032 16 64
3 3048 10 42
4 4064 8 32
5 5080 6 25
...
"chvg -t" can be used online either increasing or decreasing the value of the factor.
---------------------------------------
Changing Normal VG to Big VG:
If you reached the MAX PV limit of a Normal VG and playing with the
factor (chvg -t) is not possible anymore you can convert it to Big VG.
It is an online activity, but there must be free PPs on each physical volume, because VGDA will be expanded on all disks:
root@bb_lpar: / #
lsvg -p bbvg
bbvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk2 active 511 2 02..00..00..00..00
hdisk3 active 511 23 00..00..00..00..23
hdisk4 active 1023 0 00..00..00..00..00
root@bb_lpar: / #
chvg -B bbvg
0516-1214 chvg: Not enough free physical partitions exist on hdisk4 for the
expansion of the volume group descriptor area. Migrate/reorganize to free up
2 partitions and run chvg again.
In this case we have to migrate 2 PPs from hdisk4 to hdsik3 (so 2 PPs will be freed up on hdisk4):
root@bb_lpar: / #
lspv -M hdisk4
hdisk4:1 bblv:920
hdisk4:2 bblv:921
hdisk4:3 bblv:922
hdisk4:4 bblv:923
hdisk4:5 bblv:924
...
root@bb_lpar: / #
lspv -M hdisk3
hdisk3:484 bblv:3040
hdisk3:485 bblv:3041
hdisk3:486 bblv:3042
hdisk3:487 bblv:1
hdisk3:488 bblv:2
hdisk3:489-511
root@bb_lpar: / #
migratelp bblv/920 hdisk3/489
root@bb_lpar: / #
migratelp bblv/921 hdisk3/490
root@bb_lpar: / #
lsvg -p bbvg
bbvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk2 active 511 2 02..00..00..00..00
hdisk3 active 511 21 00..00..00..00..21
hdisk4 active 1023 2 02..00..00..00..00
If we try again changing to Big VG, now it is successful:
root@bb_lpar: / #
chvg -B bbvg
0516-1216 chvg: Physical partitions are being migrated for volume group
descriptor area expansion. Please wait.
0516-1164 chvg: Volume group bbvg2 changed. With given characteristics bbvg2
can include up to 128 physical volumes with 1016 physical partitions each.
If you check again, freed up PPs has been used:
root@bb_lpar: / #
lsvg -p bbvg
bbvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk2 active 509 0 00..00..00..00..00
hdisk3 active 509 17 00..00..00..00..17
hdisk4 active 1021 0 00..00..00..00..00
---------------------------------------
Changing Normal (or Big) VG to Scalable VG:
If you reached the MAX PV limit of a Normal or a Big VG and playing with
the factor (chvg -t) is not possible anymore you can convert that VG to
Scalable VG. A Scalable VG allows a maximum of 1024 PVs and 4096 LVs
and a very big advantage that the maximum number of PPs applies to the
entire VG and is no longer defined on a per disk basis.
!!!Converting to Scalable VG is an offline activity (varyoffvg), and
there must be free PPs on each physical volume, because VGDA will be
expanded on all disks.
root@bb_lpar: / #
chvg -G bbvg
0516-1707 chvg: The volume group must be varied off during conversion to
scalable volume group format.
root@bb_lpar: / #
varyoffvg bbvg
root@bb_lpar: / #
chvg -G bbvg
0516-1214 chvg: Not enough free physical partitions exist on hdisk2 for the
expansion of the volume group descriptor area. Migrate/reorganize to free up
18 partitions and run chvg again.
After migrating some lps to free up required PPs (in this case it was 18), then changing to Scalable VG is successful:
root@bb_lpar: / #
chvg -G bbvg
0516-1224 chvg: WARNING, once this operation is completed, volume group bbvg
cannot be imported into AIX 5.2 or lower versions. Continue (y/n) ?
...
0516-1712 chvg: Volume group bbvg changed. bbvg can include up to 1024
physical volumes with 2097152 total physical partitions in the volume
group.
---------------------------------------
0516-008 varyonvg: LVM system call returned an unknown error code (2).
solution: export LDR_CNTRL=MAXDATA=0x80000000@DSA (check
/etc/environment if LDR_CNTRL has a value, which is causing the trouble)
---------------------------------------
If VG cannot be created:
root@aix21c: / #
mkvg -y tvg hdisk29
0516-1376 mkvg: Physical volume contains a VERITAS volume group.
0516-1397 mkvg: The physical volume hdisk29, will not be added to
the volume group.
0516-862 mkvg: Unable to create volume group.
root@aixc: / #
chpv -C hdisk29 <--clears owning volume manager from a disk, after this mkvg was successful
---------------------------------------
root@aix1: /root # importvg -L testvg -n hdiskpower12
0516-022 : Illegal parameter or structure value.
0516-780 importvg: Unable to import volume group from hdiskpower12.
For me the solution was:
there was no pvid on the disk, after adding it (chdev -l hdiskpower12 -a pv=yes) it was OK
---------------------------------------
Reorgvg log files, and how it is working:
reorgvg activity is logged in
lvmcfg:
root@bb_lpar: / #
alog -ot lvmcfg | tail -3
[
S 17039512 6750244 10/23/11-12:39:05:781 reorgvg.sh 580] reorgvg bbvg bb1lv
[S 7405650 17039512 10/23/11-12:39:06:689 migfix.c 168] migfix
/tmp/.workdir.9699494.17039512_1/freemap17039512
/tmp/.workdir.9699494.17039512_1/migrate17039512
/tmp/.workdir.9699494.17039512_1/lvm_moves17039512
[
E 17039512 47:320 reorgvg.sh 23] reorgvg: exited with rc=0
Field of these lines:
S - Start, E - End; PID, PPID; TIMESTAMP
At E (end) line shows how long reorgvg was running (in second:milliseconds):
47:320 = 47s 320ms
for a long running reorgvg, you can see it's status:
1. check the working dir of reorgvg
root@aixdb2: /root #
alog -ot lvmcfg | tail -3 | grep workdir
[S 5226546 5288122 10/22/11-13:55:11:001 migfix.c 165] migfix
/tmp/.workdir.4935912.5288122_1/freemap5288122 /tmp/.workdir.4935912.5288122_1/migrate5288122 /tmp/.workdir.4935912.5288122_1/lvm_moves5288122
2. check lvm_moves file in that dir (we will need the path of this file):
root@aixdb2: /root #
ls -l /tmp/.workdir.4935912.5288122_1 | grep lvm_moves
-rw------- 1 root system 1341300 Oct 22 13:55 lvm_moves5288122
(it contains all the lp migartions, and reorgvg goes through on this file, line by line)
3. check the process of reorgvg:
root@aixdb2: /root #
ps -ef | grep reorgvg
root
5288122 5013742 0 13:52:16 pts/2 0:12 /bin/ksh /usr/sbin/reorgvg P_NAVISvg p_datlv
root@aixdb2: /root #
ps -fT 5288122
CMD
/bin/ksh /usr/sbin/reorgvg P_NAVISvg p_datlv
|\--lmigratepp
-g 00c0ad0200004c000000012ce4ad7285 -p 00c80ef201f81fa6 -n 1183 -P 00c0ad021d62f017 -N 1565
\--awk -F: {print "-p "$1" -n "$2" -P "$3" -N "$4 }
(lmigratepp shows: -g VGid -p SourcePVid -n SourcePPnumber -P DestinationPVid -N DestinationPPnumber)
lmigratepp shows the actual PP which is migrated at this moment
(if you ask few seconds later it will show the next PP which is migrated, and it uses the lvm_moves file)
4. check the line number of the PP which is being migrated at this moment:
(now the ps command in step 3 is extended with the content of the lvm_moves file)
root@aixdb2: /root #
grep -n `ps -fT 5288122|grep migr|awk '{print $12":"$14}'` /tmp/.workdir.4935912.5288122_1/lvm_moves5288122
17943:00c0ad021d66f58b:205:00c0ad021d612cda:1259
17944:00c80ef24b619875:486:00c0ad021d66f58b:205
you can compare the above line numbers (17943, 17944) to how many lines lvm_moves file has.
root@aixdb2: /root #
cat /tmp/.workdir.4935912.5288122_1/lvm_moves5288122 | wc -l
31536
It shows that from 31536 lp migrations we are at this moment at 17943.
---------------------------------------
0516-304 : Unable to find device id 00080e82dfb5a427 in the Device
Configuration Database.
If a disk has been deleted (rmdev) somehow, but from the vg it was not removed:
root@bb_lpar: / #
lsvg -p newvg
newvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk2 active 31 20 06..00..02..06..06
0516-304 : Unable to find device id 00080e82dfb5a427 in the Device
Configuration Database.
00080e82dfb5a427 removed 31 31 07..06..06..06..06
VGDA still shows the missing disk is part of the vg:
root@bb_lpar: / #
lqueryvg -tAp hdisk2
...
Physical: 00080e82dfab25bc 2 0
00080e82dfb5a427 0 4
VGDA should be updated (on hdisk2) but it is possible only, if the PVID is used with reducevg:
root@bb_lpar: / #
reducevg newvg 00080e82dfb5a427
---------------------------------------
If you run into not being able to access an hdiskpowerX disk, you may need to reset the reservation bit on it:
root@aix21: / # lqueryvg -tAp hdiskpower13
0516-024 lqueryvg: Unable to open physical volume.
Either PV was not configured or could not be opened. Run diagnostics.
root@aix21: / # lsvg -p sapvg
PTAsapvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdiskpower12 active 811 0 00..00..00..00..00
hdiskpower13 missing 811 0 00..00..00..00..00
hdiskpower14 active 811 0 00..00..00..00..00
root@aix21: / # bootinfo -s hdiskpower13
0
Possible solution could be emcpowerreset:
root@aix21: / #
/usr/lpp/EMC/Symmetrix/bin/emcpowerreset fscsi0 hdiskpower13
(after this varyonvg will bring back the disk into active state)
