== conf-sanity test 69: replace an OST with the same index ========================================================== 18:24:37 (1787783077)
start mds service on oleg224-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing load_modules_local
oleg224-server: Loading modules from /home/green/git/lustre-release/lustre
oleg224-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg224-server: detected 4 online CPUs by sysfs
oleg224-server: MODOPTS_LIBCFS=
oleg224-server: Force libcfs to create 2 CPU partitions
oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg224-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 701 (time 1787783171.02 total 10.00 last 70.07)
 - open/close 1421 (time 1787783181.03 total 20.02 last 71.90)
 - open/close 2082 (time 1787783191.03 total 30.02 last 66.10)
 - open/close 2809 (time 1787783201.05 total 40.04 last 72.56)
 - open/close 3464 (time 1787783211.06 total 50.04 last 65.46)
 - open/close 4191 (time 1787783221.07 total 60.06 last 72.62)
 - open/close 4820 (time 1787783231.08 total 70.07 last 62.82)
 - open/close 5460 (time 1787783241.10 total 80.09 last 63.88)
 - open/close 6009 (time 1787783251.10 total 90.09 last 54.88)
 - open/close 6563 (time 1787783261.11 total 100.10 last 55.38)
 - open/close 7273 (time 1787783271.12 total 110.11 last 70.92)
 - open/close 7907 (time 1787783281.12 total 120.11 last 63.37)
 - open/close 8672 (time 1787783291.12 total 130.11 last 76.50)
 - open/close 9297 (time 1787783301.14 total 140.13 last 62.41)
total: 10000 open/close in 149.47 seconds: 66.90 ops/second
 - unlinked 0 (time 1787783315 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 738 (time 1787783427.10 total 10.01 last 73.75)
 - open/close 1346 (time 1787783437.13 total 20.04 last 60.61)
 - open/close 1852 (time 1787783447.14 total 30.04 last 50.57)
 - open/close 2451 (time 1787783457.14 total 40.05 last 59.87)
 - open/close 3176 (time 1787783467.15 total 50.05 last 72.47)
 - open/close 3811 (time 1787783477.15 total 60.06 last 63.46)
 - open/close 4540 (time 1787783487.16 total 70.06 last 72.89)
 - open/close 5233 (time 1787783497.16 total 80.07 last 69.25)
 - open/close 5973 (time 1787783507.17 total 90.07 last 73.96)
 - open/close 6692 (time 1787783517.17 total 100.08 last 71.88)
 - open/close 7415 (time 1787783527.17 total 110.08 last 72.29)
 - open/close 8115 (time 1787783537.18 total 120.09 last 69.94)
 - open/close 8833 (time 1787783547.19 total 130.09 last 71.74)
 - open/close 9565 (time 1787783557.19 total 140.10 last 73.18)
total: 10000 open/close in 145.65 seconds: 68.66 ops/second
 - unlinked 0 (time 1787783567 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 725 (time 1787783665.94 total 10.00 last 72.47)
 - open/close 1419 (time 1787783675.97 total 20.04 last 69.17)
 - open/close 2127 (time 1787783686.00 total 30.06 last 70.64)
 - open/close 2913 (time 1787783696.02 total 40.08 last 78.44)
 - open/close 3673 (time 1787783706.02 total 50.09 last 75.94)
 - open/close 4365 (time 1787783716.04 total 60.10 last 69.11)
 - open/close 5001 (time 1787783726.05 total 70.11 last 63.54)
 - open/close 5527 (time 1787783736.06 total 80.12 last 52.53)
 - open/close 6025 (time 1787783746.07 total 90.13 last 49.76)
 - open/close 6633 (time 1787783756.07 total 100.13 last 60.79)
 - open/close 7227 (time 1787783766.07 total 110.14 last 59.37)
 - open/close 7867 (time 1787783776.09 total 120.15 last 63.93)
 - open/close 8527 (time 1787783786.10 total 130.16 last 65.91)
 - open/close 9123 (time 1787783796.11 total 140.17 last 59.55)
 - open/close 9635 (time 1787783806.11 total 150.17 last 51.19)
total: 10000 open/close in 156.59 seconds: 63.86 ops/second
 - unlinked 0 (time 1787783817 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 679 (time 1787783922.97 total 10.01 last 67.83)
 - open/close 1319 (time 1787783932.98 total 20.02 last 63.93)
 - open/close 2003 (time 1787783942.99 total 30.03 last 68.35)
 - open/close 2596 (time 1787783953.00 total 40.05 last 59.21)
 - open/close 3340 (time 1787783963.01 total 50.05 last 74.35)
 - open/close 4128 (time 1787783973.02 total 60.06 last 78.77)
 - open/close 4927 (time 1787783983.02 total 70.06 last 79.86)
 - open/close 5653 (time 1787783993.03 total 80.07 last 72.56)
 - open/close 6504 (time 1787784003.03 total 90.08 last 85.03)
 - open/close 7194 (time 1787784013.04 total 100.08 last 68.95)
 - open/close 7871 (time 1787784023.04 total 110.08 last 67.70)
 - open/close 8590 (time 1787784033.05 total 120.09 last 71.84)
 - open/close 9355 (time 1787784043.05 total 130.09 last 76.48)
total: 10000 open/close in 138.40 seconds: 72.25 ops/second
 - unlinked 0 (time 1787784055 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 620 (time 1787784159.15 total 10.01 last 61.93)
 - open/close 1163 (time 1787784169.17 total 20.03 last 54.18)
 - open/close 1802 (time 1787784179.18 total 30.04 last 63.83)
 - open/close 2434 (time 1787784189.18 total 40.04 last 63.20)
 - open/close 3083 (time 1787784199.18 total 50.05 last 64.88)
 - open/close 3908 (time 1787784209.19 total 60.06 last 82.41)
 - open/close 4514 (time 1787784219.20 total 70.07 last 60.53)
 - open/close 5228 (time 1787784229.21 total 80.08 last 71.35)
 - open/close 5918 (time 1787784239.22 total 90.09 last 68.93)
 - open/close 6620 (time 1787784249.23 total 100.10 last 70.13)
 - open/close 7184 (time 1787784259.23 total 110.10 last 56.39)
 - open/close 7719 (time 1787784269.24 total 120.11 last 53.46)
 - open/close 8108 (time 1787784279.25 total 130.11 last 38.86)
 - open/close 8649 (time 1787784289.32 total 140.19 last 53.69)
 - open/close 9266 (time 1787784299.33 total 150.20 last 61.66)
total: 10000 open/close in 160.10 seconds: 62.46 ops/second
 - unlinked 0 (time 1787784313 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 432 (time 1787784419.99 total 10.01 last 43.16)
 - open/close 1130 (time 1787784429.99 total 20.02 last 69.75)
 - open/close 1785 (time 1787784440.00 total 30.02 last 65.49)
 - open/close 2425 (time 1787784450.01 total 40.03 last 63.91)
 - open/close 2942 (time 1787784460.02 total 50.04 last 51.65)
 - open/close 3524 (time 1787784470.03 total 60.05 last 58.14)
 - open/close 4133 (time 1787784480.04 total 70.06 last 60.83)
 - open/close 4810 (time 1787784490.05 total 80.07 last 67.63)
 - open/close 5514 (time 1787784500.05 total 90.08 last 70.39)
 - open/close 6276 (time 1787784510.06 total 100.08 last 76.14)
 - open/close 7103 (time 1787784520.07 total 110.09 last 82.67)
 - open/close 7761 (time 1787784530.07 total 120.10 last 65.75)
 - open/close 8310 (time 1787784540.14 total 130.16 last 54.55)
 - open/close 9007 (time 1787784550.16 total 140.18 last 69.57)
 - open/close 9762 (time 1787784560.17 total 150.19 last 75.43)
total: 10000 open/close in 153.73 seconds: 65.05 ops/second
 - unlinked 0 (time 1787784568 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 653 (time 1787784666.03 total 10.01 last 65.24)
 - open/close 1233 (time 1787784676.05 total 20.02 last 57.94)
 - open/close 1841 (time 1787784686.07 total 30.04 last 60.66)
 - open/close 2556 (time 1787784696.08 total 40.06 last 71.41)
 - open/close 3315 (time 1787784706.09 total 50.07 last 75.83)
 - open/close 4111 (time 1787784716.09 total 60.07 last 79.59)
 - open/close 4796 (time 1787784726.11 total 70.08 last 68.39)
 - open/close 5467 (time 1787784736.12 total 80.09 last 67.03)
 - open/close 6178 (time 1787784746.12 total 90.10 last 71.06)
 - open/close 6772 (time 1787784756.13 total 100.11 last 59.35)
 - open/close 7424 (time 1787784766.14 total 110.11 last 65.18)
 - open/close 8135 (time 1787784776.15 total 120.12 last 71.02)
 - open/close 8766 (time 1787784786.15 total 130.13 last 63.08)
 - open/close 9450 (time 1787784796.16 total 140.14 last 68.33)
total: 10000 open/close in 148.30 seconds: 67.43 ops/second
 - unlinked 0 (time 1787784809 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
 - open/close 674 (time 1787784910.97 total 10.02 last 67.24)
 - open/close 1367 (time 1787784920.98 total 20.03 last 69.23)
 - open/close 1997 (time 1787784930.98 total 30.04 last 62.96)
 - open/close 2732 (time 1787784940.98 total 40.04 last 73.50)
 - open/close 3464 (time 1787784950.99 total 50.05 last 73.16)
 - open/close 4182 (time 1787784960.99 total 60.05 last 71.77)
 - open/close 4826 (time 1787784971.00 total 70.06 last 64.35)
 - open/close 5576 (time 1787784981.00 total 80.06 last 74.98)
 - open/close 6223 (time 1787784991.01 total 90.07 last 64.65)
 - open/close 6951 (time 1787785001.01 total 100.07 last 72.77)
 - open/close 7600 (time 1787785011.04 total 110.09 last 64.76)
 - open/close 8268 (time 1787785021.06 total 120.12 last 66.63)
 - open/close 8905 (time 1787785031.07 total 130.12 last 63.66)
 - open/close 9556 (time 1787785041.08 total 140.14 last 65.03)
total: 10000 open/close in 145.83 seconds: 68.58 ops/second
 - unlinked 0 (time 1787785052 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 716 (time 1787785157.54 total 10.01 last 71.56)
 - open/close 1421 (time 1787785167.54 total 20.01 last 70.48)
 - open/close 1978 (time 1787785177.55 total 30.02 last 55.66)
 - open/close 2623 (time 1787785187.56 total 40.02 last 64.45)
 - open/close 3248 (time 1787785197.56 total 50.03 last 62.47)
 - open/close 3823 (time 1787785207.57 total 60.04 last 57.45)
 - open/close 4516 (time 1787785217.58 total 70.05 last 69.24)
 - open/close 5262 (time 1787785227.58 total 80.05 last 74.57)
 - open/close 5959 (time 1787785237.59 total 90.06 last 69.67)
 - open/close 6585 (time 1787785247.60 total 100.06 last 62.55)
 - open/close 7335 (time 1787785257.61 total 110.07 last 74.93)
 - open/close 7984 (time 1787785267.64 total 120.11 last 64.69)
 - open/close 8639 (time 1787785277.65 total 130.12 last 65.43)
 - open/close 9387 (time 1787785287.66 total 140.12 last 74.73)
total: 10000 open/close in 148.12 seconds: 67.51 ops/second
 - unlinked 0 (time 1787785299 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 673 (time 1787785412.37 total 10.01 last 67.22)
 - open/close 1345 (time 1787785422.38 total 20.02 last 67.14)
 - open/close 2021 (time 1787785432.39 total 30.03 last 67.55)
 - open/close 2689 (time 1787785442.39 total 40.03 last 66.80)
 - open/close 3460 (time 1787785452.40 total 50.04 last 77.04)
 - open/close 4258 (time 1787785462.40 total 60.04 last 79.75)
 - open/close 4960 (time 1787785472.41 total 70.05 last 70.18)
 - open/close 5707 (time 1787785482.41 total 80.05 last 74.70)
 - open/close 6325 (time 1787785492.43 total 90.07 last 61.67)
 - open/close 7010 (time 1787785502.44 total 100.08 last 68.43)
 - open/close 7650 (time 1787785512.44 total 110.08 last 63.99)
 - open/close 8292 (time 1787785522.44 total 120.08 last 64.19)
 - open/close 8902 (time 1787785532.45 total 130.09 last 60.95)
 - open/close 9582 (time 1787785542.45 total 140.09 last 67.96)
total: 10000 open/close in 146.08 seconds: 68.46 ops/second
 - unlinked 0 (time 1787785552 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg224-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.124@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.124@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg224-server: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg224-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre
On OST0, 18150 used inodes rc=0
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg224-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.124@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.124@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.124@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.124@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.124@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg224-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1
Started lustre-OST0000
oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg224-server: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg224-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg224-client.virtnet:  -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg224-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg224-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server
stop mds service on oleg224-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server

unloading modules via unload_modules_local on: 'oleg224-server'
oleg224-server: oleg224-server.virtnet: executing unload_modules_local
oleg224-server: 
modules unloaded.
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2
pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2