== conf-sanity test 69: replace an OST with the same index ========================================================== 22:09:07 (1781230147)
start mds service on oleg446-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg446-server'
oleg446-server: oleg446-server.virtnet: executing load_modules_local
oleg446-server: Loading modules from /home/green/git/lustre-release/lustre
oleg446-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg446-server: detected 4 online CPUs by sysfs
oleg446-server: MODOPTS_LIBCFS=
oleg446-server: Force libcfs to create 2 CPU partitions
oleg446-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg446-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg446-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg446-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
Started lustre-OST0000
oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg446-client.virtnet:  -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 554 (time 1781230240.36 total 10.01 last 55.33)
 - open/close 1129 (time 1781230250.36 total 20.01 last 57.50)
 - open/close 1714 (time 1781230260.38 total 30.03 last 58.42)
 - open/close 2287 (time 1781230270.38 total 40.03 last 57.28)
 - open/close 2901 (time 1781230280.39 total 50.04 last 61.35)
 - open/close 3487 (time 1781230290.40 total 60.05 last 58.53)
 - open/close 4078 (time 1781230300.41 total 70.06 last 59.04)
 - open/close 4619 (time 1781230310.43 total 80.07 last 54.02)
 - open/close 5200 (time 1781230320.43 total 90.08 last 58.07)
 - open/close 5865 (time 1781230330.45 total 100.09 last 66.40)
 - open/close 6512 (time 1781230340.45 total 110.10 last 64.70)
 - open/close 7125 (time 1781230350.45 total 120.10 last 61.28)
 - open/close 7700 (time 1781230360.46 total 130.11 last 57.44)
 - open/close 8324 (time 1781230370.47 total 140.12 last 62.33)
 - open/close 8878 (time 1781230380.48 total 150.13 last 55.34)
 - open/close 9393 (time 1781230390.50 total 160.15 last 51.38)
 - open/close 9954 (time 1781230400.51 total 170.16 last 56.08)
total: 10000 open/close in 171.04 seconds: 58.46 ops/second
 - unlinked 0 (time 1781230406 ; total 0 ; last 0)
total: 10000 unlinks in 115 seconds: 86.956520 unlinks/second
 - open/close 559 (time 1781230536.58 total 10.02 last 55.78)
 - open/close 1123 (time 1781230546.59 total 20.03 last 56.35)
 - open/close 1712 (time 1781230556.60 total 30.03 last 58.87)
 - open/close 2285 (time 1781230566.62 total 40.06 last 57.18)
 - open/close 2803 (time 1781230576.62 total 50.06 last 51.79)
 - open/close 3400 (time 1781230586.62 total 60.06 last 59.69)
 - open/close 4014 (time 1781230596.63 total 70.07 last 61.35)
 - open/close 4628 (time 1781230606.65 total 80.09 last 61.28)
 - open/close 5170 (time 1781230616.65 total 90.09 last 54.18)
 - open/close 5724 (time 1781230626.67 total 100.11 last 55.28)
 - open/close 6253 (time 1781230636.70 total 110.14 last 52.75)
 - open/close 6827 (time 1781230646.72 total 120.16 last 57.29)
 - open/close 7396 (time 1781230656.74 total 130.18 last 56.82)
 - open/close 7950 (time 1781230666.75 total 140.19 last 55.34)
 - open/close 8555 (time 1781230676.75 total 150.19 last 60.49)
 - open/close 9183 (time 1781230686.75 total 160.19 last 62.77)
 - open/close 9754 (time 1781230696.76 total 170.20 last 57.06)
total: 10000 open/close in 174.35 seconds: 57.36 ops/second
 - unlinked 0 (time 1781230705 ; total 0 ; last 0)
total: 10000 unlinks in 109 seconds: 91.743118 unlinks/second
 - open/close 629 (time 1781230829.10 total 10.02 last 62.79)
 - open/close 1265 (time 1781230839.10 total 20.02 last 63.57)
 - open/close 1875 (time 1781230849.11 total 30.03 last 60.95)
 - open/close 2458 (time 1781230859.12 total 40.04 last 58.25)
 - open/close 3053 (time 1781230869.13 total 50.05 last 59.46)
 - open/close 3636 (time 1781230879.13 total 60.05 last 58.27)
 - open/close 4203 (time 1781230889.15 total 70.06 last 56.62)
 - open/close 4870 (time 1781230899.15 total 80.07 last 66.66)
 - open/close 5494 (time 1781230909.16 total 90.08 last 62.37)
 - open/close 6153 (time 1781230919.18 total 100.10 last 65.77)
 - open/close 6781 (time 1781230929.20 total 110.12 last 62.64)
 - open/close 7330 (time 1781230939.21 total 120.12 last 54.88)
 - open/close 7911 (time 1781230949.23 total 130.15 last 57.97)
 - open/close 8532 (time 1781230959.24 total 140.16 last 62.01)
 - open/close 9133 (time 1781230969.26 total 150.17 last 60.02)
 - open/close 9762 (time 1781230979.27 total 160.19 last 62.78)
total: 10000 open/close in 164.07 seconds: 60.95 ops/second
 - unlinked 0 (time 1781230986 ; total 0 ; last 0)
total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second
 - open/close 594 (time 1781231105.63 total 10.01 last 59.36)
 - open/close 1235 (time 1781231115.63 total 20.01 last 64.07)
 - open/close 1803 (time 1781231125.64 total 30.02 last 56.75)
 - open/close 2385 (time 1781231135.66 total 40.03 last 58.12)
 - open/close 3023 (time 1781231145.67 total 50.04 last 63.74)
 - open/close 3592 (time 1781231155.67 total 60.04 last 56.89)
 - open/close 4223 (time 1781231165.68 total 70.05 last 63.05)
 - open/close 4865 (time 1781231175.68 total 80.05 last 64.18)
 - open/close 5464 (time 1781231185.68 total 90.06 last 59.86)
 - open/close 6048 (time 1781231195.69 total 100.06 last 58.38)
 - open/close 6725 (time 1781231205.69 total 110.07 last 67.69)
 - open/close 7362 (time 1781231215.70 total 120.07 last 63.65)
 - open/close 7996 (time 1781231225.70 total 130.07 last 63.39)
 - open/close 8603 (time 1781231235.70 total 140.08 last 60.69)
 - open/close 9222 (time 1781231245.72 total 150.10 last 61.77)
 - open/close 9684 (time 1781231255.73 total 160.10 last 46.16)
total: 10000 open/close in 166.21 seconds: 60.17 ops/second
 - unlinked 0 (time 1781231266 ; total 0 ; last 0)
total: 10000 unlinks in 110 seconds: 90.909088 unlinks/second
 - open/close 568 (time 1781231391.67 total 10.01 last 56.76)
 - open/close 1150 (time 1781231401.67 total 20.01 last 58.18)
 - open/close 1720 (time 1781231411.69 total 30.03 last 56.90)
 - open/close 2312 (time 1781231421.71 total 40.05 last 59.10)
 - open/close 2933 (time 1781231431.72 total 50.06 last 61.99)
 - open/close 3523 (time 1781231441.73 total 60.07 last 58.96)
 - open/close 4116 (time 1781231451.73 total 70.07 last 59.29)
 - open/close 4656 (time 1781231461.73 total 80.07 last 53.99)
 - open/close 5138 (time 1781231471.74 total 90.08 last 48.15)
 - open/close 5679 (time 1781231481.75 total 100.09 last 54.09)
 - open/close 6220 (time 1781231491.76 total 110.10 last 54.01)
 - open/close 6752 (time 1781231501.77 total 120.11 last 53.14)
 - open/close 7283 (time 1781231511.80 total 130.14 last 52.98)
 - open/close 7828 (time 1781231521.82 total 140.16 last 54.39)
 - open/close 8383 (time 1781231531.82 total 150.16 last 55.50)
 - open/close 8987 (time 1781231541.82 total 160.16 last 60.40)
 - open/close 9570 (time 1781231551.82 total 170.16 last 58.29)
total: 10000 open/close in 176.92 seconds: 56.52 ops/second
 - unlinked 0 (time 1781231563 ; total 0 ; last 0)
total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second
 - open/close 552 (time 1781231685.49 total 10.01 last 55.14)
 - open/close 1073 (time 1781231695.50 total 20.02 last 52.04)
 - open/close 1625 (time 1781231705.51 total 30.03 last 55.15)
 - open/close 2233 (time 1781231715.51 total 40.04 last 60.78)
 - open/close 2789 (time 1781231725.51 total 50.04 last 55.59)
 - open/close 3363 (time 1781231735.51 total 60.04 last 57.39)
 - open/close 3951 (time 1781231745.53 total 70.05 last 58.72)
 - open/close 4588 (time 1781231755.55 total 80.07 last 63.58)
 - open/close 5164 (time 1781231765.55 total 90.07 last 57.58)
 - open/close 5743 (time 1781231775.55 total 100.07 last 57.89)
 - open/close 6341 (time 1781231785.56 total 110.08 last 59.75)
 - open/close 6935 (time 1781231795.58 total 120.11 last 59.26)
 - open/close 7567 (time 1781231805.60 total 130.12 last 63.10)
 - open/close 8181 (time 1781231815.62 total 140.14 last 61.30)
 - open/close 8827 (time 1781231825.62 total 150.14 last 64.60)
 - open/close 9437 (time 1781231835.62 total 160.14 last 61.00)
total: 10000 open/close in 168.51 seconds: 59.34 ops/second
 - unlinked 0 (time 1781231848 ; total 0 ; last 0)
total: 10000 unlinks in 113 seconds: 88.495575 unlinks/second
 - open/close 599 (time 1781231976.91 total 10.01 last 59.84)
 - open/close 1165 (time 1781231986.92 total 20.02 last 56.54)
 - open/close 1789 (time 1781231996.93 total 30.03 last 62.36)
 - open/close 2407 (time 1781232006.94 total 40.04 last 61.75)
 - open/close 3023 (time 1781232016.94 total 50.04 last 61.56)
 - open/close 3644 (time 1781232026.96 total 60.06 last 62.01)
 - open/close 4239 (time 1781232036.97 total 70.06 last 59.46)
 - open/close 4850 (time 1781232046.97 total 80.07 last 61.09)
 - open/close 5468 (time 1781232056.97 total 90.07 last 61.80)
 - open/close 6054 (time 1781232066.98 total 100.07 last 58.55)
 - open/close 6675 (time 1781232076.98 total 110.07 last 62.10)
 - open/close 7279 (time 1781232086.98 total 120.08 last 60.36)
 - open/close 7936 (time 1781232096.98 total 130.08 last 65.69)
 - open/close 8587 (time 1781232106.99 total 140.09 last 65.06)
 - open/close 9038 (time 1781232117.01 total 150.10 last 45.03)
 - open/close 9607 (time 1781232127.01 total 160.11 last 56.86)
total: 10000 open/close in 166.81 seconds: 59.95 ops/second
 - unlinked 0 (time 1781232138 ; total 0 ; last 0)
total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second
 - open/close 598 (time 1781232260.35 total 10.01 last 59.75)
 - open/close 1164 (time 1781232270.35 total 20.02 last 56.56)
 - open/close 1621 (time 1781232280.36 total 30.03 last 45.65)
 - open/close 2173 (time 1781232290.38 total 40.05 last 55.09)
 - open/close 2692 (time 1781232300.39 total 50.06 last 51.86)
 - open/close 3221 (time 1781232310.40 total 60.06 last 52.88)
 - open/close 3752 (time 1781232320.40 total 70.06 last 53.08)
 - open/close 4316 (time 1781232330.40 total 80.07 last 56.39)
 - open/close 4880 (time 1781232340.40 total 90.07 last 56.39)
 - open/close 5489 (time 1781232350.42 total 100.09 last 60.78)
 - open/close 6147 (time 1781232360.43 total 110.09 last 65.78)
 - open/close 6634 (time 1781232370.46 total 120.12 last 48.53)
 - open/close 6986 (time 1781232380.48 total 130.14 last 35.15)
 - open/close 7522 (time 1781232390.49 total 140.15 last 53.54)
 - open/close 8092 (time 1781232400.49 total 150.16 last 56.96)
 - open/close 8806 (time 1781232410.50 total 160.17 last 71.33)
 - open/close 9403 (time 1781232420.50 total 170.17 last 59.70)
total: 10000 open/close in 179.57 seconds: 55.69 ops/second
 - unlinked 0 (time 1781232434 ; total 0 ; last 0)
total: 10000 unlinks in 109 seconds: 91.743118 unlinks/second
 - open/close 604 (time 1781232558.82 total 10.02 last 60.31)
 - open/close 1119 (time 1781232568.83 total 20.02 last 51.46)
 - open/close 1740 (time 1781232578.83 total 30.03 last 62.06)
 - open/close 2357 (time 1781232588.87 total 40.07 last 61.47)
 - open/close 2933 (time 1781232598.87 total 50.07 last 57.58)
 - open/close 3553 (time 1781232608.88 total 60.08 last 61.97)
 - open/close 4162 (time 1781232618.89 total 70.09 last 60.83)
 - open/close 4718 (time 1781232628.90 total 80.09 last 55.57)
 - open/close 5267 (time 1781232638.91 total 90.11 last 54.82)
 - open/close 5691 (time 1781232648.92 total 100.12 last 42.37)
 - open/close 6168 (time 1781232658.93 total 110.13 last 47.65)
 - open/close 6778 (time 1781232668.95 total 120.15 last 60.88)
 - open/close 7384 (time 1781232678.97 total 130.17 last 60.50)
 - open/close 7995 (time 1781232688.99 total 140.18 last 60.98)
 - open/close 8538 (time 1781232698.99 total 150.19 last 54.26)
 - open/close 9109 (time 1781232709.00 total 160.20 last 57.07)
 - open/close 9634 (time 1781232719.02 total 170.21 last 52.40)
total: 10000 open/close in 176.16 seconds: 56.77 ops/second
 - unlinked 0 (time 1781232730 ; total 0 ; last 0)
total: 10000 unlinks in 111 seconds: 90.090088 unlinks/second
 - open/close 580 (time 1781232856.40 total 10.03 last 57.85)
 - open/close 1111 (time 1781232866.42 total 20.04 last 53.00)
 - open/close 1614 (time 1781232876.43 total 30.06 last 50.24)
 - open/close 2191 (time 1781232886.43 total 40.06 last 57.69)
 - open/close 2764 (time 1781232896.44 total 50.06 last 57.27)
 - open/close 3362 (time 1781232906.46 total 60.08 last 59.69)
 - open/close 3883 (time 1781232916.46 total 70.09 last 52.08)
 - open/close 4354 (time 1781232926.48 total 80.11 last 47.00)
 - open/close 4906 (time 1781232936.49 total 90.11 last 55.18)
 - open/close 5461 (time 1781232946.50 total 100.13 last 55.41)
 - open/close 5932 (time 1781232956.51 total 110.14 last 47.05)
 - open/close 6334 (time 1781232966.53 total 120.16 last 40.13)
 - open/close 6826 (time 1781232976.53 total 130.16 last 49.19)
 - open/close 7372 (time 1781232986.56 total 140.18 last 54.48)
 - open/close 7921 (time 1781232996.56 total 150.19 last 54.85)
 - open/close 8563 (time 1781233006.58 total 160.20 last 64.13)
 - open/close 9110 (time 1781233016.58 total 170.20 last 54.69)
 - open/close 9595 (time 1781233026.58 total 180.20 last 48.50)
total: 10000 open/close in 187.39 seconds: 53.37 ops/second
 - unlinked 0 (time 1781233039 ; total 0 ; last 0)
total: 10000 unlinks in 112 seconds: 89.285713 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg446-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg446-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg446-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.146@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.146@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg446-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
Started lustre-OST0000
oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg446-server: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg446-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg446-client.virtnet:  -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre
On OST0, 17846 used inodes rc=0
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg446-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg446-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg446-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.146@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.146@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.146@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.146@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.146@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg446-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg446-server: oleg446-server.virtnet: executing set_default_debug -1 all
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 1
Started lustre-OST0000
oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg446-server: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg446-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg446-client.virtnet:  -o user_xattr,flock 192.168.204.146@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg446-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg446-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg446-server
stop mds service on oleg446-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg446-server

unloading modules via unload_modules_local on: 'oleg446-server'
oleg446-server: oleg446-server.virtnet: executing unload_modules_local
oleg446-server: 
modules unloaded.
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 2
pdsh@oleg446-client: oleg446-client: ssh exited with exit code 2
pdsh@oleg446-client: oleg446-client: ssh exited with exit code 2
pdsh@oleg446-client: oleg446-server: ssh exited with exit code 2