== conf-sanity test 69: replace an OST with the same index ========================================================== 09:45:49 (1786628749)
start mds service on oleg110-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg110-server'
oleg110-server: oleg110-server.virtnet: executing load_modules_local
oleg110-server: Loading modules from /home/green/git/lustre-release/lustre
oleg110-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg110-server: detected 4 online CPUs by sysfs
oleg110-server: MODOPTS_LIBCFS=
oleg110-server: Force libcfs to create 2 CPU partitions
oleg110-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg110-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg110-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg110-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1
Started lustre-OST0000
oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg110-client.virtnet:  -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 717 (time 1786628845.06 total 10.00 last 71.69)
 - open/close 1339 (time 1786628855.07 total 20.01 last 62.15)
 - open/close 1975 (time 1786628865.08 total 30.02 last 63.54)
 - open/close 2578 (time 1786628875.09 total 40.03 last 60.25)
 - open/close 3218 (time 1786628885.09 total 50.03 last 63.95)
 - open/close 3856 (time 1786628895.09 total 60.03 last 63.79)
 - open/close 4599 (time 1786628905.11 total 70.04 last 74.22)
 - open/close 5260 (time 1786628915.12 total 80.05 last 66.03)
 - open/close 5928 (time 1786628925.12 total 90.06 last 66.76)
 - open/close 6629 (time 1786628935.13 total 100.07 last 70.05)
 - open/close 7305 (time 1786628945.13 total 110.07 last 67.59)
 - open/close 8036 (time 1786628955.13 total 120.07 last 73.09)
 - open/close 8706 (time 1786628965.14 total 130.08 last 66.95)
 - open/close 9374 (time 1786628975.14 total 140.08 last 66.79)
total: 10000 open/close in 149.72 seconds: 66.79 ops/second
 - unlinked 0 (time 1786628989 ; total 0 ; last 0)
total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second
 - open/close 561 (time 1786629099.01 total 10.01 last 56.02)
 - open/close 1184 (time 1786629109.02 total 20.02 last 62.23)
 - open/close 1897 (time 1786629119.04 total 30.04 last 71.18)
 - open/close 2520 (time 1786629129.05 total 40.05 last 62.25)
 - open/close 3088 (time 1786629139.05 total 50.05 last 56.77)
 - open/close 3702 (time 1786629149.07 total 60.07 last 61.32)
 - open/close 4374 (time 1786629159.07 total 70.08 last 67.15)
 - open/close 4979 (time 1786629169.08 total 80.09 last 60.44)
 - open/close 5657 (time 1786629179.09 total 90.09 last 67.77)
 - open/close 6300 (time 1786629189.10 total 100.10 last 64.21)
 - open/close 7001 (time 1786629199.11 total 110.12 last 70.01)
 - open/close 7743 (time 1786629209.12 total 120.12 last 74.18)
 - open/close 8436 (time 1786629219.13 total 130.13 last 69.25)
 - open/close 9082 (time 1786629229.13 total 140.13 last 64.58)
 - open/close 9742 (time 1786629239.14 total 150.14 last 65.94)
total: 10000 open/close in 153.50 seconds: 65.15 ops/second
 - unlinked 0 (time 1786629247 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 746 (time 1786629359.09 total 10.01 last 74.54)
 - open/close 1356 (time 1786629369.10 total 20.02 last 60.93)
 - open/close 1992 (time 1786629379.11 total 30.03 last 63.55)
 - open/close 2595 (time 1786629389.11 total 40.04 last 60.25)
 - open/close 3425 (time 1786629399.12 total 50.04 last 82.95)
 - open/close 4061 (time 1786629409.13 total 60.05 last 63.57)
 - open/close 4766 (time 1786629419.13 total 70.05 last 70.44)
 - open/close 5479 (time 1786629429.14 total 80.06 last 71.26)
 - open/close 6261 (time 1786629439.14 total 90.06 last 78.18)
 - open/close 6904 (time 1786629449.17 total 100.09 last 64.14)
 - open/close 7478 (time 1786629459.18 total 110.10 last 57.34)
 - open/close 8140 (time 1786629469.18 total 120.10 last 66.18)
 - open/close 8741 (time 1786629479.20 total 130.12 last 59.98)
 - open/close 9479 (time 1786629489.21 total 140.13 last 73.75)
total: 10000 open/close in 149.40 seconds: 66.93 ops/second
 - unlinked 0 (time 1786629502 ; total 0 ; last 0)
total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second
 - open/close 563 (time 1786629619.99 total 10.03 last 56.14)
 - open/close 1127 (time 1786629629.99 total 20.03 last 56.38)
 - open/close 1810 (time 1786629640.00 total 30.04 last 68.28)
 - open/close 2488 (time 1786629650.01 total 40.05 last 67.73)
 - open/close 3194 (time 1786629660.01 total 50.05 last 70.59)
 - open/close 3792 (time 1786629670.02 total 60.06 last 59.75)
 - open/close 4433 (time 1786629680.03 total 70.07 last 64.04)
 - open/close 5071 (time 1786629690.04 total 80.07 last 63.74)
 - open/close 5730 (time 1786629700.05 total 90.09 last 65.81)
 - open/close 6346 (time 1786629710.06 total 100.10 last 61.53)
 - open/close 6925 (time 1786629720.09 total 110.13 last 57.73)
 - open/close 7551 (time 1786629730.09 total 120.13 last 62.58)
 - open/close 8143 (time 1786629740.10 total 130.14 last 59.17)
 - open/close 8800 (time 1786629750.11 total 140.15 last 65.61)
 - open/close 9432 (time 1786629760.13 total 150.17 last 63.07)
total: 10000 open/close in 159.51 seconds: 62.69 ops/second
 - unlinked 0 (time 1786629775 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 754 (time 1786629889.12 total 10.01 last 75.34)
 - open/close 1461 (time 1786629899.13 total 20.02 last 70.63)
 - open/close 2128 (time 1786629909.13 total 30.02 last 66.67)
 - open/close 2733 (time 1786629919.14 total 40.03 last 60.44)
 - open/close 3498 (time 1786629929.15 total 50.04 last 76.44)
 - open/close 4184 (time 1786629939.15 total 60.04 last 68.58)
 - open/close 4807 (time 1786629949.15 total 70.04 last 62.30)
 - open/close 5414 (time 1786629959.16 total 80.05 last 60.66)
 - open/close 6066 (time 1786629969.16 total 90.05 last 65.17)
 - open/close 6606 (time 1786629979.17 total 100.06 last 53.98)
 - open/close 7236 (time 1786629989.17 total 110.06 last 62.98)
 - open/close 7883 (time 1786629999.17 total 120.07 last 64.67)
 - open/close 8561 (time 1786630009.19 total 130.08 last 67.72)
 - open/close 9202 (time 1786630019.19 total 140.09 last 64.05)
 - open/close 9730 (time 1786630029.20 total 150.10 last 52.75)
total: 10000 open/close in 154.78 seconds: 64.61 ops/second
 - unlinked 0 (time 1786630039 ; total 0 ; last 0)
total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second
 - open/close 647 (time 1786630156.57 total 10.01 last 64.65)
 - open/close 1271 (time 1786630166.58 total 20.01 last 62.38)
 - open/close 1890 (time 1786630176.58 total 30.01 last 61.90)
 - open/close 2526 (time 1786630186.58 total 40.01 last 63.58)
 - open/close 3119 (time 1786630196.58 total 50.01 last 59.29)
 - open/close 3748 (time 1786630206.59 total 60.02 last 62.84)
 - open/close 4431 (time 1786630216.60 total 70.03 last 68.26)
 - open/close 5077 (time 1786630226.61 total 80.04 last 64.51)
 - open/close 5739 (time 1786630236.62 total 90.05 last 66.16)
 - open/close 6323 (time 1786630246.63 total 100.07 last 58.31)
 - open/close 6960 (time 1786630256.64 total 110.08 last 63.64)
 - open/close 7595 (time 1786630266.68 total 120.11 last 63.27)
 - open/close 8213 (time 1786630276.69 total 130.13 last 61.71)
 - open/close 8735 (time 1786630286.70 total 140.13 last 52.17)
 - open/close 9181 (time 1786630296.70 total 150.13 last 44.60)
 - open/close 9912 (time 1786630306.70 total 160.14 last 73.09)
total: 10000 open/close in 161.52 seconds: 61.91 ops/second
 - unlinked 0 (time 1786630313 ; total 0 ; last 0)
total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second
 - open/close 690 (time 1786630433.40 total 10.01 last 68.93)
 - open/close 1232 (time 1786630443.43 total 20.04 last 54.04)
 - open/close 1857 (time 1786630453.44 total 30.06 last 62.39)
 - open/close 2675 (time 1786630463.45 total 40.06 last 81.74)
 - open/close 3362 (time 1786630473.45 total 50.07 last 68.69)
 - open/close 3941 (time 1786630483.46 total 60.08 last 57.84)
 - open/close 4643 (time 1786630493.47 total 70.08 last 70.16)
 - open/close 5191 (time 1786630503.48 total 80.09 last 54.74)
 - open/close 5706 (time 1786630513.49 total 90.11 last 51.44)
 - open/close 6324 (time 1786630523.50 total 100.12 last 61.73)
 - open/close 6796 (time 1786630533.52 total 110.13 last 47.14)
 - open/close 7480 (time 1786630543.52 total 120.13 last 68.40)
 - open/close 7930 (time 1786630553.52 total 130.14 last 44.97)
 - open/close 8536 (time 1786630563.53 total 140.14 last 60.56)
 - open/close 9199 (time 1786630573.56 total 150.17 last 66.11)
 - open/close 9742 (time 1786630583.60 total 160.21 last 54.09)
total: 10000 open/close in 165.04 seconds: 60.59 ops/second
 - unlinked 0 (time 1786630594 ; total 0 ; last 0)
total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second
 - open/close 542 (time 1786630705.91 total 10.01 last 54.12)
 - open/close 971 (time 1786630715.93 total 20.03 last 42.81)
 - open/close 1353 (time 1786630725.97 total 30.07 last 38.05)
 - open/close 1945 (time 1786630735.98 total 40.08 last 59.16)
 - open/close 2532 (time 1786630745.98 total 50.09 last 58.67)
 - open/close 3095 (time 1786630756.00 total 60.11 last 56.19)
 - open/close 3616 (time 1786630766.01 total 70.11 last 52.07)
 - open/close 4182 (time 1786630776.03 total 80.13 last 56.49)
 - open/close 4795 (time 1786630786.04 total 90.14 last 61.24)
 - open/close 5425 (time 1786630796.04 total 100.15 last 62.95)
 - open/close 6026 (time 1786630806.07 total 110.17 last 59.94)
 - open/close 6607 (time 1786630816.07 total 120.18 last 58.09)
 - open/close 7215 (time 1786630826.08 total 130.18 last 60.75)
 - open/close 7867 (time 1786630836.09 total 140.19 last 65.15)
 - open/close 8475 (time 1786630846.09 total 150.19 last 60.79)
 - open/close 9106 (time 1786630856.11 total 160.22 last 62.94)
 - open/close 9675 (time 1786630866.13 total 170.23 last 56.81)
total: 10000 open/close in 176.01 seconds: 56.81 ops/second
 - unlinked 0 (time 1786630878 ; total 0 ; last 0)
total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second
 - open/close 635 (time 1786630989.83 total 10.00 last 63.47)
 - open/close 1224 (time 1786630999.85 total 20.02 last 58.83)
 - open/close 1848 (time 1786631009.85 total 30.02 last 62.39)
 - open/close 2483 (time 1786631019.88 total 40.05 last 63.33)
 - open/close 3184 (time 1786631029.88 total 50.05 last 70.09)
 - open/close 3957 (time 1786631039.89 total 60.06 last 77.22)
 - open/close 4580 (time 1786631049.90 total 70.07 last 62.23)
 - open/close 5272 (time 1786631059.91 total 80.08 last 69.13)
 - open/close 5948 (time 1786631069.92 total 90.09 last 67.54)
 - open/close 6622 (time 1786631079.92 total 100.09 last 67.38)
 - open/close 7264 (time 1786631089.93 total 110.10 last 64.14)
 - open/close 7884 (time 1786631099.94 total 120.11 last 61.93)
 - open/close 8560 (time 1786631109.94 total 130.11 last 67.58)
 - open/close 9163 (time 1786631119.97 total 140.14 last 60.17)
 - open/close 9612 (time 1786631129.98 total 150.15 last 44.82)
total: 10000 open/close in 158.21 seconds: 63.21 ops/second
 - unlinked 0 (time 1786631144 ; total 0 ; last 0)
total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second
 - open/close 678 (time 1786631254.23 total 10.02 last 67.68)
 - open/close 1279 (time 1786631264.24 total 20.03 last 60.01)
 - open/close 1956 (time 1786631274.26 total 30.05 last 67.58)
 - open/close 2643 (time 1786631284.27 total 40.06 last 68.64)
 - open/close 3381 (time 1786631294.27 total 50.06 last 73.76)
 - open/close 3957 (time 1786631304.30 total 60.09 last 57.45)
 - open/close 4311 (time 1786631314.32 total 70.11 last 35.34)
 - open/close 4927 (time 1786631324.32 total 80.11 last 61.58)
 - open/close 5566 (time 1786631334.34 total 90.13 last 63.79)
 - open/close 6278 (time 1786631344.34 total 100.13 last 71.19)
 - open/close 6824 (time 1786631354.35 total 110.14 last 54.56)
 - open/close 7224 (time 1786631364.36 total 120.15 last 39.95)
 - open/close 7729 (time 1786631374.36 total 130.15 last 50.50)
 - open/close 8239 (time 1786631384.37 total 140.16 last 50.95)
 - open/close 8819 (time 1786631394.39 total 150.18 last 57.91)
 - open/close 9449 (time 1786631404.41 total 160.20 last 62.85)
total: 10000 open/close in 169.33 seconds: 59.06 ops/second
 - unlinked 0 (time 1786631418 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg110-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg110-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg110-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.110@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.110@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg110-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1
Started lustre-OST0000
oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg110-server: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg110-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg110-client.virtnet:  -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg110-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg110-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg110-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.110@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.110@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.110@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.110@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.201.110@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg110-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1
Started lustre-OST0000
oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg110-server: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg110-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg110-client.virtnet:  -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg110-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg110-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg110-server
stop mds service on oleg110-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg110-server

unloading modules via unload_modules_local on: 'oleg110-server'
oleg110-server: oleg110-server.virtnet: executing unload_modules_local
oleg110-server: 
modules unloaded.
pdsh@oleg110-client: oleg110-client: ssh exited with exit code 2
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 2
pdsh@oleg110-client: oleg110-client: ssh exited with exit code 2
pdsh@oleg110-client: oleg110-server: ssh exited with exit code 2