== conf-sanity test 69: replace an OST with the same index ========================================================== 16:01:35 (1775764895)
start mds service on oleg308-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg308-server'
oleg308-server: oleg308-server.virtnet: executing load_modules_local
oleg308-server: Loading modules from /home/green/git/lustre-release/lustre
oleg308-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy
oleg308-server: detected 4 online CPUs by sysfs
oleg308-server: MODOPTS_LIBCFS=
oleg308-server: Force libcfs to create 2 CPU partitions
oleg308-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg308-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg308-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all
pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg308-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all
pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1
Started lustre-OST0000
oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg308-client.virtnet:  -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 746 (time 1775764980.04 total 10.01 last 74.52)
 - open/close 1372 (time 1775764990.05 total 20.02 last 62.52)
 - open/close 2103 (time 1775765000.05 total 30.03 last 73.07)
 - open/close 2839 (time 1775765010.06 total 40.03 last 73.55)
 - open/close 3600 (time 1775765020.07 total 50.05 last 76.02)
 - open/close 4274 (time 1775765030.09 total 60.06 last 67.32)
 - open/close 5070 (time 1775765040.09 total 70.07 last 79.53)
 - open/close 5858 (time 1775765050.09 total 80.07 last 78.79)
 - open/close 6663 (time 1775765060.10 total 90.07 last 80.45)
 - open/close 7384 (time 1775765070.11 total 100.08 last 72.03)
 - open/close 8192 (time 1775765080.11 total 110.09 last 80.77)
 - open/close 8916 (time 1775765090.12 total 120.09 last 72.39)
 - open/close 9707 (time 1775765100.12 total 130.10 last 79.04)
total: 10000 open/close in 134.11 seconds: 74.57 ops/second
 - unlinked 0 (time 1775765108 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 691 (time 1775765209.13 total 10.01 last 69.01)
 - open/close 1463 (time 1775765219.14 total 20.03 last 77.09)
 - open/close 2108 (time 1775765229.15 total 30.04 last 64.41)
 - open/close 2849 (time 1775765239.16 total 40.05 last 74.03)
 - open/close 3602 (time 1775765249.18 total 50.06 last 75.21)
 - open/close 4360 (time 1775765259.18 total 60.06 last 75.79)
 - open/close 5119 (time 1775765269.19 total 70.08 last 75.82)
 - open/close 5887 (time 1775765279.19 total 80.08 last 76.77)
 - open/close 6753 (time 1775765289.20 total 90.09 last 86.51)
 - open/close 7492 (time 1775765299.20 total 100.09 last 73.90)
 - open/close 8233 (time 1775765309.21 total 110.10 last 74.06)
 - open/close 8871 (time 1775765319.21 total 120.10 last 63.77)
 - open/close 9483 (time 1775765329.23 total 130.12 last 61.11)
total: 10000 open/close in 137.99 seconds: 72.47 ops/second
 - unlinked 0 (time 1775765342 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 678 (time 1775765443.39 total 10.01 last 67.71)
 - open/close 1375 (time 1775765453.39 total 20.02 last 69.69)
 - open/close 2064 (time 1775765463.40 total 30.02 last 68.85)
 - open/close 2808 (time 1775765473.40 total 40.03 last 74.35)
 - open/close 3547 (time 1775765483.41 total 50.03 last 73.86)
 - open/close 4360 (time 1775765493.41 total 60.04 last 81.28)
 - open/close 5117 (time 1775765503.42 total 70.05 last 75.64)
 - open/close 5845 (time 1775765513.42 total 80.05 last 72.78)
 - open/close 6599 (time 1775765523.44 total 90.06 last 75.30)
 - open/close 7300 (time 1775765533.44 total 100.06 last 70.08)
 - open/close 8103 (time 1775765543.45 total 110.07 last 80.24)
 - open/close 8784 (time 1775765553.45 total 120.08 last 68.07)
 - open/close 9561 (time 1775765563.46 total 130.08 last 77.65)
total: 10000 open/close in 135.91 seconds: 73.58 ops/second
 - unlinked 0 (time 1775765573 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 794 (time 1775765671.95 total 10.02 last 79.27)
 - open/close 1555 (time 1775765681.95 total 20.02 last 76.08)
 - open/close 2348 (time 1775765691.96 total 30.02 last 79.28)
 - open/close 3105 (time 1775765701.96 total 40.02 last 75.70)
 - open/close 3895 (time 1775765711.97 total 50.03 last 78.90)
 - open/close 4662 (time 1775765721.98 total 60.04 last 76.64)
 - open/close 5471 (time 1775765732.01 total 70.07 last 80.67)
 - open/close 6206 (time 1775765742.01 total 80.08 last 73.46)
 - open/close 6914 (time 1775765752.03 total 90.09 last 70.68)
 - open/close 7596 (time 1775765762.04 total 100.10 last 68.14)
 - open/close 8279 (time 1775765772.05 total 110.11 last 68.21)
 - open/close 8988 (time 1775765782.05 total 120.12 last 70.89)
 - open/close 9758 (time 1775765792.06 total 130.13 last 76.89)
total: 10000 open/close in 133.00 seconds: 75.19 ops/second
 - unlinked 0 (time 1775765799 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 735 (time 1775765896.73 total 10.01 last 73.43)
 - open/close 1518 (time 1775765906.73 total 20.01 last 78.27)
 - open/close 2292 (time 1775765916.74 total 30.02 last 77.31)
 - open/close 3019 (time 1775765926.76 total 40.04 last 72.58)
 - open/close 3854 (time 1775765936.76 total 50.05 last 83.47)
 - open/close 4524 (time 1775765946.77 total 60.05 last 66.96)
 - open/close 5280 (time 1775765956.78 total 70.06 last 75.51)
 - open/close 6014 (time 1775765966.79 total 80.07 last 73.37)
 - open/close 6598 (time 1775765976.81 total 90.09 last 58.27)
 - open/close 7377 (time 1775765986.82 total 100.10 last 77.79)
 - open/close 8159 (time 1775765996.82 total 110.10 last 78.18)
 - open/close 8946 (time 1775766006.83 total 120.12 last 78.61)
 - open/close 9720 (time 1775766016.84 total 130.12 last 77.34)
total: 10000 open/close in 133.89 seconds: 74.69 ops/second
 - unlinked 0 (time 1775766025 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 733 (time 1775766129.46 total 10.00 last 73.29)
 - open/close 1479 (time 1775766139.47 total 20.01 last 74.53)
 - open/close 2102 (time 1775766149.48 total 30.02 last 62.24)
 - open/close 2890 (time 1775766159.49 total 40.03 last 78.74)
 - open/close 3633 (time 1775766169.49 total 50.03 last 74.26)
 - open/close 4418 (time 1775766179.50 total 60.04 last 78.46)
 - open/close 5111 (time 1775766189.50 total 70.04 last 69.26)
 - open/close 5831 (time 1775766199.50 total 80.05 last 72.00)
 - open/close 6560 (time 1775766209.51 total 90.05 last 72.85)
 - open/close 7319 (time 1775766219.52 total 100.06 last 75.84)
 - open/close 8032 (time 1775766229.53 total 110.07 last 71.20)
 - open/close 8680 (time 1775766239.54 total 120.08 last 64.77)
 - open/close 9322 (time 1775766249.55 total 130.10 last 64.10)
 - open/close 9877 (time 1775766259.56 total 140.10 last 55.47)
total: 10000 open/close in 142.07 seconds: 70.39 ops/second
 - unlinked 0 (time 1775766266 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
 - open/close 734 (time 1775766367.91 total 10.00 last 73.37)
 - open/close 1448 (time 1775766377.92 total 20.02 last 71.29)
 - open/close 2195 (time 1775766387.93 total 30.03 last 74.62)
 - open/close 2978 (time 1775766397.95 total 40.04 last 78.20)
 - open/close 3711 (time 1775766407.95 total 50.04 last 73.30)
 - open/close 4486 (time 1775766417.95 total 60.05 last 77.46)
 - open/close 5237 (time 1775766427.96 total 70.05 last 75.07)
 - open/close 5968 (time 1775766437.96 total 80.05 last 73.09)
 - open/close 6665 (time 1775766447.97 total 90.06 last 69.64)
 - open/close 7419 (time 1775766457.98 total 100.07 last 75.32)
 - open/close 8174 (time 1775766467.99 total 110.08 last 75.43)
 - open/close 8912 (time 1775766478.01 total 120.10 last 73.65)
 - open/close 9677 (time 1775766488.01 total 130.11 last 76.47)
total: 10000 open/close in 134.41 seconds: 74.40 ops/second
 - unlinked 0 (time 1775766496 ; total 0 ; last 0)
total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second
 - open/close 789 (time 1775766600.58 total 10.01 last 78.82)
 - open/close 1348 (time 1775766610.59 total 20.02 last 55.85)
 - open/close 2022 (time 1775766620.59 total 30.02 last 67.39)
 - open/close 2771 (time 1775766630.60 total 40.03 last 74.86)
 - open/close 3526 (time 1775766640.60 total 50.03 last 75.45)
 - open/close 4212 (time 1775766650.61 total 60.04 last 68.54)
 - open/close 4910 (time 1775766660.61 total 70.04 last 69.79)
 - open/close 5655 (time 1775766670.62 total 80.05 last 74.48)
 - open/close 6392 (time 1775766680.62 total 90.05 last 73.65)
 - open/close 7154 (time 1775766690.64 total 100.07 last 76.04)
 - open/close 7754 (time 1775766700.65 total 110.08 last 59.98)
 - open/close 8542 (time 1775766710.66 total 120.09 last 78.73)
 - open/close 9287 (time 1775766720.66 total 130.09 last 74.50)
total: 10000 open/close in 139.20 seconds: 71.84 ops/second
 - unlinked 0 (time 1775766734 ; total 0 ; last 0)
total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second
 - open/close 724 (time 1775766843.38 total 10.00 last 72.39)
 - open/close 1524 (time 1775766853.38 total 20.00 last 79.99)
 - open/close 2333 (time 1775766863.39 total 30.01 last 80.82)
 - open/close 3111 (time 1775766873.40 total 40.02 last 77.77)
 - open/close 3825 (time 1775766883.40 total 50.02 last 71.36)
 - open/close 4502 (time 1775766893.41 total 60.03 last 67.62)
 - open/close 5065 (time 1775766903.44 total 70.07 last 56.12)
 - open/close 5729 (time 1775766913.45 total 80.07 last 66.34)
 - open/close 6422 (time 1775766923.46 total 90.08 last 69.27)
 - open/close 7143 (time 1775766933.46 total 100.08 last 72.07)
 - open/close 7939 (time 1775766943.47 total 110.09 last 79.53)
 - open/close 8678 (time 1775766953.48 total 120.10 last 73.82)
 - open/close 9447 (time 1775766963.50 total 130.12 last 76.79)
total: 10000 open/close in 139.94 seconds: 71.46 ops/second
 - unlinked 0 (time 1775766977 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
 - open/close 654 (time 1775767078.50 total 10.01 last 65.32)
 - open/close 1361 (time 1775767088.54 total 20.05 last 70.43)
 - open/close 2092 (time 1775767098.54 total 30.05 last 73.08)
 - open/close 2749 (time 1775767108.55 total 40.06 last 65.66)
 - open/close 3535 (time 1775767118.55 total 50.06 last 78.58)
 - open/close 4297 (time 1775767128.56 total 60.07 last 76.17)
 - open/close 5104 (time 1775767138.58 total 70.09 last 80.53)
 - open/close 5757 (time 1775767148.59 total 80.10 last 65.23)
 - open/close 6470 (time 1775767158.59 total 90.10 last 71.28)
 - open/close 7202 (time 1775767168.59 total 100.10 last 73.19)
 - open/close 7987 (time 1775767178.60 total 110.11 last 78.44)
 - open/close 8745 (time 1775767188.61 total 120.12 last 75.70)
 - open/close 9543 (time 1775767198.63 total 130.14 last 79.67)
total: 10000 open/close in 135.96 seconds: 73.55 ops/second
 - unlinked 0 (time 1775767208 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg308-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg308-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.108@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.108@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg308-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all
pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1
Started lustre-OST0000
oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg308-server: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg308-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg308-client.virtnet:  -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre
On OST0, 20325 used inodes rc=0
pdsh@oleg308-client: oleg308-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg308-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg308-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server

   Permanent disk data:
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.108@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.108@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.203.108@tcp

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.203.108@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.203.108@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg308-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all
pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1
Started lustre-OST0000
oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg308-server: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg308-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg308-client.virtnet:  -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg308-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg308-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server
stop mds service on oleg308-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server

unloading modules via unload_modules_local on: 'oleg308-server'
oleg308-server: oleg308-server.virtnet: executing unload_modules_local
oleg308-server: 
modules unloaded.
pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2
pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2
pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2
pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2