== conf-sanity test 69: replace an OST with the same index ========================================================== 14:36:13 (1786732573)
start mds service on oleg160-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing load_modules_local
oleg160-server: Loading modules from /home/green/git/lustre-release/lustre
oleg160-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg160-server: detected 4 online CPUs by sysfs
oleg160-server: MODOPTS_LIBCFS=
oleg160-server: Force libcfs to create 2 CPU partitions
oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg160-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg160-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 864 (time 1786732652.38 total 10.01 last 86.34)
 - open/close 1813 (time 1786732662.39 total 20.01 last 94.82)
 - open/close 2744 (time 1786732672.39 total 30.02 last 93.08)
 - open/close 3570 (time 1786732682.39 total 40.02 last 82.59)
 - open/close 4299 (time 1786732692.39 total 50.02 last 72.88)
 - open/close 5041 (time 1786732702.40 total 60.02 last 74.18)
 - open/close 5659 (time 1786732712.40 total 70.03 last 61.76)
 - open/close 6497 (time 1786732722.41 total 80.04 last 83.73)
 - open/close 7384 (time 1786732732.43 total 90.05 last 88.57)
 - open/close 8312 (time 1786732742.43 total 100.06 last 92.78)
 - open/close 9193 (time 1786732752.43 total 110.06 last 88.06)
total: 10000 open/close in 118.54 seconds: 84.36 ops/second
 - unlinked 0 (time 1786732765 ; total 0 ; last 0)
total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second
 - open/close 921 (time 1786732855.55 total 10.00 last 92.07)
 - open/close 1811 (time 1786732865.57 total 20.02 last 88.86)
 - open/close 2780 (time 1786732875.57 total 30.02 last 96.89)
 - open/close 3798 (time 1786732885.58 total 40.03 last 101.73)
 - open/close 4733 (time 1786732895.59 total 50.04 last 93.40)
 - open/close 5375 (time 1786732905.60 total 60.05 last 64.13)
 - open/close 6144 (time 1786732915.61 total 70.06 last 76.83)
 - open/close 6975 (time 1786732925.61 total 80.06 last 83.09)
 - open/close 7874 (time 1786732935.62 total 90.06 last 89.85)
 - open/close 8681 (time 1786732945.62 total 100.07 last 80.67)
 - open/close 9533 (time 1786732955.63 total 110.08 last 85.09)
total: 10000 open/close in 115.00 seconds: 86.96 ops/second
 - unlinked 0 (time 1786732964 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 766 (time 1786733056.46 total 10.00 last 76.57)
 - open/close 1710 (time 1786733066.47 total 20.02 last 94.30)
 - open/close 2467 (time 1786733076.47 total 30.02 last 75.65)
 - open/close 3298 (time 1786733086.48 total 40.03 last 83.06)
 - open/close 3990 (time 1786733096.48 total 50.03 last 69.18)
 - open/close 4623 (time 1786733106.50 total 60.05 last 63.16)
 - open/close 5380 (time 1786733116.52 total 70.07 last 75.54)
 - open/close 6242 (time 1786733126.53 total 80.08 last 86.16)
 - open/close 7121 (time 1786733136.54 total 90.09 last 87.82)
 - open/close 7887 (time 1786733146.55 total 100.10 last 76.48)
 - open/close 8755 (time 1786733156.56 total 110.11 last 86.74)
 - open/close 9614 (time 1786733166.57 total 120.12 last 85.79)
total: 10000 open/close in 124.38 seconds: 80.40 ops/second
 - unlinked 0 (time 1786733175 ; total 0 ; last 0)
total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second
 - open/close 750 (time 1786733266.81 total 10.01 last 74.95)
 - open/close 1566 (time 1786733276.82 total 20.01 last 81.54)
 - open/close 2400 (time 1786733286.82 total 30.02 last 83.36)
 - open/close 3325 (time 1786733296.82 total 40.02 last 92.47)
 - open/close 4250 (time 1786733306.83 total 50.03 last 92.44)
 - open/close 5189 (time 1786733316.84 total 60.04 last 93.79)
 - open/close 6017 (time 1786733326.85 total 70.04 last 82.77)
 - open/close 6606 (time 1786733336.85 total 80.05 last 58.85)
 - open/close 7389 (time 1786733346.87 total 90.07 last 78.19)
 - open/close 8204 (time 1786733356.87 total 100.07 last 81.48)
 - open/close 9022 (time 1786733366.90 total 110.10 last 81.53)
 - open/close 9899 (time 1786733376.90 total 120.10 last 87.70)
total: 10000 open/close in 121.43 seconds: 82.35 ops/second
 - unlinked 0 (time 1786733382 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 897 (time 1786733475.17 total 10.01 last 89.64)
 - open/close 1844 (time 1786733485.17 total 20.01 last 94.68)
 - open/close 2570 (time 1786733495.18 total 30.01 last 72.56)
 - open/close 3407 (time 1786733505.18 total 40.02 last 83.68)
 - open/close 4032 (time 1786733515.19 total 50.02 last 62.48)
 - open/close 4810 (time 1786733525.19 total 60.02 last 77.78)
 - open/close 5477 (time 1786733535.21 total 70.04 last 66.59)
 - open/close 6167 (time 1786733545.21 total 80.04 last 68.99)
 - open/close 6784 (time 1786733555.21 total 90.05 last 61.66)
 - open/close 7687 (time 1786733565.22 total 100.05 last 90.26)
 - open/close 8506 (time 1786733575.22 total 110.05 last 81.89)
 - open/close 9243 (time 1786733585.22 total 120.06 last 73.66)
total: 10000 open/close in 128.93 seconds: 77.56 ops/second
 - unlinked 0 (time 1786733598 ; total 0 ; last 0)
total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second
 - open/close 897 (time 1786733689.66 total 10.01 last 89.65)
 - open/close 1599 (time 1786733699.67 total 20.01 last 70.14)
 - open/close 2390 (time 1786733709.67 total 30.02 last 79.07)
 - open/close 3190 (time 1786733719.68 total 40.02 last 79.94)
 - open/close 3975 (time 1786733729.69 total 50.03 last 78.44)
 - open/close 4674 (time 1786733739.70 total 60.04 last 69.84)
 - open/close 5495 (time 1786733749.70 total 70.05 last 82.07)
 - open/close 6302 (time 1786733759.70 total 80.05 last 80.70)
 - open/close 7170 (time 1786733769.70 total 90.05 last 86.78)
 - open/close 8051 (time 1786733779.73 total 100.07 last 87.88)
 - open/close 8932 (time 1786733789.73 total 110.08 last 88.06)
 - open/close 9746 (time 1786733799.74 total 120.08 last 81.39)
total: 10000 open/close in 123.22 seconds: 81.15 ops/second
 - unlinked 0 (time 1786733807 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 773 (time 1786733906.98 total 10.00 last 77.30)
 - open/close 1629 (time 1786733916.99 total 20.00 last 85.57)
 - open/close 2601 (time 1786733926.99 total 30.01 last 97.14)
 - open/close 3278 (time 1786733937.00 total 40.02 last 67.65)
 - open/close 3929 (time 1786733947.02 total 50.04 last 64.97)
 - open/close 4779 (time 1786733957.03 total 60.04 last 84.96)
 - open/close 5710 (time 1786733967.04 total 70.06 last 92.93)
 - open/close 6359 (time 1786733977.06 total 80.07 last 64.83)
 - open/close 7181 (time 1786733987.06 total 90.07 last 82.17)
 - open/close 8040 (time 1786733997.06 total 100.08 last 85.88)
 - open/close 8831 (time 1786734007.06 total 110.08 last 79.08)
 - open/close 9652 (time 1786734017.08 total 120.10 last 81.95)
total: 10000 open/close in 124.51 seconds: 80.31 ops/second
 - unlinked 0 (time 1786734026 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 872 (time 1786734131.66 total 10.00 last 87.18)
 - open/close 1618 (time 1786734141.68 total 20.02 last 74.46)
 - open/close 2488 (time 1786734151.68 total 30.03 last 86.96)
 - open/close 3370 (time 1786734161.69 total 40.04 last 88.11)
 - open/close 4218 (time 1786734171.70 total 50.04 last 84.76)
 - open/close 5026 (time 1786734181.71 total 60.05 last 80.71)
 - open/close 5903 (time 1786734191.71 total 70.05 last 87.67)
 - open/close 6562 (time 1786734201.73 total 80.07 last 65.81)
 - open/close 7267 (time 1786734211.74 total 90.08 last 70.42)
 - open/close 7958 (time 1786734221.75 total 100.10 last 69.00)
 - open/close 8622 (time 1786734231.77 total 110.12 last 66.27)
 - open/close 9272 (time 1786734241.77 total 120.12 last 65.00)
 - open/close 9913 (time 1786734251.79 total 130.13 last 64.02)
total: 10000 open/close in 131.97 seconds: 75.77 ops/second
 - unlinked 0 (time 1786734259 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 692 (time 1786734358.12 total 10.00 last 69.19)
 - open/close 1323 (time 1786734368.14 total 20.03 last 62.94)
 - open/close 2129 (time 1786734378.16 total 30.05 last 80.44)
 - open/close 2892 (time 1786734388.16 total 40.05 last 76.28)
 - open/close 3568 (time 1786734398.18 total 50.06 last 67.51)
 - open/close 4267 (time 1786734408.18 total 60.06 last 69.90)
 - open/close 4962 (time 1786734418.19 total 70.07 last 69.43)
 - open/close 5645 (time 1786734428.20 total 80.09 last 68.21)
 - open/close 6450 (time 1786734438.21 total 90.09 last 80.45)
 - open/close 7252 (time 1786734448.21 total 100.09 last 80.19)
 - open/close 7992 (time 1786734458.21 total 110.10 last 73.96)
 - open/close 8732 (time 1786734468.21 total 120.10 last 74.00)
 - open/close 9466 (time 1786734478.23 total 130.11 last 73.31)
total: 10000 open/close in 137.38 seconds: 72.79 ops/second
 - unlinked 0 (time 1786734490 ; total 0 ; last 0)
total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second
 - open/close 632 (time 1786734606.77 total 10.00 last 63.18)
 - open/close 1320 (time 1786734616.78 total 20.02 last 68.71)
 - open/close 2108 (time 1786734626.79 total 30.03 last 78.73)
 - open/close 2842 (time 1786734636.80 total 40.04 last 73.33)
 - open/close 3535 (time 1786734646.81 total 50.04 last 69.24)
 - open/close 4323 (time 1786734656.81 total 60.05 last 78.74)
 - open/close 4918 (time 1786734666.82 total 70.06 last 59.47)
 - open/close 5561 (time 1786734676.83 total 80.07 last 64.24)
 - open/close 6260 (time 1786734686.85 total 90.09 last 69.72)
 - open/close 6953 (time 1786734696.86 total 100.10 last 69.24)
 - open/close 7644 (time 1786734706.86 total 110.10 last 69.10)
 - open/close 8341 (time 1786734716.88 total 120.12 last 69.57)
 - open/close 9049 (time 1786734726.89 total 130.13 last 70.73)
 - open/close 9803 (time 1786734736.90 total 140.14 last 75.31)
total: 10000 open/close in 142.79 seconds: 70.03 ops/second
 - unlinked 0 (time 1786734744 ; total 0 ; last 0)
total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.160@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.160@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg160-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.160@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.160@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.160@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.160@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.201.160@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg160-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1
Started lustre-OST0000
oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg160-client.virtnet:  -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg160-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg160-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server
stop mds service on oleg160-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server

unloading modules via unload_modules_local on: 'oleg160-server'
oleg160-server: oleg160-server.virtnet: executing unload_modules_local
oleg160-server: 
modules unloaded.
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2
pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2