== conf-sanity test 69: replace an OST with the same index ========================================================== 06:41:02 (1787136062)
start mds service on oleg415-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg415-server'
oleg415-server: oleg415-server.virtnet: executing load_modules_local
oleg415-server: Loading modules from /home/green/git/lustre-release/lustre
oleg415-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg415-server: detected 4 online CPUs by sysfs
oleg415-server: MODOPTS_LIBCFS=
oleg415-server: Force libcfs to create 2 CPU partitions
oleg415-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg415-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg415-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg415-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Started lustre-OST0000
oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 527 (time 1787136162.13 total 10.01 last 52.62)
 - open/close 1104 (time 1787136172.13 total 20.02 last 57.68)
 - open/close 1715 (time 1787136182.14 total 30.03 last 61.04)
 - open/close 2320 (time 1787136192.17 total 40.05 last 60.34)
 - open/close 2988 (time 1787136202.19 total 50.08 last 66.66)
 - open/close 3638 (time 1787136212.19 total 60.08 last 64.99)
 - open/close 4341 (time 1787136222.19 total 70.08 last 70.28)
 - open/close 5019 (time 1787136232.21 total 80.09 last 67.72)
 - open/close 5689 (time 1787136242.22 total 90.10 last 66.93)
 - open/close 6366 (time 1787136252.23 total 100.11 last 67.62)
 - open/close 6955 (time 1787136262.23 total 110.11 last 58.89)
 - open/close 7606 (time 1787136272.24 total 120.13 last 65.00)
 - open/close 8188 (time 1787136282.29 total 130.18 last 57.92)
 - open/close 8807 (time 1787136292.30 total 140.18 last 61.87)
 - open/close 9404 (time 1787136302.30 total 150.19 last 59.67)
total: 10000 open/close in 159.73 seconds: 62.61 ops/second
 - unlinked 0 (time 1787136316 ; total 0 ; last 0)
total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second
 - open/close 608 (time 1787136433.59 total 10.00 last 60.77)
 - open/close 1243 (time 1787136443.59 total 20.01 last 63.46)
 - open/close 1846 (time 1787136453.60 total 30.02 last 60.24)
 - open/close 2440 (time 1787136463.61 total 40.03 last 59.37)
 - open/close 3035 (time 1787136473.63 total 50.04 last 59.40)
 - open/close 3611 (time 1787136483.63 total 60.04 last 57.59)
 - open/close 4196 (time 1787136493.63 total 70.05 last 58.49)
 - open/close 4810 (time 1787136503.63 total 80.05 last 61.38)
 - open/close 5417 (time 1787136513.63 total 90.05 last 60.68)
 - open/close 6068 (time 1787136523.64 total 100.05 last 65.07)
 - open/close 6670 (time 1787136533.65 total 110.07 last 60.13)
 - open/close 7241 (time 1787136543.65 total 120.07 last 57.10)
 - open/close 7886 (time 1787136553.65 total 130.07 last 64.47)
 - open/close 8482 (time 1787136563.66 total 140.08 last 59.56)
 - open/close 9069 (time 1787136573.67 total 150.09 last 58.63)
 - open/close 9657 (time 1787136583.68 total 160.10 last 58.73)
total: 10000 open/close in 165.46 seconds: 60.44 ops/second
 - unlinked 0 (time 1787136593 ; total 0 ; last 0)
total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second
 - open/close 632 (time 1787136715.06 total 10.01 last 63.13)
 - open/close 1260 (time 1787136725.08 total 20.03 last 62.66)
 - open/close 1866 (time 1787136735.08 total 30.03 last 60.59)
 - open/close 2489 (time 1787136745.09 total 40.04 last 62.24)
 - open/close 3106 (time 1787136755.10 total 50.06 last 61.63)
 - open/close 3681 (time 1787136765.11 total 60.06 last 57.48)
 - open/close 4241 (time 1787136775.12 total 70.07 last 55.95)
 - open/close 4857 (time 1787136785.12 total 80.07 last 61.58)
 - open/close 5424 (time 1787136795.14 total 90.09 last 56.58)
 - open/close 5989 (time 1787136805.15 total 100.11 last 56.43)
 - open/close 6390 (time 1787136815.20 total 110.15 last 39.92)
 - open/close 6835 (time 1787136825.24 total 120.19 last 44.32)
 - open/close 7388 (time 1787136835.26 total 130.21 last 55.18)
 - open/close 8053 (time 1787136845.26 total 140.22 last 66.48)
 - open/close 8712 (time 1787136855.29 total 150.24 last 65.73)
 - open/close 9327 (time 1787136865.30 total 160.25 last 61.46)
 - open/close 9948 (time 1787136875.30 total 170.26 last 62.07)
total: 10000 open/close in 170.88 seconds: 58.52 ops/second
 - unlinked 0 (time 1787136881 ; total 0 ; last 0)
total: 10000 unlinks in 106 seconds: 94.339622 unlinks/second
 - open/close 642 (time 1787137002.46 total 10.01 last 64.12)
 - open/close 1281 (time 1787137012.46 total 20.02 last 63.86)
 - open/close 1906 (time 1787137022.48 total 30.04 last 62.38)
 - open/close 2532 (time 1787137032.50 total 40.06 last 62.49)
 - open/close 3162 (time 1787137042.52 total 50.07 last 62.89)
 - open/close 3749 (time 1787137052.52 total 60.08 last 58.68)
 - open/close 4326 (time 1787137062.53 total 70.08 last 57.65)
 - open/close 4872 (time 1787137072.54 total 80.09 last 54.56)
 - open/close 5470 (time 1787137082.54 total 90.10 last 59.75)
 - open/close 6091 (time 1787137092.55 total 100.11 last 62.05)
 - open/close 6686 (time 1787137102.56 total 110.12 last 59.45)
 - open/close 7145 (time 1787137112.60 total 120.16 last 45.71)
 - open/close 7560 (time 1787137122.60 total 130.16 last 41.50)
 - open/close 8127 (time 1787137132.61 total 140.16 last 56.68)
 - open/close 8703 (time 1787137142.63 total 150.18 last 57.49)
 - open/close 9313 (time 1787137152.63 total 160.19 last 60.96)
 - open/close 9949 (time 1787137162.64 total 170.20 last 63.52)
total: 10000 open/close in 171.27 seconds: 58.39 ops/second
 - unlinked 0 (time 1787137168 ; total 0 ; last 0)
total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second
 - open/close 615 (time 1787137287.18 total 10.00 last 61.48)
 - open/close 1241 (time 1787137297.19 total 20.01 last 62.58)
 - open/close 1861 (time 1787137307.19 total 30.01 last 62.00)
 - open/close 2465 (time 1787137317.20 total 40.02 last 60.34)
 - open/close 3119 (time 1787137327.21 total 50.03 last 65.35)
 - open/close 3781 (time 1787137337.21 total 60.03 last 66.19)
 - open/close 4393 (time 1787137347.22 total 70.04 last 61.13)
 - open/close 5013 (time 1787137357.23 total 80.04 last 61.96)
 - open/close 5603 (time 1787137367.23 total 90.05 last 58.98)
 - open/close 6245 (time 1787137377.26 total 100.08 last 64.01)
 - open/close 6876 (time 1787137387.27 total 110.09 last 63.00)
 - open/close 7517 (time 1787137397.28 total 120.10 last 64.07)
 - open/close 8129 (time 1787137407.28 total 130.10 last 61.18)
 - open/close 8737 (time 1787137417.30 total 140.12 last 60.72)
 - open/close 9350 (time 1787137427.30 total 150.12 last 61.28)
 - open/close 9934 (time 1787137437.32 total 160.14 last 58.29)
total: 10000 open/close in 161.58 seconds: 61.89 ops/second
 - unlinked 0 (time 1787137443 ; total 0 ; last 0)
total: 10000 unlinks in 106 seconds: 94.339622 unlinks/second
 - open/close 545 (time 1787137563.68 total 10.01 last 54.44)
 - open/close 1140 (time 1787137573.68 total 20.01 last 59.48)
 - open/close 1713 (time 1787137583.70 total 30.03 last 57.24)
 - open/close 2285 (time 1787137593.70 total 40.03 last 57.16)
 - open/close 2931 (time 1787137603.71 total 50.04 last 64.54)
 - open/close 3527 (time 1787137613.72 total 60.05 last 59.58)
 - open/close 4121 (time 1787137623.74 total 70.07 last 59.24)
 - open/close 4831 (time 1787137633.75 total 80.08 last 70.93)
 - open/close 5447 (time 1787137643.75 total 90.08 last 61.59)
 - open/close 6107 (time 1787137653.79 total 100.11 last 65.79)
 - open/close 6785 (time 1787137663.79 total 110.12 last 67.78)
 - open/close 7233 (time 1787137673.79 total 120.12 last 44.79)
 - open/close 7879 (time 1787137683.80 total 130.13 last 64.52)
 - open/close 8521 (time 1787137693.82 total 140.15 last 64.10)
 - open/close 9181 (time 1787137703.82 total 150.15 last 65.98)
 - open/close 9804 (time 1787137713.83 total 160.16 last 62.25)
total: 10000 open/close in 163.69 seconds: 61.09 ops/second
 - unlinked 0 (time 1787137721 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 645 (time 1787137836.33 total 10.02 last 64.38)
 - open/close 1245 (time 1787137846.34 total 20.03 last 59.95)
 - open/close 1939 (time 1787137856.34 total 30.03 last 69.40)
 - open/close 2556 (time 1787137866.36 total 40.05 last 61.58)
 - open/close 3198 (time 1787137876.36 total 50.05 last 64.19)
 - open/close 3797 (time 1787137886.37 total 60.05 last 59.86)
 - open/close 4380 (time 1787137896.41 total 70.09 last 58.07)
 - open/close 4935 (time 1787137906.42 total 80.10 last 55.45)
 - open/close 5472 (time 1787137916.43 total 90.11 last 53.66)
 - open/close 6069 (time 1787137926.45 total 100.13 last 59.56)
 - open/close 6662 (time 1787137936.46 total 110.14 last 59.27)
 - open/close 7281 (time 1787137946.46 total 120.14 last 61.87)
 - open/close 7871 (time 1787137956.47 total 130.15 last 58.95)
 - open/close 8427 (time 1787137966.47 total 140.16 last 55.57)
 - open/close 9030 (time 1787137976.47 total 150.16 last 60.30)
 - open/close 9558 (time 1787137986.48 total 160.16 last 52.79)
total: 10000 open/close in 167.81 seconds: 59.59 ops/second
 - unlinked 0 (time 1787137999 ; total 0 ; last 0)
total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second
 - open/close 592 (time 1787138118.55 total 10.01 last 59.15)
 - open/close 1174 (time 1787138128.55 total 20.01 last 58.20)
 - open/close 1619 (time 1787138138.57 total 30.03 last 44.40)
 - open/close 2124 (time 1787138148.58 total 40.04 last 50.46)
 - open/close 2621 (time 1787138158.59 total 50.05 last 49.63)
 - open/close 3199 (time 1787138168.60 total 60.06 last 57.74)
 - open/close 3836 (time 1787138178.61 total 70.07 last 63.65)
 - open/close 4479 (time 1787138188.61 total 80.07 last 64.28)
 - open/close 5158 (time 1787138198.62 total 90.08 last 67.84)
 - open/close 5829 (time 1787138208.63 total 100.09 last 67.08)
 - open/close 6375 (time 1787138218.64 total 110.10 last 54.55)
 - open/close 6942 (time 1787138228.65 total 120.11 last 56.60)
 - open/close 7522 (time 1787138238.66 total 130.12 last 57.96)
 - open/close 8116 (time 1787138248.67 total 140.13 last 59.34)
 - open/close 8704 (time 1787138258.69 total 150.15 last 58.68)
 - open/close 9314 (time 1787138268.70 total 160.16 last 60.93)
 - open/close 9853 (time 1787138278.71 total 170.17 last 53.86)
total: 10000 open/close in 172.66 seconds: 57.92 ops/second
 - unlinked 0 (time 1787138286 ; total 0 ; last 0)
total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second
 - open/close 594 (time 1787138407.08 total 10.02 last 59.26)
 - open/close 1199 (time 1787138417.09 total 20.03 last 60.46)
 - open/close 1850 (time 1787138427.10 total 30.04 last 65.03)
 - open/close 2496 (time 1787138437.10 total 40.04 last 64.59)
 - open/close 3030 (time 1787138447.11 total 50.05 last 53.37)
 - open/close 3546 (time 1787138457.12 total 60.06 last 51.54)
 - open/close 4038 (time 1787138467.12 total 70.06 last 49.19)
 - open/close 4551 (time 1787138477.16 total 80.10 last 51.11)
 - open/close 5116 (time 1787138487.16 total 90.11 last 56.46)
 - open/close 5731 (time 1787138497.18 total 100.13 last 61.37)
 - open/close 6337 (time 1787138507.20 total 110.14 last 60.52)
 - open/close 6913 (time 1787138517.23 total 120.17 last 57.41)
 - open/close 7524 (time 1787138527.25 total 130.19 last 61.01)
 - open/close 8129 (time 1787138537.25 total 140.19 last 60.48)
 - open/close 8792 (time 1787138547.25 total 150.19 last 66.29)
 - open/close 9373 (time 1787138557.26 total 160.20 last 58.05)
total: 10000 open/close in 170.11 seconds: 58.79 ops/second
 - unlinked 0 (time 1787138572 ; total 0 ; last 0)
total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second
 - open/close 605 (time 1787138695.44 total 10.01 last 60.43)
 - open/close 1170 (time 1787138705.45 total 20.02 last 56.46)
 - open/close 1781 (time 1787138715.45 total 30.02 last 61.06)
 - open/close 2462 (time 1787138725.46 total 40.03 last 68.04)
 - open/close 3148 (time 1787138735.47 total 50.04 last 68.58)
 - open/close 3829 (time 1787138745.47 total 60.04 last 68.06)
 - open/close 4491 (time 1787138755.49 total 70.06 last 66.09)
 - open/close 5147 (time 1787138765.51 total 80.08 last 65.47)
 - open/close 5747 (time 1787138775.51 total 90.08 last 60.00)
 - open/close 6391 (time 1787138785.52 total 100.09 last 64.32)
 - open/close 7026 (time 1787138795.53 total 110.10 last 63.44)
 - open/close 7643 (time 1787138805.53 total 120.10 last 61.70)
 - open/close 8253 (time 1787138815.54 total 130.11 last 60.95)
 - open/close 8748 (time 1787138825.54 total 140.11 last 49.48)
 - open/close 9305 (time 1787138835.55 total 150.12 last 55.67)
 - open/close 9950 (time 1787138845.56 total 160.14 last 64.40)
total: 10000 open/close in 160.98 seconds: 62.12 ops/second
 - unlinked 0 (time 1787138851 ; total 0 ; last 0)
total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg415-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg415-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg415-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.115@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.115@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg415-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Started lustre-OST0000
oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg415-server: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg415-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
On OST0, 9782 used inodes rc=0
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg415-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg415-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg415-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.115@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.115@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.204.115@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.204.115@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.204.115@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg415-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Started lustre-OST0000
oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg415-server: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg415-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg415-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg415-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg415-server
stop mds service on oleg415-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg415-server

unloading modules via unload_modules_local on: 'oleg415-server'
oleg415-server: oleg415-server.virtnet: executing unload_modules_local
oleg415-server: 
modules unloaded.
pdsh@oleg415-client: oleg415-client: ssh exited with exit code 2
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 2
pdsh@oleg415-client: oleg415-client: ssh exited with exit code 2
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 2