== conf-sanity test 69: replace an OST with the same index ========================================================== 17:25:02 (1788470702)
start mds service on oleg415-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1921056 Sep  3 03:55 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg415-server'
oleg415-server: oleg415-server.virtnet: executing load_modules_local
oleg415-server: kmemleak disabled
oleg415-server: Loading modules from /home/green/git/lustre-release/lustre
oleg415-server: detected 4 online CPUs by sysfs
oleg415-server: MODOPTS_LIBCFS=
oleg415-server: Force libcfs to create 2 CPU partitions
oleg415-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg415-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg415-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg415-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg415-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Started lustre-OST0000
oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 711 (time 1788470815.29 total 10.00 last 71.09)
 - open/close 1352 (time 1788470825.29 total 20.01 last 64.06)
 - open/close 1957 (time 1788470835.31 total 30.03 last 60.39)
 - open/close 2693 (time 1788470845.31 total 40.03 last 73.58)
 - open/close 3426 (time 1788470855.32 total 50.04 last 73.24)
 - open/close 4114 (time 1788470865.33 total 60.04 last 68.76)
 - open/close 4827 (time 1788470875.33 total 70.04 last 71.28)
 - open/close 5503 (time 1788470885.34 total 80.05 last 67.55)
 - open/close 6189 (time 1788470895.34 total 90.06 last 68.56)
 - open/close 6927 (time 1788470905.34 total 100.06 last 73.77)
 - open/close 7613 (time 1788470915.37 total 110.08 last 68.45)
 - open/close 8318 (time 1788470925.39 total 120.10 last 70.36)
 - open/close 8922 (time 1788470935.42 total 130.13 last 60.21)
 - open/close 9464 (time 1788470945.42 total 140.14 last 54.17)
total: 10000 open/close in 149.68 seconds: 66.81 ops/second
 - unlinked 0 (time 1788470959 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 735 (time 1788471066.37 total 10.00 last 73.50)
 - open/close 1447 (time 1788471076.38 total 20.00 last 71.17)
 - open/close 2292 (time 1788471086.38 total 30.01 last 84.49)
 - open/close 3006 (time 1788471096.38 total 40.01 last 71.38)
 - open/close 3709 (time 1788471106.39 total 50.02 last 70.23)
 - open/close 4299 (time 1788471116.40 total 60.03 last 58.92)
 - open/close 4867 (time 1788471126.42 total 70.05 last 56.71)
 - open/close 5576 (time 1788471136.45 total 80.07 last 70.71)
 - open/close 6294 (time 1788471146.45 total 90.08 last 71.77)
 - open/close 6912 (time 1788471156.45 total 100.08 last 61.78)
 - open/close 7520 (time 1788471166.46 total 110.09 last 60.77)
 - open/close 8193 (time 1788471176.46 total 120.09 last 67.28)
 - open/close 8872 (time 1788471186.47 total 130.09 last 67.88)
 - open/close 9566 (time 1788471196.47 total 140.09 last 69.40)
total: 10000 open/close in 147.71 seconds: 67.70 ops/second
 - unlinked 0 (time 1788471210 ; total 0 ; last 0)
total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second
 - open/close 729 (time 1788471318.93 total 10.00 last 72.90)
 - open/close 1415 (time 1788471328.93 total 20.00 last 68.60)
 - open/close 2061 (time 1788471338.94 total 30.01 last 64.56)
 - open/close 2788 (time 1788471348.95 total 40.02 last 72.63)
 - open/close 3492 (time 1788471358.95 total 50.02 last 70.37)
 - open/close 4140 (time 1788471368.96 total 60.02 last 64.80)
 - open/close 4797 (time 1788471378.96 total 70.03 last 65.65)
 - open/close 5545 (time 1788471388.98 total 80.05 last 74.67)
 - open/close 6345 (time 1788471398.98 total 90.05 last 79.99)
 - open/close 7065 (time 1788471408.99 total 100.06 last 71.94)
 - open/close 7788 (time 1788471419.00 total 110.06 last 72.25)
 - open/close 8553 (time 1788471429.00 total 120.06 last 76.49)
 - open/close 9318 (time 1788471439.00 total 130.07 last 76.44)
 - open/close 9934 (time 1788471449.01 total 140.08 last 61.57)
total: 10000 open/close in 140.90 seconds: 70.97 ops/second
 - unlinked 0 (time 1788471454 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 734 (time 1788471560.07 total 10.00 last 73.37)
 - open/close 1537 (time 1788471570.08 total 20.01 last 80.24)
 - open/close 2260 (time 1788471580.09 total 30.02 last 72.24)
 - open/close 3005 (time 1788471590.10 total 40.03 last 74.41)
 - open/close 3709 (time 1788471600.10 total 50.03 last 70.38)
 - open/close 4335 (time 1788471610.11 total 60.05 last 62.51)
 - open/close 4921 (time 1788471620.13 total 70.06 last 58.51)
 - open/close 5626 (time 1788471630.14 total 80.08 last 70.40)
 - open/close 6400 (time 1788471640.16 total 90.09 last 77.29)
 - open/close 7150 (time 1788471650.17 total 100.10 last 74.94)
 - open/close 7929 (time 1788471660.18 total 110.11 last 77.82)
 - open/close 8630 (time 1788471670.18 total 120.11 last 70.09)
 - open/close 9329 (time 1788471680.19 total 130.13 last 69.79)
total: 10000 open/close in 138.60 seconds: 72.15 ops/second
 - unlinked 0 (time 1788471694 ; total 1 ; last 1)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 589 (time 1788471799.19 total 10.01 last 58.83)
 - open/close 1348 (time 1788471809.19 total 20.01 last 75.90)
 - open/close 2227 (time 1788471819.19 total 30.02 last 87.84)
 - open/close 2913 (time 1788471829.20 total 40.02 last 68.59)
 - open/close 3669 (time 1788471839.20 total 50.02 last 75.58)
 - open/close 4414 (time 1788471849.21 total 60.03 last 74.42)
 - open/close 5164 (time 1788471859.24 total 70.06 last 74.80)
 - open/close 5919 (time 1788471869.26 total 80.08 last 75.36)
 - open/close 6656 (time 1788471879.28 total 90.11 last 73.51)
 - open/close 7123 (time 1788471889.29 total 100.11 last 46.66)
 - open/close 7866 (time 1788471899.30 total 110.12 last 74.24)
 - open/close 8642 (time 1788471909.32 total 120.14 last 77.47)
 - open/close 9396 (time 1788471919.32 total 130.14 last 75.39)
total: 10000 open/close in 137.27 seconds: 72.85 ops/second
 - unlinked 0 (time 1788471930 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 579 (time 1788472035.68 total 10.01 last 57.85)
 - open/close 1274 (time 1788472045.68 total 20.01 last 69.48)
 - open/close 2028 (time 1788472055.68 total 30.02 last 75.37)
 - open/close 2778 (time 1788472065.69 total 40.03 last 74.92)
 - open/close 3479 (time 1788472075.70 total 50.03 last 70.09)
 - open/close 4030 (time 1788472085.71 total 60.04 last 55.03)
 - open/close 4530 (time 1788472095.72 total 70.05 last 49.94)
 - open/close 5259 (time 1788472105.73 total 80.06 last 72.82)
 - open/close 6026 (time 1788472115.74 total 90.07 last 76.63)
 - open/close 6750 (time 1788472125.74 total 100.07 last 72.39)
 - open/close 7431 (time 1788472135.75 total 110.08 last 68.06)
 - open/close 8043 (time 1788472145.76 total 120.09 last 61.14)
 - open/close 8776 (time 1788472155.76 total 130.09 last 73.25)
 - open/close 9488 (time 1788472165.77 total 140.10 last 71.18)
total: 10000 open/close in 147.12 seconds: 67.97 ops/second
 - unlinked 0 (time 1788472176 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
 - open/close 638 (time 1788472278.06 total 10.01 last 63.73)
 - open/close 1328 (time 1788472288.06 total 20.01 last 69.00)
 - open/close 2125 (time 1788472298.08 total 30.02 last 79.61)
 - open/close 2879 (time 1788472308.08 total 40.03 last 75.37)
 - open/close 3564 (time 1788472318.09 total 50.04 last 68.40)
 - open/close 4318 (time 1788472328.10 total 60.05 last 75.32)
 - open/close 5117 (time 1788472338.11 total 70.06 last 79.84)
 - open/close 5904 (time 1788472348.13 total 80.08 last 78.53)
 - open/close 6606 (time 1788472358.13 total 90.08 last 70.19)
 - open/close 7399 (time 1788472368.14 total 100.09 last 79.27)
 - open/close 8057 (time 1788472378.14 total 110.09 last 65.78)
 - open/close 8816 (time 1788472388.14 total 120.09 last 75.89)
 - open/close 9590 (time 1788472398.17 total 130.12 last 77.20)
total: 10000 open/close in 135.62 seconds: 73.74 ops/second
 - unlinked 0 (time 1788472407 ; total 0 ; last 0)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
 - open/close 739 (time 1788472506.79 total 10.00 last 73.89)
 - open/close 1444 (time 1788472516.80 total 20.01 last 70.45)
 - open/close 2172 (time 1788472526.80 total 30.01 last 72.79)
 - open/close 2923 (time 1788472536.81 total 40.01 last 75.08)
 - open/close 3666 (time 1788472546.82 total 50.03 last 74.19)
 - open/close 4402 (time 1788472556.83 total 60.04 last 73.52)
 - open/close 5144 (time 1788472566.85 total 70.05 last 74.09)
 - open/close 5894 (time 1788472576.85 total 80.06 last 74.95)
 - open/close 6652 (time 1788472586.86 total 90.06 last 75.77)
 - open/close 7420 (time 1788472596.87 total 100.07 last 76.72)
 - open/close 8171 (time 1788472606.87 total 110.08 last 75.07)
 - open/close 8905 (time 1788472616.88 total 120.08 last 73.37)
 - open/close 9672 (time 1788472626.88 total 130.09 last 76.64)
total: 10000 open/close in 134.70 seconds: 74.24 ops/second
 - unlinked 0 (time 1788472635 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 746 (time 1788472732.12 total 10.01 last 74.53)
 - open/close 1520 (time 1788472742.13 total 20.02 last 77.31)
 - open/close 2269 (time 1788472752.14 total 30.03 last 74.83)
 - open/close 3039 (time 1788472762.14 total 40.04 last 76.96)
 - open/close 3795 (time 1788472772.15 total 50.04 last 75.54)
 - open/close 4571 (time 1788472782.16 total 60.05 last 77.57)
 - open/close 5371 (time 1788472792.16 total 70.05 last 80.00)
 - open/close 6116 (time 1788472802.16 total 80.05 last 74.46)
 - open/close 6858 (time 1788472812.16 total 90.05 last 74.20)
 - open/close 7577 (time 1788472822.17 total 100.06 last 71.83)
 - open/close 8321 (time 1788472832.19 total 110.08 last 74.30)
 - open/close 9105 (time 1788472842.20 total 120.09 last 78.32)
 - open/close 9849 (time 1788472852.21 total 130.10 last 74.28)
total: 10000 open/close in 132.46 seconds: 75.49 ops/second
 - unlinked 0 (time 1788472859 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
 - open/close 695 (time 1788472961.33 total 10.01 last 69.46)
 - open/close 1430 (time 1788472971.33 total 20.01 last 73.46)
 - open/close 2015 (time 1788472981.34 total 30.02 last 58.43)
 - open/close 2723 (time 1788472991.37 total 40.05 last 70.65)
 - open/close 3486 (time 1788473001.37 total 50.05 last 76.29)
 - open/close 4224 (time 1788473011.39 total 60.07 last 73.66)
 - open/close 4901 (time 1788473021.41 total 70.08 last 67.58)
 - open/close 5634 (time 1788473031.41 total 80.09 last 73.23)
 - open/close 6420 (time 1788473041.42 total 90.10 last 78.55)
 - open/close 7217 (time 1788473051.43 total 100.10 last 79.67)
 - open/close 8000 (time 1788473061.43 total 110.11 last 78.26)
 - open/close 8769 (time 1788473071.44 total 120.12 last 76.86)
 - open/close 9547 (time 1788473081.44 total 130.12 last 77.74)
total: 10000 open/close in 135.89 seconds: 73.59 ops/second
 - unlinked 0 (time 1788473091 ; total 0 ; last 0)
total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg415-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg415-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg415-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.115@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.115@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg415-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Started lustre-OST0000
oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg415-server: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg415-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg415-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg415-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg415-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.115@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.115@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.115@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.115@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.115@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg415-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1
Started lustre-OST0000
oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg415-server: oleg415-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg415-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg415-client.virtnet:  -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg415-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg415-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg415-server
stop mds service on oleg415-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg415-server
unloading modules via unload_modules_local on: 'oleg415-server'
oleg415-server: oleg415-server.virtnet: executing unload_modules_local
oleg415-server: kmemleak disabled
modules unloaded.
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 2
pdsh@oleg415-client: oleg415-client: ssh exited with exit code 2
pdsh@oleg415-client: oleg415-server: ssh exited with exit code 2
pdsh@oleg415-client: oleg415-client: ssh exited with exit code 2