== conf-sanity test 69: replace an OST with the same index ========================================================== 08:15:20 (1788437720)
start mds service on oleg636-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x  1 green green 1921056 Sep  3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg636-server'
oleg636-server: oleg636-server.virtnet: executing load_modules_local
oleg636-server: kmemleak disabled
oleg636-server: Loading modules from /home/green/git/lustre-release/lustre
oleg636-server: detected 4 online CPUs by sysfs
oleg636-server: MODOPTS_LIBCFS=
oleg636-server: Force libcfs to create 2 CPU partitions
oleg636-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg636-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg636-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg636-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all
pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg636-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all
pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1
Started lustre-OST0000
oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg636-client.virtnet:  -o user_xattr,flock 192.168.206.136@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 872 (time 1788437791.96 total 10.00 last 87.17)
 - open/close 1747 (time 1788437801.97 total 20.02 last 87.38)
 - open/close 2643 (time 1788437811.98 total 30.02 last 89.53)
 - open/close 3498 (time 1788437821.98 total 40.03 last 85.47)
 - open/close 4359 (time 1788437831.99 total 50.03 last 86.05)
 - open/close 5220 (time 1788437841.99 total 60.04 last 86.05)
 - open/close 6092 (time 1788437852.00 total 70.04 last 87.17)
 - open/close 6975 (time 1788437862.00 total 80.04 last 88.30)
 - open/close 7840 (time 1788437872.00 total 90.05 last 86.44)
 - open/close 8720 (time 1788437882.01 total 100.06 last 87.95)
 - open/close 9608 (time 1788437892.01 total 110.06 last 88.78)
total: 10000 open/close in 114.63 seconds: 87.24 ops/second
 - unlinked 0 (time 1788437899 ; total 0 ; last 0)
total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second
 - open/close 807 (time 1788437989.36 total 10.00 last 80.67)
 - open/close 1651 (time 1788437999.36 total 20.01 last 84.36)
 - open/close 2556 (time 1788438009.37 total 30.02 last 90.41)
 - open/close 3340 (time 1788438019.38 total 40.02 last 78.36)
 - open/close 4081 (time 1788438029.39 total 50.03 last 74.03)
 - open/close 4786 (time 1788438039.39 total 60.03 last 70.50)
 - open/close 5456 (time 1788438049.40 total 70.04 last 66.95)
 - open/close 6353 (time 1788438059.40 total 80.05 last 89.64)
 - open/close 7227 (time 1788438069.40 total 90.05 last 87.38)
 - open/close 8112 (time 1788438079.45 total 100.09 last 88.10)
 - open/close 8996 (time 1788438089.46 total 110.10 last 88.36)
 - open/close 9839 (time 1788438099.46 total 120.10 last 84.28)
total: 10000 open/close in 122.28 seconds: 81.78 ops/second
 - unlinked 0 (time 1788438105 ; total 0 ; last 0)
total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second
 - open/close 868 (time 1788438192.87 total 10.00 last 86.77)
 - open/close 1702 (time 1788438202.87 total 20.01 last 83.36)
 - open/close 2569 (time 1788438212.87 total 30.01 last 86.69)
 - open/close 3406 (time 1788438222.88 total 40.02 last 83.65)
 - open/close 4223 (time 1788438232.89 total 50.03 last 81.62)
 - open/close 5159 (time 1788438242.90 total 60.04 last 93.48)
 - open/close 5959 (time 1788438252.92 total 70.05 last 79.89)
 - open/close 6821 (time 1788438262.92 total 80.06 last 86.13)
 - open/close 7653 (time 1788438272.93 total 90.07 last 83.16)
 - open/close 8460 (time 1788438282.93 total 100.07 last 80.69)
 - open/close 9254 (time 1788438292.94 total 110.07 last 79.34)
total: 10000 open/close in 118.97 seconds: 84.06 ops/second
 - unlinked 0 (time 1788438305 ; total 0 ; last 0)
total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second
 - open/close 814 (time 1788438390.63 total 10.00 last 81.36)
 - open/close 1705 (time 1788438400.63 total 20.01 last 89.05)
 - open/close 2563 (time 1788438410.64 total 30.01 last 85.76)
 - open/close 3403 (time 1788438420.64 total 40.02 last 83.99)
 - open/close 4185 (time 1788438430.65 total 50.02 last 78.13)
 - open/close 5046 (time 1788438440.66 total 60.03 last 86.03)
 - open/close 5907 (time 1788438450.66 total 70.04 last 86.07)
 - open/close 6711 (time 1788438460.67 total 80.05 last 80.30)
 - open/close 7603 (time 1788438470.68 total 90.05 last 89.17)
 - open/close 8463 (time 1788438480.68 total 100.06 last 85.93)
 - open/close 9278 (time 1788438490.70 total 110.08 last 81.35)
total: 10000 open/close in 119.00 seconds: 84.04 ops/second
 - unlinked 0 (time 1788438504 ; total 0 ; last 0)
total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second
 - open/close 885 (time 1788438591.81 total 10.00 last 88.47)
 - open/close 1753 (time 1788438601.82 total 20.01 last 86.75)
 - open/close 2633 (time 1788438611.82 total 30.01 last 87.97)
 - open/close 3408 (time 1788438621.83 total 40.02 last 77.46)
 - open/close 4104 (time 1788438631.83 total 50.02 last 69.59)
 - open/close 4790 (time 1788438641.84 total 60.03 last 68.50)
 - open/close 5442 (time 1788438651.85 total 70.04 last 65.13)
 - open/close 6300 (time 1788438661.86 total 80.05 last 85.74)
 - open/close 7138 (time 1788438671.86 total 90.05 last 83.78)
 - open/close 7911 (time 1788438681.88 total 100.07 last 77.19)
 - open/close 8777 (time 1788438691.88 total 110.07 last 86.56)
 - open/close 9541 (time 1788438701.88 total 120.08 last 76.38)
total: 10000 open/close in 125.21 seconds: 79.87 ops/second
 - unlinked 0 (time 1788438710 ; total 0 ; last 0)
total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second
 - open/close 792 (time 1788438801.95 total 10.00 last 79.17)
 - open/close 1576 (time 1788438811.96 total 20.01 last 78.34)
 - open/close 2391 (time 1788438821.96 total 30.02 last 81.46)
 - open/close 3292 (time 1788438831.97 total 40.02 last 90.04)
 - open/close 4163 (time 1788438841.97 total 50.03 last 87.08)
 - open/close 4986 (time 1788438851.98 total 60.03 last 82.27)
 - open/close 5819 (time 1788438861.98 total 70.03 last 83.29)
 - open/close 6617 (time 1788438871.99 total 80.05 last 79.69)
 - open/close 7511 (time 1788438882.00 total 90.05 last 89.34)
 - open/close 8345 (time 1788438892.00 total 100.06 last 83.37)
 - open/close 9198 (time 1788438902.02 total 110.07 last 85.14)
total: 10000 open/close in 119.92 seconds: 83.39 ops/second
 - unlinked 0 (time 1788438915 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 836 (time 1788439007.66 total 10.00 last 83.57)
 - open/close 1617 (time 1788439017.69 total 20.03 last 77.91)
 - open/close 2442 (time 1788439027.69 total 30.03 last 82.48)
 - open/close 3261 (time 1788439037.70 total 40.04 last 81.83)
 - open/close 4083 (time 1788439047.71 total 50.05 last 82.12)
 - open/close 4918 (time 1788439057.71 total 60.06 last 83.43)
 - open/close 5634 (time 1788439067.73 total 70.07 last 71.48)
 - open/close 6480 (time 1788439077.74 total 80.08 last 84.57)
 - open/close 7348 (time 1788439087.74 total 90.08 last 86.79)
 - open/close 8280 (time 1788439097.75 total 100.09 last 93.09)
 - open/close 9150 (time 1788439107.75 total 110.09 last 86.99)
total: 10000 open/close in 119.62 seconds: 83.60 ops/second
 - unlinked 0 (time 1788439121 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
 - open/close 827 (time 1788439214.52 total 10.01 last 82.63)
 - open/close 1624 (time 1788439224.53 total 20.02 last 79.64)
 - open/close 2373 (time 1788439234.54 total 30.02 last 74.84)
 - open/close 3151 (time 1788439244.54 total 40.03 last 77.74)
 - open/close 3916 (time 1788439254.55 total 50.04 last 76.45)
 - open/close 4687 (time 1788439264.55 total 60.04 last 77.07)
 - open/close 5518 (time 1788439274.57 total 70.06 last 82.95)
 - open/close 6316 (time 1788439284.58 total 80.07 last 79.74)
 - open/close 7167 (time 1788439294.59 total 90.07 last 85.03)
 - open/close 7877 (time 1788439304.61 total 100.10 last 70.81)
 - open/close 8607 (time 1788439314.62 total 110.10 last 72.98)
 - open/close 9246 (time 1788439324.63 total 120.12 last 63.78)
 - open/close 9827 (time 1788439334.64 total 130.13 last 58.05)
total: 10000 open/close in 132.48 seconds: 75.48 ops/second
 - unlinked 0 (time 1788439341 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 828 (time 1788439437.09 total 10.01 last 82.73)
 - open/close 1685 (time 1788439447.10 total 20.01 last 85.67)
 - open/close 2521 (time 1788439457.14 total 30.05 last 83.26)
 - open/close 3296 (time 1788439467.14 total 40.06 last 77.45)
 - open/close 4085 (time 1788439477.16 total 50.08 last 78.75)
 - open/close 4842 (time 1788439487.16 total 60.08 last 75.69)
 - open/close 5689 (time 1788439497.16 total 70.08 last 84.69)
 - open/close 6471 (time 1788439507.17 total 80.09 last 78.16)
 - open/close 7248 (time 1788439517.18 total 90.09 last 77.63)
 - open/close 8056 (time 1788439527.19 total 100.10 last 80.74)
 - open/close 8875 (time 1788439537.20 total 110.11 last 81.81)
 - open/close 9724 (time 1788439547.20 total 120.11 last 84.89)
total: 10000 open/close in 123.75 seconds: 80.81 ops/second
 - unlinked 0 (time 1788439555 ; total 0 ; last 0)
total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second
 - open/close 879 (time 1788439646.76 total 10.00 last 87.88)
 - open/close 1673 (time 1788439656.76 total 20.00 last 79.38)
 - open/close 2522 (time 1788439666.76 total 30.01 last 84.87)
 - open/close 3281 (time 1788439676.78 total 40.03 last 75.77)
 - open/close 3998 (time 1788439686.79 total 50.04 last 71.62)
 - open/close 4824 (time 1788439696.80 total 60.05 last 82.52)
 - open/close 5598 (time 1788439706.80 total 70.05 last 77.40)
 - open/close 6443 (time 1788439716.81 total 80.06 last 84.40)
 - open/close 7306 (time 1788439726.83 total 90.07 last 86.19)
 - open/close 8156 (time 1788439736.84 total 100.08 last 84.93)
 - open/close 9062 (time 1788439746.84 total 110.08 last 90.56)
 - open/close 9837 (time 1788439756.85 total 120.09 last 77.43)
total: 10000 open/close in 122.40 seconds: 81.70 ops/second
 - unlinked 0 (time 1788439763 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg636-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg636-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg636-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.136@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.136@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg636-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all
pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1
Started lustre-OST0000
oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg636-server: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg636-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg636-client.virtnet:  -o user_xattr,flock 192.168.206.136@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg636-client: oleg636-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg636-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg636-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg636-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.136@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.136@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on mgsnode=192.168.206.136@tcp sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on mgsnode=192.168.206.136@tcp sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:mgsnode=192.168.206.136@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg636-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all
pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1
Started lustre-OST0000
oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg636-server: oleg636-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg636-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg636-client.virtnet:  -o user_xattr,flock 192.168.206.136@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg636-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg636-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg636-server
stop mds service on oleg636-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg636-server
unloading modules via unload_modules_local on: 'oleg636-server'
oleg636-server: oleg636-server.virtnet: executing unload_modules_local
oleg636-server: kmemleak disabled
modules unloaded.
pdsh@oleg636-client: oleg636-client: ssh exited with exit code 2
pdsh@oleg636-client: oleg636-server: ssh exited with exit code 2
pdsh@oleg636-client: oleg636-server: ssh exited with exit code 2
pdsh@oleg636-client: oleg636-client: ssh exited with exit code 2