== conf-sanity test 69: replace an OST with the same index ========================================================== 00:01:39 (1789444899)
start mds service on oleg209-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1933448 Sep 14 13:54 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg209-server'
oleg209-server: oleg209-server.virtnet: executing load_modules_local
oleg209-server: kmemleak disabled
oleg209-server: Loading modules from /home/green/git/lustre-release/lustre
oleg209-server: detected 4 online CPUs by sysfs
oleg209-server: MODOPTS_LIBCFS=
oleg209-server: Force libcfs to create 2 CPU partitions
oleg209-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg209-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg209-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg209-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all
pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg209-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all
pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1
Started lustre-OST0000
oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg209-client.virtnet:  -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 723 (time 1789444986.05 total 10.01 last 72.24)
 - open/close 1620 (time 1789444996.07 total 20.03 last 89.52)
 - open/close 2479 (time 1789445006.08 total 30.04 last 85.81)
 - open/close 3346 (time 1789445016.08 total 40.04 last 86.70)
 - open/close 4212 (time 1789445026.08 total 50.04 last 86.56)
 - open/close 5012 (time 1789445036.09 total 60.05 last 79.96)
 - open/close 5805 (time 1789445046.09 total 70.05 last 79.28)
 - open/close 6569 (time 1789445056.09 total 80.05 last 76.39)
 - open/close 7344 (time 1789445066.10 total 90.06 last 77.47)
 - open/close 8051 (time 1789445076.10 total 100.06 last 70.65)
 - open/close 8760 (time 1789445086.10 total 110.06 last 70.90)
 - open/close 9472 (time 1789445096.11 total 120.07 last 71.14)
total: 10000 open/close in 127.11 seconds: 78.67 ops/second
 - unlinked 0 (time 1789445108 ; total 0 ; last 0)
total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second
 - open/close 787 (time 1789445216.34 total 10.00 last 78.69)
 - open/close 1433 (time 1789445226.34 total 20.00 last 64.59)
 - open/close 2161 (time 1789445236.36 total 30.01 last 72.72)
 - open/close 2863 (time 1789445246.36 total 40.02 last 70.14)
 - open/close 3563 (time 1789445256.37 total 50.03 last 69.98)
 - open/close 4252 (time 1789445266.37 total 60.03 last 68.88)
 - open/close 4953 (time 1789445276.39 total 70.05 last 69.94)
 - open/close 5569 (time 1789445286.40 total 80.06 last 61.56)
 - open/close 6244 (time 1789445296.40 total 90.06 last 67.47)
 - open/close 6973 (time 1789445306.41 total 100.07 last 72.84)
 - open/close 7560 (time 1789445316.41 total 110.07 last 58.69)
 - open/close 8197 (time 1789445326.43 total 120.09 last 63.61)
 - open/close 8884 (time 1789445336.43 total 130.09 last 68.68)
 - open/close 9518 (time 1789445346.45 total 140.11 last 63.29)
total: 10000 open/close in 148.11 seconds: 67.52 ops/second
 - unlinked 0 (time 1789445359 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 733 (time 1789445461.99 total 10.00 last 73.29)
 - open/close 1433 (time 1789445471.99 total 20.00 last 69.98)
 - open/close 2038 (time 1789445482.01 total 30.02 last 60.39)
 - open/close 2751 (time 1789445492.02 total 40.03 last 71.23)
 - open/close 3477 (time 1789445502.03 total 50.04 last 72.53)
 - open/close 4133 (time 1789445512.05 total 60.06 last 65.49)
 - open/close 4815 (time 1789445522.05 total 70.06 last 68.18)
 - open/close 5482 (time 1789445532.06 total 80.07 last 66.65)
 - open/close 6176 (time 1789445542.08 total 90.09 last 69.23)
 - open/close 6992 (time 1789445552.10 total 100.11 last 81.50)
 - open/close 7735 (time 1789445562.10 total 110.11 last 74.26)
 - open/close 8405 (time 1789445572.11 total 120.12 last 66.94)
 - open/close 9092 (time 1789445582.12 total 130.13 last 68.63)
 - open/close 9757 (time 1789445592.13 total 140.14 last 66.42)
total: 10000 open/close in 144.19 seconds: 69.35 ops/second
 - unlinked 0 (time 1789445601 ; total 0 ; last 0)
total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second
 - open/close 592 (time 1789445710.03 total 10.01 last 59.15)
 - open/close 1220 (time 1789445720.03 total 20.01 last 62.79)
 - open/close 1865 (time 1789445730.04 total 30.02 last 64.47)
 - open/close 2486 (time 1789445740.04 total 40.02 last 62.09)
 - open/close 3180 (time 1789445750.06 total 50.04 last 69.25)
 - open/close 3978 (time 1789445760.07 total 60.05 last 79.74)
 - open/close 4755 (time 1789445770.08 total 70.05 last 77.64)
 - open/close 5413 (time 1789445780.08 total 80.06 last 65.76)
 - open/close 6177 (time 1789445790.09 total 90.06 last 76.37)
 - open/close 6969 (time 1789445800.09 total 100.07 last 79.18)
 - open/close 7702 (time 1789445810.10 total 110.08 last 73.23)
 - open/close 8362 (time 1789445820.12 total 120.09 last 65.89)
 - open/close 9085 (time 1789445830.12 total 130.09 last 72.29)
 - open/close 9787 (time 1789445840.14 total 140.12 last 70.02)
total: 10000 open/close in 142.74 seconds: 70.06 ops/second
 - unlinked 0 (time 1789445847 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 737 (time 1789445946.34 total 10.01 last 73.66)
 - open/close 1464 (time 1789445956.35 total 20.02 last 72.61)
 - open/close 2291 (time 1789445966.37 total 30.03 last 82.58)
 - open/close 2935 (time 1789445976.38 total 40.04 last 64.35)
 - open/close 3606 (time 1789445986.38 total 50.04 last 67.08)
 - open/close 4335 (time 1789445996.38 total 60.05 last 72.88)
 - open/close 5084 (time 1789446006.39 total 70.06 last 74.82)
 - open/close 5740 (time 1789446016.39 total 80.06 last 65.60)
 - open/close 6480 (time 1789446026.40 total 90.06 last 73.99)
 - open/close 7231 (time 1789446036.40 total 100.07 last 75.05)
 - open/close 7770 (time 1789446046.41 total 110.07 last 53.86)
 - open/close 8495 (time 1789446056.45 total 120.11 last 72.20)
 - open/close 9154 (time 1789446066.45 total 130.12 last 65.89)
 - open/close 9814 (time 1789446076.46 total 140.12 last 65.96)
total: 10000 open/close in 142.56 seconds: 70.15 ops/second
 - unlinked 0 (time 1789446084 ; total 0 ; last 0)
total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second
 - open/close 827 (time 1789446198.13 total 10.02 last 82.54)
 - open/close 1646 (time 1789446208.13 total 20.02 last 81.89)
 - open/close 2372 (time 1789446218.13 total 30.03 last 72.55)
 - open/close 3081 (time 1789446228.14 total 40.04 last 70.84)
 - open/close 3750 (time 1789446238.14 total 50.04 last 66.89)
 - open/close 4514 (time 1789446248.15 total 60.04 last 76.39)
 - open/close 5205 (time 1789446258.15 total 70.05 last 69.05)
 - open/close 5835 (time 1789446268.15 total 80.05 last 62.99)
 - open/close 6588 (time 1789446278.16 total 90.05 last 75.29)
 - open/close 7330 (time 1789446288.16 total 100.05 last 74.20)
 - open/close 7974 (time 1789446298.17 total 110.06 last 64.33)
 - open/close 8597 (time 1789446308.17 total 120.06 last 62.28)
 - open/close 9260 (time 1789446318.17 total 130.07 last 66.28)
 - open/close 9866 (time 1789446328.18 total 140.08 last 60.54)
total: 10000 open/close in 142.35 seconds: 70.25 ops/second
 - unlinked 0 (time 1789446335 ; total 0 ; last 0)
total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second
 - open/close 660 (time 1789446445.84 total 10.00 last 65.98)
 - open/close 1438 (time 1789446455.85 total 20.01 last 77.78)
 - open/close 2250 (time 1789446465.85 total 30.01 last 81.16)
 - open/close 2957 (time 1789446475.86 total 40.02 last 70.65)
 - open/close 3621 (time 1789446485.87 total 50.02 last 66.36)
 - open/close 4275 (time 1789446495.87 total 60.03 last 65.37)
 - open/close 4933 (time 1789446505.88 total 70.03 last 65.76)
 - open/close 5593 (time 1789446515.89 total 80.05 last 65.91)
 - open/close 6283 (time 1789446525.89 total 90.05 last 68.98)
 - open/close 7096 (time 1789446535.89 total 100.05 last 81.28)
 - open/close 7838 (time 1789446545.91 total 110.07 last 74.06)
 - open/close 8554 (time 1789446555.92 total 120.08 last 71.58)
 - open/close 9272 (time 1789446565.93 total 130.09 last 71.72)
 - open/close 9940 (time 1789446575.93 total 140.09 last 66.78)
total: 10000 open/close in 140.92 seconds: 70.96 ops/second
 - unlinked 0 (time 1789446580 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 632 (time 1789446681.33 total 10.01 last 63.13)
 - open/close 1305 (time 1789446691.34 total 20.02 last 67.23)
 - open/close 2051 (time 1789446701.35 total 30.03 last 74.52)
 - open/close 2806 (time 1789446711.36 total 40.04 last 75.43)
 - open/close 3583 (time 1789446721.40 total 50.08 last 77.40)
 - open/close 4290 (time 1789446731.41 total 60.09 last 70.63)
 - open/close 5002 (time 1789446741.42 total 70.11 last 71.09)
 - open/close 5660 (time 1789446751.44 total 80.12 last 65.69)
 - open/close 6325 (time 1789446761.45 total 90.13 last 66.42)
 - open/close 6980 (time 1789446771.47 total 100.15 last 65.39)
 - open/close 7515 (time 1789446781.47 total 110.16 last 53.47)
 - open/close 8114 (time 1789446791.48 total 120.16 last 59.85)
 - open/close 8759 (time 1789446801.49 total 130.18 last 64.41)
 - open/close 9397 (time 1789446811.49 total 140.18 last 63.80)
total: 10000 open/close in 148.96 seconds: 67.13 ops/second
 - unlinked 0 (time 1789446825 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 551 (time 1789446932.42 total 10.00 last 55.09)
 - open/close 1183 (time 1789446942.45 total 20.03 last 63.00)
 - open/close 1893 (time 1789446952.46 total 30.05 last 70.90)
 - open/close 2704 (time 1789446962.47 total 40.05 last 81.06)
 - open/close 3538 (time 1789446972.47 total 50.06 last 83.33)
 - open/close 4296 (time 1789446982.48 total 60.07 last 75.75)
 - open/close 5039 (time 1789446992.49 total 70.08 last 74.24)
 - open/close 5772 (time 1789447002.49 total 80.08 last 73.29)
 - open/close 6497 (time 1789447012.49 total 90.08 last 72.49)
 - open/close 7242 (time 1789447022.51 total 100.10 last 74.34)
 - open/close 7940 (time 1789447032.52 total 110.11 last 69.75)
 - open/close 8720 (time 1789447042.53 total 120.11 last 77.96)
 - open/close 9483 (time 1789447052.53 total 130.12 last 76.25)
total: 10000 open/close in 136.59 seconds: 73.21 ops/second
 - unlinked 0 (time 1789447063 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 675 (time 1789447166.24 total 10.01 last 67.41)
 - open/close 1423 (time 1789447176.25 total 20.02 last 74.78)
 - open/close 2075 (time 1789447186.27 total 30.04 last 65.03)
 - open/close 2707 (time 1789447196.29 total 40.06 last 63.11)
 - open/close 3382 (time 1789447206.29 total 50.06 last 67.47)
 - open/close 4091 (time 1789447216.30 total 60.07 last 70.83)
 - open/close 4794 (time 1789447226.31 total 70.08 last 70.27)
 - open/close 5510 (time 1789447236.31 total 80.08 last 71.55)
 - open/close 6167 (time 1789447246.32 total 90.09 last 65.64)
 - open/close 6863 (time 1789447256.32 total 100.09 last 69.59)
 - open/close 7437 (time 1789447266.32 total 110.09 last 57.39)
 - open/close 8161 (time 1789447276.33 total 120.10 last 72.34)
 - open/close 8941 (time 1789447286.34 total 130.11 last 77.98)
 - open/close 9567 (time 1789447296.34 total 140.11 last 62.55)
total: 10000 open/close in 146.59 seconds: 68.22 ops/second
 - unlinked 0 (time 1789447307 ; total 0 ; last 0)
total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg209-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg209-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.109@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.109@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg209-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all
pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1
Started lustre-OST0000
oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg209-server: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg209-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg209-client.virtnet:  -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg209-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg209-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.109@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.109@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.109@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.109@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.202.109@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg209-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all
pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1
Started lustre-OST0000
oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg209-server: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg209-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg209-client.virtnet:  -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg209-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg209-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server
stop mds service on oleg209-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server
unloading modules via unload_modules_local on: 'oleg209-server'
oleg209-server: oleg209-server.virtnet: executing unload_modules_local
oleg209-server: kmemleak disabled
modules unloaded.
pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2
pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2
pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2
pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2