== conf-sanity test 69: replace an OST with the same index ========================================================== 16:56:00 (1788468960)
start mds service on oleg220-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1921072 Sep  3 15:45 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg220-server'
oleg220-server: oleg220-server.virtnet: executing load_modules_local
oleg220-server: kmemleak disabled
oleg220-server: Loading modules from /home/green/git/lustre-release/lustre
oleg220-server: detected 4 online CPUs by sysfs
oleg220-server: MODOPTS_LIBCFS=
oleg220-server: Force libcfs to create 2 CPU partitions
oleg220-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg220-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg220-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg220-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg220-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg220-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Started lustre-OST0000
oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 620 (time 1788469058.28 total 10.00 last 62.00)
 - open/close 1202 (time 1788469068.28 total 20.00 last 58.18)
 - open/close 1778 (time 1788469078.28 total 30.00 last 57.60)
 - open/close 2377 (time 1788469088.30 total 40.02 last 59.83)
 - open/close 2928 (time 1788469098.31 total 50.03 last 55.03)
 - open/close 3487 (time 1788469108.32 total 60.04 last 55.83)
 - open/close 4177 (time 1788469118.34 total 70.06 last 68.86)
 - open/close 4783 (time 1788469128.34 total 80.06 last 60.58)
 - open/close 5288 (time 1788469138.35 total 90.07 last 50.47)
 - open/close 5767 (time 1788469148.36 total 100.08 last 47.87)
 - open/close 6198 (time 1788469158.36 total 110.08 last 43.09)
 - open/close 6740 (time 1788469168.37 total 120.09 last 54.15)
 - open/close 7318 (time 1788469178.38 total 130.10 last 57.75)
 - open/close 7765 (time 1788469188.38 total 140.10 last 44.68)
 - open/close 8121 (time 1788469198.40 total 150.12 last 35.55)
 - open/close 8636 (time 1788469208.40 total 160.12 last 51.46)
 - open/close 9273 (time 1788469218.41 total 170.13 last 63.68)
 - open/close 9920 (time 1788469228.42 total 180.14 last 64.64)
total: 10000 open/close in 181.28 seconds: 55.16 ops/second
 - unlinked 0 (time 1788469234 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 650 (time 1788469347.49 total 10.01 last 64.94)
 - open/close 1251 (time 1788469357.50 total 20.02 last 60.03)
 - open/close 1742 (time 1788469367.51 total 30.03 last 49.04)
 - open/close 2198 (time 1788469377.53 total 40.05 last 45.54)
 - open/close 2700 (time 1788469387.54 total 50.06 last 50.12)
 - open/close 3307 (time 1788469397.54 total 60.07 last 60.70)
 - open/close 3856 (time 1788469407.56 total 70.08 last 54.82)
 - open/close 4446 (time 1788469417.57 total 80.09 last 58.92)
 - open/close 5077 (time 1788469427.58 total 90.10 last 63.05)
 - open/close 5722 (time 1788469437.58 total 100.11 last 64.47)
 - open/close 6313 (time 1788469447.60 total 110.12 last 59.03)
 - open/close 6909 (time 1788469457.63 total 120.16 last 59.38)
 - open/close 7449 (time 1788469467.66 total 130.18 last 53.86)
 - open/close 8165 (time 1788469477.67 total 140.19 last 71.54)
 - open/close 8776 (time 1788469487.68 total 150.20 last 61.02)
 - open/close 9285 (time 1788469497.69 total 160.21 last 50.85)
 - open/close 9748 (time 1788469507.69 total 170.21 last 46.30)
total: 10000 open/close in 176.34 seconds: 56.71 ops/second
 - unlinked 0 (time 1788469521 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 664 (time 1788469627.64 total 10.00 last 66.39)
 - open/close 1272 (time 1788469637.67 total 20.04 last 60.59)
 - open/close 1749 (time 1788469647.67 total 30.04 last 47.70)
 - open/close 2496 (time 1788469657.68 total 40.05 last 74.63)
 - open/close 3189 (time 1788469667.69 total 50.05 last 69.24)
 - open/close 3808 (time 1788469677.69 total 60.06 last 61.88)
 - open/close 4513 (time 1788469687.69 total 70.06 last 70.48)
 - open/close 5258 (time 1788469697.70 total 80.06 last 74.49)
 - open/close 5942 (time 1788469707.71 total 90.07 last 68.32)
 - open/close 6668 (time 1788469717.71 total 100.08 last 72.57)
 - open/close 7358 (time 1788469727.72 total 110.08 last 68.96)
 - open/close 7951 (time 1788469737.72 total 120.09 last 59.27)
 - open/close 8456 (time 1788469747.72 total 130.09 last 50.50)
 - open/close 9058 (time 1788469757.73 total 140.10 last 60.16)
 - open/close 9719 (time 1788469767.73 total 150.10 last 66.08)
total: 10000 open/close in 154.38 seconds: 64.78 ops/second
 - unlinked 0 (time 1788469777 ; total 0 ; last 0)
total: 10000 unlinks in 118 seconds: 84.745766 unlinks/second
 - open/close 571 (time 1788469911.71 total 10.02 last 56.99)
 - open/close 1186 (time 1788469921.72 total 20.03 last 61.44)
 - open/close 1820 (time 1788469931.74 total 30.06 last 63.23)
 - open/close 2558 (time 1788469941.75 total 40.06 last 73.77)
 - open/close 3270 (time 1788469951.76 total 50.07 last 71.11)
 - open/close 4064 (time 1788469961.76 total 60.07 last 79.38)
 - open/close 4686 (time 1788469971.77 total 70.08 last 62.17)
 - open/close 5120 (time 1788469981.79 total 80.10 last 43.31)
 - open/close 5562 (time 1788469991.80 total 90.11 last 44.16)
 - open/close 6120 (time 1788470001.82 total 100.14 last 55.65)
 - open/close 6811 (time 1788470011.83 total 110.15 last 69.02)
 - open/close 7482 (time 1788470021.85 total 120.16 last 67.01)
 - open/close 7973 (time 1788470031.86 total 130.17 last 49.05)
 - open/close 8606 (time 1788470041.86 total 140.18 last 63.28)
 - open/close 9171 (time 1788470051.87 total 150.18 last 56.47)
 - open/close 9630 (time 1788470061.87 total 160.18 last 45.89)
total: 10000 open/close in 167.76 seconds: 59.61 ops/second
 - unlinked 0 (time 1788470075 ; total 0 ; last 0)
total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second
 - open/close 498 (time 1788470182.40 total 10.01 last 49.77)
 - open/close 1180 (time 1788470192.41 total 20.02 last 68.13)
 - open/close 1897 (time 1788470202.41 total 30.02 last 71.67)
 - open/close 2291 (time 1788470212.43 total 40.03 last 39.36)
 - open/close 2579 (time 1788470222.46 total 50.07 last 28.70)
 - open/close 3120 (time 1788470232.47 total 60.08 last 54.06)
 - open/close 3494 (time 1788470242.48 total 70.09 last 37.37)
 - open/close 3915 (time 1788470252.49 total 80.10 last 42.03)
 - open/close 4377 (time 1788470262.50 total 90.11 last 46.18)
 - open/close 4758 (time 1788470272.54 total 100.15 last 37.95)
 - open/close 5117 (time 1788470282.54 total 110.15 last 35.89)
 - open/close 5631 (time 1788470292.55 total 120.16 last 51.33)
 - open/close 6197 (time 1788470302.56 total 130.17 last 56.57)
 - open/close 6634 (time 1788470312.58 total 140.18 last 43.63)
 - open/close 7274 (time 1788470322.60 total 150.21 last 63.82)
 - open/close 7872 (time 1788470332.61 total 160.21 last 59.79)
 - open/close 8428 (time 1788470342.61 total 170.22 last 55.58)
 - open/close 9045 (time 1788470352.61 total 180.22 last 61.70)
 - open/close 9552 (time 1788470362.64 total 190.24 last 50.56)
total: 10000 open/close in 197.31 seconds: 50.68 ops/second
 - unlinked 0 (time 1788470374 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 807 (time 1788470474.86 total 10.01 last 80.64)
 - open/close 1556 (time 1788470484.87 total 20.02 last 74.84)
 - open/close 2205 (time 1788470494.89 total 30.04 last 64.75)
 - open/close 2835 (time 1788470504.90 total 40.05 last 62.95)
 - open/close 3547 (time 1788470514.90 total 50.05 last 71.17)
 - open/close 4196 (time 1788470524.92 total 60.07 last 64.77)
 - open/close 4902 (time 1788470534.95 total 70.09 last 70.44)
 - open/close 5628 (time 1788470544.96 total 80.10 last 72.53)
 - open/close 6379 (time 1788470554.96 total 90.11 last 75.06)
 - open/close 7070 (time 1788470564.97 total 100.11 last 69.06)
 - open/close 7774 (time 1788470574.97 total 110.12 last 70.37)
 - open/close 8446 (time 1788470584.97 total 120.12 last 67.17)
 - open/close 9179 (time 1788470595.00 total 130.14 last 73.14)
 - open/close 9903 (time 1788470605.00 total 140.15 last 72.33)
total: 10000 open/close in 141.99 seconds: 70.43 ops/second
 - unlinked 0 (time 1788470611 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 700 (time 1788470722.78 total 10.01 last 69.94)
 - open/close 1394 (time 1788470732.79 total 20.02 last 69.33)
 - open/close 2068 (time 1788470742.80 total 30.03 last 67.32)
 - open/close 2665 (time 1788470752.80 total 40.03 last 59.69)
 - open/close 3339 (time 1788470762.81 total 50.04 last 67.35)
 - open/close 4110 (time 1788470772.82 total 60.05 last 77.04)
 - open/close 4676 (time 1788470782.82 total 70.05 last 56.58)
 - open/close 5375 (time 1788470792.83 total 80.06 last 69.86)
 - open/close 6173 (time 1788470802.83 total 90.07 last 79.73)
 - open/close 6783 (time 1788470812.85 total 100.08 last 60.93)
 - open/close 7523 (time 1788470822.85 total 110.08 last 73.98)
 - open/close 8327 (time 1788470832.87 total 120.11 last 80.20)
 - open/close 9070 (time 1788470842.91 total 130.14 last 74.04)
 - open/close 9643 (time 1788470852.92 total 140.16 last 57.21)
total: 10000 open/close in 145.13 seconds: 68.90 ops/second
 - unlinked 0 (time 1788470862 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
 - open/close 738 (time 1788470964.13 total 10.01 last 73.69)
 - open/close 1380 (time 1788470974.14 total 20.02 last 64.14)
 - open/close 2102 (time 1788470984.15 total 30.03 last 72.16)
 - open/close 2792 (time 1788470994.15 total 40.03 last 69.00)
 - open/close 3447 (time 1788471004.16 total 50.04 last 65.46)
 - open/close 4150 (time 1788471014.16 total 60.04 last 70.28)
 - open/close 4805 (time 1788471024.17 total 70.05 last 65.45)
 - open/close 5489 (time 1788471034.19 total 80.07 last 68.21)
 - open/close 6199 (time 1788471044.20 total 90.08 last 70.98)
 - open/close 6920 (time 1788471054.20 total 100.08 last 72.10)
 - open/close 7673 (time 1788471064.20 total 110.08 last 75.28)
 - open/close 8447 (time 1788471074.21 total 120.09 last 77.34)
 - open/close 9257 (time 1788471084.21 total 130.09 last 80.98)
total: 10000 open/close in 138.73 seconds: 72.08 ops/second
 - unlinked 0 (time 1788471097 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 678 (time 1788471204.80 total 10.00 last 67.79)
 - open/close 1288 (time 1788471214.80 total 20.00 last 61.00)
 - open/close 1943 (time 1788471224.81 total 30.02 last 65.39)
 - open/close 2541 (time 1788471234.84 total 40.04 last 59.66)
 - open/close 3189 (time 1788471244.85 total 50.06 last 64.69)
 - open/close 3798 (time 1788471254.87 total 60.08 last 60.79)
 - open/close 4454 (time 1788471264.90 total 70.11 last 65.42)
 - open/close 5115 (time 1788471274.91 total 80.11 last 66.05)
 - open/close 5850 (time 1788471284.92 total 90.12 last 73.44)
 - open/close 6464 (time 1788471294.93 total 100.14 last 61.30)
 - open/close 7073 (time 1788471304.94 total 110.15 last 60.85)
 - open/close 7548 (time 1788471314.96 total 120.17 last 47.41)
 - open/close 8116 (time 1788471324.97 total 130.18 last 56.74)
 - open/close 8599 (time 1788471335.00 total 140.20 last 48.18)
 - open/close 9281 (time 1788471345.01 total 150.21 last 68.12)
 - open/close 9988 (time 1788471355.01 total 160.22 last 70.66)
total: 10000 open/close in 160.42 seconds: 62.33 ops/second
 - unlinked 0 (time 1788471360 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 688 (time 1788471465.83 total 10.01 last 68.76)
 - open/close 1322 (time 1788471475.84 total 20.02 last 63.34)
 - open/close 1985 (time 1788471485.85 total 30.02 last 66.27)
 - open/close 2781 (time 1788471495.85 total 40.03 last 79.54)
 - open/close 3574 (time 1788471505.86 total 50.04 last 79.24)
 - open/close 4291 (time 1788471515.87 total 60.04 last 71.64)
 - open/close 5024 (time 1788471525.88 total 70.06 last 73.22)
 - open/close 5677 (time 1788471535.90 total 80.07 last 65.21)
 - open/close 6336 (time 1788471545.90 total 90.08 last 65.85)
 - open/close 7013 (time 1788471555.91 total 100.09 last 67.62)
 - open/close 7707 (time 1788471565.92 total 110.09 last 69.38)
 - open/close 8353 (time 1788471575.93 total 120.10 last 64.53)
 - open/close 9019 (time 1788471585.95 total 130.13 last 66.45)
 - open/close 9721 (time 1788471595.96 total 140.14 last 70.14)
total: 10000 open/close in 144.18 seconds: 69.36 ops/second
 - unlinked 0 (time 1788471605 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg220-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg220-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg220-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.120@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.120@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg220-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Started lustre-OST0000
oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg220-server: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg220-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg220-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg220-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg220-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.120@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.120@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.120@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.120@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.120@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg220-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg220-server: oleg220-server.virtnet: executing set_default_debug -1 all
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 1
Started lustre-OST0000
oleg220-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg220-server: oleg220-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg220-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg220-client.virtnet:  -o user_xattr,flock 192.168.202.120@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg220-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg220-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg220-server
stop mds service on oleg220-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg220-server
unloading modules via unload_modules_local on: 'oleg220-server'
oleg220-server: oleg220-server.virtnet: executing unload_modules_local
oleg220-server: kmemleak disabled
modules unloaded.
pdsh@oleg220-client: oleg220-client: ssh exited with exit code 2
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 2
pdsh@oleg220-client: oleg220-client: ssh exited with exit code 2
pdsh@oleg220-client: oleg220-server: ssh exited with exit code 2