== conf-sanity test 69: replace an OST with the same index ========================================================== 18:11:02 (1790892662)
start mds service on oleg458-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1940568 Oct  1 17:41 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg458-server'
oleg458-server: oleg458-server.virtnet: executing load_modules_local
oleg458-server: kmemleak disabled
oleg458-server: Loading modules from /home/green/git/lustre-release/lustre
oleg458-server: detected 4 online CPUs by sysfs
oleg458-server: MODOPTS_LIBCFS=
oleg458-server: Force libcfs to create 2 CPU partitions
oleg458-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg458-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg458-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg458-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg458-server: oleg458-server.virtnet: executing set_default_debug -1 all
pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg458-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg458-server: oleg458-server.virtnet: executing set_default_debug -1 all
pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1
Started lustre-OST0000
oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg458-client.virtnet:  -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 658 (time 1790892753.99 total 10.01 last 65.73)
 - open/close 1278 (time 1790892764.01 total 20.03 last 61.89)
 - open/close 1965 (time 1790892774.01 total 30.04 last 68.65)
 - open/close 2597 (time 1790892784.04 total 40.06 last 63.05)
 - open/close 3279 (time 1790892794.05 total 50.07 last 68.12)
 - open/close 3874 (time 1790892804.06 total 60.08 last 59.43)
 - open/close 4440 (time 1790892814.06 total 70.08 last 56.60)
 - open/close 5037 (time 1790892824.09 total 80.11 last 59.54)
 - open/close 5649 (time 1790892834.11 total 90.13 last 61.10)
 - open/close 6342 (time 1790892844.11 total 100.14 last 69.24)
 - open/close 6953 (time 1790892854.12 total 110.14 last 61.07)
 - open/close 7567 (time 1790892864.13 total 120.15 last 61.33)
 - open/close 8210 (time 1790892874.14 total 130.16 last 64.25)
 - open/close 8866 (time 1790892884.16 total 140.19 last 65.43)
 - open/close 9537 (time 1790892894.17 total 150.19 last 67.09)
total: 10000 open/close in 157.75 seconds: 63.39 ops/second
 - unlinked 0 (time 1790892905 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 546 (time 1790893018.33 total 10.02 last 54.51)
 - open/close 1111 (time 1790893028.34 total 20.03 last 56.44)
 - open/close 1668 (time 1790893038.35 total 30.04 last 55.65)
 - open/close 2231 (time 1790893048.35 total 40.04 last 56.28)
 - open/close 2765 (time 1790893058.36 total 50.04 last 53.38)
 - open/close 3337 (time 1790893068.36 total 60.05 last 57.17)
 - open/close 3909 (time 1790893078.37 total 70.06 last 57.14)
 - open/close 4501 (time 1790893088.38 total 80.07 last 59.14)
 - open/close 5127 (time 1790893098.38 total 90.07 last 62.60)
 - open/close 5733 (time 1790893108.39 total 100.07 last 60.57)
 - open/close 6343 (time 1790893118.39 total 110.08 last 60.98)
 - open/close 6971 (time 1790893128.40 total 120.08 last 62.74)
 - open/close 7572 (time 1790893138.40 total 130.09 last 60.06)
 - open/close 8211 (time 1790893148.41 total 140.09 last 63.88)
 - open/close 8888 (time 1790893158.41 total 150.10 last 67.68)
 - open/close 9488 (time 1790893168.42 total 160.11 last 59.91)
total: 10000 open/close in 168.92 seconds: 59.20 ops/second
 - unlinked 0 (time 1790893182 ; total 0 ; last 0)
total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second
 - open/close 653 (time 1790893298.00 total 10.01 last 65.23)
 - open/close 1231 (time 1790893308.01 total 20.03 last 57.69)
 - open/close 1818 (time 1790893318.04 total 30.05 last 58.54)
 - open/close 2379 (time 1790893328.04 total 40.06 last 56.09)
 - open/close 2960 (time 1790893338.06 total 50.07 last 58.02)
 - open/close 3568 (time 1790893348.06 total 60.07 last 60.78)
 - open/close 4194 (time 1790893358.06 total 70.08 last 62.58)
 - open/close 4722 (time 1790893368.08 total 80.09 last 52.70)
 - open/close 5326 (time 1790893378.09 total 90.11 last 60.33)
 - open/close 5904 (time 1790893388.10 total 100.11 last 57.77)
 - open/close 6478 (time 1790893398.11 total 110.12 last 57.35)
 - open/close 7063 (time 1790893408.11 total 120.13 last 58.45)
 - open/close 7687 (time 1790893418.13 total 130.14 last 62.30)
 - open/close 8325 (time 1790893428.14 total 140.15 last 63.73)
 - open/close 8916 (time 1790893438.15 total 150.16 last 59.05)
 - open/close 9543 (time 1790893448.16 total 160.17 last 62.63)
total: 10000 open/close in 168.16 seconds: 59.47 ops/second
 - unlinked 0 (time 1790893460 ; total 0 ; last 0)
total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second
 - open/close 621 (time 1790893578.12 total 10.00 last 62.09)
 - open/close 1248 (time 1790893588.13 total 20.00 last 62.69)
 - open/close 1861 (time 1790893598.13 total 30.01 last 61.28)
 - open/close 2474 (time 1790893608.14 total 40.02 last 61.23)
 - open/close 3007 (time 1790893618.15 total 50.02 last 53.26)
 - open/close 3587 (time 1790893628.18 total 60.05 last 57.83)
 - open/close 4228 (time 1790893638.18 total 70.05 last 64.10)
 - open/close 4892 (time 1790893648.19 total 80.07 last 66.29)
 - open/close 5464 (time 1790893658.20 total 90.07 last 57.19)
 - open/close 6011 (time 1790893668.22 total 100.10 last 54.56)
 - open/close 6598 (time 1790893678.23 total 110.11 last 58.65)
 - open/close 7224 (time 1790893688.26 total 120.14 last 62.41)
 - open/close 7837 (time 1790893698.26 total 130.14 last 61.29)
 - open/close 8429 (time 1790893708.27 total 140.14 last 59.18)
 - open/close 8999 (time 1790893718.29 total 150.16 last 56.88)
 - open/close 9642 (time 1790893728.29 total 160.17 last 64.29)
total: 10000 open/close in 166.38 seconds: 60.10 ops/second
 - unlinked 0 (time 1790893739 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 608 (time 1790893853.37 total 10.01 last 60.71)
 - open/close 1186 (time 1790893863.38 total 20.02 last 57.74)
 - open/close 1838 (time 1790893873.39 total 30.03 last 65.18)
 - open/close 2470 (time 1790893883.39 total 40.03 last 63.19)
 - open/close 3145 (time 1790893893.39 total 50.03 last 67.48)
 - open/close 3759 (time 1790893903.41 total 60.05 last 61.27)
 - open/close 4375 (time 1790893913.41 total 70.05 last 61.59)
 - open/close 4977 (time 1790893923.43 total 80.07 last 60.08)
 - open/close 5590 (time 1790893933.43 total 90.08 last 61.28)
 - open/close 6210 (time 1790893943.45 total 100.09 last 61.91)
 - open/close 6858 (time 1790893953.48 total 110.12 last 64.58)
 - open/close 7435 (time 1790893963.50 total 120.14 last 57.62)
 - open/close 7860 (time 1790893973.52 total 130.16 last 42.39)
 - open/close 8422 (time 1790893983.54 total 140.18 last 56.12)
 - open/close 8967 (time 1790893993.55 total 150.19 last 54.42)
 - open/close 9467 (time 1790894003.58 total 160.22 last 49.85)
total: 10000 open/close in 169.77 seconds: 58.90 ops/second
 - unlinked 0 (time 1790894017 ; total 0 ; last 0)
total: 10000 unlinks in 106 seconds: 94.339622 unlinks/second
 - open/close 653 (time 1790894137.30 total 10.03 last 65.08)
 - open/close 1250 (time 1790894147.32 total 20.06 last 59.57)
 - open/close 1774 (time 1790894157.32 total 30.06 last 52.39)
 - open/close 2364 (time 1790894167.33 total 40.07 last 58.95)
 - open/close 2838 (time 1790894177.33 total 50.07 last 47.39)
 - open/close 3396 (time 1790894187.34 total 60.07 last 55.77)
 - open/close 3967 (time 1790894197.35 total 70.09 last 57.01)
 - open/close 4586 (time 1790894207.36 total 80.10 last 61.84)
 - open/close 5236 (time 1790894217.36 total 90.10 last 65.00)
 - open/close 5860 (time 1790894227.37 total 100.11 last 62.35)
 - open/close 6307 (time 1790894237.38 total 110.11 last 44.68)
 - open/close 6807 (time 1790894247.38 total 120.12 last 49.97)
 - open/close 7342 (time 1790894257.39 total 130.13 last 53.45)
 - open/close 7866 (time 1790894267.40 total 140.13 last 52.36)
 - open/close 8385 (time 1790894277.41 total 150.14 last 51.86)
 - open/close 8952 (time 1790894287.42 total 160.15 last 56.63)
 - open/close 9569 (time 1790894297.42 total 170.16 last 61.66)
total: 10000 open/close in 177.55 seconds: 56.32 ops/second
 - unlinked 0 (time 1790894309 ; total 0 ; last 0)
total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second
 - open/close 583 (time 1790894431.22 total 10.00 last 58.28)
 - open/close 1143 (time 1790894441.23 total 20.01 last 55.94)
 - open/close 1667 (time 1790894451.23 total 30.02 last 52.37)
 - open/close 2217 (time 1790894461.23 total 40.02 last 54.99)
 - open/close 2795 (time 1790894471.25 total 50.03 last 57.73)
 - open/close 3370 (time 1790894481.26 total 60.05 last 57.42)
 - open/close 3908 (time 1790894491.27 total 70.05 last 53.78)
 - open/close 4474 (time 1790894501.28 total 80.06 last 56.54)
 - open/close 5048 (time 1790894511.28 total 90.07 last 57.38)
 - open/close 5594 (time 1790894521.29 total 100.08 last 54.53)
 - open/close 6135 (time 1790894531.30 total 110.08 last 54.07)
 - open/close 6768 (time 1790894541.31 total 120.10 last 63.21)
 - open/close 7386 (time 1790894551.32 total 130.11 last 61.74)
 - open/close 8034 (time 1790894561.34 total 140.13 last 64.67)
 - open/close 8614 (time 1790894571.36 total 150.15 last 57.89)
 - open/close 9168 (time 1790894581.38 total 160.16 last 55.31)
 - open/close 9735 (time 1790894591.40 total 170.18 last 56.59)
total: 10000 open/close in 174.38 seconds: 57.35 ops/second
 - unlinked 0 (time 1790894600 ; total 0 ; last 0)
total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second
 - open/close 606 (time 1790894724.32 total 10.01 last 60.55)
 - open/close 1168 (time 1790894734.32 total 20.01 last 56.18)
 - open/close 1732 (time 1790894744.33 total 30.01 last 56.38)
 - open/close 2336 (time 1790894754.34 total 40.02 last 60.34)
 - open/close 2918 (time 1790894764.34 total 50.03 last 58.18)
 - open/close 3448 (time 1790894774.35 total 60.03 last 52.97)
 - open/close 3959 (time 1790894784.35 total 70.04 last 51.08)
 - open/close 4558 (time 1790894794.36 total 80.05 last 59.80)
 - open/close 5145 (time 1790894804.39 total 90.08 last 58.56)
 - open/close 5736 (time 1790894814.40 total 100.09 last 59.04)
 - open/close 6302 (time 1790894824.40 total 110.09 last 56.58)
 - open/close 6890 (time 1790894834.42 total 120.10 last 58.71)
 - open/close 7506 (time 1790894844.42 total 130.10 last 61.60)
 - open/close 8122 (time 1790894854.42 total 140.11 last 61.57)
 - open/close 8751 (time 1790894864.44 total 150.13 last 62.79)
 - open/close 9347 (time 1790894874.45 total 160.14 last 59.54)
 - open/close 9958 (time 1790894884.47 total 170.15 last 60.99)
total: 10000 open/close in 170.84 seconds: 58.53 ops/second
 - unlinked 0 (time 1790894889 ; total 0 ; last 0)
total: 10000 unlinks in 109 seconds: 91.743118 unlinks/second
 - open/close 615 (time 1790895013.52 total 10.00 last 61.49)
 - open/close 1222 (time 1790895023.54 total 20.02 last 60.61)
 - open/close 1812 (time 1790895033.54 total 30.02 last 58.99)
 - open/close 2462 (time 1790895043.55 total 40.03 last 64.91)
 - open/close 3012 (time 1790895053.57 total 50.05 last 54.90)
 - open/close 3554 (time 1790895063.57 total 60.05 last 54.20)
 - open/close 4061 (time 1790895073.58 total 70.06 last 50.65)
 - open/close 4696 (time 1790895083.62 total 80.10 last 63.23)
 - open/close 5266 (time 1790895093.63 total 90.10 last 56.99)
 - open/close 5842 (time 1790895103.64 total 100.12 last 57.49)
 - open/close 6472 (time 1790895113.65 total 110.13 last 62.96)
 - open/close 7071 (time 1790895123.67 total 120.14 last 59.82)
 - open/close 7707 (time 1790895133.68 total 130.16 last 63.52)
 - open/close 8251 (time 1790895143.68 total 140.16 last 54.37)
 - open/close 8858 (time 1790895153.68 total 150.16 last 60.70)
 - open/close 9415 (time 1790895163.69 total 160.17 last 55.66)
total: 10000 open/close in 169.56 seconds: 58.98 ops/second
 - unlinked 0 (time 1790895177 ; total 0 ; last 0)
total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second
 - open/close 553 (time 1790895296.47 total 10.00 last 55.30)
 - open/close 1175 (time 1790895306.47 total 20.00 last 62.20)
 - open/close 1673 (time 1790895316.48 total 30.01 last 49.73)
 - open/close 2232 (time 1790895326.51 total 40.04 last 55.77)
 - open/close 2809 (time 1790895336.52 total 50.05 last 57.65)
 - open/close 3429 (time 1790895346.53 total 60.06 last 61.92)
 - open/close 4037 (time 1790895356.53 total 70.06 last 60.78)
 - open/close 4636 (time 1790895366.54 total 80.07 last 59.87)
 - open/close 5247 (time 1790895376.55 total 90.08 last 61.05)
 - open/close 5878 (time 1790895386.56 total 100.09 last 63.02)
 - open/close 6461 (time 1790895396.56 total 110.09 last 58.27)
 - open/close 7066 (time 1790895406.56 total 120.09 last 60.50)
 - open/close 7686 (time 1790895416.57 total 130.10 last 61.93)
 - open/close 8330 (time 1790895426.58 total 140.11 last 64.36)
 - open/close 8923 (time 1790895436.59 total 150.12 last 59.26)
 - open/close 9564 (time 1790895446.60 total 160.13 last 64.03)
total: 10000 open/close in 166.68 seconds: 59.99 ops/second
 - unlinked 0 (time 1790895458 ; total 0 ; last 0)
total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg458-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg458-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg458-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.158@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.158@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg458-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg458-server: oleg458-server.virtnet: executing set_default_debug -1 all
pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1
Started lustre-OST0000
oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg458-server: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg458-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg458-client.virtnet:  -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg458-client: oleg458-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg458-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg458-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg458-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.158@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.158@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.158@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.158@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.158@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg458-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg458-server: oleg458-server.virtnet: executing set_default_debug -1 all
pdsh@oleg458-client: oleg458-server: ssh exited with exit code 1
Started lustre-OST0000
oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg458-server: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg458-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg458-client.virtnet:  -o user_xattr,flock 192.168.204.158@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg458-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg458-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg458-server
stop mds service on oleg458-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg458-server
unloading modules via unload_modules_local on: 'oleg458-server'
oleg458-server: oleg458-server.virtnet: executing unload_modules_local
oleg458-server: kmemleak disabled
modules unloaded.
pdsh@oleg458-client: oleg458-server: ssh exited with exit code 2
pdsh@oleg458-client: oleg458-client: ssh exited with exit code 2
pdsh@oleg458-client: oleg458-server: ssh exited with exit code 2
pdsh@oleg458-client: oleg458-client: ssh exited with exit code 2