== conf-sanity test 69: replace an OST with the same index ========================================================== 21:33:03 (1788744783)
start mds service on oleg250-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1931560 Sep  6 20:25 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg250-server'
oleg250-server: oleg250-server.virtnet: executing load_modules_local
oleg250-server: kmemleak disabled
oleg250-server: Loading modules from /home/green/git/lustre-release/lustre
oleg250-server: detected 4 online CPUs by sysfs
oleg250-server: MODOPTS_LIBCFS=
oleg250-server: Force libcfs to create 2 CPU partitions
oleg250-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg250-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg250-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg250-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all
pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg250-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all
pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1
Started lustre-OST0000
oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg250-client.virtnet:  -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 636 (time 1788744878.04 total 10.01 last 63.55)
 - open/close 1314 (time 1788744888.06 total 20.03 last 67.67)
 - open/close 1919 (time 1788744898.07 total 30.03 last 60.47)
 - open/close 2491 (time 1788744908.07 total 40.04 last 57.16)
 - open/close 3164 (time 1788744918.10 total 50.06 last 67.13)
 - open/close 3758 (time 1788744928.11 total 60.07 last 59.34)
 - open/close 4482 (time 1788744938.11 total 70.08 last 72.37)
 - open/close 5209 (time 1788744948.11 total 80.08 last 72.68)
 - open/close 5936 (time 1788744958.12 total 90.08 last 72.70)
 - open/close 6597 (time 1788744968.12 total 100.08 last 66.08)
 - open/close 7194 (time 1788744978.14 total 110.10 last 59.59)
 - open/close 7731 (time 1788744988.15 total 120.11 last 53.66)
 - open/close 8161 (time 1788744998.16 total 130.13 last 42.93)
 - open/close 8620 (time 1788745008.18 total 140.15 last 45.80)
 - open/close 9228 (time 1788745018.19 total 150.16 last 60.75)
 - open/close 9824 (time 1788745028.21 total 160.18 last 59.47)
total: 10000 open/close in 162.08 seconds: 61.70 ops/second
 - unlinked 0 (time 1788745035 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 697 (time 1788745143.17 total 10.01 last 69.63)
 - open/close 1399 (time 1788745153.18 total 20.02 last 70.16)
 - open/close 2080 (time 1788745163.20 total 30.04 last 67.93)
 - open/close 2705 (time 1788745173.22 total 40.06 last 62.40)
 - open/close 3301 (time 1788745183.24 total 50.08 last 59.48)
 - open/close 4065 (time 1788745193.24 total 60.08 last 76.37)
 - open/close 4761 (time 1788745203.26 total 70.10 last 69.49)
 - open/close 5494 (time 1788745213.26 total 80.10 last 73.27)
 - open/close 6072 (time 1788745223.27 total 90.11 last 57.75)
 - open/close 6666 (time 1788745233.29 total 100.12 last 59.32)
 - open/close 7326 (time 1788745243.30 total 110.13 last 65.93)
 - open/close 8045 (time 1788745253.30 total 120.14 last 71.85)
 - open/close 8632 (time 1788745263.32 total 130.16 last 58.61)
 - open/close 9211 (time 1788745273.33 total 140.16 last 57.86)
 - open/close 9906 (time 1788745283.34 total 150.17 last 69.43)
total: 10000 open/close in 151.27 seconds: 66.11 ops/second
 - unlinked 0 (time 1788745288 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 632 (time 1788745387.09 total 10.01 last 63.11)
 - open/close 1273 (time 1788745397.10 total 20.02 last 64.04)
 - open/close 2014 (time 1788745407.12 total 30.04 last 74.00)
 - open/close 2639 (time 1788745417.13 total 40.05 last 62.43)
 - open/close 3263 (time 1788745427.13 total 50.06 last 62.36)
 - open/close 3780 (time 1788745437.15 total 60.07 last 51.61)
 - open/close 4301 (time 1788745447.19 total 70.11 last 51.90)
 - open/close 4930 (time 1788745457.20 total 80.12 last 62.86)
 - open/close 5679 (time 1788745467.20 total 90.12 last 74.88)
 - open/close 6269 (time 1788745477.22 total 100.14 last 58.88)
 - open/close 6846 (time 1788745487.24 total 110.17 last 57.57)
 - open/close 7366 (time 1788745497.26 total 120.18 last 51.93)
 - open/close 7922 (time 1788745507.27 total 130.19 last 55.55)
 - open/close 8543 (time 1788745517.27 total 140.19 last 62.09)
 - open/close 9144 (time 1788745527.28 total 150.20 last 60.06)
 - open/close 9803 (time 1788745537.28 total 160.20 last 65.89)
total: 10000 open/close in 163.36 seconds: 61.22 ops/second
 - unlinked 0 (time 1788745545 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 753 (time 1788745657.71 total 10.02 last 75.18)
 - open/close 1554 (time 1788745667.71 total 20.02 last 80.09)
 - open/close 2353 (time 1788745677.71 total 30.02 last 79.86)
 - open/close 3096 (time 1788745687.72 total 40.02 last 74.28)
 - open/close 3725 (time 1788745697.72 total 50.03 last 62.89)
 - open/close 4485 (time 1788745707.72 total 60.03 last 75.96)
 - open/close 5113 (time 1788745717.72 total 70.03 last 62.79)
 - open/close 5858 (time 1788745727.73 total 80.04 last 74.45)
 - open/close 6549 (time 1788745737.74 total 90.05 last 69.06)
 - open/close 7218 (time 1788745747.74 total 100.05 last 66.87)
 - open/close 7961 (time 1788745757.75 total 110.05 last 74.27)
 - open/close 8638 (time 1788745767.76 total 120.07 last 67.57)
 - open/close 9280 (time 1788745777.78 total 130.08 last 64.12)
 - open/close 9958 (time 1788745787.80 total 140.11 last 67.65)
total: 10000 open/close in 140.75 seconds: 71.05 ops/second
 - unlinked 0 (time 1788745793 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 517 (time 1788745908.00 total 10.02 last 51.62)
 - open/close 1110 (time 1788745918.02 total 20.03 last 59.19)
 - open/close 1726 (time 1788745928.03 total 30.04 last 61.54)
 - open/close 2477 (time 1788745938.03 total 40.05 last 75.05)
 - open/close 3129 (time 1788745948.04 total 50.06 last 65.16)
 - open/close 3821 (time 1788745958.04 total 60.06 last 69.16)
 - open/close 4577 (time 1788745968.05 total 70.07 last 75.55)
 - open/close 5173 (time 1788745978.05 total 80.07 last 59.58)
 - open/close 5806 (time 1788745988.06 total 90.08 last 63.26)
 - open/close 6534 (time 1788745998.07 total 100.08 last 72.75)
 - open/close 7209 (time 1788746008.08 total 110.09 last 67.44)
 - open/close 7826 (time 1788746018.09 total 120.11 last 61.59)
 - open/close 8539 (time 1788746028.11 total 130.12 last 71.22)
 - open/close 9186 (time 1788746038.11 total 140.13 last 64.67)
 - open/close 9882 (time 1788746048.12 total 150.14 last 69.53)
total: 10000 open/close in 151.90 seconds: 65.83 ops/second
 - unlinked 0 (time 1788746054 ; total 0 ; last 0)
total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second
 - open/close 652 (time 1788746157.79 total 10.00 last 65.20)
 - open/close 1300 (time 1788746167.80 total 20.01 last 64.73)
 - open/close 1981 (time 1788746177.81 total 30.02 last 68.04)
 - open/close 2711 (time 1788746187.82 total 40.03 last 72.92)
 - open/close 3443 (time 1788746197.82 total 50.03 last 73.18)
 - open/close 4051 (time 1788746207.83 total 60.04 last 60.75)
 - open/close 4798 (time 1788746217.83 total 70.05 last 74.67)
 - open/close 5501 (time 1788746227.86 total 80.07 last 70.13)
 - open/close 6173 (time 1788746237.87 total 90.08 last 67.13)
 - open/close 6856 (time 1788746247.88 total 100.09 last 68.23)
 - open/close 7582 (time 1788746257.90 total 110.11 last 72.48)
 - open/close 8334 (time 1788746267.90 total 120.11 last 75.17)
 - open/close 9063 (time 1788746277.90 total 130.12 last 72.87)
 - open/close 9902 (time 1788746287.91 total 140.12 last 83.83)
total: 10000 open/close in 141.63 seconds: 70.61 ops/second
 - unlinked 0 (time 1788746294 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 730 (time 1788746407.51 total 10.01 last 72.90)
 - open/close 1477 (time 1788746417.53 total 20.04 last 74.54)
 - open/close 2175 (time 1788746427.53 total 30.04 last 69.79)
 - open/close 2934 (time 1788746437.54 total 40.04 last 75.88)
 - open/close 3752 (time 1788746447.54 total 50.04 last 81.78)
 - open/close 4494 (time 1788746457.54 total 60.04 last 74.19)
 - open/close 5124 (time 1788746467.54 total 70.05 last 62.98)
 - open/close 5793 (time 1788746477.55 total 80.05 last 66.84)
 - open/close 6489 (time 1788746487.56 total 90.06 last 69.54)
 - open/close 7349 (time 1788746497.57 total 100.07 last 85.95)
 - open/close 8051 (time 1788746507.57 total 110.07 last 70.17)
 - open/close 8722 (time 1788746517.58 total 120.08 last 67.05)
 - open/close 9394 (time 1788746527.58 total 130.09 last 67.17)
total: 10000 open/close in 138.80 seconds: 72.05 ops/second
 - unlinked 0 (time 1788746540 ; total 0 ; last 0)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
 - open/close 654 (time 1788746640.15 total 10.01 last 65.36)
 - open/close 1320 (time 1788746650.16 total 20.01 last 66.54)
 - open/close 2046 (time 1788746660.16 total 30.02 last 72.54)
 - open/close 2876 (time 1788746670.17 total 40.03 last 82.94)
 - open/close 3644 (time 1788746680.18 total 50.04 last 76.74)
 - open/close 4362 (time 1788746690.18 total 60.04 last 71.78)
 - open/close 5134 (time 1788746700.19 total 70.05 last 77.14)
 - open/close 5902 (time 1788746710.20 total 80.06 last 76.73)
 - open/close 6706 (time 1788746720.20 total 90.06 last 80.39)
 - open/close 7495 (time 1788746730.20 total 100.06 last 78.86)
 - open/close 8320 (time 1788746740.21 total 110.07 last 82.48)
 - open/close 9128 (time 1788746750.21 total 120.07 last 80.78)
 - open/close 9839 (time 1788746760.21 total 130.07 last 71.09)
total: 10000 open/close in 132.17 seconds: 75.66 ops/second
 - unlinked 0 (time 1788746767 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 837 (time 1788746863.02 total 10.01 last 83.62)
 - open/close 1712 (time 1788746873.02 total 20.02 last 87.44)
 - open/close 2609 (time 1788746883.03 total 30.02 last 89.66)
 - open/close 3338 (time 1788746893.04 total 40.03 last 72.84)
 - open/close 4150 (time 1788746903.05 total 50.04 last 81.12)
 - open/close 5014 (time 1788746913.05 total 60.05 last 86.33)
 - open/close 5820 (time 1788746923.06 total 70.05 last 80.59)
 - open/close 6616 (time 1788746933.06 total 80.05 last 79.56)
 - open/close 7473 (time 1788746943.06 total 90.06 last 85.68)
 - open/close 8265 (time 1788746953.07 total 100.06 last 79.17)
 - open/close 9113 (time 1788746963.08 total 110.07 last 84.70)
total: 10000 open/close in 119.60 seconds: 83.61 ops/second
 - unlinked 0 (time 1788746976 ; total 0 ; last 0)
total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second
 - open/close 843 (time 1788747063.13 total 10.01 last 84.20)
 - open/close 1653 (time 1788747073.13 total 20.01 last 80.99)
 - open/close 2368 (time 1788747083.16 total 30.04 last 71.34)
 - open/close 3244 (time 1788747093.16 total 40.04 last 87.58)
 - open/close 4113 (time 1788747103.16 total 50.04 last 86.90)
 - open/close 5000 (time 1788747113.17 total 60.05 last 88.62)
 - open/close 5932 (time 1788747123.17 total 70.05 last 93.20)
 - open/close 6829 (time 1788747133.17 total 80.05 last 89.67)
 - open/close 7619 (time 1788747143.18 total 90.06 last 78.96)
 - open/close 8523 (time 1788747153.18 total 100.06 last 90.33)
 - open/close 9362 (time 1788747163.19 total 110.07 last 83.85)
total: 10000 open/close in 118.01 seconds: 84.74 ops/second
 - unlinked 0 (time 1788747174 ; total 0 ; last 0)
total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg250-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg250-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.150@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.150@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg250-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all
pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1
Started lustre-OST0000
oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg250-server: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg250-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg250-client.virtnet:  -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg250-client: oleg250-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg250-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg250-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.150@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.150@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.150@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.150@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.202.150@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg250-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all
pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1
Started lustre-OST0000
oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg250-server: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg250-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg250-client.virtnet:  -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg250-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg250-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server
stop mds service on oleg250-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg250-server
unloading modules via unload_modules_local on: 'oleg250-server'
oleg250-server: oleg250-server.virtnet: executing unload_modules_local
oleg250-server: kmemleak disabled
modules unloaded.
pdsh@oleg250-client: oleg250-client: ssh exited with exit code 2
pdsh@oleg250-client: oleg250-server: ssh exited with exit code 2
pdsh@oleg250-client: oleg250-client: ssh exited with exit code 2
pdsh@oleg250-client: oleg250-server: ssh exited with exit code 2