== conf-sanity test 69: replace an OST with the same index ========================================================== 21:19:23 (1789607963)
start mds service on oleg313-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1931640 Sep 16 13:21 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg313-server'
oleg313-server: oleg313-server.virtnet: executing load_modules_local
oleg313-server: kmemleak disabled
oleg313-server: Loading modules from /home/green/git/lustre-release/lustre
oleg313-server: detected 4 online CPUs by sysfs
oleg313-server: MODOPTS_LIBCFS=
oleg313-server: Force libcfs to create 2 CPU partitions
oleg313-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg313-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg313-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg313-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all
pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg313-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all
pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1
Started lustre-OST0000
oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg313-client.virtnet:  -o user_xattr,flock 192.168.203.113@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 661 (time 1789608051.41 total 10.01 last 66.05)
 - open/close 1287 (time 1789608061.42 total 20.01 last 62.56)
 - open/close 1914 (time 1789608071.43 total 30.03 last 62.61)
 - open/close 2531 (time 1789608081.45 total 40.04 last 61.63)
 - open/close 3195 (time 1789608091.46 total 50.05 last 66.31)
 - open/close 3865 (time 1789608101.47 total 60.06 last 66.95)
 - open/close 4528 (time 1789608111.47 total 70.07 last 66.26)
 - open/close 5181 (time 1789608121.47 total 80.07 last 65.29)
 - open/close 5808 (time 1789608131.48 total 90.07 last 62.67)
 - open/close 6508 (time 1789608141.49 total 100.08 last 69.94)
 - open/close 7228 (time 1789608151.50 total 110.09 last 71.92)
 - open/close 7866 (time 1789608161.51 total 120.11 last 63.72)
 - open/close 8326 (time 1789608171.51 total 130.11 last 45.98)
 - open/close 8725 (time 1789608181.56 total 140.15 last 39.74)
 - open/close 9244 (time 1789608191.58 total 150.18 last 51.76)
 - open/close 9724 (time 1789608201.61 total 160.21 last 47.87)
total: 10000 open/close in 165.27 seconds: 60.51 ops/second
 - unlinked 0 (time 1789608212 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 617 (time 1789608323.77 total 10.01 last 61.65)
 - open/close 1200 (time 1789608333.78 total 20.02 last 58.25)
 - open/close 1755 (time 1789608343.82 total 30.05 last 55.29)
 - open/close 2308 (time 1789608353.82 total 40.06 last 55.28)
 - open/close 2940 (time 1789608363.83 total 50.07 last 63.15)
 - open/close 3552 (time 1789608373.84 total 60.07 last 61.16)
 - open/close 4203 (time 1789608383.85 total 70.08 last 65.03)
 - open/close 4886 (time 1789608393.86 total 80.10 last 68.21)
 - open/close 5544 (time 1789608403.87 total 90.11 last 65.73)
 - open/close 6153 (time 1789608413.88 total 100.12 last 60.84)
 - open/close 6807 (time 1789608423.89 total 110.13 last 65.34)
 - open/close 7399 (time 1789608433.90 total 120.14 last 59.12)
 - open/close 8045 (time 1789608443.91 total 130.15 last 64.52)
 - open/close 8709 (time 1789608453.92 total 140.16 last 66.33)
 - open/close 9388 (time 1789608463.93 total 150.17 last 67.86)
total: 10000 open/close in 158.49 seconds: 63.09 ops/second
 - unlinked 0 (time 1789608476 ; total 0 ; last 0)
total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second
 - open/close 593 (time 1789608585.28 total 10.00 last 59.28)
 - open/close 1153 (time 1789608595.29 total 20.01 last 55.97)
 - open/close 1739 (time 1789608605.30 total 30.02 last 58.52)
 - open/close 2279 (time 1789608615.31 total 40.03 last 53.95)
 - open/close 2975 (time 1789608625.33 total 50.05 last 69.49)
 - open/close 3648 (time 1789608635.34 total 60.06 last 67.21)
 - open/close 4312 (time 1789608645.34 total 70.06 last 66.38)
 - open/close 4866 (time 1789608655.35 total 80.07 last 55.38)
 - open/close 5525 (time 1789608665.35 total 90.07 last 65.88)
 - open/close 6126 (time 1789608675.36 total 100.08 last 60.07)
 - open/close 6702 (time 1789608685.36 total 110.08 last 57.58)
 - open/close 7377 (time 1789608695.37 total 120.09 last 67.46)
 - open/close 8062 (time 1789608705.37 total 130.09 last 68.47)
 - open/close 8805 (time 1789608715.38 total 140.10 last 74.23)
 - open/close 9485 (time 1789608725.39 total 150.11 last 67.92)
total: 10000 open/close in 157.40 seconds: 63.53 ops/second
 - unlinked 0 (time 1789608737 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 647 (time 1789608844.10 total 10.00 last 64.69)
 - open/close 1260 (time 1789608854.12 total 20.01 last 61.23)
 - open/close 1853 (time 1789608864.12 total 30.02 last 59.29)
 - open/close 2527 (time 1789608874.12 total 40.02 last 67.38)
 - open/close 3210 (time 1789608884.13 total 50.03 last 68.25)
 - open/close 3844 (time 1789608894.13 total 60.03 last 63.37)
 - open/close 4488 (time 1789608904.13 total 70.03 last 64.38)
 - open/close 5166 (time 1789608914.14 total 80.04 last 67.75)
 - open/close 5881 (time 1789608924.15 total 90.05 last 71.41)
 - open/close 6538 (time 1789608934.16 total 100.05 last 65.69)
 - open/close 7246 (time 1789608944.17 total 110.06 last 70.73)
 - open/close 7834 (time 1789608954.18 total 120.08 last 58.74)
 - open/close 8496 (time 1789608964.19 total 130.09 last 66.10)
 - open/close 9041 (time 1789608974.23 total 140.12 last 54.31)
 - open/close 9523 (time 1789608984.23 total 150.13 last 48.18)
total: 10000 open/close in 157.92 seconds: 63.32 ops/second
 - unlinked 0 (time 1789608996 ; total 0 ; last 0)
total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second
 - open/close 647 (time 1789609105.74 total 10.01 last 64.64)
 - open/close 1333 (time 1789609115.74 total 20.01 last 68.59)
 - open/close 1998 (time 1789609125.75 total 30.02 last 66.43)
 - open/close 2684 (time 1789609135.75 total 40.02 last 68.59)
 - open/close 3377 (time 1789609145.75 total 50.03 last 69.29)
 - open/close 4055 (time 1789609155.77 total 60.04 last 67.71)
 - open/close 4765 (time 1789609165.77 total 70.05 last 70.95)
 - open/close 5441 (time 1789609175.78 total 80.05 last 67.56)
 - open/close 6087 (time 1789609185.79 total 90.06 last 64.52)
 - open/close 6725 (time 1789609195.80 total 100.07 last 63.76)
 - open/close 7371 (time 1789609205.80 total 110.08 last 64.57)
 - open/close 8029 (time 1789609215.80 total 120.08 last 65.79)
 - open/close 8696 (time 1789609225.80 total 130.08 last 66.69)
 - open/close 9384 (time 1789609235.81 total 140.08 last 68.77)
total: 10000 open/close in 149.50 seconds: 66.89 ops/second
 - unlinked 0 (time 1789609250 ; total 0 ; last 0)
total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second
 - open/close 704 (time 1789609368.68 total 10.02 last 70.28)
 - open/close 1282 (time 1789609378.70 total 20.03 last 57.73)
 - open/close 1870 (time 1789609388.70 total 30.03 last 58.77)
 - open/close 2494 (time 1789609398.70 total 40.04 last 62.39)
 - open/close 3170 (time 1789609408.71 total 50.04 last 67.55)
 - open/close 3873 (time 1789609418.72 total 60.05 last 70.26)
 - open/close 4578 (time 1789609428.72 total 70.06 last 70.46)
 - open/close 5225 (time 1789609438.73 total 80.07 last 64.63)
 - open/close 5870 (time 1789609448.76 total 90.10 last 64.29)
 - open/close 6571 (time 1789609458.77 total 100.10 last 70.06)
 - open/close 7136 (time 1789609468.78 total 110.12 last 56.41)
 - open/close 7763 (time 1789609478.80 total 120.13 last 62.63)
 - open/close 8464 (time 1789609488.80 total 130.13 last 70.09)
 - open/close 9122 (time 1789609498.81 total 140.14 last 65.73)
 - open/close 9810 (time 1789609508.82 total 150.15 last 68.72)
total: 10000 open/close in 153.13 seconds: 65.31 ops/second
 - unlinked 0 (time 1789609515 ; total 0 ; last 0)
total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second
 - open/close 647 (time 1789609633.58 total 10.00 last 64.67)
 - open/close 1338 (time 1789609643.59 total 20.02 last 69.00)
 - open/close 2003 (time 1789609653.60 total 30.02 last 66.47)
 - open/close 2645 (time 1789609663.61 total 40.03 last 64.16)
 - open/close 3284 (time 1789609673.61 total 50.04 last 63.86)
 - open/close 3899 (time 1789609683.62 total 60.04 last 61.47)
 - open/close 4528 (time 1789609693.62 total 70.05 last 62.86)
 - open/close 5160 (time 1789609703.63 total 80.05 last 63.16)
 - open/close 5791 (time 1789609713.63 total 90.06 last 63.09)
 - open/close 6460 (time 1789609723.63 total 100.06 last 66.87)
 - open/close 7151 (time 1789609733.65 total 110.07 last 69.00)
 - open/close 7763 (time 1789609743.65 total 120.08 last 61.17)
 - open/close 8402 (time 1789609753.67 total 130.09 last 63.83)
 - open/close 9012 (time 1789609763.67 total 140.09 last 61.00)
 - open/close 9682 (time 1789609773.67 total 150.10 last 66.97)
total: 10000 open/close in 155.51 seconds: 64.30 ops/second
 - unlinked 0 (time 1789609783 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 646 (time 1789609898.54 total 10.02 last 64.47)
 - open/close 1232 (time 1789609908.55 total 20.03 last 58.52)
 - open/close 1769 (time 1789609918.55 total 30.03 last 53.70)
 - open/close 2333 (time 1789609928.57 total 40.05 last 56.29)
 - open/close 2991 (time 1789609938.58 total 50.06 last 65.78)
 - open/close 3684 (time 1789609948.58 total 60.06 last 69.28)
 - open/close 4392 (time 1789609958.58 total 70.06 last 70.79)
 - open/close 5055 (time 1789609968.58 total 80.06 last 66.27)
 - open/close 5657 (time 1789609978.59 total 90.07 last 60.14)
 - open/close 6292 (time 1789609988.60 total 100.08 last 63.49)
 - open/close 6942 (time 1789609998.61 total 110.09 last 64.91)
 - open/close 7531 (time 1789610008.62 total 120.10 last 58.84)
 - open/close 8093 (time 1789610018.64 total 130.12 last 56.10)
 - open/close 8706 (time 1789610028.64 total 140.12 last 61.25)
 - open/close 9283 (time 1789610038.65 total 150.13 last 57.66)
 - open/close 9960 (time 1789610048.66 total 160.14 last 67.64)
total: 10000 open/close in 160.72 seconds: 62.22 ops/second
 - unlinked 0 (time 1789610054 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 580 (time 1789610170.86 total 10.00 last 58.00)
 - open/close 1320 (time 1789610180.87 total 20.02 last 73.89)
 - open/close 1965 (time 1789610190.88 total 30.02 last 64.49)
 - open/close 2653 (time 1789610200.89 total 40.03 last 68.72)
 - open/close 3305 (time 1789610210.89 total 50.03 last 65.20)
 - open/close 3978 (time 1789610220.90 total 60.04 last 67.21)
 - open/close 4627 (time 1789610230.92 total 70.06 last 64.79)
 - open/close 5278 (time 1789610240.93 total 80.07 last 65.00)
 - open/close 5923 (time 1789610250.95 total 90.10 last 64.35)
 - open/close 6566 (time 1789610260.96 total 100.10 last 64.27)
 - open/close 7221 (time 1789610270.96 total 110.11 last 65.47)
 - open/close 7884 (time 1789610280.98 total 120.12 last 66.20)
 - open/close 8478 (time 1789610290.98 total 130.12 last 59.39)
 - open/close 9150 (time 1789610300.99 total 140.14 last 67.12)
 - open/close 9783 (time 1789610311.00 total 150.15 last 63.24)
total: 10000 open/close in 153.30 seconds: 65.23 ops/second
 - unlinked 0 (time 1789610318 ; total 0 ; last 0)
total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second
 - open/close 647 (time 1789610429.07 total 10.01 last 64.67)
 - open/close 1319 (time 1789610439.08 total 20.02 last 67.13)
 - open/close 2046 (time 1789610449.09 total 30.02 last 72.68)
 - open/close 2764 (time 1789610459.09 total 40.02 last 71.77)
 - open/close 3380 (time 1789610469.10 total 50.03 last 61.55)
 - open/close 4002 (time 1789610479.11 total 60.04 last 62.12)
 - open/close 4638 (time 1789610489.12 total 70.05 last 63.56)
 - open/close 5292 (time 1789610499.13 total 80.06 last 65.31)
 - open/close 5954 (time 1789610509.14 total 90.07 last 66.15)
 - open/close 6599 (time 1789610519.15 total 100.08 last 64.44)
 - open/close 7234 (time 1789610529.15 total 110.09 last 63.45)
 - open/close 7812 (time 1789610539.16 total 120.09 last 57.79)
 - open/close 8508 (time 1789610549.17 total 130.10 last 69.52)
 - open/close 9139 (time 1789610559.17 total 140.10 last 63.09)
 - open/close 9782 (time 1789610569.19 total 150.13 last 64.14)
total: 10000 open/close in 153.41 seconds: 65.18 ops/second
 - unlinked 0 (time 1789610577 ; total 0 ; last 0)
total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg313-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg313-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg313-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.113@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.113@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg313-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all
pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1
Started lustre-OST0000
oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg313-server: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg313-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg313-client.virtnet:  -o user_xattr,flock 192.168.203.113@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg313-client: oleg313-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg313-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg313-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg313-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.113@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.113@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.113@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.113@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.113@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg313-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all
pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1
Started lustre-OST0000
oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg313-server: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg313-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg313-client.virtnet:  -o user_xattr,flock 192.168.203.113@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg313-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg313-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg313-server
stop mds service on oleg313-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg313-server
unloading modules via unload_modules_local on: 'oleg313-server'
oleg313-server: oleg313-server.virtnet: executing unload_modules_local
oleg313-server: kmemleak disabled
modules unloaded.
pdsh@oleg313-client: oleg313-client: ssh exited with exit code 2
pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2
pdsh@oleg313-client: oleg313-client: ssh exited with exit code 2
pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2