== conf-sanity test 69: replace an OST with the same index ========================================================== 13:58:26 (1790963906)
start mds service on oleg449-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1921056 Oct  2 13:44 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg449-server'
oleg449-server: oleg449-server.virtnet: executing load_modules_local
oleg449-server: kmemleak disabled
oleg449-server: Loading modules from /home/green/git/lustre-release/lustre
oleg449-server: detected 4 online CPUs by sysfs
oleg449-server: MODOPTS_LIBCFS=
oleg449-server: Force libcfs to create 2 CPU partitions
oleg449-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg449-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg449-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg449-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all
pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg449-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all
pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1
Started lustre-OST0000
oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg449-client.virtnet:  -o user_xattr,flock 192.168.204.149@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 756 (time 1790964008.70 total 10.02 last 75.47)
 - open/close 1419 (time 1790964018.70 total 20.02 last 66.28)
 - open/close 2102 (time 1790964028.71 total 30.03 last 68.24)
 - open/close 2729 (time 1790964038.72 total 40.03 last 62.68)
 - open/close 3446 (time 1790964048.72 total 50.04 last 71.64)
 - open/close 4195 (time 1790964058.73 total 60.05 last 74.83)
 - open/close 4812 (time 1790964068.74 total 70.06 last 61.67)
 - open/close 5471 (time 1790964078.74 total 80.06 last 65.88)
 - open/close 6059 (time 1790964088.75 total 90.07 last 58.75)
 - open/close 6721 (time 1790964098.75 total 100.07 last 66.17)
 - open/close 7391 (time 1790964108.76 total 110.07 last 66.98)
 - open/close 8068 (time 1790964118.76 total 120.08 last 67.65)
 - open/close 8731 (time 1790964128.77 total 130.08 last 66.29)
 - open/close 9396 (time 1790964138.77 total 140.08 last 66.50)
total: 10000 open/close in 148.90 seconds: 67.16 ops/second
 - unlinked 0 (time 1790964152 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 676 (time 1790964255.10 total 10.01 last 67.56)
 - open/close 1312 (time 1790964265.11 total 20.02 last 63.52)
 - open/close 1855 (time 1790964275.12 total 30.03 last 54.23)
 - open/close 2604 (time 1790964285.14 total 40.05 last 74.77)
 - open/close 3424 (time 1790964295.14 total 50.05 last 81.97)
 - open/close 4182 (time 1790964305.16 total 60.07 last 75.71)
 - open/close 4996 (time 1790964315.17 total 70.07 last 81.32)
 - open/close 5712 (time 1790964325.18 total 80.09 last 71.52)
 - open/close 6500 (time 1790964335.18 total 90.09 last 78.76)
 - open/close 7150 (time 1790964345.18 total 100.09 last 64.98)
 - open/close 7824 (time 1790964355.19 total 110.10 last 67.34)
 - open/close 8603 (time 1790964365.20 total 120.11 last 77.84)
 - open/close 9343 (time 1790964375.20 total 130.11 last 73.97)
total: 10000 open/close in 138.58 seconds: 72.16 ops/second
 - unlinked 0 (time 1790964387 ; total 0 ; last 0)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
 - open/close 716 (time 1790964486.95 total 10.00 last 71.56)
 - open/close 1414 (time 1790964496.95 total 20.01 last 69.80)
 - open/close 2124 (time 1790964506.97 total 30.02 last 70.89)
 - open/close 2726 (time 1790964516.97 total 40.02 last 60.19)
 - open/close 3418 (time 1790964526.98 total 50.03 last 69.13)
 - open/close 4243 (time 1790964536.99 total 60.04 last 82.40)
 - open/close 5063 (time 1790964547.00 total 70.05 last 81.98)
 - open/close 5844 (time 1790964557.01 total 80.06 last 78.02)
 - open/close 6646 (time 1790964567.02 total 90.07 last 80.11)
 - open/close 7402 (time 1790964577.03 total 100.08 last 75.54)
 - open/close 8151 (time 1790964587.03 total 110.08 last 74.90)
 - open/close 8917 (time 1790964597.04 total 120.09 last 76.53)
 - open/close 9736 (time 1790964607.04 total 130.09 last 81.86)
total: 10000 open/close in 133.42 seconds: 74.95 ops/second
 - unlinked 0 (time 1790964614 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
 - open/close 732 (time 1790964717.14 total 10.00 last 73.18)
 - open/close 1453 (time 1790964727.14 total 20.00 last 72.09)
 - open/close 2129 (time 1790964737.14 total 30.01 last 67.57)
 - open/close 2762 (time 1790964747.16 total 40.03 last 63.19)
 - open/close 3409 (time 1790964757.16 total 50.03 last 64.69)
 - open/close 4091 (time 1790964767.17 total 60.03 last 68.19)
 - open/close 4811 (time 1790964777.18 total 70.04 last 71.93)
 - open/close 5556 (time 1790964787.20 total 80.06 last 74.33)
 - open/close 6271 (time 1790964797.20 total 90.06 last 71.49)
 - open/close 7003 (time 1790964807.21 total 100.07 last 73.14)
 - open/close 7736 (time 1790964817.21 total 110.07 last 73.29)
 - open/close 8377 (time 1790964827.21 total 120.08 last 64.09)
 - open/close 9043 (time 1790964837.22 total 130.08 last 66.55)
 - open/close 9749 (time 1790964847.23 total 140.09 last 70.53)
total: 10000 open/close in 144.02 seconds: 69.43 ops/second
 - unlinked 0 (time 1790964858 ; total 1 ; last 1)
total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second
 - open/close 664 (time 1790964971.76 total 10.01 last 66.30)
 - open/close 1371 (time 1790964981.78 total 20.03 last 70.58)
 - open/close 1997 (time 1790964991.79 total 30.04 last 62.54)
 - open/close 2583 (time 1790965001.80 total 40.06 last 58.51)
 - open/close 3222 (time 1790965011.82 total 50.07 last 63.81)
 - open/close 3880 (time 1790965021.82 total 60.07 last 65.80)
 - open/close 4621 (time 1790965031.83 total 70.09 last 73.99)
 - open/close 5353 (time 1790965041.83 total 80.09 last 73.19)
 - open/close 5995 (time 1790965051.84 total 90.10 last 64.15)
 - open/close 6597 (time 1790965061.85 total 100.10 last 60.15)
 - open/close 7201 (time 1790965071.86 total 110.12 last 60.33)
 - open/close 7875 (time 1790965081.86 total 120.12 last 67.38)
 - open/close 8643 (time 1790965091.87 total 130.12 last 76.77)
 - open/close 9342 (time 1790965101.87 total 140.13 last 69.85)
total: 10000 open/close in 149.44 seconds: 66.92 ops/second
 - unlinked 0 (time 1790965116 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 725 (time 1790965216.10 total 10.01 last 72.46)
 - open/close 1478 (time 1790965226.11 total 20.02 last 75.22)
 - open/close 2215 (time 1790965236.11 total 30.02 last 73.65)
 - open/close 2905 (time 1790965246.12 total 40.03 last 68.95)
 - open/close 3518 (time 1790965256.13 total 50.04 last 61.22)
 - open/close 4177 (time 1790965266.14 total 60.05 last 65.85)
 - open/close 4916 (time 1790965276.14 total 70.05 last 73.88)
 - open/close 5701 (time 1790965286.16 total 80.07 last 78.40)
 - open/close 6426 (time 1790965296.16 total 90.08 last 72.43)
 - open/close 7152 (time 1790965306.17 total 100.08 last 72.58)
 - open/close 7788 (time 1790965316.17 total 110.08 last 63.59)
 - open/close 8545 (time 1790965326.18 total 120.09 last 75.62)
 - open/close 9216 (time 1790965336.18 total 130.09 last 67.07)
 - open/close 9908 (time 1790965346.22 total 140.13 last 68.96)
total: 10000 open/close in 141.33 seconds: 70.76 ops/second
 - unlinked 0 (time 1790965351 ; total 0 ; last 0)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
 - open/close 755 (time 1790965451.52 total 10.03 last 75.29)
 - open/close 1479 (time 1790965461.53 total 20.05 last 72.26)
 - open/close 2272 (time 1790965471.54 total 30.05 last 79.29)
 - open/close 2992 (time 1790965481.54 total 40.05 last 71.99)
 - open/close 3721 (time 1790965491.57 total 50.09 last 72.63)
 - open/close 4409 (time 1790965501.59 total 60.10 last 68.73)
 - open/close 5080 (time 1790965511.60 total 70.12 last 66.98)
 - open/close 5543 (time 1790965521.61 total 80.13 last 46.25)
 - open/close 6009 (time 1790965531.62 total 90.13 last 46.57)
 - open/close 6555 (time 1790965541.64 total 100.16 last 54.47)
 - open/close 7221 (time 1790965551.65 total 110.16 last 66.56)
 - open/close 8009 (time 1790965561.66 total 120.17 last 78.75)
 - open/close 8802 (time 1790965571.67 total 130.19 last 79.18)
 - open/close 9636 (time 1790965581.67 total 140.19 last 83.39)
total: 10000 open/close in 144.88 seconds: 69.02 ops/second
 - unlinked 0 (time 1790965590 ; total 0 ; last 0)
total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second
 - open/close 736 (time 1790965694.27 total 10.01 last 73.53)
 - open/close 1434 (time 1790965704.28 total 20.02 last 69.70)
 - open/close 2082 (time 1790965714.30 total 30.04 last 64.71)
 - open/close 2766 (time 1790965724.30 total 40.04 last 68.35)
 - open/close 3488 (time 1790965734.31 total 50.06 last 72.12)
 - open/close 4173 (time 1790965744.35 total 60.09 last 68.25)
 - open/close 4980 (time 1790965754.35 total 70.09 last 80.69)
 - open/close 5731 (time 1790965764.37 total 80.11 last 75.00)
 - open/close 6459 (time 1790965774.37 total 90.11 last 72.78)
 - open/close 7226 (time 1790965784.38 total 100.12 last 76.62)
 - open/close 7925 (time 1790965794.39 total 110.13 last 69.84)
 - open/close 8507 (time 1790965804.39 total 120.13 last 58.17)
 - open/close 9222 (time 1790965814.40 total 130.14 last 71.48)
 - open/close 9945 (time 1790965824.40 total 140.14 last 72.30)
total: 10000 open/close in 140.90 seconds: 70.97 ops/second
 - unlinked 0 (time 1790965830 ; total 0 ; last 0)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
 - open/close 800 (time 1790965930.22 total 10.00 last 79.99)
 - open/close 1512 (time 1790965940.22 total 20.00 last 71.19)
 - open/close 2264 (time 1790965950.23 total 30.01 last 75.15)
 - open/close 2974 (time 1790965960.24 total 40.02 last 70.94)
 - open/close 3686 (time 1790965970.25 total 50.02 last 71.16)
 - open/close 4515 (time 1790965980.26 total 60.04 last 82.80)
 - open/close 5260 (time 1790965990.26 total 70.04 last 74.49)
 - open/close 6018 (time 1790966000.27 total 80.05 last 75.70)
 - open/close 6745 (time 1790966010.30 total 90.08 last 72.49)
 - open/close 7460 (time 1790966020.30 total 100.08 last 71.48)
 - open/close 8239 (time 1790966030.31 total 110.09 last 77.85)
 - open/close 9001 (time 1790966040.32 total 120.10 last 76.15)
 - open/close 9787 (time 1790966050.32 total 130.10 last 78.56)
total: 10000 open/close in 133.06 seconds: 75.15 ops/second
 - unlinked 0 (time 1790966057 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 649 (time 1790966163.21 total 10.01 last 64.86)
 - open/close 1388 (time 1790966173.23 total 20.03 last 73.74)
 - open/close 2087 (time 1790966183.25 total 30.05 last 69.77)
 - open/close 2793 (time 1790966193.26 total 40.06 last 70.48)
 - open/close 3597 (time 1790966203.28 total 50.08 last 80.26)
 - open/close 4361 (time 1790966213.28 total 60.08 last 76.39)
 - open/close 5098 (time 1790966223.28 total 70.08 last 73.70)
 - open/close 5830 (time 1790966233.29 total 80.09 last 73.14)
 - open/close 6620 (time 1790966243.30 total 90.10 last 78.90)
 - open/close 7394 (time 1790966253.31 total 100.11 last 77.37)
 - open/close 8070 (time 1790966263.32 total 110.12 last 67.53)
 - open/close 8777 (time 1790966273.34 total 120.14 last 70.57)
 - open/close 9328 (time 1790966283.35 total 130.15 last 55.03)
 - open/close 9946 (time 1790966293.35 total 140.15 last 61.78)
total: 10000 open/close in 141.26 seconds: 70.79 ops/second
 - unlinked 0 (time 1790966299 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg449-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg449-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg449-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.149@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.149@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg449-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all
pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1
Started lustre-OST0000
oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg449-server: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg449-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg449-client.virtnet:  -o user_xattr,flock 192.168.204.149@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg449-client: oleg449-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg449-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg449-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg449-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.149@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.149@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.149@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.149@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.149@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg449-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all
pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1
Started lustre-OST0000
oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg449-server: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg449-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg449-client.virtnet:  -o user_xattr,flock 192.168.204.149@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg449-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg449-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg449-server
stop mds service on oleg449-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg449-server
unloading modules via unload_modules_local on: 'oleg449-server'
oleg449-server: oleg449-server.virtnet: executing unload_modules_local
oleg449-server: kmemleak disabled
modules unloaded.
pdsh@oleg449-client: oleg449-client: ssh exited with exit code 2
pdsh@oleg449-client: oleg449-server: ssh exited with exit code 2
pdsh@oleg449-client: oleg449-client: ssh exited with exit code 2
pdsh@oleg449-client: oleg449-server: ssh exited with exit code 2