== conf-sanity test 69: replace an OST with the same index ========================================================== 11:13:20 (1788534800)
start mds service on oleg424-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
-rwxr-xr-x. 1 green green 1923184 Sep  3 20:45 /home/green/git/lustre-release/lustre/utils/mount.lustre
-rwxr-xr-x. 1 root  root       12 Jan 31  2019 /usr/sbin/mount.lustre
mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre
loading modules on: 'oleg424-server'
oleg424-server: oleg424-server.virtnet: executing load_modules_local
oleg424-server: kmemleak disabled
oleg424-server: Loading modules from /home/green/git/lustre-release/lustre
oleg424-server: detected 4 online CPUs by sysfs
oleg424-server: MODOPTS_LIBCFS=
oleg424-server: Force libcfs to create 2 CPU partitions
oleg424-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg424-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg424-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
oleg424-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all
pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg424-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all
pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1
Started lustre-OST0000
oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg424-client.virtnet:  -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 560 (time 1788534891.65 total 10.01 last 55.95)
 - open/close 1160 (time 1788534901.66 total 20.03 last 59.90)
 - open/close 1729 (time 1788534911.67 total 30.03 last 56.88)
 - open/close 2204 (time 1788534921.69 total 40.05 last 47.41)
 - open/close 2597 (time 1788534931.69 total 50.05 last 39.29)
 - open/close 3041 (time 1788534941.69 total 60.06 last 44.39)
 - open/close 3417 (time 1788534951.71 total 70.07 last 37.54)
 - open/close 3937 (time 1788534961.72 total 80.08 last 51.94)
 - open/close 4469 (time 1788534971.73 total 90.09 last 53.16)
 - open/close 5038 (time 1788534981.74 total 100.10 last 56.85)
 - open/close 5698 (time 1788534991.74 total 110.10 last 66.00)
 - open/close 6255 (time 1788535001.76 total 120.13 last 55.55)
 - open/close 6848 (time 1788535011.77 total 130.14 last 59.25)
 - open/close 7417 (time 1788535021.77 total 140.14 last 56.90)
 - open/close 7980 (time 1788535031.78 total 150.14 last 56.26)
 - open/close 8525 (time 1788535041.78 total 160.15 last 54.49)
 - open/close 9075 (time 1788535051.78 total 170.15 last 54.99)
 - open/close 9563 (time 1788535061.79 total 180.15 last 48.79)
total: 10000 open/close in 186.94 seconds: 53.49 ops/second
 - unlinked 0 (time 1788535073 ; total 0 ; last 0)
total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second
 - open/close 594 (time 1788535196.00 total 10.00 last 59.40)
 - open/close 1222 (time 1788535206.02 total 20.02 last 62.70)
 - open/close 1835 (time 1788535216.03 total 30.03 last 61.21)
 - open/close 2309 (time 1788535226.04 total 40.04 last 47.35)
 - open/close 2773 (time 1788535236.05 total 50.05 last 46.38)
 - open/close 3204 (time 1788535246.05 total 60.05 last 43.09)
 - open/close 3776 (time 1788535256.05 total 70.05 last 57.19)
 - open/close 4358 (time 1788535266.06 total 80.05 last 58.18)
 - open/close 4895 (time 1788535276.06 total 90.06 last 53.67)
 - open/close 5455 (time 1788535286.07 total 100.07 last 55.93)
 - open/close 6036 (time 1788535296.08 total 110.08 last 58.05)
 - open/close 6569 (time 1788535306.09 total 120.09 last 53.26)
 - open/close 7006 (time 1788535316.09 total 130.09 last 43.68)
 - open/close 7646 (time 1788535326.11 total 140.11 last 63.89)
 - open/close 8251 (time 1788535336.12 total 150.12 last 60.44)
 - open/close 8765 (time 1788535346.13 total 160.13 last 51.34)
 - open/close 9373 (time 1788535356.15 total 170.14 last 60.73)
 - open/close 9950 (time 1788535366.18 total 180.17 last 57.52)
total: 10000 open/close in 181.21 seconds: 55.18 ops/second
 - unlinked 0 (time 1788535372 ; total 0 ; last 0)
total: 10000 unlinks in 119 seconds: 84.033615 unlinks/second
 - open/close 577 (time 1788535508.00 total 10.01 last 57.62)
 - open/close 1154 (time 1788535518.01 total 20.02 last 57.66)
 - open/close 1637 (time 1788535528.02 total 30.04 last 48.23)
 - open/close 2133 (time 1788535538.04 total 40.05 last 49.50)
 - open/close 2663 (time 1788535548.04 total 50.06 last 52.99)
 - open/close 3147 (time 1788535558.07 total 60.08 last 48.30)
 - open/close 3696 (time 1788535568.11 total 70.12 last 54.68)
 - open/close 4151 (time 1788535578.13 total 80.15 last 45.38)
 - open/close 4637 (time 1788535588.14 total 90.15 last 48.56)
 - open/close 5152 (time 1788535598.17 total 100.18 last 51.37)
 - open/close 5610 (time 1788535608.18 total 110.20 last 45.73)
 - open/close 6076 (time 1788535618.19 total 120.20 last 46.58)
 - open/close 6544 (time 1788535628.22 total 130.23 last 46.64)
 - open/close 7055 (time 1788535638.23 total 140.24 last 51.06)
 - open/close 7571 (time 1788535648.25 total 150.26 last 51.49)
 - open/close 8044 (time 1788535658.25 total 160.26 last 47.29)
 - open/close 8593 (time 1788535668.27 total 170.28 last 54.79)
 - open/close 9091 (time 1788535678.28 total 180.29 last 49.75)
 - open/close 9615 (time 1788535688.29 total 190.30 last 52.34)
total: 10000 open/close in 197.55 seconds: 50.62 ops/second
 - unlinked 0 (time 1788535700 ; total 0 ; last 0)
total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second
 - open/close 657 (time 1788535820.06 total 10.00 last 65.67)
 - open/close 1335 (time 1788535830.07 total 20.01 last 67.75)
 - open/close 1968 (time 1788535840.07 total 30.02 last 63.26)
 - open/close 2542 (time 1788535850.08 total 40.02 last 57.36)
 - open/close 3113 (time 1788535860.08 total 50.03 last 57.07)
 - open/close 3754 (time 1788535870.09 total 60.03 last 64.09)
 - open/close 4420 (time 1788535880.09 total 70.03 last 66.60)
 - open/close 5017 (time 1788535890.09 total 80.04 last 59.66)
 - open/close 5493 (time 1788535900.11 total 90.05 last 47.53)
 - open/close 6046 (time 1788535910.12 total 100.06 last 55.25)
 - open/close 6607 (time 1788535920.12 total 110.07 last 56.07)
 - open/close 7209 (time 1788535930.14 total 120.08 last 60.10)
 - open/close 7880 (time 1788535940.17 total 130.11 last 66.92)
 - open/close 8510 (time 1788535950.17 total 140.11 last 62.99)
 - open/close 9136 (time 1788535960.19 total 150.13 last 62.48)
 - open/close 9667 (time 1788535970.20 total 160.14 last 53.05)
total: 10000 open/close in 164.96 seconds: 60.62 ops/second
 - unlinked 0 (time 1788535979 ; total 0 ; last 0)
total: 10000 unlinks in 106 seconds: 94.339622 unlinks/second
 - open/close 582 (time 1788536100.29 total 10.01 last 58.12)
 - open/close 1220 (time 1788536110.30 total 20.02 last 63.79)
 - open/close 1773 (time 1788536120.31 total 30.03 last 55.22)
 - open/close 2300 (time 1788536130.33 total 40.05 last 52.59)
 - open/close 2798 (time 1788536140.33 total 50.05 last 49.77)
 - open/close 3391 (time 1788536150.36 total 60.08 last 59.16)
 - open/close 3908 (time 1788536160.40 total 70.12 last 51.46)
 - open/close 4449 (time 1788536170.42 total 80.14 last 54.02)
 - open/close 5052 (time 1788536180.42 total 90.14 last 60.28)
 - open/close 5582 (time 1788536190.44 total 100.16 last 52.89)
 - open/close 6072 (time 1788536200.45 total 110.17 last 48.98)
 - open/close 6557 (time 1788536210.47 total 120.19 last 48.39)
 - open/close 7070 (time 1788536220.47 total 130.19 last 51.29)
 - open/close 7699 (time 1788536230.47 total 140.19 last 62.89)
 - open/close 8307 (time 1788536240.48 total 150.20 last 60.75)
 - open/close 8956 (time 1788536250.49 total 160.21 last 64.83)
 - open/close 9567 (time 1788536260.50 total 170.22 last 61.04)
total: 10000 open/close in 177.32 seconds: 56.40 ops/second
 - unlinked 0 (time 1788536272 ; total 0 ; last 0)
total: 10000 unlinks in 111 seconds: 90.090088 unlinks/second
 - open/close 505 (time 1788536398.01 total 10.02 last 50.41)
 - open/close 1032 (time 1788536408.02 total 20.02 last 52.66)
 - open/close 1617 (time 1788536418.03 total 30.03 last 58.45)
 - open/close 2103 (time 1788536428.04 total 40.05 last 48.54)
 - open/close 2529 (time 1788536438.05 total 50.06 last 42.55)
 - open/close 2897 (time 1788536448.06 total 60.06 last 36.79)
 - open/close 3335 (time 1788536458.08 total 70.08 last 43.71)
 - open/close 3842 (time 1788536468.09 total 80.09 last 50.65)
 - open/close 4460 (time 1788536478.11 total 90.11 last 61.66)
 - open/close 5062 (time 1788536488.13 total 100.13 last 60.10)
 - open/close 5586 (time 1788536498.15 total 110.15 last 52.29)
 - open/close 6164 (time 1788536508.16 total 120.17 last 57.71)
 - open/close 6666 (time 1788536518.17 total 130.17 last 50.19)
 - open/close 7295 (time 1788536528.17 total 140.18 last 62.86)
 - open/close 7901 (time 1788536538.17 total 150.18 last 60.58)
 - open/close 8493 (time 1788536548.18 total 160.18 last 59.17)
 - open/close 9128 (time 1788536558.18 total 170.19 last 63.48)
 - open/close 9733 (time 1788536568.19 total 180.20 last 60.44)
total: 10000 open/close in 184.29 seconds: 54.26 ops/second
 - unlinked 0 (time 1788536577 ; total 0 ; last 0)
total: 10000 unlinks in 106 seconds: 94.339622 unlinks/second
 - open/close 620 (time 1788536700.04 total 10.03 last 61.84)
 - open/close 1200 (time 1788536710.05 total 20.04 last 57.90)
 - open/close 1739 (time 1788536720.05 total 30.04 last 53.90)
 - open/close 2394 (time 1788536730.07 total 40.06 last 65.43)
 - open/close 2995 (time 1788536740.08 total 50.07 last 60.04)
 - open/close 3603 (time 1788536750.08 total 60.07 last 60.75)
 - open/close 4223 (time 1788536760.09 total 70.08 last 61.96)
 - open/close 4910 (time 1788536770.10 total 80.09 last 68.66)
 - open/close 5487 (time 1788536780.12 total 90.11 last 57.58)
 - open/close 5997 (time 1788536790.17 total 100.16 last 50.75)
 - open/close 6534 (time 1788536800.17 total 110.16 last 53.70)
 - open/close 7075 (time 1788536810.19 total 120.18 last 53.99)
 - open/close 7586 (time 1788536820.19 total 130.18 last 51.08)
 - open/close 8096 (time 1788536830.22 total 140.21 last 50.87)
 - open/close 8652 (time 1788536840.23 total 150.22 last 55.53)
 - open/close 9310 (time 1788536850.24 total 160.23 last 65.74)
 - open/close 9975 (time 1788536860.24 total 170.23 last 66.45)
total: 10000 open/close in 170.65 seconds: 58.60 ops/second
 - unlinked 0 (time 1788536864 ; total 0 ; last 0)
total: 10000 unlinks in 109 seconds: 91.743118 unlinks/second
 - open/close 600 (time 1788536989.17 total 10.01 last 59.95)
 - open/close 1278 (time 1788536999.21 total 20.04 last 67.56)
 - open/close 1894 (time 1788537009.23 total 30.06 last 61.50)
 - open/close 2455 (time 1788537019.23 total 40.06 last 56.09)
 - open/close 3037 (time 1788537029.23 total 50.07 last 58.18)
 - open/close 3649 (time 1788537039.24 total 60.08 last 61.13)
 - open/close 4259 (time 1788537049.25 total 70.09 last 60.95)
 - open/close 4896 (time 1788537059.27 total 80.10 last 63.59)
 - open/close 5535 (time 1788537069.27 total 90.11 last 63.89)
 - open/close 6124 (time 1788537079.28 total 100.12 last 58.84)
 - open/close 6709 (time 1788537089.28 total 110.12 last 58.49)
 - open/close 7322 (time 1788537099.28 total 120.12 last 61.28)
 - open/close 7937 (time 1788537109.31 total 130.15 last 61.34)
 - open/close 8619 (time 1788537119.33 total 140.17 last 68.07)
 - open/close 9203 (time 1788537129.34 total 150.18 last 58.33)
 - open/close 9830 (time 1788537139.34 total 160.18 last 62.70)
total: 10000 open/close in 162.90 seconds: 61.39 ops/second
 - unlinked 0 (time 1788537147 ; total 0 ; last 0)
total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second
 - open/close 505 (time 1788537266.63 total 10.01 last 50.45)
 - open/close 1044 (time 1788537276.64 total 20.02 last 53.82)
 - open/close 1492 (time 1788537286.65 total 30.03 last 44.77)
 - open/close 2072 (time 1788537296.66 total 40.04 last 57.94)
 - open/close 2609 (time 1788537306.67 total 50.05 last 53.67)
 - open/close 3199 (time 1788537316.67 total 60.05 last 58.96)
 - open/close 3830 (time 1788537326.67 total 70.05 last 63.09)
 - open/close 4412 (time 1788537336.74 total 80.12 last 57.84)
 - open/close 4951 (time 1788537346.76 total 90.14 last 53.79)
 - open/close 5469 (time 1788537356.78 total 100.16 last 51.69)
 - open/close 6060 (time 1788537366.79 total 110.17 last 59.02)
 - open/close 6676 (time 1788537376.80 total 120.18 last 61.55)
 - open/close 7275 (time 1788537386.80 total 130.19 last 59.87)
 - open/close 7865 (time 1788537396.82 total 140.20 last 58.91)
 - open/close 8521 (time 1788537406.83 total 150.21 last 65.51)
 - open/close 9157 (time 1788537416.84 total 160.22 last 63.55)
 - open/close 9764 (time 1788537426.85 total 170.23 last 60.66)
total: 10000 open/close in 174.38 seconds: 57.35 ops/second
 - unlinked 0 (time 1788537435 ; total 0 ; last 0)
total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second
 - open/close 499 (time 1788537553.17 total 10.05 last 49.64)
 - open/close 1108 (time 1788537563.18 total 20.06 last 60.85)
 - open/close 1735 (time 1788537573.19 total 30.07 last 62.66)
 - open/close 2318 (time 1788537583.20 total 40.08 last 58.22)
 - open/close 2974 (time 1788537593.21 total 50.09 last 65.56)
 - open/close 3691 (time 1788537603.22 total 60.10 last 71.63)
 - open/close 4311 (time 1788537613.22 total 70.10 last 62.00)
 - open/close 4984 (time 1788537623.23 total 80.11 last 67.23)
 - open/close 5589 (time 1788537633.23 total 90.11 last 60.48)
 - open/close 6219 (time 1788537643.25 total 100.13 last 62.86)
 - open/close 6869 (time 1788537653.26 total 110.14 last 64.99)
 - open/close 7502 (time 1788537663.27 total 120.15 last 63.22)
 - open/close 8111 (time 1788537673.27 total 130.15 last 60.89)
 - open/close 8773 (time 1788537683.27 total 140.15 last 66.19)
 - open/close 9440 (time 1788537693.30 total 150.18 last 66.52)
total: 10000 open/close in 158.67 seconds: 63.02 ops/second
 - unlinked 0 (time 1788537705 ; total 0 ; last 0)
total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg424-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg424-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.124@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.124@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg424-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all
pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1
Started lustre-OST0000
oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg424-server: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg424-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg424-client.virtnet:  -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg424-client: oleg424-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg424-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg424-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.124@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.124@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.124@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.124@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.124@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg424-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all
pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1
Started lustre-OST0000
oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg424-server: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg424-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg424-client.virtnet:  -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg424-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg424-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server
stop mds service on oleg424-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server
unloading modules via unload_modules_local on: 'oleg424-server'
oleg424-server: oleg424-server.virtnet: executing unload_modules_local
oleg424-server: kmemleak disabled
modules unloaded.
pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2
pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2
pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2
pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2