== conf-sanity test 69: replace an OST with the same index ========================================================== 21:33:07 (1787707987)
start mds service on oleg219-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg219-server'
oleg219-server: oleg219-server.virtnet: executing load_modules_local
oleg219-server: Loading modules from /home/green/git/lustre-release/lustre
oleg219-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg219-server: detected 4 online CPUs by sysfs
oleg219-server: MODOPTS_LIBCFS=
oleg219-server: Force libcfs to create 2 CPU partitions
oleg219-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg219-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg219-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all
pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg219-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all
pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1
Started lustre-OST0000
oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg219-client.virtnet:  -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 616 (time 1787708083.23 total 10.00 last 61.59)
 - open/close 1223 (time 1787708093.24 total 20.01 last 60.65)
 - open/close 1889 (time 1787708103.26 total 30.03 last 66.46)
 - open/close 2567 (time 1787708113.26 total 40.03 last 67.78)
 - open/close 3286 (time 1787708123.28 total 50.05 last 71.76)
 - open/close 3830 (time 1787708133.30 total 60.07 last 54.31)
 - open/close 4453 (time 1787708143.31 total 70.08 last 62.24)
 - open/close 5093 (time 1787708153.32 total 80.09 last 63.96)
 - open/close 5789 (time 1787708163.32 total 90.09 last 69.59)
 - open/close 6380 (time 1787708173.33 total 100.10 last 59.03)
 - open/close 7059 (time 1787708183.34 total 110.11 last 67.82)
 - open/close 7657 (time 1787708193.35 total 120.12 last 59.72)
 - open/close 8230 (time 1787708203.36 total 130.13 last 57.29)
 - open/close 8896 (time 1787708213.36 total 140.13 last 66.55)
 - open/close 9524 (time 1787708223.37 total 150.14 last 62.77)
total: 10000 open/close in 159.17 seconds: 62.82 ops/second
 - unlinked 0 (time 1787708237 ; total 0 ; last 0)
total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second
 - open/close 706 (time 1787708345.83 total 10.00 last 70.59)
 - open/close 1447 (time 1787708355.84 total 20.01 last 74.04)
 - open/close 2051 (time 1787708365.86 total 30.03 last 60.26)
 - open/close 2648 (time 1787708375.87 total 40.05 last 59.62)
 - open/close 3266 (time 1787708385.88 total 50.05 last 61.76)
 - open/close 3916 (time 1787708395.90 total 60.07 last 64.89)
 - open/close 4577 (time 1787708405.92 total 70.09 last 65.94)
 - open/close 5333 (time 1787708415.92 total 80.09 last 75.59)
 - open/close 5981 (time 1787708425.93 total 90.10 last 64.77)
 - open/close 6591 (time 1787708435.95 total 100.12 last 60.87)
 - open/close 7235 (time 1787708445.95 total 110.12 last 64.37)
 - open/close 7892 (time 1787708455.95 total 120.13 last 65.69)
 - open/close 8657 (time 1787708465.95 total 130.13 last 76.50)
 - open/close 9302 (time 1787708475.97 total 140.14 last 64.42)
 - open/close 9890 (time 1787708485.99 total 150.16 last 58.66)
total: 10000 open/close in 151.66 seconds: 65.94 ops/second
 - unlinked 0 (time 1787708492 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 598 (time 1787708599.31 total 10.00 last 59.78)
 - open/close 1215 (time 1787708609.32 total 20.01 last 61.64)
 - open/close 1845 (time 1787708619.33 total 30.02 last 62.96)
 - open/close 2500 (time 1787708629.34 total 40.03 last 65.41)
 - open/close 3152 (time 1787708639.35 total 50.04 last 65.17)
 - open/close 3793 (time 1787708649.35 total 60.04 last 64.09)
 - open/close 4430 (time 1787708659.36 total 70.05 last 63.61)
 - open/close 5079 (time 1787708669.36 total 80.05 last 64.89)
 - open/close 5804 (time 1787708679.38 total 90.07 last 72.39)
 - open/close 6448 (time 1787708689.38 total 100.07 last 64.40)
 - open/close 7217 (time 1787708699.39 total 110.08 last 76.85)
 - open/close 7936 (time 1787708709.39 total 120.08 last 71.85)
 - open/close 8630 (time 1787708719.40 total 130.09 last 69.32)
 - open/close 9366 (time 1787708729.41 total 140.10 last 73.56)
total: 10000 open/close in 149.79 seconds: 66.76 ops/second
 - unlinked 0 (time 1787708745 ; total 0 ; last 0)
total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second
 - open/close 764 (time 1787708864.28 total 10.01 last 76.36)
 - open/close 1482 (time 1787708874.29 total 20.02 last 71.72)
 - open/close 2218 (time 1787708884.31 total 30.04 last 73.43)
 - open/close 2948 (time 1787708894.34 total 40.06 last 72.84)
 - open/close 3635 (time 1787708904.34 total 50.07 last 68.65)
 - open/close 4312 (time 1787708914.35 total 60.07 last 67.67)
 - open/close 5063 (time 1787708924.35 total 70.08 last 75.04)
 - open/close 5866 (time 1787708934.38 total 80.11 last 80.07)
 - open/close 6566 (time 1787708944.40 total 90.13 last 69.87)
 - open/close 7225 (time 1787708954.40 total 100.13 last 65.90)
 - open/close 7853 (time 1787708964.41 total 110.14 last 62.73)
 - open/close 8547 (time 1787708974.42 total 120.15 last 69.36)
 - open/close 9341 (time 1787708984.43 total 130.16 last 79.32)
 - open/close 9901 (time 1787708994.44 total 140.16 last 55.96)
total: 10000 open/close in 141.59 seconds: 70.62 ops/second
 - unlinked 0 (time 1787709001 ; total 0 ; last 0)
total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second
 - open/close 710 (time 1787709111.86 total 10.00 last 71.00)
 - open/close 1410 (time 1787709121.89 total 20.03 last 69.77)
 - open/close 2019 (time 1787709131.90 total 30.05 last 60.81)
 - open/close 2721 (time 1787709141.91 total 40.05 last 70.17)
 - open/close 3504 (time 1787709151.91 total 50.06 last 78.25)
 - open/close 4281 (time 1787709161.92 total 60.07 last 77.63)
 - open/close 5059 (time 1787709171.94 total 70.09 last 77.63)
 - open/close 5662 (time 1787709181.96 total 80.10 last 60.21)
 - open/close 6248 (time 1787709191.97 total 90.12 last 58.52)
 - open/close 6837 (time 1787709201.98 total 100.12 last 58.88)
 - open/close 7478 (time 1787709211.98 total 110.13 last 64.06)
 - open/close 8040 (time 1787709221.99 total 120.13 last 56.16)
 - open/close 8697 (time 1787709231.99 total 130.14 last 65.68)
 - open/close 9435 (time 1787709242.00 total 140.14 last 73.74)
total: 10000 open/close in 148.76 seconds: 67.22 ops/second
 - unlinked 0 (time 1787709254 ; total 0 ; last 0)
total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second
 - open/close 607 (time 1787709365.40 total 10.00 last 60.70)
 - open/close 1226 (time 1787709375.42 total 20.02 last 61.76)
 - open/close 1855 (time 1787709385.43 total 30.03 last 62.84)
 - open/close 2507 (time 1787709395.43 total 40.04 last 65.17)
 - open/close 3157 (time 1787709405.43 total 50.04 last 65.00)
 - open/close 3834 (time 1787709415.44 total 60.04 last 67.67)
 - open/close 4498 (time 1787709425.44 total 70.05 last 66.38)
 - open/close 5151 (time 1787709435.44 total 80.05 last 65.28)
 - open/close 5880 (time 1787709445.46 total 90.06 last 72.79)
 - open/close 6557 (time 1787709455.46 total 100.07 last 67.67)
 - open/close 7208 (time 1787709465.46 total 110.07 last 65.09)
 - open/close 7879 (time 1787709475.47 total 120.07 last 67.07)
 - open/close 8592 (time 1787709485.47 total 130.08 last 71.26)
 - open/close 9242 (time 1787709495.48 total 140.08 last 64.98)
 - open/close 9973 (time 1787709505.51 total 150.11 last 72.87)
total: 10000 open/close in 150.74 seconds: 66.34 ops/second
 - unlinked 0 (time 1787709510 ; total 0 ; last 0)
total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second
 - open/close 651 (time 1787709614.62 total 10.02 last 64.99)
 - open/close 1311 (time 1787709624.62 total 20.02 last 65.99)
 - open/close 1947 (time 1787709634.63 total 30.03 last 63.55)
 - open/close 2700 (time 1787709644.64 total 40.04 last 75.20)
 - open/close 3263 (time 1787709654.65 total 50.05 last 56.24)
 - open/close 3728 (time 1787709664.66 total 60.06 last 46.47)
 - open/close 4273 (time 1787709674.66 total 70.06 last 54.49)
 - open/close 4825 (time 1787709684.66 total 80.06 last 55.20)
 - open/close 5468 (time 1787709694.68 total 90.08 last 64.18)
 - open/close 6086 (time 1787709704.68 total 100.08 last 61.77)
 - open/close 6699 (time 1787709714.69 total 110.09 last 61.27)
 - open/close 7208 (time 1787709724.70 total 120.10 last 50.84)
 - open/close 7715 (time 1787709734.71 total 130.11 last 50.64)
 - open/close 8314 (time 1787709744.72 total 140.12 last 59.82)
 - open/close 8993 (time 1787709754.72 total 150.13 last 67.87)
 - open/close 9762 (time 1787709764.73 total 160.13 last 76.87)
total: 10000 open/close in 163.75 seconds: 61.07 ops/second
 - unlinked 0 (time 1787709773 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 711 (time 1787709885.81 total 10.01 last 71.00)
 - open/close 1231 (time 1787709895.82 total 20.03 last 51.94)
 - open/close 1810 (time 1787709905.83 total 30.03 last 57.86)
 - open/close 2453 (time 1787709915.84 total 40.04 last 64.23)
 - open/close 3217 (time 1787709925.84 total 50.05 last 76.37)
 - open/close 3864 (time 1787709935.85 total 60.06 last 64.62)
 - open/close 4499 (time 1787709945.86 total 70.07 last 63.44)
 - open/close 5212 (time 1787709955.87 total 80.08 last 71.24)
 - open/close 5886 (time 1787709965.87 total 90.08 last 67.38)
 - open/close 6547 (time 1787709975.88 total 100.09 last 66.06)
 - open/close 7278 (time 1787709985.88 total 110.09 last 73.08)
 - open/close 7922 (time 1787709995.89 total 120.09 last 64.36)
 - open/close 8507 (time 1787710005.90 total 130.10 last 58.46)
 - open/close 9093 (time 1787710015.91 total 140.11 last 58.53)
 - open/close 9666 (time 1787710025.91 total 150.12 last 57.27)
total: 10000 open/close in 156.05 seconds: 64.08 ops/second
 - unlinked 0 (time 1787710037 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 645 (time 1787710144.20 total 10.00 last 64.48)
 - open/close 1309 (time 1787710154.20 total 20.01 last 66.36)
 - open/close 2070 (time 1787710164.22 total 30.03 last 75.98)
 - open/close 2827 (time 1787710174.23 total 40.04 last 75.62)
 - open/close 3631 (time 1787710184.24 total 50.05 last 80.31)
 - open/close 4390 (time 1787710194.25 total 60.05 last 75.87)
 - open/close 5048 (time 1787710204.26 total 70.07 last 65.70)
 - open/close 5564 (time 1787710214.28 total 80.08 last 51.52)
 - open/close 6123 (time 1787710224.29 total 90.09 last 55.84)
 - open/close 6794 (time 1787710234.29 total 100.10 last 67.06)
 - open/close 7407 (time 1787710244.30 total 110.10 last 61.28)
 - open/close 8041 (time 1787710254.30 total 120.11 last 63.37)
 - open/close 8617 (time 1787710264.32 total 130.13 last 57.51)
 - open/close 9069 (time 1787710274.33 total 140.13 last 45.16)
 - open/close 9534 (time 1787710284.35 total 150.16 last 46.39)
 - open/close 9988 (time 1787710294.38 total 160.19 last 45.27)
total: 10000 open/close in 160.45 seconds: 62.33 ops/second
 - unlinked 0 (time 1787710301 ; total 0 ; last 0)
total: 10000 unlinks in 109 seconds: 91.743118 unlinks/second
 - open/close 691 (time 1787710424.54 total 10.01 last 69.04)
 - open/close 1435 (time 1787710434.55 total 20.02 last 74.33)
 - open/close 2058 (time 1787710444.55 total 30.02 last 62.29)
 - open/close 2914 (time 1787710454.56 total 40.03 last 85.53)
 - open/close 3693 (time 1787710464.56 total 50.04 last 77.83)
 - open/close 4316 (time 1787710474.58 total 60.05 last 62.22)
 - open/close 4896 (time 1787710484.59 total 70.06 last 57.92)
 - open/close 5613 (time 1787710494.61 total 80.08 last 71.60)
 - open/close 6273 (time 1787710504.62 total 90.09 last 65.93)
 - open/close 6874 (time 1787710514.62 total 100.09 last 60.10)
 - open/close 7607 (time 1787710524.62 total 110.10 last 73.26)
 - open/close 8286 (time 1787710534.63 total 120.10 last 67.88)
 - open/close 9115 (time 1787710544.63 total 130.10 last 82.85)
 - open/close 9875 (time 1787710554.64 total 140.11 last 75.96)
total: 10000 open/close in 142.17 seconds: 70.34 ops/second
 - unlinked 0 (time 1787710561 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg219-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg219-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.119@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.119@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg219-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all
pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1
Started lustre-OST0000
oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg219-server: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg219-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg219-client.virtnet:  -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre
On OST0, 18334 used inodes rc=0
pdsh@oleg219-client: oleg219-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg219-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg219-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.119@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.119@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.119@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.119@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.202.119@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg219-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all
pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1
Started lustre-OST0000
oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg219-server: oleg219-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg219-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg219-client.virtnet:  -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg219-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg219-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server
stop mds service on oleg219-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server

unloading modules via unload_modules_local on: 'oleg219-server'
oleg219-server: oleg219-server.virtnet: executing unload_modules_local
oleg219-server: 
modules unloaded.
pdsh@oleg219-client: oleg219-client: ssh exited with exit code 2
pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2
pdsh@oleg219-client: oleg219-client: ssh exited with exit code 2
pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2