== conf-sanity test 69: replace an OST with the same index ========================================================== 08:28:55 (1787574535)
start mds service on oleg632-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg632-server'
oleg632-server: oleg632-server.virtnet: executing load_modules_local
oleg632-server: Loading modules from /home/green/git/lustre-release/lustre
oleg632-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg632-server: detected 4 online CPUs by sysfs
oleg632-server: MODOPTS_LIBCFS=
oleg632-server: Force libcfs to create 2 CPU partitions
oleg632-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg632-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg632-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg632-server: oleg632-server.virtnet: executing set_default_debug -1 all
pdsh@oleg632-client: oleg632-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg632-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg632-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg632-server: oleg632-server.virtnet: executing set_default_debug -1 all
pdsh@oleg632-client: oleg632-server: ssh exited with exit code 1
Started lustre-OST0000
oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg632-client.virtnet:  -o user_xattr,flock 192.168.206.132@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 621 (time 1787574624.08 total 10.00 last 62.09)
 - open/close 1271 (time 1787574634.09 total 20.01 last 64.96)
 - open/close 1983 (time 1787574644.13 total 30.05 last 70.89)
 - open/close 2741 (time 1787574654.14 total 40.06 last 75.73)
 - open/close 3485 (time 1787574664.14 total 50.06 last 74.38)
 - open/close 4202 (time 1787574674.15 total 60.07 last 71.63)
 - open/close 4913 (time 1787574684.16 total 70.08 last 71.07)
 - open/close 5721 (time 1787574694.16 total 80.08 last 80.77)
 - open/close 6399 (time 1787574704.16 total 90.08 last 67.78)
 - open/close 7032 (time 1787574714.18 total 100.09 last 63.23)
 - open/close 7642 (time 1787574724.19 total 110.11 last 60.93)
 - open/close 8344 (time 1787574734.19 total 120.11 last 70.16)
 - open/close 9005 (time 1787574744.20 total 130.12 last 66.03)
 - open/close 9672 (time 1787574754.22 total 140.14 last 66.61)
total: 10000 open/close in 145.76 seconds: 68.61 ops/second
 - unlinked 0 (time 1787574764 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 732 (time 1787574869.16 total 10.01 last 73.10)
 - open/close 1470 (time 1787574879.16 total 20.02 last 73.79)
 - open/close 2161 (time 1787574889.17 total 30.02 last 69.05)
 - open/close 2862 (time 1787574899.18 total 40.04 last 70.01)
 - open/close 3568 (time 1787574909.19 total 50.05 last 70.50)
 - open/close 4287 (time 1787574919.21 total 60.07 last 71.76)
 - open/close 4983 (time 1787574929.23 total 70.09 last 69.48)
 - open/close 5700 (time 1787574939.24 total 80.09 last 71.65)
 - open/close 6431 (time 1787574949.24 total 90.10 last 73.06)
 - open/close 7139 (time 1787574959.25 total 100.11 last 70.76)
 - open/close 7811 (time 1787574969.25 total 110.11 last 67.17)
 - open/close 8508 (time 1787574979.26 total 120.11 last 69.69)
 - open/close 9150 (time 1787574989.26 total 130.12 last 64.17)
 - open/close 9813 (time 1787574999.26 total 140.12 last 66.27)
total: 10000 open/close in 142.91 seconds: 69.97 ops/second
 - unlinked 0 (time 1787575006 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
 - open/close 700 (time 1787575107.31 total 10.01 last 69.94)
 - open/close 1420 (time 1787575117.31 total 20.01 last 71.98)
 - open/close 2072 (time 1787575127.32 total 30.02 last 65.11)
 - open/close 2749 (time 1787575137.33 total 40.03 last 67.65)
 - open/close 3423 (time 1787575147.34 total 50.04 last 67.37)
 - open/close 4128 (time 1787575157.34 total 60.04 last 70.45)
 - open/close 4861 (time 1787575167.35 total 70.05 last 73.23)
 - open/close 5591 (time 1787575177.36 total 80.06 last 72.93)
 - open/close 6298 (time 1787575187.37 total 90.07 last 70.65)
 - open/close 6993 (time 1787575197.38 total 100.08 last 69.41)
 - open/close 7740 (time 1787575207.40 total 110.10 last 74.58)
 - open/close 8510 (time 1787575217.41 total 120.11 last 76.90)
 - open/close 9254 (time 1787575227.43 total 130.13 last 74.25)
total: 10000 open/close in 139.91 seconds: 71.48 ops/second
 - unlinked 0 (time 1787575240 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 673 (time 1787575337.64 total 10.01 last 67.23)
 - open/close 1447 (time 1787575347.66 total 20.03 last 77.25)
 - open/close 2197 (time 1787575357.66 total 30.04 last 74.95)
 - open/close 2933 (time 1787575367.67 total 40.04 last 73.56)
 - open/close 3668 (time 1787575377.68 total 50.05 last 73.42)
 - open/close 4451 (time 1787575387.69 total 60.06 last 78.25)
 - open/close 5194 (time 1787575397.69 total 70.07 last 74.27)
 - open/close 5932 (time 1787575407.69 total 80.07 last 73.78)
 - open/close 6656 (time 1787575417.71 total 90.08 last 72.31)
 - open/close 7432 (time 1787575427.72 total 100.09 last 77.50)
 - open/close 8138 (time 1787575437.74 total 110.12 last 70.43)
 - open/close 8855 (time 1787575447.74 total 120.12 last 71.69)
 - open/close 9595 (time 1787575457.75 total 130.12 last 73.96)
total: 10000 open/close in 135.07 seconds: 74.04 ops/second
 - unlinked 0 (time 1787575466 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
 - open/close 677 (time 1787575560.84 total 10.00 last 67.68)
 - open/close 1437 (time 1787575570.86 total 20.02 last 75.84)
 - open/close 2137 (time 1787575580.87 total 30.03 last 69.95)
 - open/close 2846 (time 1787575590.87 total 40.03 last 70.88)
 - open/close 3541 (time 1787575600.87 total 50.04 last 69.49)
 - open/close 4237 (time 1787575610.88 total 60.05 last 69.52)
 - open/close 4946 (time 1787575620.90 total 70.06 last 70.80)
 - open/close 5706 (time 1787575630.91 total 80.07 last 75.94)
 - open/close 6401 (time 1787575640.92 total 90.09 last 69.37)
 - open/close 6923 (time 1787575650.93 total 100.10 last 52.15)
 - open/close 7532 (time 1787575660.97 total 110.14 last 60.66)
 - open/close 8223 (time 1787575670.98 total 120.15 last 69.03)
 - open/close 8877 (time 1787575680.99 total 130.15 last 65.39)
 - open/close 9507 (time 1787575691.00 total 140.16 last 62.91)
total: 10000 open/close in 147.02 seconds: 68.02 ops/second
 - unlinked 0 (time 1787575702 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 708 (time 1787575800.66 total 10.01 last 70.75)
 - open/close 1496 (time 1787575810.66 total 20.02 last 78.73)
 - open/close 2223 (time 1787575820.67 total 30.02 last 72.67)
 - open/close 2951 (time 1787575830.67 total 40.02 last 72.77)
 - open/close 3662 (time 1787575840.68 total 50.03 last 71.07)
 - open/close 4279 (time 1787575850.68 total 60.03 last 61.66)
 - open/close 4956 (time 1787575860.69 total 70.04 last 67.66)
 - open/close 5736 (time 1787575870.69 total 80.04 last 77.98)
 - open/close 6441 (time 1787575880.71 total 90.07 last 70.33)
 - open/close 7075 (time 1787575890.73 total 100.08 last 63.33)
 - open/close 7757 (time 1787575900.74 total 110.10 last 68.07)
 - open/close 8426 (time 1787575910.75 total 120.10 last 66.89)
 - open/close 9023 (time 1787575920.75 total 130.10 last 59.66)
 - open/close 9701 (time 1787575930.76 total 140.11 last 67.73)
total: 10000 open/close in 144.97 seconds: 68.98 ops/second
 - unlinked 0 (time 1787575940 ; total 1 ; last 1)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
 - open/close 703 (time 1787576038.51 total 10.02 last 70.13)
 - open/close 1439 (time 1787576048.51 total 20.03 last 73.57)
 - open/close 2133 (time 1787576058.52 total 30.03 last 69.37)
 - open/close 2815 (time 1787576068.53 total 40.04 last 68.16)
 - open/close 3535 (time 1787576078.54 total 50.05 last 71.89)
 - open/close 4231 (time 1787576088.55 total 60.06 last 69.57)
 - open/close 4928 (time 1787576098.55 total 70.06 last 69.68)
 - open/close 5646 (time 1787576108.55 total 80.06 last 71.79)
 - open/close 6373 (time 1787576118.55 total 90.07 last 72.67)
 - open/close 7066 (time 1787576128.56 total 100.08 last 69.25)
 - open/close 7786 (time 1787576138.57 total 110.09 last 71.91)
 - open/close 8488 (time 1787576148.61 total 120.12 last 69.95)
 - open/close 9175 (time 1787576158.62 total 130.13 last 68.66)
 - open/close 9923 (time 1787576168.62 total 140.14 last 74.73)
total: 10000 open/close in 141.19 seconds: 70.83 ops/second
 - unlinked 0 (time 1787576173 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 650 (time 1787576278.71 total 10.01 last 64.93)
 - open/close 1155 (time 1787576288.73 total 20.02 last 50.43)
 - open/close 1746 (time 1787576298.73 total 30.03 last 59.06)
 - open/close 2485 (time 1787576308.74 total 40.03 last 73.88)
 - open/close 3235 (time 1787576318.75 total 50.04 last 74.93)
 - open/close 3907 (time 1787576328.76 total 60.05 last 67.13)
 - open/close 4636 (time 1787576338.76 total 70.06 last 72.85)
 - open/close 5312 (time 1787576348.77 total 80.06 last 67.57)
 - open/close 5882 (time 1787576358.78 total 90.07 last 56.95)
 - open/close 6601 (time 1787576368.78 total 100.08 last 71.84)
 - open/close 7344 (time 1787576378.79 total 110.09 last 74.25)
 - open/close 8074 (time 1787576388.80 total 120.10 last 72.94)
 - open/close 8801 (time 1787576398.80 total 130.10 last 72.69)
 - open/close 9532 (time 1787576408.81 total 140.10 last 73.05)
total: 10000 open/close in 147.25 seconds: 67.91 ops/second
 - unlinked 0 (time 1787576420 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 559 (time 1787576525.20 total 10.00 last 55.89)
 - open/close 1203 (time 1787576535.22 total 20.02 last 64.26)
 - open/close 1862 (time 1787576545.22 total 30.03 last 65.88)
 - open/close 2547 (time 1787576555.23 total 40.04 last 68.44)
 - open/close 3301 (time 1787576565.24 total 50.04 last 75.35)
 - open/close 3983 (time 1787576575.25 total 60.06 last 68.10)
 - open/close 4664 (time 1787576585.26 total 70.06 last 68.06)
 - open/close 5256 (time 1787576595.26 total 80.07 last 59.16)
 - open/close 5853 (time 1787576605.27 total 90.08 last 59.66)
 - open/close 6467 (time 1787576615.29 total 100.10 last 61.28)
 - open/close 7054 (time 1787576625.31 total 110.12 last 58.59)
 - open/close 7734 (time 1787576635.32 total 120.13 last 67.92)
 - open/close 8399 (time 1787576645.34 total 130.14 last 66.41)
 - open/close 9023 (time 1787576655.34 total 140.14 last 62.38)
 - open/close 9522 (time 1787576665.34 total 150.15 last 49.88)
total: 10000 open/close in 157.23 seconds: 63.60 ops/second
 - unlinked 0 (time 1787576676 ; total 0 ; last 0)
total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second
 - open/close 612 (time 1787576786.03 total 10.00 last 61.20)
 - open/close 1257 (time 1787576796.05 total 20.02 last 64.38)
 - open/close 1768 (time 1787576806.06 total 30.03 last 51.05)
 - open/close 2528 (time 1787576816.07 total 40.04 last 75.91)
 - open/close 3337 (time 1787576826.10 total 50.07 last 80.70)
 - open/close 4093 (time 1787576836.11 total 60.08 last 75.52)
 - open/close 4816 (time 1787576846.11 total 70.08 last 72.27)
 - open/close 5499 (time 1787576856.13 total 80.10 last 68.20)
 - open/close 6226 (time 1787576866.13 total 90.10 last 72.68)
 - open/close 6922 (time 1787576876.13 total 100.10 last 69.59)
 - open/close 7633 (time 1787576886.14 total 110.11 last 71.04)
 - open/close 8405 (time 1787576896.14 total 120.11 last 77.19)
 - open/close 9086 (time 1787576906.15 total 130.12 last 68.04)
 - open/close 9832 (time 1787576916.15 total 140.12 last 74.56)
total: 10000 open/close in 142.97 seconds: 69.94 ops/second
 - unlinked 0 (time 1787576923 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg632-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg632-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg632-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.132@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.132@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg632-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg632-server: oleg632-server.virtnet: executing set_default_debug -1 all
pdsh@oleg632-client: oleg632-server: ssh exited with exit code 1
Started lustre-OST0000
oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg632-server: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg632-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg632-client.virtnet:  -o user_xattr,flock 192.168.206.132@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg632-client: oleg632-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg632-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg632-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg632-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.132@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.132@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.132@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.132@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.132@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg632-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg632-server: oleg632-server.virtnet: executing set_default_debug -1 all
pdsh@oleg632-client: oleg632-server: ssh exited with exit code 1
Started lustre-OST0000
oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg632-server: oleg632-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg632-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg632-client.virtnet:  -o user_xattr,flock 192.168.206.132@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg632-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg632-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg632-server
stop mds service on oleg632-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg632-server

unloading modules via unload_modules_local on: 'oleg632-server'
oleg632-server: oleg632-server.virtnet: executing unload_modules_local
oleg632-server: 
modules unloaded.
pdsh@oleg632-client: oleg632-client: ssh exited with exit code 2
pdsh@oleg632-client: oleg632-server: ssh exited with exit code 2
pdsh@oleg632-client: oleg632-server: ssh exited with exit code 2
pdsh@oleg632-client: oleg632-client: ssh exited with exit code 2