== conf-sanity test 69: replace an OST with the same index ========================================================== 08:31:32 (1748867492)
start mds service on oleg437-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg437-server'
oleg437-server: oleg437-server.virtnet: executing load_modules_local
oleg437-server: Loading modules from /home/green/git/lustre-release/lustre
oleg437-server: detected 4 online CPUs by sysfs
oleg437-server: MODOPTS_LIBCFS=
oleg437-server: Force libcfs to create 2 CPU partitions
oleg437-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg437-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all
pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg437-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all
pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1
Started lustre-OST0000
oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg437-client.virtnet:  -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11872 inodes available. Want 99872. rc=0
 - open/close 1340 (time 1748867532.75 total 10.00 last 133.99)
 - open/close 2727 (time 1748867542.75 total 20.01 last 138.63)
 - open/close 4048 (time 1748867552.77 total 30.02 last 131.90)
 - open/close 5414 (time 1748867562.77 total 40.02 last 136.56)
 - open/close 7210 (time 1748867572.77 total 50.02 last 179.59)
 - open/close 9167 (time 1748867582.77 total 60.03 last 195.63)
total: 10000 open/close in 65.28 seconds: 153.18 ops/second
 - unlinked 0 (time 1748867589 ; total 0 ; last 0)
total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second
 - open/close 1672 (time 1748867636.56 total 10.00 last 167.15)
 - open/close 3492 (time 1748867646.56 total 20.00 last 181.97)
 - open/close 5185 (time 1748867656.56 total 30.01 last 169.26)
 - open/close 6871 (time 1748867666.57 total 40.01 last 168.46)
 - open/close 8657 (time 1748867676.57 total 50.02 last 178.50)
total: 10000 open/close in 57.71 seconds: 173.28 ops/second
 - unlinked 0 (time 1748867685 ; total 0 ; last 0)
total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second
 - open/close 1266 (time 1748867732.73 total 10.01 last 126.50)
 - open/close 2509 (time 1748867742.73 total 20.01 last 124.23)
 - open/close 3813 (time 1748867752.74 total 30.02 last 130.30)
 - open/close 5001 (time 1748867762.75 total 40.03 last 118.70)
 - open/close 6061 (time 1748867772.75 total 50.03 last 105.96)
 - open/close 7193 (time 1748867782.75 total 60.03 last 113.18)
 - open/close 8445 (time 1748867792.76 total 70.04 last 125.08)
 - open/close 9695 (time 1748867802.77 total 80.05 last 124.95)
total: 10000 open/close in 82.26 seconds: 121.57 ops/second
 - unlinked 0 (time 1748867806 ; total 0 ; last 0)
total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second
 - open/close 1577 (time 1748867857.45 total 10.01 last 157.57)
 - open/close 3205 (time 1748867867.45 total 20.01 last 162.79)
 - open/close 4814 (time 1748867877.45 total 30.01 last 160.87)
 - open/close 6411 (time 1748867887.45 total 40.01 last 159.66)
 - open/close 8128 (time 1748867897.46 total 50.02 last 171.63)
 - open/close 9928 (time 1748867907.46 total 60.02 last 179.97)
total: 10000 open/close in 60.38 seconds: 165.62 ops/second
 - unlinked 0 (time 1748867909 ; total 1 ; last 1)
total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second
 - open/close 1657 (time 1748867950.50 total 10.01 last 165.60)
 - open/close 3716 (time 1748867960.50 total 20.01 last 205.87)
 - open/close 5375 (time 1748867970.51 total 30.01 last 165.84)
 - open/close 6978 (time 1748867980.51 total 40.01 last 160.27)
 - open/close 8697 (time 1748867990.51 total 50.01 last 171.86)
total: 10000 open/close in 55.27 seconds: 180.92 ops/second
 - unlinked 0 (time 1748867996 ; total 0 ; last 0)
total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second
 - open/close 3301 (time 1748868030.68 total 10.00 last 330.03)
 - open/close 6063 (time 1748868040.68 total 20.00 last 276.17)
 - open/close 8596 (time 1748868050.68 total 30.01 last 253.20)
total: 10000 open/close in 35.36 seconds: 282.77 ops/second
 - unlinked 0 (time 1748868057 ; total 0 ; last 0)
total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second
 - open/close 2393 (time 1748868088.88 total 10.00 last 239.22)
 - open/close 4287 (time 1748868098.89 total 20.01 last 189.30)
 - open/close 6762 (time 1748868108.89 total 30.01 last 247.48)
 - open/close 9666 (time 1748868118.89 total 40.01 last 290.37)
total: 10000 open/close in 40.98 seconds: 244.00 ops/second
 - unlinked 0 (time 1748868120 ; total 0 ; last 0)
total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second
 - open/close 2976 (time 1748868150.79 total 10.00 last 297.58)
 - open/close 7093 (time 1748868160.79 total 20.00 last 411.67)
total: 10000 open/close in 29.33 seconds: 340.98 ops/second
 - unlinked 0 (time 1748868171 ; total 0 ; last 0)
total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second
 - open/close 3104 (time 1748868201.67 total 10.00 last 310.31)
 - open/close 6794 (time 1748868211.67 total 20.00 last 369.00)
 - open/close 9818 (time 1748868221.67 total 30.00 last 302.36)
total: 10000 open/close in 30.52 seconds: 327.69 ops/second
 - unlinked 0 (time 1748868223 ; total 0 ; last 0)
total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second
 - open/close 4720 (time 1748868249.93 total 10.00 last 471.89)
 - open/close 9653 (time 1748868259.93 total 20.00 last 493.21)
total: 10000 open/close in 20.66 seconds: 484.10 ops/second
 - unlinked 0 (time 1748868261 ; total 0 ; last 0)
total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg437-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg437-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.137@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.137@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg437-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all
pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1
Started lustre-OST0000
oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg437-server: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg437-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg437-client.virtnet:  -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre
On OST0, 12877 used inodes rc=0
umount lustre on /mnt/lustre.....
Stopping client oleg437-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg437-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server
stop mds service on oleg437-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg437-server
unloading modules via unload_modules_local on: 'oleg437-server'
oleg437-server: oleg437-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2
pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2
pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2
pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2