== conf-sanity test 69: replace an OST with the same index ========================================================== 23:07:29 (1735099649)
start mds service on oleg414-server
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
loading modules on: 'oleg414-server'
oleg414-server: oleg414-server.virtnet: executing load_modules_local
oleg414-server: Loading modules from /home/green/git/lustre-release/lustre
oleg414-server: detected 4 online CPUs by sysfs
oleg414-server: MODOPTS_LIBCFS=
oleg414-server: Force libcfs to create 2 CPU partitions
oleg414-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg414-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all
pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg414-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all
pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1
Started lustre-OST0000
oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg414-client.virtnet:  -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11872 inodes available. Want 99872. rc=0
 - open/close 1596 (time 1735099690.56 total 10.00 last 159.56)
 - open/close 3360 (time 1735099700.57 total 20.00 last 176.37)
 - open/close 5034 (time 1735099710.57 total 30.01 last 167.38)
 - open/close 6714 (time 1735099720.57 total 40.01 last 167.92)
 - open/close 8328 (time 1735099730.57 total 50.01 last 161.39)
total: 10000 open/close in 58.94 seconds: 169.67 ops/second
 - unlinked 0 (time 1735099740 ; total 0 ; last 0)
total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second
 - open/close 2233 (time 1735099777.23 total 10.00 last 223.19)
 - open/close 4382 (time 1735099787.23 total 20.01 last 214.84)
 - open/close 6675 (time 1735099797.23 total 30.01 last 229.29)
 - open/close 9113 (time 1735099807.23 total 40.01 last 243.74)
total: 10000 open/close in 44.31 seconds: 225.68 ops/second
 - unlinked 0 (time 1735099812 ; total 0 ; last 0)
total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second
 - open/close 2642 (time 1735099845.08 total 10.00 last 264.19)
 - open/close 5250 (time 1735099855.08 total 20.00 last 260.79)
 - open/close 7904 (time 1735099865.09 total 30.01 last 265.29)
total: 10000 open/close in 38.30 seconds: 261.10 ops/second
 - unlinked 0 (time 1735099874 ; total 0 ; last 0)
total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second
 - open/close 2370 (time 1735099906.17 total 10.00 last 236.94)
 - open/close 4668 (time 1735099916.18 total 20.01 last 229.66)
 - open/close 6929 (time 1735099926.18 total 30.01 last 226.02)
 - open/close 9468 (time 1735099936.19 total 40.01 last 253.86)
total: 10000 open/close in 42.22 seconds: 236.85 ops/second
 - unlinked 0 (time 1735099939 ; total 0 ; last 0)
total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second
 - open/close 2121 (time 1735099980.43 total 10.00 last 212.09)
 - open/close 4183 (time 1735099990.43 total 20.00 last 206.14)
 - open/close 5330 (time 1735100000.43 total 30.00 last 114.69)
 - open/close 6138 (time 1735100010.44 total 40.01 last 80.79)
 - open/close 7052 (time 1735100020.44 total 50.02 last 91.31)
 - open/close 8179 (time 1735100030.45 total 60.02 last 112.64)
 - open/close 9274 (time 1735100040.48 total 70.05 last 109.14)
total: 10000 open/close in 76.43 seconds: 130.85 ops/second
 - unlinked 0 (time 1735100048 ; total 0 ; last 0)
total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second
 - open/close 1365 (time 1735100105.60 total 10.00 last 136.45)
 - open/close 3227 (time 1735100115.60 total 20.00 last 186.18)
 - open/close 5023 (time 1735100125.61 total 30.01 last 179.50)
 - open/close 6820 (time 1735100135.61 total 40.01 last 179.65)
 - open/close 8595 (time 1735100145.62 total 50.02 last 177.41)
total: 10000 open/close in 57.61 seconds: 173.57 ops/second
 - unlinked 0 (time 1735100154 ; total 0 ; last 0)
total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second
 - open/close 1473 (time 1735100197.25 total 10.00 last 147.28)
 - open/close 2943 (time 1735100207.26 total 20.01 last 146.92)
 - open/close 4420 (time 1735100217.26 total 30.01 last 147.68)
 - open/close 5904 (time 1735100227.26 total 40.01 last 148.39)
 - open/close 7456 (time 1735100237.26 total 50.01 last 155.17)
 - open/close 8904 (time 1735100247.27 total 60.02 last 144.72)
total: 10000 open/close in 68.25 seconds: 146.53 ops/second
 - unlinked 0 (time 1735100257 ; total 0 ; last 0)
total: 10000 unlinks in 37 seconds: 270.270264 unlinks/second
 - open/close 1581 (time 1735100306.01 total 10.00 last 158.08)
 - open/close 3439 (time 1735100316.01 total 20.00 last 185.76)
 - open/close 5366 (time 1735100326.02 total 30.01 last 192.62)
 - open/close 7148 (time 1735100336.02 total 40.01 last 178.16)
 - open/close 9552 (time 1735100346.02 total 50.01 last 240.35)
total: 10000 open/close in 51.52 seconds: 194.12 ops/second
 - unlinked 0 (time 1735100348 ; total 0 ; last 0)
total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second
 - open/close 2968 (time 1735100377.41 total 10.00 last 296.76)
 - open/close 6226 (time 1735100387.41 total 20.00 last 325.69)
 - open/close 9127 (time 1735100397.41 total 30.01 last 290.05)
total: 10000 open/close in 32.46 seconds: 308.04 ops/second
 - unlinked 0 (time 1735100400 ; total 0 ; last 0)
total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second
 - open/close 3875 (time 1735100427.18 total 10.00 last 387.38)
 - open/close 7645 (time 1735100437.19 total 20.01 last 376.92)
total: 10000 open/close in 27.60 seconds: 362.29 ops/second
 - unlinked 0 (time 1735100445 ; total 0 ; last 0)
total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg414-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg414-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.114@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.114@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg414-server
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all
pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1
Started lustre-OST0000
oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg414-server: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg414-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 16 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg414-client.virtnet:  -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre
On OST0, 20325 used inodes rc=0
umount lustre on /mnt/lustre.....
Stopping client oleg414-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg414-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server
stop mds service on oleg414-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server
unloading modules via unload_modules_local on: 'oleg414-server'
oleg414-server: oleg414-server.virtnet: executing unload_modules_local
modules unloaded.
pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2
pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2
pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2
pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2