== conf-sanity test 69: replace an OST with the same index ========================================================== 02:38:19 (1785220699)
start mds service on oleg154-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg154-server'
oleg154-server: oleg154-server.virtnet: executing load_modules_local
oleg154-server: Loading modules from /home/green/git/lustre-release/lustre
oleg154-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg154-server: detected 4 online CPUs by sysfs
oleg154-server: MODOPTS_LIBCFS=
oleg154-server: Force libcfs to create 2 CPU partitions
oleg154-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg154-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg154-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all
pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg154-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg154-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all
pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1
Started lustre-OST0000
oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg154-client.virtnet:  -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 1057 (time 1785220775.59 total 10.00 last 105.67)
 - open/close 2052 (time 1785220785.60 total 20.01 last 99.42)
 - open/close 3131 (time 1785220795.61 total 30.02 last 107.83)
 - open/close 4049 (time 1785220805.62 total 40.02 last 91.73)
 - open/close 5054 (time 1785220815.62 total 50.03 last 100.45)
 - open/close 6197 (time 1785220825.63 total 60.04 last 114.20)
 - open/close 7221 (time 1785220835.64 total 70.04 last 102.34)
 - open/close 8202 (time 1785220845.64 total 80.05 last 98.06)
 - open/close 9211 (time 1785220855.65 total 90.06 last 100.82)
total: 10000 open/close in 97.86 seconds: 102.19 ops/second
 - unlinked 0 (time 1785220866 ; total 0 ; last 0)
total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second
 - open/close 1021 (time 1785220932.00 total 10.01 last 102.04)
 - open/close 2208 (time 1785220942.00 total 20.01 last 118.66)
 - open/close 3327 (time 1785220952.00 total 30.01 last 111.90)
 - open/close 4334 (time 1785220962.01 total 40.01 last 100.69)
 - open/close 5393 (time 1785220972.01 total 50.01 last 105.85)
 - open/close 6394 (time 1785220982.02 total 60.02 last 100.03)
 - open/close 7306 (time 1785220992.02 total 70.02 last 91.18)
 - open/close 8326 (time 1785221002.02 total 80.03 last 101.98)
 - open/close 9367 (time 1785221012.03 total 90.04 last 103.97)
total: 10000 open/close in 96.20 seconds: 103.95 ops/second
 - unlinked 0 (time 1785221021 ; total 0 ; last 0)
total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second
 - open/close 888 (time 1785221095.01 total 10.00 last 88.77)
 - open/close 1648 (time 1785221105.02 total 20.02 last 75.89)
 - open/close 2589 (time 1785221115.03 total 30.03 last 94.00)
 - open/close 3740 (time 1785221125.03 total 40.03 last 115.08)
 - open/close 4776 (time 1785221135.04 total 50.03 last 103.56)
 - open/close 5896 (time 1785221145.04 total 60.04 last 111.96)
 - open/close 6839 (time 1785221155.04 total 70.04 last 94.28)
 - open/close 7677 (time 1785221165.05 total 80.05 last 83.74)
 - open/close 8783 (time 1785221175.05 total 90.05 last 110.59)
total: 10000 open/close in 99.90 seconds: 100.10 ops/second
 - unlinked 0 (time 1785221187 ; total 0 ; last 0)
total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second
 - open/close 1445 (time 1785221251.22 total 10.02 last 144.24)
 - open/close 2650 (time 1785221261.23 total 20.03 last 120.40)
 - open/close 3755 (time 1785221271.23 total 30.03 last 110.44)
 - open/close 4944 (time 1785221281.24 total 40.04 last 118.84)
 - open/close 6231 (time 1785221291.24 total 50.04 last 128.65)
 - open/close 7383 (time 1785221301.25 total 60.05 last 115.08)
 - open/close 8615 (time 1785221311.26 total 70.06 last 123.07)
 - open/close 9874 (time 1785221321.27 total 80.07 last 125.83)
total: 10000 open/close in 81.21 seconds: 123.14 ops/second
 - unlinked 0 (time 1785221324 ; total 0 ; last 0)
total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second
 - open/close 1260 (time 1785221387.29 total 10.00 last 126.00)
 - open/close 2482 (time 1785221397.30 total 20.01 last 122.11)
 - open/close 3682 (time 1785221407.30 total 30.02 last 119.90)
 - open/close 4969 (time 1785221417.31 total 40.02 last 128.64)
 - open/close 6263 (time 1785221427.31 total 50.03 last 129.33)
 - open/close 7412 (time 1785221437.32 total 60.03 last 114.86)
 - open/close 8570 (time 1785221447.33 total 70.04 last 115.62)
 - open/close 9730 (time 1785221457.34 total 80.05 last 115.93)
total: 10000 open/close in 82.56 seconds: 121.12 ops/second
 - unlinked 0 (time 1785221462 ; total 0 ; last 0)
total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second
 - open/close 1183 (time 1785221530.10 total 10.01 last 118.20)
 - open/close 2543 (time 1785221540.10 total 20.01 last 135.96)
 - open/close 3754 (time 1785221550.11 total 30.01 last 121.07)
 - open/close 4977 (time 1785221560.11 total 40.02 last 122.24)
 - open/close 6223 (time 1785221570.12 total 50.02 last 124.54)
 - open/close 7409 (time 1785221580.12 total 60.03 last 118.54)
 - open/close 8675 (time 1785221590.12 total 70.03 last 126.58)
 - open/close 9971 (time 1785221600.13 total 80.03 last 129.54)
total: 10000 open/close in 80.23 seconds: 124.64 ops/second
 - unlinked 0 (time 1785221602 ; total 0 ; last 0)
total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second
 - open/close 1257 (time 1785221662.24 total 10.00 last 125.66)
 - open/close 2617 (time 1785221672.24 total 20.00 last 135.98)
 - open/close 3887 (time 1785221682.25 total 30.01 last 126.98)
 - open/close 5208 (time 1785221692.25 total 40.01 last 132.09)
 - open/close 6486 (time 1785221702.25 total 50.01 last 127.71)
 - open/close 7871 (time 1785221712.26 total 60.02 last 138.45)
 - open/close 9036 (time 1785221722.26 total 70.02 last 116.46)
total: 10000 open/close in 77.44 seconds: 129.13 ops/second
 - unlinked 0 (time 1785221731 ; total 0 ; last 0)
total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second
 - open/close 1644 (time 1785221787.99 total 10.00 last 164.38)
 - open/close 3102 (time 1785221797.99 total 20.00 last 145.76)
 - open/close 4372 (time 1785221808.00 total 30.01 last 126.92)
 - open/close 5688 (time 1785221818.00 total 40.01 last 131.56)
 - open/close 7119 (time 1785221828.00 total 50.01 last 143.08)
 - open/close 8536 (time 1785221838.00 total 60.02 last 141.64)
 - open/close 9946 (time 1785221848.01 total 70.02 last 140.94)
total: 10000 open/close in 70.55 seconds: 141.73 ops/second
 - unlinked 0 (time 1785221850 ; total 0 ; last 0)
total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second
 - open/close 1555 (time 1785221903.60 total 10.00 last 155.49)
 - open/close 3237 (time 1785221913.60 total 20.00 last 168.15)
 - open/close 4377 (time 1785221923.60 total 30.01 last 113.97)
 - open/close 5006 (time 1785221933.60 total 40.01 last 62.88)
 - open/close 5750 (time 1785221943.61 total 50.01 last 74.40)
 - open/close 6351 (time 1785221955.12 total 61.53 last 52.18)
 - open/close 6884 (time 1785221965.51 total 71.92 last 51.30)
 - open/close 7483 (time 1785221975.52 total 81.92 last 59.86)
 - open/close 8459 (time 1785221985.53 total 91.93 last 97.53)
 - open/close 9182 (time 1785221995.53 total 101.94 last 72.26)
 - open/close 9783 (time 1785222005.56 total 111.97 last 59.90)
total: 10000 open/close in 115.85 seconds: 86.32 ops/second
 - unlinked 0 (time 1785222015 ; total 0 ; last 0)
total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second
 - open/close 980 (time 1785222099.38 total 10.01 last 97.94)
 - open/close 1980 (time 1785222109.39 total 20.01 last 99.98)
 - open/close 3042 (time 1785222119.39 total 30.01 last 106.14)
 - open/close 3960 (time 1785222129.39 total 40.02 last 91.76)
 - open/close 4876 (time 1785222139.40 total 50.02 last 91.59)
 - open/close 5851 (time 1785222149.40 total 60.02 last 97.47)
 - open/close 6753 (time 1785222159.40 total 70.03 last 90.15)
 - open/close 7640 (time 1785222169.41 total 80.03 last 88.66)
 - open/close 8448 (time 1785222179.42 total 90.04 last 80.74)
 - open/close 9341 (time 1785222189.42 total 100.04 last 89.28)
total: 10000 open/close in 107.94 seconds: 92.64 ops/second
 - unlinked 0 (time 1785222201 ; total 0 ; last 0)
total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg154-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg154-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg154-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.154@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.154@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg154-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all
pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1
Started lustre-OST0000
oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg154-server: oleg154-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg154-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg154-client.virtnet:  -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre
On OST0, 11798 used inodes rc=0
pdsh@oleg154-client: oleg154-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg154-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg154-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg154-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.154@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.154@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.154@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.154@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.154@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg154-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg154-server: oleg154-server.virtnet: executing set_default_debug -1 all
pdsh@oleg154-client: oleg154-server: ssh exited with exit code 1
Started lustre-OST0000
oleg154-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg154-server: oleg154-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg154-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg154-client.virtnet:  -o user_xattr,flock 192.168.201.154@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg154-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg154-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg154-server
stop mds service on oleg154-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg154-server

unloading modules via unload_modules_local on: 'oleg154-server'
oleg154-server: oleg154-server.virtnet: executing unload_modules_local
oleg154-server: 
modules unloaded.
pdsh@oleg154-client: oleg154-client: ssh exited with exit code 2
pdsh@oleg154-client: oleg154-server: ssh exited with exit code 2
pdsh@oleg154-client: oleg154-client: ssh exited with exit code 2
pdsh@oleg154-client: oleg154-server: ssh exited with exit code 2