== conf-sanity test 69: replace an OST with the same index ========================================================== 01:00:13 (1786338013)
start mds service on oleg356-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg356-server'
oleg356-server: oleg356-server.virtnet: executing load_modules_local
oleg356-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg356-server: Loading modules from /home/green/git/lustre-release/lustre
oleg356-server: detected 4 online CPUs by sysfs
oleg356-server: MODOPTS_LIBCFS=
oleg356-server: Force libcfs to create 2 CPU partitions
oleg356-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg356-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg356-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all
pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg356-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all
pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1
Started lustre-OST0000
oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg356-client.virtnet:  -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 851 (time 1786338098.57 total 10.01 last 85.05)
 - open/close 1677 (time 1786338108.57 total 20.01 last 82.57)
 - open/close 2486 (time 1786338118.61 total 30.05 last 80.59)
 - open/close 3327 (time 1786338128.62 total 40.05 last 84.07)
 - open/close 4207 (time 1786338138.62 total 50.06 last 87.95)
 - open/close 5089 (time 1786338148.63 total 60.06 last 88.14)
 - open/close 6050 (time 1786338158.63 total 70.07 last 96.09)
 - open/close 6922 (time 1786338168.63 total 80.07 last 87.19)
 - open/close 7790 (time 1786338178.66 total 90.10 last 86.54)
 - open/close 8681 (time 1786338188.66 total 100.10 last 89.06)
 - open/close 9549 (time 1786338198.67 total 110.10 last 86.80)
total: 10000 open/close in 115.00 seconds: 86.96 ops/second
 - unlinked 0 (time 1786338207 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 930 (time 1786338303.30 total 10.00 last 92.96)
 - open/close 1850 (time 1786338313.31 total 20.01 last 91.97)
 - open/close 2691 (time 1786338323.32 total 30.02 last 84.01)
 - open/close 3526 (time 1786338333.33 total 40.03 last 83.44)
 - open/close 4448 (time 1786338343.33 total 50.03 last 92.20)
 - open/close 5277 (time 1786338353.33 total 60.03 last 82.85)
 - open/close 6124 (time 1786338363.33 total 70.04 last 84.68)
 - open/close 7014 (time 1786338373.34 total 80.04 last 89.00)
 - open/close 7893 (time 1786338383.34 total 90.04 last 87.84)
 - open/close 8735 (time 1786338393.35 total 100.05 last 84.13)
 - open/close 9604 (time 1786338403.35 total 110.05 last 86.89)
total: 10000 open/close in 114.87 seconds: 87.05 ops/second
 - unlinked 0 (time 1786338412 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 882 (time 1786338510.94 total 10.00 last 88.16)
 - open/close 1673 (time 1786338520.95 total 20.02 last 79.00)
 - open/close 2527 (time 1786338530.96 total 30.02 last 85.35)
 - open/close 3392 (time 1786338540.96 total 40.03 last 86.45)
 - open/close 4240 (time 1786338550.98 total 50.04 last 84.67)
 - open/close 5094 (time 1786338560.98 total 60.04 last 85.40)
 - open/close 5909 (time 1786338570.98 total 70.05 last 81.50)
 - open/close 6705 (time 1786338581.00 total 80.06 last 79.47)
 - open/close 7561 (time 1786338591.00 total 90.07 last 85.54)
 - open/close 8380 (time 1786338601.01 total 100.08 last 81.82)
 - open/close 9138 (time 1786338611.02 total 110.09 last 75.74)
 - open/close 9964 (time 1786338621.03 total 120.09 last 82.55)
total: 10000 open/close in 120.50 seconds: 82.98 ops/second
 - unlinked 0 (time 1786338625 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 928 (time 1786338720.99 total 10.01 last 92.74)
 - open/close 1888 (time 1786338731.00 total 20.01 last 95.99)
 - open/close 2811 (time 1786338741.02 total 30.03 last 92.09)
 - open/close 3670 (time 1786338751.02 total 40.04 last 85.84)
 - open/close 4510 (time 1786338761.03 total 50.04 last 83.96)
 - open/close 5319 (time 1786338771.04 total 60.05 last 80.83)
 - open/close 6145 (time 1786338781.05 total 70.06 last 82.54)
 - open/close 6934 (time 1786338791.05 total 80.06 last 78.88)
 - open/close 7788 (time 1786338801.07 total 90.08 last 85.23)
 - open/close 8672 (time 1786338811.08 total 100.09 last 88.32)
 - open/close 9528 (time 1786338821.10 total 110.11 last 85.43)
total: 10000 open/close in 115.92 seconds: 86.26 ops/second
 - unlinked 0 (time 1786338830 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 889 (time 1786338929.22 total 10.00 last 88.87)
 - open/close 1689 (time 1786338939.23 total 20.01 last 79.91)
 - open/close 2572 (time 1786338949.24 total 30.02 last 88.22)
 - open/close 3525 (time 1786338959.24 total 40.03 last 95.23)
 - open/close 4424 (time 1786338969.25 total 50.04 last 89.84)
 - open/close 5425 (time 1786338979.26 total 60.04 last 100.04)
 - open/close 6336 (time 1786338989.26 total 70.04 last 91.09)
 - open/close 7108 (time 1786338999.26 total 80.05 last 77.18)
 - open/close 8020 (time 1786339009.26 total 90.05 last 91.19)
 - open/close 8759 (time 1786339019.28 total 100.07 last 73.75)
 - open/close 9557 (time 1786339029.29 total 110.08 last 79.74)
total: 10000 open/close in 116.14 seconds: 86.10 ops/second
 - unlinked 0 (time 1786339040 ; total 0 ; last 0)
total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second
 - open/close 948 (time 1786339131.62 total 10.01 last 94.69)
 - open/close 1883 (time 1786339141.63 total 20.02 last 93.45)
 - open/close 2795 (time 1786339151.64 total 30.03 last 91.11)
 - open/close 3582 (time 1786339161.69 total 40.08 last 78.29)
 - open/close 4502 (time 1786339171.70 total 50.09 last 91.94)
 - open/close 5387 (time 1786339181.70 total 60.09 last 88.47)
 - open/close 6293 (time 1786339191.71 total 70.10 last 90.49)
 - open/close 7191 (time 1786339201.72 total 80.11 last 89.74)
 - open/close 8021 (time 1786339211.72 total 90.11 last 82.99)
 - open/close 8885 (time 1786339221.72 total 100.11 last 86.40)
 - open/close 9707 (time 1786339231.73 total 110.12 last 82.15)
total: 10000 open/close in 113.48 seconds: 88.12 ops/second
 - unlinked 0 (time 1786339239 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
 - open/close 965 (time 1786339333.38 total 10.00 last 96.47)
 - open/close 1817 (time 1786339343.38 total 20.00 last 85.19)
 - open/close 2666 (time 1786339353.38 total 30.01 last 84.87)
 - open/close 3548 (time 1786339363.39 total 40.01 last 88.18)
 - open/close 4449 (time 1786339373.39 total 50.01 last 90.09)
 - open/close 5271 (time 1786339383.39 total 60.01 last 82.19)
 - open/close 6092 (time 1786339393.39 total 70.02 last 82.06)
 - open/close 6979 (time 1786339403.39 total 80.02 last 88.70)
 - open/close 7885 (time 1786339413.40 total 90.03 last 90.50)
 - open/close 8736 (time 1786339423.41 total 100.03 last 85.05)
 - open/close 9670 (time 1786339433.41 total 110.04 last 93.35)
total: 10000 open/close in 114.01 seconds: 87.71 ops/second
 - unlinked 0 (time 1786339441 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 897 (time 1786339539.34 total 10.01 last 89.63)
 - open/close 1773 (time 1786339549.34 total 20.01 last 87.60)
 - open/close 2609 (time 1786339559.35 total 30.01 last 83.55)
 - open/close 3498 (time 1786339569.36 total 40.03 last 88.79)
 - open/close 4382 (time 1786339579.36 total 50.03 last 88.38)
 - open/close 5275 (time 1786339589.36 total 60.03 last 89.27)
 - open/close 6129 (time 1786339599.37 total 70.04 last 85.35)
 - open/close 6852 (time 1786339609.37 total 80.04 last 72.27)
 - open/close 7760 (time 1786339619.37 total 90.04 last 90.79)
 - open/close 8602 (time 1786339629.38 total 100.05 last 84.16)
 - open/close 9444 (time 1786339639.38 total 110.05 last 84.19)
total: 10000 open/close in 116.27 seconds: 86.01 ops/second
 - unlinked 0 (time 1786339649 ; total 0 ; last 0)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
 - open/close 791 (time 1786339750.14 total 10.00 last 79.07)
 - open/close 1520 (time 1786339760.15 total 20.01 last 72.84)
 - open/close 2370 (time 1786339770.16 total 30.02 last 84.95)
 - open/close 3234 (time 1786339780.16 total 40.02 last 86.38)
 - open/close 4093 (time 1786339790.16 total 50.03 last 85.85)
 - open/close 4913 (time 1786339800.17 total 60.04 last 81.92)
 - open/close 5705 (time 1786339810.19 total 70.06 last 79.04)
 - open/close 6597 (time 1786339820.20 total 80.06 last 89.17)
 - open/close 7473 (time 1786339830.26 total 90.12 last 87.07)
 - open/close 8337 (time 1786339840.26 total 100.12 last 86.40)
 - open/close 9045 (time 1786339850.26 total 110.12 last 70.77)
 - open/close 9689 (time 1786339860.27 total 120.13 last 64.35)
total: 10000 open/close in 123.57 seconds: 80.92 ops/second
 - unlinked 0 (time 1786339868 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 906 (time 1786339967.15 total 10.01 last 90.47)
 - open/close 1759 (time 1786339977.15 total 20.02 last 85.24)
 - open/close 2677 (time 1786339987.15 total 30.02 last 91.79)
 - open/close 3595 (time 1786339997.16 total 40.03 last 91.76)
 - open/close 4443 (time 1786340007.16 total 50.03 last 84.75)
 - open/close 5351 (time 1786340017.17 total 60.04 last 90.76)
 - open/close 6117 (time 1786340027.18 total 70.05 last 76.53)
 - open/close 6938 (time 1786340037.19 total 80.05 last 82.02)
 - open/close 7700 (time 1786340047.20 total 90.07 last 76.12)
 - open/close 8501 (time 1786340057.20 total 100.07 last 80.05)
 - open/close 9426 (time 1786340067.20 total 110.07 last 92.49)
total: 10000 open/close in 115.97 seconds: 86.23 ops/second
 - unlinked 0 (time 1786340077 ; total 0 ; last 0)
total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg356-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg356-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.156@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg356-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all
pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1
Started lustre-OST0000
oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg356-server: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg356-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg356-client.virtnet:  -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg356-client: oleg356-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg356-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg356-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.156@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 autodegrade=on

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.156@tcp sys.timeout=20 autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.156@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg356-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all
pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1
Started lustre-OST0000
oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg356-server: oleg356-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg356-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg356-client.virtnet:  -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg356-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg356-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server
stop mds service on oleg356-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server

unloading modules via unload_modules_local on: 'oleg356-server'
oleg356-server: oleg356-server.virtnet: executing unload_modules_local
oleg356-server: 
modules unloaded.
pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2
pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2
pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2
pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2