== conf-sanity test 69: replace an OST with the same index ========================================================== 22:36:31 (1786415791)
start mds service on oleg653-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg653-server'
oleg653-server: oleg653-server.virtnet: executing load_modules_local
oleg653-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg653-server: Loading modules from /home/green/git/lustre-release/lustre
oleg653-server: detected 4 online CPUs by sysfs
oleg653-server: MODOPTS_LIBCFS=
oleg653-server: Force libcfs to create 2 CPU partitions
oleg653-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg653-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg653-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg653-server: oleg653-server.virtnet: executing set_default_debug -1 all
pdsh@oleg653-client: oleg653-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg653-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg653-server: oleg653-server.virtnet: executing set_default_debug -1 all
pdsh@oleg653-client: oleg653-server: ssh exited with exit code 1
Started lustre-OST0000
oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg653-client.virtnet:  -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 837 (time 1786415874.32 total 10.00 last 83.68)
 - open/close 1662 (time 1786415884.33 total 20.01 last 82.46)
 - open/close 2468 (time 1786415894.34 total 30.02 last 80.52)
 - open/close 3244 (time 1786415904.35 total 40.03 last 77.50)
 - open/close 4007 (time 1786415914.35 total 50.04 last 76.25)
 - open/close 4726 (time 1786415924.37 total 60.05 last 71.82)
 - open/close 5502 (time 1786415934.37 total 70.06 last 77.53)
 - open/close 6327 (time 1786415944.38 total 80.06 last 82.49)
 - open/close 7122 (time 1786415954.38 total 90.06 last 79.49)
 - open/close 7893 (time 1786415964.38 total 100.06 last 77.09)
 - open/close 8733 (time 1786415974.38 total 110.06 last 83.99)
 - open/close 9582 (time 1786415984.39 total 120.07 last 84.84)
total: 10000 open/close in 125.09 seconds: 79.94 ops/second
 - unlinked 0 (time 1786415993 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 798 (time 1786416088.57 total 10.00 last 79.78)
 - open/close 1593 (time 1786416098.58 total 20.01 last 79.47)
 - open/close 2409 (time 1786416108.59 total 30.02 last 81.47)
 - open/close 3250 (time 1786416118.60 total 40.03 last 84.03)
 - open/close 4123 (time 1786416128.61 total 50.04 last 87.25)
 - open/close 4919 (time 1786416138.62 total 60.04 last 79.54)
 - open/close 5744 (time 1786416148.62 total 70.05 last 82.46)
 - open/close 6494 (time 1786416158.64 total 80.07 last 74.85)
 - open/close 7247 (time 1786416168.65 total 90.07 last 75.27)
 - open/close 7948 (time 1786416178.65 total 100.08 last 70.07)
 - open/close 8748 (time 1786416188.67 total 110.10 last 79.86)
 - open/close 9580 (time 1786416198.67 total 120.10 last 83.16)
total: 10000 open/close in 125.66 seconds: 79.58 ops/second
 - unlinked 0 (time 1786416208 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 755 (time 1786416305.16 total 10.00 last 75.46)
 - open/close 1558 (time 1786416315.17 total 20.01 last 80.24)
 - open/close 2338 (time 1786416325.17 total 30.02 last 77.95)
 - open/close 3108 (time 1786416335.18 total 40.02 last 76.98)
 - open/close 3904 (time 1786416345.18 total 50.03 last 79.56)
 - open/close 4728 (time 1786416355.19 total 60.04 last 82.32)
 - open/close 5469 (time 1786416365.19 total 70.04 last 74.10)
 - open/close 6141 (time 1786416375.19 total 80.04 last 67.17)
 - open/close 6964 (time 1786416385.22 total 90.06 last 82.13)
 - open/close 7776 (time 1786416395.22 total 100.06 last 81.19)
 - open/close 8590 (time 1786416405.22 total 110.07 last 81.34)
 - open/close 9377 (time 1786416415.23 total 120.08 last 78.67)
total: 10000 open/close in 128.34 seconds: 77.92 ops/second
 - unlinked 0 (time 1786416427 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 823 (time 1786416524.13 total 10.00 last 82.26)
 - open/close 1694 (time 1786416534.14 total 20.01 last 87.02)
 - open/close 2493 (time 1786416544.15 total 30.03 last 79.75)
 - open/close 3284 (time 1786416554.16 total 40.04 last 79.08)
 - open/close 4062 (time 1786416564.16 total 50.04 last 77.79)
 - open/close 4768 (time 1786416574.19 total 60.07 last 70.37)
 - open/close 5629 (time 1786416584.19 total 70.07 last 86.10)
 - open/close 6429 (time 1786416594.20 total 80.08 last 79.92)
 - open/close 7166 (time 1786416604.21 total 90.09 last 73.65)
 - open/close 7960 (time 1786416614.22 total 100.10 last 79.31)
 - open/close 8597 (time 1786416624.22 total 110.10 last 63.67)
 - open/close 9366 (time 1786416634.25 total 120.12 last 76.72)
total: 10000 open/close in 127.74 seconds: 78.28 ops/second
 - unlinked 0 (time 1786416645 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 780 (time 1786416743.42 total 10.01 last 77.95)
 - open/close 1661 (time 1786416753.42 total 20.01 last 88.09)
 - open/close 2424 (time 1786416763.43 total 30.02 last 76.22)
 - open/close 3175 (time 1786416773.43 total 40.02 last 75.09)
 - open/close 3904 (time 1786416783.45 total 50.04 last 72.78)
 - open/close 4767 (time 1786416793.46 total 60.05 last 86.23)
 - open/close 5627 (time 1786416803.47 total 70.05 last 85.92)
 - open/close 6474 (time 1786416813.48 total 80.06 last 84.63)
 - open/close 7338 (time 1786416823.48 total 90.07 last 86.36)
 - open/close 8190 (time 1786416833.50 total 100.08 last 85.08)
 - open/close 9035 (time 1786416843.50 total 110.09 last 84.47)
 - open/close 9842 (time 1786416853.50 total 120.09 last 80.66)
total: 10000 open/close in 122.07 seconds: 81.92 ops/second
 - unlinked 0 (time 1786416859 ; total 0 ; last 0)
total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second
 - open/close 768 (time 1786416953.58 total 10.01 last 76.76)
 - open/close 1579 (time 1786416963.59 total 20.01 last 81.03)
 - open/close 2404 (time 1786416973.61 total 30.04 last 82.31)
 - open/close 3237 (time 1786416983.61 total 40.04 last 83.26)
 - open/close 4044 (time 1786416993.62 total 50.05 last 80.65)
 - open/close 4831 (time 1786417003.62 total 60.05 last 78.69)
 - open/close 5686 (time 1786417013.62 total 70.05 last 85.47)
 - open/close 6515 (time 1786417023.64 total 80.06 last 82.81)
 - open/close 7383 (time 1786417033.65 total 90.08 last 86.66)
 - open/close 8225 (time 1786417043.66 total 100.09 last 84.13)
 - open/close 9051 (time 1786417053.67 total 110.10 last 82.48)
 - open/close 9894 (time 1786417063.69 total 120.11 last 84.20)
total: 10000 open/close in 121.37 seconds: 82.39 ops/second
 - unlinked 0 (time 1786417068 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 844 (time 1786417166.06 total 10.00 last 84.38)
 - open/close 1604 (time 1786417176.07 total 20.02 last 75.90)
 - open/close 2443 (time 1786417186.07 total 30.02 last 83.89)
 - open/close 3222 (time 1786417196.08 total 40.02 last 77.88)
 - open/close 4018 (time 1786417206.09 total 50.04 last 79.48)
 - open/close 4669 (time 1786417216.10 total 60.05 last 65.02)
 - open/close 5488 (time 1786417226.12 total 70.06 last 81.81)
 - open/close 6261 (time 1786417236.12 total 80.06 last 77.26)
 - open/close 6920 (time 1786417246.13 total 90.07 last 65.86)
 - open/close 7657 (time 1786417256.13 total 100.07 last 73.69)
 - open/close 8431 (time 1786417266.13 total 110.07 last 77.40)
 - open/close 9202 (time 1786417276.14 total 120.08 last 77.03)
total: 10000 open/close in 129.96 seconds: 76.95 ops/second
 - unlinked 0 (time 1786417290 ; total 0 ; last 0)
total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second
 - open/close 787 (time 1786417384.71 total 10.02 last 78.56)
 - open/close 1625 (time 1786417394.72 total 20.02 last 83.76)
 - open/close 2433 (time 1786417404.73 total 30.03 last 80.70)
 - open/close 3228 (time 1786417414.74 total 40.04 last 79.43)
 - open/close 3904 (time 1786417424.74 total 50.05 last 67.56)
 - open/close 4765 (time 1786417434.74 total 60.05 last 86.09)
 - open/close 5580 (time 1786417444.75 total 70.06 last 81.46)
 - open/close 6368 (time 1786417454.76 total 80.06 last 78.74)
 - open/close 7249 (time 1786417464.76 total 90.07 last 88.05)
 - open/close 8088 (time 1786417474.77 total 100.07 last 83.88)
 - open/close 8954 (time 1786417484.77 total 110.08 last 86.52)
 - open/close 9698 (time 1786417494.78 total 120.09 last 74.34)
total: 10000 open/close in 123.83 seconds: 80.76 ops/second
 - unlinked 0 (time 1786417502 ; total 0 ; last 0)
total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second
 - open/close 807 (time 1786417593.92 total 10.01 last 80.58)
 - open/close 1672 (time 1786417603.93 total 20.03 last 86.40)
 - open/close 2515 (time 1786417613.94 total 30.04 last 84.17)
 - open/close 3355 (time 1786417623.95 total 40.05 last 83.91)
 - open/close 4168 (time 1786417633.96 total 50.06 last 81.24)
 - open/close 4905 (time 1786417643.97 total 60.07 last 73.66)
 - open/close 5568 (time 1786417653.97 total 70.07 last 66.29)
 - open/close 6269 (time 1786417663.97 total 80.07 last 70.08)
 - open/close 6998 (time 1786417673.98 total 90.08 last 72.85)
 - open/close 7724 (time 1786417683.98 total 100.08 last 72.59)
 - open/close 8390 (time 1786417693.99 total 110.09 last 66.52)
 - open/close 8990 (time 1786417703.99 total 120.09 last 59.99)
 - open/close 9701 (time 1786417714.01 total 130.11 last 70.99)
total: 10000 open/close in 134.05 seconds: 74.60 ops/second
 - unlinked 0 (time 1786417722 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 756 (time 1786417817.76 total 10.00 last 75.56)
 - open/close 1454 (time 1786417827.77 total 20.01 last 69.78)
 - open/close 2286 (time 1786417837.77 total 30.01 last 83.16)
 - open/close 3135 (time 1786417847.78 total 40.02 last 84.81)
 - open/close 3940 (time 1786417857.79 total 50.03 last 80.45)
 - open/close 4769 (time 1786417867.79 total 60.03 last 82.88)
 - open/close 5559 (time 1786417877.82 total 70.06 last 78.80)
 - open/close 6372 (time 1786417887.82 total 80.06 last 81.27)
 - open/close 7202 (time 1786417897.84 total 90.08 last 82.84)
 - open/close 8065 (time 1786417907.86 total 100.10 last 86.16)
 - open/close 8928 (time 1786417917.87 total 110.11 last 86.20)
 - open/close 9712 (time 1786417927.87 total 120.11 last 78.36)
total: 10000 open/close in 123.63 seconds: 80.89 ops/second
 - unlinked 0 (time 1786417936 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg653-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg653-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg653-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.153@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.153@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg653-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg653-server: oleg653-server.virtnet: executing set_default_debug -1 all
pdsh@oleg653-client: oleg653-server: ssh exited with exit code 1
Started lustre-OST0000
oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg653-server: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg653-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg653-client.virtnet:  -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre
On OST0, 17222 used inodes rc=0
pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg653-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg653-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg653-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.153@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.153@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.153@tcp autodegrade=on sys.timeout=20

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.206.153@tcp autodegrade=on sys.timeout=20 autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.206.153@tcp
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg653-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg653-server: oleg653-server.virtnet: executing set_default_debug -1 all
pdsh@oleg653-client: oleg653-server: ssh exited with exit code 1
Started lustre-OST0000
oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg653-server: oleg653-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg653-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg653-client.virtnet:  -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg653-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg653-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg653-server
stop mds service on oleg653-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg653-server

unloading modules via unload_modules_local on: 'oleg653-server'
oleg653-server: oleg653-server.virtnet: executing unload_modules_local
oleg653-server: 
modules unloaded.
pdsh@oleg653-client: oleg653-client: ssh exited with exit code 2
pdsh@oleg653-client: oleg653-server: ssh exited with exit code 2
pdsh@oleg653-client: oleg653-server: ssh exited with exit code 2
pdsh@oleg653-client: oleg653-client: ssh exited with exit code 2