== conf-sanity test 69: replace an OST with the same index ========================================================== 10:57:12 (1779202632)
start mds service on oleg228-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg228-server'
oleg228-server: oleg228-server.virtnet: executing load_modules_local
oleg228-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy
oleg228-server: Loading modules from /home/green/git/lustre-release/lustre
oleg228-server: detected 4 online CPUs by sysfs
oleg228-server: MODOPTS_LIBCFS=
oleg228-server: Force libcfs to create 2 CPU partitions
oleg228-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg228-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg228-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all
pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg228-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all
pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1
Started lustre-OST0000
oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg228-client.virtnet:  -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 783 (time 1779202715.96 total 10.00 last 78.27)
 - open/close 1525 (time 1779202725.96 total 20.01 last 74.16)
 - open/close 2290 (time 1779202735.97 total 30.02 last 76.44)
 - open/close 3071 (time 1779202745.97 total 40.02 last 78.08)
 - open/close 3865 (time 1779202755.98 total 50.03 last 79.34)
 - open/close 4643 (time 1779202765.98 total 60.03 last 77.78)
 - open/close 5260 (time 1779202775.99 total 70.04 last 61.66)
 - open/close 5769 (time 1779202785.99 total 80.04 last 50.88)
 - open/close 6371 (time 1779202796.01 total 90.06 last 60.09)
 - open/close 7100 (time 1779202806.02 total 100.06 last 72.86)
 - open/close 7784 (time 1779202816.02 total 110.07 last 68.38)
 - open/close 8464 (time 1779202826.03 total 120.07 last 67.93)
 - open/close 9194 (time 1779202836.04 total 130.08 last 72.93)
 - open/close 9873 (time 1779202846.05 total 140.10 last 67.81)
total: 10000 open/close in 141.75 seconds: 70.55 ops/second
 - unlinked 0 (time 1779202852 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 721 (time 1779202958.71 total 10.01 last 72.03)
 - open/close 1228 (time 1779202968.71 total 20.01 last 50.68)
 - open/close 1908 (time 1779202978.72 total 30.02 last 67.99)
 - open/close 2613 (time 1779202988.72 total 40.02 last 70.49)
 - open/close 3389 (time 1779202998.72 total 50.02 last 77.58)
 - open/close 4161 (time 1779203008.73 total 60.03 last 77.12)
 - open/close 5043 (time 1779203018.73 total 70.03 last 88.17)
 - open/close 5732 (time 1779203028.75 total 80.05 last 68.80)
 - open/close 6510 (time 1779203038.76 total 90.06 last 77.72)
 - open/close 7263 (time 1779203048.76 total 100.06 last 75.30)
 - open/close 8029 (time 1779203058.77 total 110.07 last 76.54)
 - open/close 8693 (time 1779203068.78 total 120.08 last 66.31)
 - open/close 9447 (time 1779203078.78 total 130.08 last 75.37)
total: 10000 open/close in 136.52 seconds: 73.25 ops/second
 - unlinked 0 (time 1779203090 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
 - open/close 727 (time 1779203192.74 total 10.00 last 72.67)
 - open/close 1443 (time 1779203202.74 total 20.01 last 71.59)
 - open/close 2195 (time 1779203212.76 total 30.02 last 75.08)
 - open/close 3008 (time 1779203222.78 total 40.04 last 81.13)
 - open/close 3732 (time 1779203232.79 total 50.05 last 72.33)
 - open/close 4420 (time 1779203242.80 total 60.06 last 68.71)
 - open/close 5081 (time 1779203252.81 total 70.07 last 66.07)
 - open/close 5769 (time 1779203262.81 total 80.07 last 68.80)
 - open/close 6527 (time 1779203272.81 total 90.07 last 75.77)
 - open/close 7217 (time 1779203282.83 total 100.09 last 68.86)
 - open/close 7973 (time 1779203292.84 total 110.10 last 75.58)
 - open/close 8617 (time 1779203302.84 total 120.10 last 64.40)
 - open/close 9374 (time 1779203312.84 total 130.11 last 75.64)
total: 10000 open/close in 137.61 seconds: 72.67 ops/second
 - unlinked 0 (time 1779203325 ; total 0 ; last 0)
total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second
 - open/close 580 (time 1779203428.31 total 10.01 last 57.92)
 - open/close 1257 (time 1779203438.31 total 20.01 last 67.70)
 - open/close 2033 (time 1779203448.32 total 30.02 last 77.55)
 - open/close 2926 (time 1779203458.32 total 40.02 last 89.30)
 - open/close 3752 (time 1779203468.33 total 50.03 last 82.55)
 - open/close 4476 (time 1779203478.34 total 60.04 last 72.34)
 - open/close 5142 (time 1779203488.35 total 70.05 last 66.48)
 - open/close 5697 (time 1779203498.36 total 80.06 last 55.45)
 - open/close 6358 (time 1779203508.40 total 90.10 last 65.85)
 - open/close 7129 (time 1779203518.41 total 100.11 last 77.00)
 - open/close 7838 (time 1779203528.42 total 110.12 last 70.84)
 - open/close 8373 (time 1779203538.42 total 120.12 last 53.50)
 - open/close 8894 (time 1779203548.44 total 130.14 last 52.02)
 - open/close 9498 (time 1779203558.44 total 140.14 last 60.37)
total: 10000 open/close in 148.27 seconds: 67.44 ops/second
 - unlinked 0 (time 1779203572 ; total 0 ; last 0)
total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second
 - open/close 593 (time 1779203695.40 total 10.02 last 59.21)
 - open/close 1309 (time 1779203705.42 total 20.03 last 71.47)
 - open/close 1894 (time 1779203715.42 total 30.04 last 58.48)
 - open/close 2564 (time 1779203725.43 total 40.05 last 66.93)
 - open/close 3303 (time 1779203735.46 total 50.08 last 73.66)
 - open/close 3899 (time 1779203745.49 total 60.10 last 59.47)
 - open/close 4349 (time 1779203755.50 total 70.12 last 44.93)
 - open/close 4972 (time 1779203765.51 total 80.13 last 62.25)
 - open/close 5675 (time 1779203775.53 total 90.15 last 70.14)
 - open/close 6234 (time 1779203785.53 total 100.15 last 55.90)
 - open/close 6883 (time 1779203795.55 total 110.16 last 64.82)
 - open/close 7530 (time 1779203805.56 total 120.18 last 64.62)
 - open/close 8154 (time 1779203815.61 total 130.22 last 62.11)
 - open/close 8857 (time 1779203825.61 total 140.23 last 70.28)
 - open/close 9555 (time 1779203835.61 total 150.23 last 69.79)
total: 10000 open/close in 156.74 seconds: 63.80 ops/second
 - unlinked 0 (time 1779203847 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 797 (time 1779203961.20 total 10.00 last 79.69)
 - open/close 1570 (time 1779203971.22 total 20.02 last 77.16)
 - open/close 2327 (time 1779203981.22 total 30.03 last 75.63)
 - open/close 3119 (time 1779203991.22 total 40.03 last 79.19)
 - open/close 3803 (time 1779204001.23 total 50.03 last 68.40)
 - open/close 4592 (time 1779204011.23 total 60.03 last 78.88)
 - open/close 5411 (time 1779204021.23 total 70.03 last 81.89)
 - open/close 6190 (time 1779204031.24 total 80.05 last 77.79)
 - open/close 6841 (time 1779204041.25 total 90.05 last 65.08)
 - open/close 7655 (time 1779204051.25 total 100.05 last 81.40)
 - open/close 8467 (time 1779204061.25 total 110.05 last 81.19)
 - open/close 9362 (time 1779204071.26 total 120.06 last 89.42)
total: 10000 open/close in 127.95 seconds: 78.15 ops/second
 - unlinked 0 (time 1779204083 ; total 0 ; last 0)
total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second
 - open/close 616 (time 1779204199.50 total 10.01 last 61.53)
 - open/close 1324 (time 1779204209.51 total 20.02 last 70.75)
 - open/close 1906 (time 1779204219.52 total 30.03 last 58.16)
 - open/close 2670 (time 1779204229.52 total 40.03 last 76.34)
 - open/close 3590 (time 1779204239.53 total 50.04 last 91.93)
 - open/close 4498 (time 1779204249.54 total 60.05 last 90.72)
 - open/close 5345 (time 1779204259.55 total 70.06 last 84.63)
 - open/close 6070 (time 1779204269.56 total 80.07 last 72.44)
 - open/close 6601 (time 1779204279.58 total 90.08 last 53.01)
 - open/close 7105 (time 1779204289.58 total 100.09 last 50.39)
 - open/close 7689 (time 1779204299.58 total 110.09 last 58.38)
 - open/close 8352 (time 1779204309.59 total 120.09 last 66.27)
 - open/close 9056 (time 1779204319.61 total 130.12 last 70.25)
 - open/close 9589 (time 1779204329.62 total 140.13 last 53.23)
total: 10000 open/close in 146.71 seconds: 68.16 ops/second
 - unlinked 0 (time 1779204341 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 437 (time 1779204447.28 total 10.02 last 43.63)
 - open/close 782 (time 1779204457.28 total 20.02 last 34.50)
 - open/close 1138 (time 1779204467.29 total 30.02 last 35.59)
 - open/close 1529 (time 1779204477.66 total 40.39 last 37.71)
 - open/close 1951 (time 1779204487.66 total 50.39 last 42.19)
 - open/close 2620 (time 1779204497.66 total 60.39 last 66.88)
 - open/close 3264 (time 1779204507.66 total 70.40 last 64.38)
 - open/close 3889 (time 1779204517.69 total 80.42 last 62.36)
 - open/close 4576 (time 1779204527.69 total 90.42 last 68.70)
 - open/close 5058 (time 1779204537.70 total 100.43 last 48.15)
 - open/close 5620 (time 1779204547.73 total 110.46 last 56.01)
 - open/close 6286 (time 1779204557.75 total 120.49 last 66.46)
 - open/close 6929 (time 1779204567.77 total 130.50 last 64.19)
 - open/close 7530 (time 1779204577.77 total 140.50 last 60.10)
 - open/close 8280 (time 1779204587.78 total 150.51 last 74.94)
 - open/close 8995 (time 1779204597.79 total 160.53 last 71.38)
 - open/close 9667 (time 1779204607.79 total 170.53 last 67.19)
total: 10000 open/close in 175.05 seconds: 57.13 ops/second
 - unlinked 0 (time 1779204616 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 869 (time 1779204712.67 total 10.01 last 86.85)
 - open/close 1629 (time 1779204722.67 total 20.01 last 75.97)
 - open/close 2451 (time 1779204732.68 total 30.02 last 82.12)
 - open/close 3244 (time 1779204742.69 total 40.03 last 79.23)
 - open/close 4006 (time 1779204752.70 total 50.04 last 76.10)
 - open/close 4595 (time 1779204762.70 total 60.04 last 58.89)
 - open/close 5329 (time 1779204772.71 total 70.05 last 73.35)
 - open/close 6009 (time 1779204782.72 total 80.06 last 67.96)
 - open/close 6713 (time 1779204792.72 total 90.06 last 70.40)
 - open/close 7497 (time 1779204802.73 total 100.07 last 78.34)
 - open/close 8302 (time 1779204812.73 total 110.07 last 80.45)
 - open/close 9109 (time 1779204822.73 total 120.08 last 80.68)
 - open/close 9863 (time 1779204832.75 total 130.09 last 75.31)
total: 10000 open/close in 131.68 seconds: 75.94 ops/second
 - unlinked 0 (time 1779204838 ; total 0 ; last 0)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
 - open/close 537 (time 1779204937.29 total 10.02 last 53.59)
 - open/close 1065 (time 1779204947.30 total 20.03 last 52.76)
 - open/close 1812 (time 1779204957.34 total 30.07 last 74.41)
 - open/close 2521 (time 1779204967.36 total 40.09 last 70.76)
 - open/close 3217 (time 1779204977.36 total 50.09 last 69.58)
 - open/close 3929 (time 1779204987.38 total 60.11 last 71.07)
 - open/close 4604 (time 1779204997.40 total 70.13 last 67.37)
 - open/close 5398 (time 1779205007.40 total 80.13 last 79.38)
 - open/close 6229 (time 1779205017.41 total 90.14 last 83.04)
 - open/close 6892 (time 1779205027.42 total 100.14 last 66.24)
 - open/close 7586 (time 1779205037.43 total 110.15 last 69.34)
 - open/close 8320 (time 1779205047.43 total 120.16 last 73.35)
 - open/close 8884 (time 1779205057.44 total 130.17 last 56.35)
 - open/close 9578 (time 1779205067.45 total 140.17 last 69.36)
total: 10000 open/close in 146.52 seconds: 68.25 ops/second
 - unlinked 0 (time 1779205078 ; total 0 ; last 0)
total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg228-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg228-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg228-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.128@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.128@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg228-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all
pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1
Started lustre-OST0000
oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg228-server: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg228-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg228-client.virtnet:  -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg228-client: oleg228-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg228-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg228-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg228-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.128@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.128@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.128@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.128@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.202.128@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg228-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all
pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1
Started lustre-OST0000
oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg228-server: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg228-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg228-client.virtnet:  -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg228-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg228-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg228-server
stop mds service on oleg228-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg228-server

unloading modules via unload_modules_local on: 'oleg228-server'
oleg228-server: oleg228-server.virtnet: executing unload_modules_local
oleg228-server: 
modules unloaded.
pdsh@oleg228-client: oleg228-client: ssh exited with exit code 2
pdsh@oleg228-client: oleg228-server: ssh exited with exit code 2
pdsh@oleg228-client: oleg228-client: ssh exited with exit code 2
pdsh@oleg228-client: oleg228-server: ssh exited with exit code 2