== conf-sanity test 69: replace an OST with the same index ========================================================== 02:11:06 (1786169466)
start mds service on oleg135-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg135-server'
oleg135-server: oleg135-server.virtnet: executing load_modules_local
oleg135-server: Loading modules from /home/green/git/lustre-release/lustre
oleg135-server: detected 4 online CPUs by sysfs
oleg135-server: MODOPTS_LIBCFS=
oleg135-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg135-server: Force libcfs to create 2 CPU partitions
oleg135-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg135-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg135-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all
pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg135-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all
pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1
Started lustre-OST0000
oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg135-client.virtnet:  -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 881 (time 1786169555.29 total 10.01 last 88.05)
 - open/close 1566 (time 1786169565.31 total 20.02 last 68.37)
 - open/close 2358 (time 1786169575.31 total 30.03 last 79.16)
 - open/close 3164 (time 1786169585.32 total 40.04 last 80.51)
 - open/close 3882 (time 1786169595.32 total 50.04 last 71.80)
 - open/close 4622 (time 1786169605.33 total 60.05 last 73.93)
 - open/close 5344 (time 1786169615.33 total 70.05 last 72.19)
 - open/close 6054 (time 1786169625.34 total 80.05 last 70.99)
 - open/close 6844 (time 1786169635.34 total 90.06 last 78.95)
 - open/close 7695 (time 1786169645.34 total 100.06 last 85.08)
 - open/close 8511 (time 1786169655.35 total 110.07 last 81.52)
 - open/close 9319 (time 1786169665.36 total 120.08 last 80.75)
total: 10000 open/close in 128.86 seconds: 77.60 ops/second
 - unlinked 0 (time 1786169679 ; total 0 ; last 0)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 692 (time 1786169777.63 total 10.01 last 69.14)
 - open/close 1478 (time 1786169787.64 total 20.01 last 78.58)
 - open/close 2254 (time 1786169797.65 total 30.03 last 77.47)
 - open/close 3004 (time 1786169807.66 total 40.03 last 74.97)
 - open/close 3660 (time 1786169817.68 total 50.05 last 65.49)
 - open/close 4323 (time 1786169827.68 total 60.06 last 66.26)
 - open/close 5059 (time 1786169837.70 total 70.08 last 73.46)
 - open/close 5820 (time 1786169847.75 total 80.13 last 75.72)
 - open/close 6572 (time 1786169857.75 total 90.13 last 75.20)
 - open/close 7295 (time 1786169867.76 total 100.14 last 72.22)
 - open/close 8032 (time 1786169877.77 total 110.15 last 73.64)
 - open/close 8730 (time 1786169887.77 total 120.15 last 69.79)
 - open/close 9504 (time 1786169897.79 total 130.16 last 77.27)
total: 10000 open/close in 137.09 seconds: 72.95 ops/second
 - unlinked 0 (time 1786169909 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 729 (time 1786170010.17 total 10.01 last 72.82)
 - open/close 1385 (time 1786170020.18 total 20.02 last 65.54)
 - open/close 2058 (time 1786170030.20 total 30.04 last 67.19)
 - open/close 2793 (time 1786170040.20 total 40.04 last 73.48)
 - open/close 3479 (time 1786170050.22 total 50.06 last 68.45)
 - open/close 4176 (time 1786170060.25 total 60.09 last 69.53)
 - open/close 4848 (time 1786170070.26 total 70.10 last 67.12)
 - open/close 5500 (time 1786170080.27 total 80.10 last 65.16)
 - open/close 6133 (time 1786170090.28 total 90.11 last 63.25)
 - open/close 6887 (time 1786170100.28 total 100.12 last 75.35)
 - open/close 7663 (time 1786170110.29 total 110.12 last 77.56)
 - open/close 8436 (time 1786170120.29 total 120.13 last 77.25)
 - open/close 9234 (time 1786170130.30 total 130.14 last 79.73)
 - open/close 9943 (time 1786170140.31 total 140.15 last 70.84)
total: 10000 open/close in 140.90 seconds: 70.97 ops/second
 - unlinked 0 (time 1786170146 ; total 1 ; last 1)
total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second
 - open/close 707 (time 1786170244.62 total 10.01 last 70.64)
 - open/close 1362 (time 1786170254.62 total 20.01 last 65.48)
 - open/close 1949 (time 1786170264.62 total 30.01 last 58.68)
 - open/close 2553 (time 1786170274.63 total 40.02 last 60.36)
 - open/close 3280 (time 1786170284.64 total 50.03 last 72.65)
 - open/close 4061 (time 1786170294.64 total 60.03 last 78.09)
 - open/close 4809 (time 1786170304.64 total 70.03 last 74.78)
 - open/close 5498 (time 1786170314.65 total 80.04 last 68.83)
 - open/close 6038 (time 1786170324.66 total 90.05 last 53.95)
 - open/close 6693 (time 1786170334.67 total 100.06 last 65.46)
 - open/close 7467 (time 1786170344.67 total 110.06 last 77.39)
 - open/close 8292 (time 1786170354.67 total 120.06 last 82.49)
 - open/close 8961 (time 1786170364.68 total 130.07 last 66.84)
 - open/close 9712 (time 1786170374.68 total 140.07 last 75.09)
total: 10000 open/close in 144.10 seconds: 69.40 ops/second
 - unlinked 0 (time 1786170383 ; total 0 ; last 0)
total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second
 - open/close 696 (time 1786170487.27 total 10.01 last 69.54)
 - open/close 1478 (time 1786170497.28 total 20.02 last 78.12)
 - open/close 2232 (time 1786170507.29 total 30.02 last 75.37)
 - open/close 3104 (time 1786170517.29 total 40.03 last 87.14)
 - open/close 3874 (time 1786170527.31 total 50.05 last 76.88)
 - open/close 4626 (time 1786170537.31 total 60.05 last 75.18)
 - open/close 5323 (time 1786170547.32 total 70.06 last 69.62)
 - open/close 5991 (time 1786170557.34 total 80.08 last 66.68)
 - open/close 6730 (time 1786170567.36 total 90.10 last 73.75)
 - open/close 7477 (time 1786170577.38 total 100.11 last 74.59)
 - open/close 8192 (time 1786170587.39 total 110.12 last 71.44)
 - open/close 8845 (time 1786170597.39 total 120.13 last 65.26)
 - open/close 9488 (time 1786170607.40 total 130.14 last 64.22)
total: 10000 open/close in 137.70 seconds: 72.62 ops/second
 - unlinked 0 (time 1786170619 ; total 0 ; last 0)
total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second
 - open/close 697 (time 1786170732.80 total 10.00 last 69.69)
 - open/close 1321 (time 1786170742.81 total 20.01 last 62.34)
 - open/close 2064 (time 1786170752.82 total 30.02 last 74.25)
 - open/close 2821 (time 1786170762.83 total 40.03 last 75.63)
 - open/close 3414 (time 1786170772.83 total 50.04 last 59.26)
 - open/close 3976 (time 1786170782.84 total 60.04 last 56.17)
 - open/close 4752 (time 1786170792.84 total 70.04 last 77.60)
 - open/close 5438 (time 1786170802.85 total 80.05 last 68.53)
 - open/close 6140 (time 1786170812.85 total 90.06 last 70.16)
 - open/close 6762 (time 1786170822.88 total 100.08 last 62.05)
 - open/close 7338 (time 1786170832.88 total 110.08 last 57.58)
 - open/close 8055 (time 1786170842.89 total 120.09 last 71.67)
 - open/close 8685 (time 1786170852.89 total 130.09 last 62.96)
 - open/close 9357 (time 1786170862.90 total 140.10 last 67.17)
total: 10000 open/close in 148.58 seconds: 67.30 ops/second
 - unlinked 0 (time 1786170876 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 585 (time 1786170984.35 total 10.02 last 58.39)
 - open/close 1201 (time 1786170994.37 total 20.04 last 61.49)
 - open/close 1866 (time 1786171004.37 total 30.04 last 66.47)
 - open/close 2479 (time 1786171014.39 total 40.06 last 61.21)
 - open/close 3048 (time 1786171024.40 total 50.07 last 56.82)
 - open/close 3691 (time 1786171034.41 total 60.08 last 64.25)
 - open/close 4414 (time 1786171044.42 total 70.09 last 72.24)
 - open/close 5066 (time 1786171054.42 total 80.09 last 65.20)
 - open/close 5783 (time 1786171064.43 total 90.10 last 71.61)
 - open/close 6431 (time 1786171074.43 total 100.10 last 64.80)
 - open/close 7089 (time 1786171084.43 total 110.10 last 65.78)
 - open/close 7767 (time 1786171094.44 total 120.11 last 67.73)
 - open/close 8381 (time 1786171104.46 total 130.13 last 61.27)
 - open/close 8992 (time 1786171114.47 total 140.14 last 61.04)
 - open/close 9517 (time 1786171124.48 total 150.15 last 52.48)
total: 10000 open/close in 157.34 seconds: 63.56 ops/second
 - unlinked 0 (time 1786171137 ; total 0 ; last 0)
total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second
 - open/close 596 (time 1786171246.85 total 10.00 last 59.60)
 - open/close 1243 (time 1786171256.87 total 20.02 last 64.58)
 - open/close 1918 (time 1786171266.87 total 30.03 last 67.46)
 - open/close 2620 (time 1786171276.88 total 40.03 last 70.18)
 - open/close 3305 (time 1786171286.88 total 50.03 last 68.48)
 - open/close 3963 (time 1786171296.89 total 60.04 last 65.74)
 - open/close 4570 (time 1786171306.89 total 70.04 last 60.67)
 - open/close 5170 (time 1786171316.90 total 80.05 last 59.96)
 - open/close 5892 (time 1786171326.90 total 90.05 last 72.20)
 - open/close 6648 (time 1786171336.91 total 100.06 last 75.51)
 - open/close 7383 (time 1786171346.92 total 110.07 last 73.46)
 - open/close 7975 (time 1786171356.92 total 120.07 last 59.20)
 - open/close 8687 (time 1786171366.92 total 130.07 last 71.20)
 - open/close 9485 (time 1786171376.93 total 140.08 last 79.72)
total: 10000 open/close in 148.59 seconds: 67.30 ops/second
 - unlinked 0 (time 1786171391 ; total 0 ; last 0)
total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second
 - open/close 637 (time 1786171500.73 total 10.01 last 63.64)
 - open/close 1331 (time 1786171510.74 total 20.02 last 69.34)
 - open/close 1944 (time 1786171520.77 total 30.05 last 61.12)
 - open/close 2530 (time 1786171530.78 total 40.06 last 58.53)
 - open/close 3250 (time 1786171540.79 total 50.07 last 71.94)
 - open/close 3921 (time 1786171550.80 total 60.08 last 67.04)
 - open/close 4598 (time 1786171560.80 total 70.08 last 67.66)
 - open/close 5267 (time 1786171570.81 total 80.09 last 66.84)
 - open/close 5868 (time 1786171580.83 total 90.11 last 59.99)
 - open/close 6408 (time 1786171590.84 total 100.11 last 53.97)
 - open/close 7147 (time 1786171600.85 total 110.13 last 73.81)
 - open/close 7686 (time 1786171610.86 total 120.14 last 53.82)
 - open/close 8274 (time 1786171620.86 total 130.14 last 58.78)
 - open/close 8831 (time 1786171630.89 total 140.17 last 55.57)
 - open/close 9556 (time 1786171640.89 total 150.17 last 72.49)
total: 10000 open/close in 156.56 seconds: 63.87 ops/second
 - unlinked 0 (time 1786171654 ; total 0 ; last 0)
total: 10000 unlinks in 109 seconds: 91.743118 unlinks/second
 - open/close 624 (time 1786171779.26 total 10.01 last 62.35)
 - open/close 1157 (time 1786171789.27 total 20.02 last 53.27)
 - open/close 1723 (time 1786171799.28 total 30.02 last 56.55)
 - open/close 2335 (time 1786171809.29 total 40.04 last 61.11)
 - open/close 2917 (time 1786171819.29 total 50.04 last 58.18)
 - open/close 3681 (time 1786171829.31 total 60.05 last 76.31)
 - open/close 4406 (time 1786171839.31 total 70.06 last 72.46)
 - open/close 5102 (time 1786171849.31 total 80.06 last 69.59)
 - open/close 5760 (time 1786171859.31 total 90.06 last 65.79)
 - open/close 6396 (time 1786171869.32 total 100.07 last 63.56)
 - open/close 7058 (time 1786171879.36 total 110.11 last 65.94)
 - open/close 7825 (time 1786171889.36 total 120.11 last 76.70)
 - open/close 8673 (time 1786171899.37 total 130.12 last 84.73)
 - open/close 9511 (time 1786171909.37 total 140.12 last 83.77)
total: 10000 open/close in 147.12 seconds: 67.97 ops/second
 - unlinked 0 (time 1786171920 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg135-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg135-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.135@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.135@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg135-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all
pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1
Started lustre-OST0000
oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg135-server: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg135-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg135-client.virtnet:  -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg135-client: oleg135-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg135-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg135-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.135@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.135@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.135@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.135@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.201.135@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg135-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all
pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1
Started lustre-OST0000
oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg135-server: oleg135-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg135-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg135-client.virtnet:  -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg135-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg135-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server
stop mds service on oleg135-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server

unloading modules via unload_modules_local on: 'oleg135-server'
oleg135-server: oleg135-server.virtnet: executing unload_modules_local
oleg135-server: 
modules unloaded.
pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2
pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2
pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2
pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2