== conf-sanity test 69: replace an OST with the same index ========================================================== 15:43:05 (1786563785)
start mds service on oleg210-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg210-server'
oleg210-server: oleg210-server.virtnet: executing load_modules_local
oleg210-server: Loading modules from /home/green/git/lustre-release/lustre
oleg210-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg210-server: detected 4 online CPUs by sysfs
oleg210-server: MODOPTS_LIBCFS=
oleg210-server: Force libcfs to create 2 CPU partitions
oleg210-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg210-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg210-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg210-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1
Started lustre-OST0000
oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg210-client.virtnet:  -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 521 (time 1786563895.93 total 10.01 last 52.07)
 - open/close 1111 (time 1786563905.95 total 20.03 last 58.88)
 - open/close 1725 (time 1786563915.95 total 30.03 last 61.38)
 - open/close 2184 (time 1786563925.95 total 40.03 last 45.89)
 - open/close 2728 (time 1786563935.97 total 50.05 last 54.32)
 - open/close 3281 (time 1786563945.98 total 60.05 last 55.26)
 - open/close 3835 (time 1786563955.98 total 70.06 last 55.35)
 - open/close 4502 (time 1786563965.98 total 80.06 last 66.70)
 - open/close 5102 (time 1786563976.00 total 90.08 last 59.91)
 - open/close 5833 (time 1786563986.00 total 100.08 last 73.07)
 - open/close 6466 (time 1786563996.03 total 110.11 last 63.11)
 - open/close 7031 (time 1786564006.05 total 120.12 last 56.43)
 - open/close 7681 (time 1786564016.05 total 130.13 last 64.98)
 - open/close 8246 (time 1786564026.06 total 140.14 last 56.43)
 - open/close 9018 (time 1786564036.06 total 150.14 last 77.18)
 - open/close 9632 (time 1786564046.08 total 160.16 last 61.31)
total: 10000 open/close in 166.64 seconds: 60.01 ops/second
 - unlinked 0 (time 1786564057 ; total 0 ; last 0)
total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second
 - open/close 669 (time 1786564171.57 total 10.02 last 66.77)
 - open/close 1296 (time 1786564181.58 total 20.03 last 62.61)
 - open/close 1974 (time 1786564191.63 total 30.09 last 67.45)
 - open/close 2670 (time 1786564201.64 total 40.09 last 69.56)
 - open/close 3282 (time 1786564211.65 total 50.10 last 61.14)
 - open/close 3910 (time 1786564221.65 total 60.11 last 62.78)
 - open/close 4603 (time 1786564231.65 total 70.11 last 69.29)
 - open/close 5187 (time 1786564241.65 total 80.11 last 58.39)
 - open/close 5813 (time 1786564251.66 total 90.12 last 62.54)
 - open/close 6525 (time 1786564261.67 total 100.13 last 71.14)
 - open/close 7192 (time 1786564271.69 total 110.14 last 66.61)
 - open/close 7882 (time 1786564281.70 total 120.15 last 68.93)
 - open/close 8610 (time 1786564291.70 total 130.15 last 72.80)
 - open/close 9261 (time 1786564301.70 total 140.16 last 65.06)
total: 10000 open/close in 149.82 seconds: 66.75 ops/second
 - unlinked 0 (time 1786564316 ; total 0 ; last 0)
total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second
 - open/close 586 (time 1786564439.25 total 10.01 last 58.54)
 - open/close 1291 (time 1786564449.26 total 20.02 last 70.43)
 - open/close 1986 (time 1786564459.27 total 30.03 last 69.41)
 - open/close 2656 (time 1786564469.29 total 40.05 last 66.90)
 - open/close 3372 (time 1786564479.29 total 50.05 last 71.59)
 - open/close 3993 (time 1786564489.30 total 60.06 last 62.02)
 - open/close 4701 (time 1786564499.31 total 70.07 last 70.73)
 - open/close 5343 (time 1786564509.33 total 80.09 last 64.11)
 - open/close 5930 (time 1786564519.33 total 90.09 last 58.69)
 - open/close 6687 (time 1786564529.33 total 100.09 last 75.68)
 - open/close 7405 (time 1786564539.34 total 110.10 last 71.72)
 - open/close 8007 (time 1786564549.35 total 120.11 last 60.16)
 - open/close 8543 (time 1786564559.36 total 130.12 last 53.55)
 - open/close 9217 (time 1786564569.36 total 140.13 last 67.35)
 - open/close 9932 (time 1786564579.37 total 150.13 last 71.48)
total: 10000 open/close in 150.96 seconds: 66.24 ops/second
 - unlinked 0 (time 1786564584 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 593 (time 1786564697.42 total 10.00 last 59.27)
 - open/close 1378 (time 1786564707.44 total 20.02 last 78.41)
 - open/close 2133 (time 1786564717.44 total 30.02 last 75.46)
 - open/close 2648 (time 1786564727.44 total 40.02 last 51.50)
 - open/close 3196 (time 1786564737.46 total 50.04 last 54.70)
 - open/close 3708 (time 1786564747.47 total 60.05 last 51.13)
 - open/close 4391 (time 1786564757.49 total 70.07 last 68.22)
 - open/close 5035 (time 1786564767.50 total 80.08 last 64.33)
 - open/close 5627 (time 1786564777.50 total 90.08 last 59.17)
 - open/close 6368 (time 1786564787.50 total 100.08 last 74.09)
 - open/close 7016 (time 1786564797.50 total 110.08 last 64.80)
 - open/close 7635 (time 1786564807.51 total 120.09 last 61.85)
 - open/close 8281 (time 1786564817.52 total 130.10 last 64.55)
 - open/close 8939 (time 1786564827.54 total 140.12 last 65.66)
 - open/close 9656 (time 1786564837.55 total 150.13 last 71.61)
total: 10000 open/close in 154.84 seconds: 64.58 ops/second
 - unlinked 0 (time 1786564848 ; total 0 ; last 0)
total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second
 - open/close 555 (time 1786564964.49 total 10.01 last 55.44)
 - open/close 1231 (time 1786564974.51 total 20.03 last 67.49)
 - open/close 1833 (time 1786564984.52 total 30.04 last 60.14)
 - open/close 2497 (time 1786564994.53 total 40.05 last 66.32)
 - open/close 3198 (time 1786565004.55 total 50.06 last 69.99)
 - open/close 3821 (time 1786565014.59 total 60.11 last 62.01)
 - open/close 4609 (time 1786565024.59 total 70.11 last 78.79)
 - open/close 5261 (time 1786565034.60 total 80.12 last 65.13)
 - open/close 6002 (time 1786565044.61 total 90.13 last 74.07)
 - open/close 6597 (time 1786565054.61 total 100.13 last 59.47)
 - open/close 7224 (time 1786565064.62 total 110.14 last 62.67)
 - open/close 7827 (time 1786565074.63 total 120.15 last 60.25)
 - open/close 8486 (time 1786565084.64 total 130.16 last 65.83)
 - open/close 9062 (time 1786565094.64 total 140.16 last 57.57)
 - open/close 9644 (time 1786565104.66 total 150.18 last 58.11)
total: 10000 open/close in 155.21 seconds: 64.43 ops/second
 - unlinked 0 (time 1786565115 ; total 0 ; last 0)
total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second
 - open/close 609 (time 1786565220.64 total 10.01 last 60.87)
 - open/close 1330 (time 1786565230.65 total 20.02 last 72.02)
 - open/close 2045 (time 1786565240.66 total 30.03 last 71.42)
 - open/close 2661 (time 1786565250.67 total 40.03 last 61.58)
 - open/close 3382 (time 1786565260.67 total 50.04 last 72.03)
 - open/close 4000 (time 1786565270.69 total 60.05 last 61.73)
 - open/close 4716 (time 1786565280.69 total 70.06 last 71.56)
 - open/close 5338 (time 1786565290.70 total 80.06 last 62.18)
 - open/close 5960 (time 1786565300.70 total 90.06 last 62.20)
 - open/close 6515 (time 1786565310.70 total 100.06 last 55.49)
 - open/close 7197 (time 1786565320.73 total 110.09 last 68.00)
 - open/close 7849 (time 1786565330.73 total 120.10 last 65.16)
 - open/close 8523 (time 1786565340.75 total 130.12 last 67.28)
 - open/close 9170 (time 1786565350.77 total 140.13 last 64.59)
 - open/close 9781 (time 1786565360.78 total 150.14 last 61.04)
total: 10000 open/close in 153.78 seconds: 65.03 ops/second
 - unlinked 0 (time 1786565369 ; total 0 ; last 0)
total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second
 - open/close 601 (time 1786565486.84 total 10.01 last 60.04)
 - open/close 1254 (time 1786565496.85 total 20.02 last 65.25)
 - open/close 1863 (time 1786565506.86 total 30.03 last 60.85)
 - open/close 2465 (time 1786565516.87 total 40.04 last 60.12)
 - open/close 3043 (time 1786565526.89 total 50.05 last 57.71)
 - open/close 3578 (time 1786565536.89 total 60.06 last 53.49)
 - open/close 4231 (time 1786565546.89 total 70.06 last 65.29)
 - open/close 4935 (time 1786565556.90 total 80.07 last 70.34)
 - open/close 5622 (time 1786565566.91 total 90.08 last 68.63)
 - open/close 6284 (time 1786565576.95 total 100.11 last 65.96)
 - open/close 6953 (time 1786565586.95 total 110.12 last 66.88)
 - open/close 7596 (time 1786565596.96 total 120.13 last 64.23)
 - open/close 8276 (time 1786565606.97 total 130.14 last 67.94)
 - open/close 9010 (time 1786565616.98 total 140.15 last 73.33)
 - open/close 9770 (time 1786565626.99 total 150.15 last 75.96)
total: 10000 open/close in 153.61 seconds: 65.10 ops/second
 - unlinked 0 (time 1786565635 ; total 0 ; last 0)
total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second
 - open/close 705 (time 1786565740.96 total 10.01 last 70.44)
 - open/close 1430 (time 1786565750.97 total 20.01 last 72.45)
 - open/close 2066 (time 1786565760.99 total 30.04 last 63.46)
 - open/close 2716 (time 1786565770.99 total 40.04 last 64.97)
 - open/close 3368 (time 1786565781.00 total 50.05 last 65.14)
 - open/close 3945 (time 1786565791.02 total 60.06 last 57.63)
 - open/close 4615 (time 1786565801.03 total 70.07 last 66.93)
 - open/close 5279 (time 1786565811.03 total 80.08 last 66.37)
 - open/close 5925 (time 1786565821.04 total 90.09 last 64.54)
 - open/close 6576 (time 1786565831.06 total 100.10 last 65.01)
 - open/close 7176 (time 1786565841.06 total 110.11 last 59.98)
 - open/close 7744 (time 1786565851.07 total 120.12 last 56.72)
 - open/close 8403 (time 1786565861.08 total 130.13 last 65.84)
 - open/close 9055 (time 1786565871.09 total 140.14 last 65.15)
 - open/close 9700 (time 1786565881.10 total 150.15 last 64.41)
total: 10000 open/close in 154.44 seconds: 64.75 ops/second
 - unlinked 0 (time 1786565890 ; total 0 ; last 0)
total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second
 - open/close 671 (time 1786565999.68 total 10.00 last 67.08)
 - open/close 1287 (time 1786566009.70 total 20.02 last 61.51)
 - open/close 1996 (time 1786566019.71 total 30.03 last 70.81)
 - open/close 2681 (time 1786566029.72 total 40.04 last 68.45)
 - open/close 3475 (time 1786566039.72 total 50.04 last 79.36)
 - open/close 4184 (time 1786566049.73 total 60.05 last 70.82)
 - open/close 4899 (time 1786566059.74 total 70.06 last 71.45)
 - open/close 5604 (time 1786566069.77 total 80.09 last 70.32)
 - open/close 6242 (time 1786566079.78 total 90.10 last 63.73)
 - open/close 6968 (time 1786566089.78 total 100.10 last 72.56)
 - open/close 7640 (time 1786566099.79 total 110.12 last 67.13)
 - open/close 8389 (time 1786566109.80 total 120.12 last 74.87)
 - open/close 9032 (time 1786566119.80 total 130.12 last 64.26)
 - open/close 9627 (time 1786566129.84 total 140.16 last 59.32)
total: 10000 open/close in 145.76 seconds: 68.60 ops/second
 - unlinked 0 (time 1786566139 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 616 (time 1786566246.21 total 10.01 last 61.53)
 - open/close 1325 (time 1786566256.21 total 20.02 last 70.85)
 - open/close 2036 (time 1786566266.22 total 30.02 last 71.08)
 - open/close 2655 (time 1786566276.22 total 40.02 last 61.90)
 - open/close 3303 (time 1786566286.23 total 50.03 last 64.74)
 - open/close 3995 (time 1786566296.24 total 60.04 last 69.13)
 - open/close 4664 (time 1786566306.25 total 70.05 last 66.83)
 - open/close 5353 (time 1786566316.25 total 80.05 last 68.89)
 - open/close 6015 (time 1786566326.25 total 90.05 last 66.19)
 - open/close 6673 (time 1786566336.25 total 100.06 last 65.78)
 - open/close 7346 (time 1786566346.26 total 110.06 last 67.28)
 - open/close 7988 (time 1786566356.26 total 120.06 last 64.19)
 - open/close 8725 (time 1786566366.31 total 130.11 last 73.31)
 - open/close 9472 (time 1786566376.31 total 140.12 last 74.69)
total: 10000 open/close in 148.50 seconds: 67.34 ops/second
 - unlinked 0 (time 1786566389 ; total 0 ; last 0)
total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg210-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg210-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.110@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.110@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg210-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1
Started lustre-OST0000
oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg210-client.virtnet:  -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre
On OST0, 16206 used inodes rc=0
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg210-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg210-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.202.110@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.202.110@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.110@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.110@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.202.110@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg210-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1
Started lustre-OST0000
oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg210-client.virtnet:  -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg210-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg210-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server
stop mds service on oleg210-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server

unloading modules via unload_modules_local on: 'oleg210-server'
oleg210-server: oleg210-server.virtnet: executing unload_modules_local
oleg210-server: 
modules unloaded.
pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2
pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2
pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2