== conf-sanity test 69: replace an OST with the same index ========================================================== 03:35:03 (1784187303)
start mds service on oleg406-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg406-server'
oleg406-server: oleg406-server.virtnet: executing load_modules_local
oleg406-server: Loading modules from /home/green/git/lustre-release/lustre
oleg406-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg406-server: detected 4 online CPUs by sysfs
oleg406-server: MODOPTS_LIBCFS=
oleg406-server: Force libcfs to create 2 CPU partitions
oleg406-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg406-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg406-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all
pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg406-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all
pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1
Started lustre-OST0000
oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg406-client.virtnet:  -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 626 (time 1784187387.79 total 10.00 last 62.60)
 - open/close 1252 (time 1784187397.81 total 20.02 last 62.49)
 - open/close 1840 (time 1784187407.84 total 30.04 last 58.64)
 - open/close 2491 (time 1784187417.85 total 40.05 last 65.03)
 - open/close 3134 (time 1784187427.85 total 50.06 last 64.28)
 - open/close 3783 (time 1784187437.86 total 60.07 last 64.85)
 - open/close 4376 (time 1784187447.87 total 70.07 last 59.26)
 - open/close 4954 (time 1784187457.87 total 80.08 last 57.76)
 - open/close 5655 (time 1784187467.89 total 90.10 last 69.96)
 - open/close 6213 (time 1784187477.90 total 100.10 last 55.78)
 - open/close 6842 (time 1784187487.92 total 110.13 last 62.76)
 - open/close 7429 (time 1784187497.93 total 120.14 last 58.65)
 - open/close 8044 (time 1784187507.93 total 130.14 last 61.48)
 - open/close 8681 (time 1784187517.94 total 140.15 last 63.65)
 - open/close 9285 (time 1784187527.95 total 150.15 last 60.35)
 - open/close 9926 (time 1784187537.96 total 160.17 last 64.00)
total: 10000 open/close in 161.25 seconds: 62.01 ops/second
 - unlinked 0 (time 1784187543 ; total 0 ; last 0)
total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second
 - open/close 685 (time 1784187658.64 total 10.01 last 68.44)
 - open/close 1308 (time 1784187668.64 total 20.01 last 62.26)
 - open/close 1962 (time 1784187678.64 total 30.02 last 65.39)
 - open/close 2630 (time 1784187688.65 total 40.02 last 66.74)
 - open/close 3279 (time 1784187698.65 total 50.02 last 64.90)
 - open/close 3955 (time 1784187708.65 total 60.03 last 67.59)
 - open/close 4562 (time 1784187718.66 total 70.03 last 60.66)
 - open/close 5211 (time 1784187728.67 total 80.04 last 64.86)
 - open/close 5873 (time 1784187738.67 total 90.04 last 66.16)
 - open/close 6486 (time 1784187748.69 total 100.07 last 61.16)
 - open/close 7061 (time 1784187758.70 total 110.07 last 57.49)
 - open/close 7618 (time 1784187768.71 total 120.08 last 55.62)
 - open/close 8115 (time 1784187778.71 total 130.09 last 49.68)
 - open/close 8653 (time 1784187788.73 total 140.10 last 53.71)
 - open/close 9174 (time 1784187798.75 total 150.12 last 52.01)
 - open/close 9751 (time 1784187808.76 total 160.13 last 57.62)
total: 10000 open/close in 164.77 seconds: 60.69 ops/second
 - unlinked 0 (time 1784187817 ; total 0 ; last 0)
total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second
 - open/close 536 (time 1784187934.83 total 10.01 last 53.55)
 - open/close 1140 (time 1784187944.84 total 20.02 last 60.32)
 - open/close 1774 (time 1784187954.85 total 30.03 last 63.35)
 - open/close 2382 (time 1784187964.85 total 40.03 last 60.77)
 - open/close 2960 (time 1784187974.86 total 50.04 last 57.77)
 - open/close 3556 (time 1784187984.86 total 60.04 last 59.59)
 - open/close 4222 (time 1784187994.87 total 70.05 last 66.57)
 - open/close 4861 (time 1784188004.87 total 80.05 last 63.87)
 - open/close 5399 (time 1784188014.87 total 90.05 last 53.79)
 - open/close 6057 (time 1784188024.90 total 100.08 last 65.60)
 - open/close 6665 (time 1784188034.91 total 110.09 last 60.74)
 - open/close 7260 (time 1784188044.91 total 120.09 last 59.48)
 - open/close 7870 (time 1784188054.92 total 130.10 last 60.96)
 - open/close 8529 (time 1784188064.93 total 140.11 last 65.84)
 - open/close 9123 (time 1784188074.95 total 150.13 last 59.26)
 - open/close 9783 (time 1784188084.96 total 160.14 last 65.94)
total: 10000 open/close in 163.56 seconds: 61.14 ops/second
 - unlinked 0 (time 1784188093 ; total 0 ; last 0)
total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second
 - open/close 654 (time 1784188209.95 total 10.01 last 65.36)
 - open/close 1239 (time 1784188219.97 total 20.02 last 58.40)
 - open/close 1867 (time 1784188229.97 total 30.03 last 62.78)
 - open/close 2556 (time 1784188239.98 total 40.03 last 68.86)
 - open/close 3218 (time 1784188249.98 total 50.03 last 66.19)
 - open/close 3761 (time 1784188259.98 total 60.04 last 54.29)
 - open/close 4419 (time 1784188269.99 total 70.04 last 65.79)
 - open/close 5030 (time 1784188279.99 total 80.05 last 61.04)
 - open/close 5624 (time 1784188290.01 total 90.06 last 59.32)
 - open/close 6260 (time 1784188300.02 total 100.07 last 63.52)
 - open/close 6904 (time 1784188310.03 total 110.08 last 64.37)
 - open/close 7587 (time 1784188320.03 total 120.09 last 68.24)
 - open/close 8211 (time 1784188330.05 total 130.11 last 62.28)
 - open/close 8833 (time 1784188340.07 total 140.12 last 62.09)
 - open/close 9466 (time 1784188350.08 total 150.13 last 63.23)
total: 10000 open/close in 158.34 seconds: 63.15 ops/second
 - unlinked 0 (time 1784188363 ; total 0 ; last 0)
total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second
 - open/close 567 (time 1784188486.52 total 10.00 last 56.68)
 - open/close 1165 (time 1784188496.55 total 20.03 last 59.67)
 - open/close 1773 (time 1784188506.56 total 30.04 last 60.70)
 - open/close 2343 (time 1784188516.57 total 40.05 last 56.93)
 - open/close 2880 (time 1784188526.59 total 50.07 last 53.63)
 - open/close 3400 (time 1784188536.59 total 60.07 last 51.99)
 - open/close 3974 (time 1784188546.59 total 70.07 last 57.38)
 - open/close 4629 (time 1784188556.59 total 80.07 last 65.49)
 - open/close 5214 (time 1784188566.60 total 90.08 last 58.46)
 - open/close 5828 (time 1784188576.60 total 100.08 last 61.36)
 - open/close 6537 (time 1784188586.61 total 110.09 last 70.88)
 - open/close 7193 (time 1784188596.62 total 120.10 last 65.51)
 - open/close 7781 (time 1784188606.63 total 130.11 last 58.77)
 - open/close 8437 (time 1784188616.67 total 140.15 last 65.31)
 - open/close 8961 (time 1784188626.68 total 150.16 last 52.35)
 - open/close 9501 (time 1784188636.69 total 160.17 last 53.97)
total: 10000 open/close in 168.61 seconds: 59.31 ops/second
 - unlinked 0 (time 1784188652 ; total 0 ; last 0)
total: 10000 unlinks in 110 seconds: 90.909088 unlinks/second
 - open/close 596 (time 1784188777.47 total 10.01 last 59.55)
 - open/close 1223 (time 1784188787.50 total 20.04 last 62.52)
 - open/close 1831 (time 1784188797.51 total 30.04 last 60.76)
 - open/close 2474 (time 1784188807.52 total 40.06 last 64.21)
 - open/close 3128 (time 1784188817.53 total 50.07 last 65.32)
 - open/close 3699 (time 1784188827.54 total 60.08 last 57.04)
 - open/close 4254 (time 1784188837.55 total 70.09 last 55.46)
 - open/close 4854 (time 1784188847.55 total 80.09 last 59.99)
 - open/close 5437 (time 1784188857.56 total 90.10 last 58.25)
 - open/close 6046 (time 1784188867.59 total 100.13 last 60.72)
 - open/close 6533 (time 1784188877.61 total 110.14 last 48.61)
 - open/close 7077 (time 1784188887.63 total 120.16 last 54.30)
 - open/close 7686 (time 1784188897.63 total 130.17 last 60.87)
 - open/close 8297 (time 1784188907.65 total 140.18 last 61.00)
 - open/close 8905 (time 1784188917.67 total 150.20 last 60.69)
 - open/close 9489 (time 1784188927.68 total 160.21 last 58.33)
total: 10000 open/close in 168.42 seconds: 59.38 ops/second
 - unlinked 0 (time 1784188940 ; total 0 ; last 0)
total: 10000 unlinks in 108 seconds: 92.592590 unlinks/second
 - open/close 616 (time 1784189061.77 total 10.00 last 61.59)
 - open/close 1242 (time 1784189071.77 total 20.00 last 62.58)
 - open/close 1838 (time 1784189081.78 total 30.01 last 59.55)
 - open/close 2416 (time 1784189091.78 total 40.01 last 57.79)
 - open/close 3013 (time 1784189101.79 total 50.02 last 59.66)
 - open/close 3659 (time 1784189111.79 total 60.02 last 64.59)
 - open/close 4275 (time 1784189121.81 total 70.05 last 61.45)
 - open/close 4852 (time 1784189131.83 total 80.06 last 57.63)
 - open/close 5486 (time 1784189141.84 total 90.07 last 63.33)
 - open/close 6060 (time 1784189151.85 total 100.08 last 57.35)
 - open/close 6643 (time 1784189161.86 total 110.09 last 58.24)
 - open/close 7309 (time 1784189171.86 total 120.10 last 66.55)
 - open/close 7823 (time 1784189181.88 total 130.11 last 51.32)
 - open/close 8326 (time 1784189191.93 total 140.16 last 50.05)
 - open/close 8723 (time 1784189201.95 total 150.18 last 39.63)
 - open/close 9268 (time 1784189211.95 total 160.18 last 54.50)
 - open/close 9855 (time 1784189221.96 total 170.19 last 58.64)
total: 10000 open/close in 172.29 seconds: 58.04 ops/second
 - unlinked 0 (time 1784189228 ; total 1 ; last 1)
total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second
 - open/close 613 (time 1784189347.65 total 10.01 last 61.22)
 - open/close 1224 (time 1784189357.65 total 20.02 last 61.08)
 - open/close 1748 (time 1784189367.65 total 30.02 last 52.39)
 - open/close 2377 (time 1784189377.66 total 40.02 last 62.87)
 - open/close 2974 (time 1784189387.68 total 50.04 last 59.59)
 - open/close 3582 (time 1784189397.68 total 60.05 last 60.76)
 - open/close 4050 (time 1784189407.69 total 70.05 last 46.78)
 - open/close 4669 (time 1784189417.71 total 80.08 last 61.77)
 - open/close 5240 (time 1784189427.72 total 90.09 last 57.01)
 - open/close 5873 (time 1784189437.75 total 100.11 last 63.17)
 - open/close 6473 (time 1784189447.76 total 110.13 last 59.90)
 - open/close 7051 (time 1784189457.77 total 120.13 last 57.78)
 - open/close 7645 (time 1784189467.78 total 130.14 last 59.34)
 - open/close 8219 (time 1784189477.78 total 140.14 last 57.40)
 - open/close 8866 (time 1784189487.79 total 150.15 last 64.64)
 - open/close 9434 (time 1784189497.79 total 160.16 last 56.75)
 - open/close 9952 (time 1784189507.81 total 170.17 last 51.73)
total: 10000 open/close in 170.79 seconds: 58.55 ops/second
 - unlinked 0 (time 1784189513 ; total 0 ; last 0)
total: 10000 unlinks in 111 seconds: 90.090088 unlinks/second
 - open/close 552 (time 1784189640.49 total 10.01 last 55.12)
 - open/close 1103 (time 1784189650.49 total 20.02 last 55.09)
 - open/close 1616 (time 1784189660.49 total 30.02 last 51.29)
 - open/close 2165 (time 1784189670.49 total 40.02 last 54.89)
 - open/close 2688 (time 1784189680.50 total 50.03 last 52.25)
 - open/close 3275 (time 1784189690.50 total 60.03 last 58.68)
 - open/close 3863 (time 1784189700.51 total 70.04 last 58.78)
 - open/close 4446 (time 1784189710.53 total 80.06 last 58.17)
 - open/close 4992 (time 1784189720.53 total 90.06 last 54.60)
 - open/close 5472 (time 1784189730.54 total 100.07 last 47.96)
 - open/close 5934 (time 1784189740.54 total 110.07 last 46.19)
 - open/close 6459 (time 1784189750.54 total 120.07 last 52.49)
 - open/close 6973 (time 1784189760.55 total 130.07 last 51.38)
 - open/close 7535 (time 1784189770.56 total 140.09 last 56.12)
 - open/close 8096 (time 1784189780.56 total 150.09 last 56.09)
 - open/close 8580 (time 1784189790.59 total 160.12 last 48.27)
 - open/close 9151 (time 1784189800.60 total 170.13 last 57.01)
 - open/close 9740 (time 1784189810.61 total 180.14 last 58.88)
total: 10000 open/close in 184.49 seconds: 54.20 ops/second
 - unlinked 0 (time 1784189820 ; total 0 ; last 0)
total: 10000 unlinks in 130 seconds: 76.923080 unlinks/second
 - open/close 579 (time 1784189965.71 total 10.01 last 57.84)
 - open/close 1186 (time 1784189975.72 total 20.02 last 60.63)
 - open/close 1750 (time 1784189985.74 total 30.04 last 56.29)
 - open/close 2312 (time 1784189995.75 total 40.05 last 56.13)
 - open/close 2912 (time 1784190005.78 total 50.08 last 59.85)
 - open/close 3476 (time 1784190015.80 total 60.10 last 56.29)
 - open/close 4049 (time 1784190025.82 total 70.12 last 57.15)
 - open/close 4615 (time 1784190035.83 total 80.13 last 56.53)
 - open/close 5113 (time 1784190045.84 total 90.14 last 49.78)
 - open/close 5640 (time 1784190055.85 total 100.15 last 52.64)
 - open/close 6200 (time 1784190065.87 total 110.17 last 55.87)
 - open/close 6818 (time 1784190075.88 total 120.18 last 61.75)
 - open/close 7411 (time 1784190085.89 total 130.19 last 59.25)
 - open/close 7974 (time 1784190095.89 total 140.19 last 56.28)
 - open/close 8582 (time 1784190105.90 total 150.20 last 60.75)
 - open/close 9161 (time 1784190115.91 total 160.21 last 57.86)
 - open/close 9746 (time 1784190125.92 total 170.22 last 58.43)
total: 10000 open/close in 175.20 seconds: 57.08 ops/second
 - unlinked 0 (time 1784190135 ; total 0 ; last 0)
total: 10000 unlinks in 113 seconds: 88.495575 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg406-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg406-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.106@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.106@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg406-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all
pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1
Started lustre-OST0000
oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg406-server: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg406-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg406-client.virtnet:  -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg406-client: oleg406-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg406-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg406-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.106@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.106@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.204.106@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.204.106@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.204.106@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg406-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all
pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1
Started lustre-OST0000
oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg406-server: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg406-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg406-client.virtnet:  -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg406-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg406-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server
stop mds service on oleg406-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg406-server

unloading modules via unload_modules_local on: 'oleg406-server'
oleg406-server: oleg406-server.virtnet: executing unload_modules_local
oleg406-server: 
modules unloaded.
pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2
pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2
pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2
pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2