== conf-sanity test 69: replace an OST with the same index ========================================================== 19:01:03 (1781218863)
start mds service on oleg312-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing load_modules_local
oleg312-server: Loading modules from /home/green/git/lustre-release/lustre
oleg312-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy
oleg312-server: detected 4 online CPUs by sysfs
oleg312-server: MODOPTS_LIBCFS=
oleg312-server: Force libcfs to create 2 CPU partitions
oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg312-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg312-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 624 (time 1781218946.72 total 10.01 last 62.36)
 - open/close 1268 (time 1781218956.74 total 20.03 last 64.26)
 - open/close 1969 (time 1781218966.74 total 30.03 last 70.09)
 - open/close 2604 (time 1781218976.75 total 40.03 last 63.49)
 - open/close 3251 (time 1781218986.76 total 50.05 last 64.60)
 - open/close 3974 (time 1781218996.77 total 60.06 last 72.22)
 - open/close 4594 (time 1781219006.81 total 70.09 last 61.79)
 - open/close 5253 (time 1781219016.82 total 80.11 last 65.81)
 - open/close 5896 (time 1781219026.85 total 90.13 last 64.14)
 - open/close 6506 (time 1781219036.85 total 100.13 last 60.99)
 - open/close 7194 (time 1781219046.87 total 110.16 last 68.62)
 - open/close 7796 (time 1781219056.89 total 120.17 last 60.10)
 - open/close 8415 (time 1781219066.90 total 130.19 last 61.82)
 - open/close 9000 (time 1781219076.90 total 140.19 last 58.49)
 - open/close 9649 (time 1781219086.91 total 150.19 last 64.87)
total: 10000 open/close in 157.11 seconds: 63.65 ops/second
 - unlinked 0 (time 1781219098 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 672 (time 1781219211.26 total 10.01 last 67.16)
 - open/close 1382 (time 1781219221.27 total 20.02 last 70.93)
 - open/close 2042 (time 1781219231.27 total 30.02 last 65.95)
 - open/close 2660 (time 1781219241.28 total 40.03 last 61.76)
 - open/close 3204 (time 1781219251.29 total 50.04 last 54.35)
 - open/close 3735 (time 1781219261.29 total 60.04 last 53.09)
 - open/close 4203 (time 1781219271.30 total 70.05 last 46.78)
 - open/close 4805 (time 1781219281.30 total 80.05 last 60.17)
 - open/close 5388 (time 1781219291.31 total 90.06 last 58.26)
 - open/close 6050 (time 1781219301.31 total 100.06 last 66.17)
 - open/close 6733 (time 1781219311.31 total 110.06 last 68.29)
 - open/close 7430 (time 1781219321.32 total 120.07 last 69.67)
 - open/close 8063 (time 1781219331.32 total 130.07 last 63.29)
 - open/close 8726 (time 1781219341.33 total 140.08 last 66.25)
 - open/close 9424 (time 1781219351.34 total 150.09 last 69.73)
total: 10000 open/close in 159.40 seconds: 62.74 ops/second
 - unlinked 0 (time 1781219365 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 619 (time 1781219478.22 total 10.01 last 61.82)
 - open/close 1258 (time 1781219488.23 total 20.02 last 63.86)
 - open/close 2008 (time 1781219498.23 total 30.02 last 74.98)
 - open/close 2688 (time 1781219508.23 total 40.02 last 67.99)
 - open/close 3364 (time 1781219518.25 total 50.04 last 67.51)
 - open/close 4044 (time 1781219528.25 total 60.04 last 67.96)
 - open/close 4711 (time 1781219538.27 total 70.06 last 66.58)
 - open/close 5311 (time 1781219548.28 total 80.07 last 59.92)
 - open/close 5925 (time 1781219558.29 total 90.08 last 61.34)
 - open/close 6586 (time 1781219568.32 total 100.11 last 65.94)
 - open/close 7195 (time 1781219578.32 total 110.11 last 60.85)
 - open/close 7820 (time 1781219588.33 total 120.12 last 62.45)
 - open/close 8397 (time 1781219598.34 total 130.12 last 57.68)
 - open/close 9072 (time 1781219608.34 total 140.13 last 67.45)
 - open/close 9760 (time 1781219618.35 total 150.14 last 68.73)
total: 10000 open/close in 153.37 seconds: 65.20 ops/second
 - unlinked 0 (time 1781219625 ; total 0 ; last 0)
total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second
 - open/close 628 (time 1781219744.07 total 10.00 last 62.77)
 - open/close 1259 (time 1781219754.07 total 20.00 last 63.10)
 - open/close 1973 (time 1781219764.07 total 30.01 last 71.37)
 - open/close 2615 (time 1781219774.08 total 40.02 last 64.14)
 - open/close 3229 (time 1781219784.09 total 50.03 last 61.33)
 - open/close 3879 (time 1781219794.10 total 60.04 last 64.94)
 - open/close 4527 (time 1781219804.11 total 70.04 last 64.76)
 - open/close 5150 (time 1781219814.14 total 80.08 last 62.09)
 - open/close 5739 (time 1781219824.15 total 90.09 last 58.85)
 - open/close 6368 (time 1781219834.16 total 100.10 last 62.81)
 - open/close 7034 (time 1781219844.17 total 110.11 last 66.55)
 - open/close 7733 (time 1781219854.17 total 120.11 last 69.90)
 - open/close 8429 (time 1781219864.17 total 130.11 last 69.57)
 - open/close 9074 (time 1781219874.18 total 140.12 last 64.47)
 - open/close 9747 (time 1781219884.19 total 150.12 last 67.25)
total: 10000 open/close in 154.70 seconds: 64.64 ops/second
 - unlinked 0 (time 1781219893 ; total 0 ; last 0)
total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second
 - open/close 591 (time 1781220009.24 total 10.04 last 58.87)
 - open/close 1237 (time 1781220019.26 total 20.06 last 64.46)
 - open/close 1878 (time 1781220029.27 total 30.07 last 64.06)
 - open/close 2542 (time 1781220039.28 total 40.08 last 66.30)
 - open/close 3177 (time 1781220049.30 total 50.10 last 63.34)
 - open/close 3785 (time 1781220059.31 total 60.11 last 60.77)
 - open/close 4384 (time 1781220069.32 total 70.12 last 59.86)
 - open/close 4887 (time 1781220079.32 total 80.12 last 50.28)
 - open/close 5512 (time 1781220089.32 total 90.12 last 62.47)
 - open/close 6096 (time 1781220099.33 total 100.13 last 58.37)
 - open/close 6685 (time 1781220109.35 total 110.15 last 58.79)
 - open/close 7315 (time 1781220119.40 total 120.20 last 62.70)
 - open/close 7987 (time 1781220129.40 total 130.20 last 67.14)
 - open/close 8637 (time 1781220139.41 total 140.21 last 64.98)
 - open/close 9282 (time 1781220149.41 total 150.21 last 64.48)
 - open/close 9892 (time 1781220159.42 total 160.21 last 60.98)
total: 10000 open/close in 162.07 seconds: 61.70 ops/second
 - unlinked 0 (time 1781220165 ; total 0 ; last 0)
total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second
 - open/close 468 (time 1781220281.52 total 10.02 last 46.71)
 - open/close 1114 (time 1781220291.53 total 20.02 last 64.60)
 - open/close 1778 (time 1781220301.53 total 30.02 last 66.40)
 - open/close 2411 (time 1781220311.53 total 40.02 last 63.27)
 - open/close 2981 (time 1781220321.54 total 50.03 last 56.96)
 - open/close 3616 (time 1781220331.57 total 60.06 last 63.30)
 - open/close 4279 (time 1781220341.59 total 70.09 last 66.15)
 - open/close 4904 (time 1781220351.60 total 80.09 last 62.46)
 - open/close 5497 (time 1781220361.60 total 90.10 last 59.27)
 - open/close 6170 (time 1781220371.62 total 100.11 last 67.21)
 - open/close 6866 (time 1781220381.62 total 110.11 last 69.58)
 - open/close 7497 (time 1781220391.63 total 120.12 last 63.03)
 - open/close 8091 (time 1781220401.63 total 130.13 last 59.38)
 - open/close 8748 (time 1781220411.64 total 140.14 last 65.64)
 - open/close 9379 (time 1781220421.65 total 150.15 last 63.02)
total: 10000 open/close in 159.42 seconds: 62.73 ops/second
 - unlinked 0 (time 1781220435 ; total 0 ; last 0)
total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second
 - open/close 678 (time 1781220552.60 total 10.01 last 67.76)
 - open/close 1321 (time 1781220562.63 total 20.03 last 64.14)
 - open/close 1918 (time 1781220572.63 total 30.04 last 59.65)
 - open/close 2584 (time 1781220582.65 total 40.06 last 66.46)
 - open/close 3171 (time 1781220592.66 total 50.07 last 58.65)
 - open/close 3811 (time 1781220602.67 total 60.07 last 63.96)
 - open/close 4465 (time 1781220612.68 total 70.08 last 65.35)
 - open/close 5137 (time 1781220622.69 total 80.10 last 67.10)
 - open/close 5797 (time 1781220632.70 total 90.11 last 65.94)
 - open/close 6368 (time 1781220642.70 total 100.11 last 57.09)
 - open/close 6983 (time 1781220652.71 total 110.12 last 61.45)
 - open/close 7672 (time 1781220662.72 total 120.13 last 68.83)
 - open/close 8332 (time 1781220672.73 total 130.13 last 65.97)
 - open/close 8953 (time 1781220682.74 total 140.14 last 62.04)
 - open/close 9546 (time 1781220692.75 total 150.15 last 59.23)
total: 10000 open/close in 157.86 seconds: 63.35 ops/second
 - unlinked 0 (time 1781220704 ; total 0 ; last 0)
total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second
 - open/close 543 (time 1781220815.24 total 10.00 last 54.28)
 - open/close 1152 (time 1781220825.26 total 20.02 last 60.81)
 - open/close 1809 (time 1781220835.27 total 30.03 last 65.64)
 - open/close 2420 (time 1781220845.28 total 40.04 last 60.99)
 - open/close 3027 (time 1781220855.29 total 50.05 last 60.64)
 - open/close 3705 (time 1781220865.30 total 60.06 last 67.74)
 - open/close 4346 (time 1781220875.32 total 70.08 last 63.97)
 - open/close 5031 (time 1781220885.33 total 80.09 last 68.44)
 - open/close 5759 (time 1781220895.33 total 90.10 last 72.78)
 - open/close 6351 (time 1781220905.34 total 100.10 last 59.17)
 - open/close 7003 (time 1781220915.35 total 110.11 last 65.16)
 - open/close 7692 (time 1781220925.36 total 120.12 last 68.83)
 - open/close 8367 (time 1781220935.38 total 130.14 last 67.33)
 - open/close 8996 (time 1781220945.39 total 140.15 last 62.85)
 - open/close 9604 (time 1781220955.40 total 150.16 last 60.72)
total: 10000 open/close in 156.67 seconds: 63.83 ops/second
 - unlinked 0 (time 1781220966 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 587 (time 1781221081.06 total 10.00 last 58.69)
 - open/close 1256 (time 1781221091.06 total 20.01 last 66.86)
 - open/close 1903 (time 1781221101.07 total 30.01 last 64.67)
 - open/close 2587 (time 1781221111.08 total 40.02 last 68.32)
 - open/close 3227 (time 1781221121.08 total 50.02 last 63.99)
 - open/close 3903 (time 1781221131.08 total 60.03 last 67.59)
 - open/close 4535 (time 1781221141.09 total 70.03 last 63.18)
 - open/close 5193 (time 1781221151.09 total 80.03 last 65.78)
 - open/close 5821 (time 1781221161.09 total 90.04 last 62.76)
 - open/close 6441 (time 1781221171.10 total 100.04 last 61.99)
 - open/close 7101 (time 1781221181.10 total 110.05 last 65.94)
 - open/close 7728 (time 1781221191.11 total 120.05 last 62.68)
 - open/close 8418 (time 1781221201.12 total 130.06 last 68.94)
 - open/close 8971 (time 1781221211.12 total 140.07 last 55.26)
 - open/close 9705 (time 1781221221.14 total 150.08 last 73.30)
total: 10000 open/close in 155.95 seconds: 64.12 ops/second
 - unlinked 0 (time 1781221231 ; total 0 ; last 0)
total: 10000 unlinks in 98 seconds: 102.040817 unlinks/second
 - open/close 660 (time 1781221343.86 total 10.00 last 65.98)
 - open/close 1316 (time 1781221353.88 total 20.02 last 65.47)
 - open/close 1950 (time 1781221363.90 total 30.04 last 63.28)
 - open/close 2648 (time 1781221373.91 total 40.04 last 69.79)
 - open/close 3337 (time 1781221383.92 total 50.06 last 68.77)
 - open/close 3960 (time 1781221393.94 total 60.07 last 62.23)
 - open/close 4637 (time 1781221403.96 total 70.09 last 67.57)
 - open/close 5387 (time 1781221413.96 total 80.10 last 74.96)
 - open/close 6050 (time 1781221423.97 total 90.11 last 66.21)
 - open/close 6722 (time 1781221433.98 total 100.12 last 67.15)
 - open/close 7418 (time 1781221444.00 total 110.14 last 69.49)
 - open/close 8106 (time 1781221454.02 total 120.16 last 68.66)
 - open/close 8715 (time 1781221464.04 total 130.18 last 60.77)
 - open/close 9316 (time 1781221474.05 total 140.19 last 60.04)
 - open/close 9996 (time 1781221484.08 total 150.22 last 67.80)
total: 10000 open/close in 150.26 seconds: 66.55 ops/second
 - unlinked 0 (time 1781221489 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.112@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg312-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
On OST0, 17126 used inodes rc=0
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.112@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.112@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.112@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.112@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.203.112@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg312-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1
Started lustre-OST0000
oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg312-server: oleg312-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg312-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg312-client.virtnet:  -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg312-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg312-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server
stop mds service on oleg312-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server

unloading modules via unload_modules_local on: 'oleg312-server'
oleg312-server: oleg312-server.virtnet: executing unload_modules_local
oleg312-server: 
modules unloaded.
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2
pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2