== conf-sanity test 69: replace an OST with the same index ========================================================== 18:42:38 (1787697758)
start mds service on oleg328-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg328-server'
oleg328-server: oleg328-server.virtnet: executing load_modules_local
oleg328-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg328-server: Loading modules from /home/green/git/lustre-release/lustre
oleg328-server: detected 4 online CPUs by sysfs
oleg328-server: MODOPTS_LIBCFS=
oleg328-server: Force libcfs to create 2 CPU partitions
oleg328-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg328-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg328-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg328-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1
Started lustre-OST0000
oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg328-client.virtnet:  -o user_xattr,flock 192.168.203.128@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 563 (time 1787697848.46 total 10.00 last 56.29)
 - open/close 1054 (time 1787697858.47 total 20.01 last 49.04)
 - open/close 1619 (time 1787697868.48 total 30.02 last 56.46)
 - open/close 2216 (time 1787697878.48 total 40.03 last 59.67)
 - open/close 2821 (time 1787697888.49 total 50.03 last 60.45)
 - open/close 3406 (time 1787697898.50 total 60.05 last 58.44)
 - open/close 4055 (time 1787697908.51 total 70.05 last 64.84)
 - open/close 4708 (time 1787697918.52 total 80.06 last 65.26)
 - open/close 5364 (time 1787697928.52 total 90.07 last 65.56)
 - open/close 6000 (time 1787697938.53 total 100.08 last 63.53)
 - open/close 6767 (time 1787697948.55 total 110.09 last 76.60)
 - open/close 7429 (time 1787697958.56 total 120.10 last 66.14)
 - open/close 8061 (time 1787697968.57 total 130.12 last 63.08)
 - open/close 8709 (time 1787697978.58 total 140.13 last 64.75)
 - open/close 9325 (time 1787697988.59 total 150.13 last 61.55)
 - open/close 9951 (time 1787697998.60 total 160.15 last 62.51)
total: 10000 open/close in 160.97 seconds: 62.12 ops/second
 - unlinked 0 (time 1787698004 ; total 1 ; last 1)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 593 (time 1787698115.75 total 10.01 last 59.23)
 - open/close 1260 (time 1787698125.75 total 20.01 last 66.69)
 - open/close 1883 (time 1787698135.75 total 30.01 last 62.30)
 - open/close 2496 (time 1787698145.76 total 40.02 last 61.28)
 - open/close 3165 (time 1787698155.76 total 50.02 last 66.85)
 - open/close 3761 (time 1787698165.78 total 60.05 last 59.47)
 - open/close 4346 (time 1787698175.81 total 70.07 last 58.37)
 - open/close 4792 (time 1787698185.81 total 80.07 last 44.59)
 - open/close 5356 (time 1787698195.82 total 90.08 last 56.34)
 - open/close 5984 (time 1787698205.83 total 100.09 last 62.73)
 - open/close 6584 (time 1787698215.83 total 110.09 last 59.98)
 - open/close 7102 (time 1787698225.87 total 120.13 last 51.62)
 - open/close 7464 (time 1787698235.94 total 130.20 last 35.94)
 - open/close 7952 (time 1787698245.94 total 140.20 last 48.79)
 - open/close 8427 (time 1787698255.96 total 150.22 last 47.42)
 - open/close 9081 (time 1787698265.96 total 160.22 last 65.39)
 - open/close 9672 (time 1787698275.97 total 170.23 last 59.05)
total: 10000 open/close in 175.14 seconds: 57.10 ops/second
 - unlinked 0 (time 1787698285 ; total 0 ; last 0)
total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second
 - open/close 700 (time 1787698396.88 total 10.00 last 69.98)
 - open/close 1363 (time 1787698406.89 total 20.01 last 66.25)
 - open/close 2041 (time 1787698416.90 total 30.02 last 67.76)
 - open/close 2728 (time 1787698426.91 total 40.03 last 68.63)
 - open/close 3308 (time 1787698436.91 total 50.03 last 57.98)
 - open/close 3926 (time 1787698446.92 total 60.04 last 61.74)
 - open/close 4596 (time 1787698456.93 total 70.05 last 66.95)
 - open/close 5238 (time 1787698466.93 total 80.05 last 64.19)
 - open/close 5975 (time 1787698476.93 total 90.05 last 73.67)
 - open/close 6689 (time 1787698486.94 total 100.06 last 71.37)
 - open/close 7388 (time 1787698496.94 total 110.06 last 69.85)
 - open/close 8039 (time 1787698506.95 total 120.07 last 65.08)
 - open/close 8639 (time 1787698516.95 total 130.08 last 59.95)
 - open/close 9247 (time 1787698526.96 total 140.08 last 60.77)
 - open/close 9867 (time 1787698536.97 total 150.09 last 61.96)
total: 10000 open/close in 153.07 seconds: 65.33 ops/second
 - unlinked 0 (time 1787698545 ; total 0 ; last 0)
total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second
 - open/close 673 (time 1787698660.43 total 10.00 last 67.29)
 - open/close 1274 (time 1787698670.44 total 20.00 last 60.08)
 - open/close 1937 (time 1787698680.44 total 30.01 last 66.25)
 - open/close 2612 (time 1787698690.45 total 40.02 last 67.45)
 - open/close 3243 (time 1787698700.45 total 50.02 last 63.10)
 - open/close 3850 (time 1787698710.45 total 60.02 last 60.68)
 - open/close 4514 (time 1787698720.45 total 70.02 last 66.40)
 - open/close 5182 (time 1787698730.46 total 80.03 last 66.78)
 - open/close 5811 (time 1787698740.46 total 90.03 last 62.87)
 - open/close 6469 (time 1787698750.48 total 100.04 last 65.72)
 - open/close 7094 (time 1787698760.48 total 110.05 last 62.49)
 - open/close 7668 (time 1787698770.49 total 120.06 last 57.35)
 - open/close 8416 (time 1787698780.49 total 130.06 last 74.76)
 - open/close 9160 (time 1787698790.52 total 140.09 last 74.16)
 - open/close 9870 (time 1787698800.53 total 150.10 last 70.98)
total: 10000 open/close in 151.91 seconds: 65.83 ops/second
 - unlinked 0 (time 1787698806 ; total 0 ; last 0)
total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second
 - open/close 667 (time 1787698917.47 total 10.00 last 66.69)
 - open/close 1370 (time 1787698927.47 total 20.01 last 70.24)
 - open/close 2087 (time 1787698937.48 total 30.01 last 71.68)
 - open/close 2837 (time 1787698947.49 total 40.03 last 74.90)
 - open/close 3559 (time 1787698957.49 total 50.03 last 72.17)
 - open/close 4246 (time 1787698967.50 total 60.04 last 68.66)
 - open/close 4853 (time 1787698977.50 total 70.04 last 60.69)
 - open/close 5536 (time 1787698987.51 total 80.05 last 68.24)
 - open/close 6199 (time 1787698997.52 total 90.06 last 66.22)
 - open/close 6838 (time 1787699007.54 total 100.08 last 63.79)
 - open/close 7441 (time 1787699017.55 total 110.08 last 60.27)
 - open/close 8086 (time 1787699027.55 total 120.09 last 64.45)
 - open/close 8779 (time 1787699037.57 total 130.11 last 69.16)
 - open/close 9335 (time 1787699047.58 total 140.12 last 55.57)
 - open/close 9979 (time 1787699057.58 total 150.12 last 64.37)
total: 10000 open/close in 150.40 seconds: 66.49 ops/second
 - unlinked 0 (time 1787699062 ; total 0 ; last 0)
total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second
 - open/close 695 (time 1787699164.96 total 10.00 last 69.49)
 - open/close 1426 (time 1787699174.98 total 20.03 last 72.90)
 - open/close 2167 (time 1787699185.00 total 30.04 last 73.99)
 - open/close 2802 (time 1787699195.00 total 40.05 last 63.49)
 - open/close 3519 (time 1787699205.04 total 50.08 last 71.42)
 - open/close 4084 (time 1787699215.05 total 60.09 last 56.45)
 - open/close 4726 (time 1787699225.05 total 70.10 last 64.18)
 - open/close 5331 (time 1787699235.05 total 80.10 last 60.49)
 - open/close 5925 (time 1787699245.07 total 90.12 last 59.30)
 - open/close 6531 (time 1787699255.07 total 100.12 last 60.59)
 - open/close 7168 (time 1787699265.08 total 110.12 last 63.65)
 - open/close 7840 (time 1787699275.08 total 120.13 last 67.16)
 - open/close 8571 (time 1787699285.08 total 130.13 last 73.10)
 - open/close 9292 (time 1787699295.10 total 140.14 last 72.00)
 - open/close 9956 (time 1787699305.10 total 150.15 last 66.38)
total: 10000 open/close in 150.76 seconds: 66.33 ops/second
 - unlinked 0 (time 1787699310 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
 - open/close 646 (time 1787699417.66 total 10.00 last 64.57)
 - open/close 1242 (time 1787699427.67 total 20.01 last 59.55)
 - open/close 1762 (time 1787699437.67 total 30.02 last 51.98)
 - open/close 2271 (time 1787699447.68 total 40.03 last 50.84)
 - open/close 2814 (time 1787699457.69 total 50.04 last 54.25)
 - open/close 3438 (time 1787699467.69 total 60.04 last 62.40)
 - open/close 4065 (time 1787699477.70 total 70.05 last 62.65)
 - open/close 4700 (time 1787699487.70 total 80.05 last 63.47)
 - open/close 5369 (time 1787699497.72 total 90.06 last 66.83)
 - open/close 5975 (time 1787699507.73 total 100.07 last 60.53)
 - open/close 6667 (time 1787699517.74 total 110.09 last 69.08)
 - open/close 7324 (time 1787699527.78 total 120.13 last 65.46)
 - open/close 8017 (time 1787699537.78 total 130.13 last 69.29)
 - open/close 8566 (time 1787699547.79 total 140.13 last 54.86)
 - open/close 9153 (time 1787699557.79 total 150.14 last 58.67)
 - open/close 9809 (time 1787699567.80 total 160.14 last 65.60)
total: 10000 open/close in 162.85 seconds: 61.41 ops/second
 - unlinked 0 (time 1787699574 ; total 0 ; last 0)
total: 10000 unlinks in 95 seconds: 105.263161 unlinks/second
 - open/close 566 (time 1787699684.71 total 10.01 last 56.55)
 - open/close 1185 (time 1787699694.72 total 20.02 last 61.86)
 - open/close 1866 (time 1787699704.73 total 30.02 last 68.06)
 - open/close 2478 (time 1787699714.73 total 40.03 last 61.16)
 - open/close 3129 (time 1787699724.74 total 50.04 last 65.04)
 - open/close 3833 (time 1787699734.75 total 60.05 last 70.34)
 - open/close 4447 (time 1787699744.75 total 70.05 last 61.39)
 - open/close 5043 (time 1787699754.77 total 80.06 last 59.51)
 - open/close 5719 (time 1787699764.78 total 90.08 last 67.48)
 - open/close 6444 (time 1787699774.79 total 100.09 last 72.46)
 - open/close 7125 (time 1787699784.79 total 110.09 last 68.09)
 - open/close 7871 (time 1787699794.79 total 120.09 last 74.57)
 - open/close 8591 (time 1787699804.80 total 130.10 last 71.95)
 - open/close 9300 (time 1787699814.80 total 140.10 last 70.88)
total: 10000 open/close in 149.53 seconds: 66.88 ops/second
 - unlinked 0 (time 1787699829 ; total 0 ; last 0)
total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second
 - open/close 595 (time 1787699940.52 total 10.00 last 59.50)
 - open/close 1273 (time 1787699950.54 total 20.03 last 67.62)
 - open/close 1941 (time 1787699960.56 total 30.04 last 66.69)
 - open/close 2588 (time 1787699970.56 total 40.05 last 64.66)
 - open/close 3240 (time 1787699980.57 total 50.06 last 65.14)
 - open/close 3858 (time 1787699990.58 total 60.06 last 61.78)
 - open/close 4520 (time 1787700000.59 total 70.07 last 66.15)
 - open/close 5191 (time 1787700010.59 total 80.08 last 67.05)
 - open/close 5855 (time 1787700020.60 total 90.08 last 66.36)
 - open/close 6589 (time 1787700030.61 total 100.09 last 73.36)
 - open/close 7292 (time 1787700040.61 total 110.09 last 70.27)
 - open/close 7995 (time 1787700050.62 total 120.10 last 70.26)
 - open/close 8662 (time 1787700060.63 total 130.11 last 66.61)
 - open/close 9313 (time 1787700070.63 total 140.12 last 65.09)
total: 10000 open/close in 149.93 seconds: 66.70 ops/second
 - unlinked 0 (time 1787700085 ; total 0 ; last 0)
total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second
 - open/close 647 (time 1787700195.53 total 10.02 last 64.54)
 - open/close 1338 (time 1787700205.53 total 20.03 last 69.09)
 - open/close 2019 (time 1787700215.56 total 30.05 last 67.92)
 - open/close 2714 (time 1787700225.56 total 40.06 last 69.45)
 - open/close 3436 (time 1787700235.56 total 50.06 last 72.20)
 - open/close 4134 (time 1787700245.57 total 60.07 last 69.73)
 - open/close 4827 (time 1787700255.59 total 70.08 last 69.21)
 - open/close 5457 (time 1787700265.60 total 80.09 last 62.93)
 - open/close 6192 (time 1787700275.60 total 90.10 last 73.46)
 - open/close 6859 (time 1787700285.65 total 100.15 last 66.37)
 - open/close 7615 (time 1787700295.66 total 110.16 last 75.54)
 - open/close 8244 (time 1787700305.68 total 120.17 last 62.80)
 - open/close 8778 (time 1787700315.68 total 130.18 last 53.36)
 - open/close 9495 (time 1787700325.70 total 140.20 last 71.57)
total: 10000 open/close in 147.85 seconds: 67.64 ops/second
 - unlinked 0 (time 1787700337 ; total 0 ; last 0)
total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg328-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg328-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.128@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.128@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg328-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1
Started lustre-OST0000
oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg328-server: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg328-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg328-client.virtnet:  -o user_xattr,flock 192.168.203.128@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg328-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg328-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.203.128@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.203.128@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 mgsnode=192.168.203.128@tcp autodegrade=on

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 mgsnode=192.168.203.128@tcp autodegrade=on autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.203.128@tcp
  lustre:autodegrade=on
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg328-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1
Started lustre-OST0000
oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg328-server: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg328-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg328-client.virtnet:  -o user_xattr,flock 192.168.203.128@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg328-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg328-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server
stop mds service on oleg328-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server

unloading modules via unload_modules_local on: 'oleg328-server'
oleg328-server: oleg328-server.virtnet: executing unload_modules_local
oleg328-server: 
modules unloaded.
pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2
pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2
pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2