== conf-sanity test 69: replace an OST with the same index ========================================================== 08:36:12 (1787142972)
start mds service on oleg115-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg115-server'
oleg115-server: oleg115-server.virtnet: executing load_modules_local
oleg115-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy
oleg115-server: Loading modules from /home/green/git/lustre-release/lustre
oleg115-server: detected 4 online CPUs by sysfs
oleg115-server: MODOPTS_LIBCFS=
oleg115-server: Force libcfs to create 2 CPU partitions
oleg115-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg115-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg115-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all
pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0
start ost1 service on oleg115-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all
pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1
Started lustre-OST0000
oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
mount lustre  on /mnt/lustre.....
Starting client: oleg115-client.virtnet:  -o user_xattr,flock 192.168.201.115@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 793 (time 1787143064.67 total 10.01 last 79.21)
 - open/close 1682 (time 1787143074.67 total 20.01 last 88.89)
 - open/close 2532 (time 1787143084.68 total 30.02 last 84.95)
 - open/close 3388 (time 1787143094.69 total 40.03 last 85.52)
 - open/close 4329 (time 1787143104.69 total 50.03 last 94.09)
 - open/close 5313 (time 1787143114.70 total 60.04 last 98.32)
 - open/close 6308 (time 1787143124.70 total 70.04 last 99.43)
 - open/close 7328 (time 1787143134.71 total 80.05 last 101.96)
 - open/close 8356 (time 1787143144.71 total 90.05 last 102.78)
 - open/close 9433 (time 1787143154.72 total 100.06 last 107.61)
total: 10000 open/close in 106.28 seconds: 94.09 ops/second
 - unlinked 0 (time 1787143165 ; total 0 ; last 0)
total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second
 - open/close 942 (time 1787143248.23 total 10.01 last 94.10)
 - open/close 1841 (time 1787143258.24 total 20.01 last 89.87)
 - open/close 2879 (time 1787143268.24 total 30.02 last 103.75)
 - open/close 3780 (time 1787143278.25 total 40.02 last 90.06)
 - open/close 4489 (time 1787143288.31 total 50.08 last 70.48)
 - open/close 5462 (time 1787143298.32 total 60.10 last 97.12)
 - open/close 6088 (time 1787143308.33 total 70.10 last 62.58)
 - open/close 7080 (time 1787143318.33 total 80.11 last 99.13)
 - open/close 8093 (time 1787143328.34 total 90.11 last 101.27)
 - open/close 9093 (time 1787143338.34 total 100.12 last 99.94)
total: 10000 open/close in 109.26 seconds: 91.52 ops/second
 - unlinked 0 (time 1787143351 ; total 0 ; last 0)
total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second
 - open/close 992 (time 1787143439.15 total 10.00 last 99.20)
 - open/close 1886 (time 1787143449.16 total 20.00 last 89.36)
 - open/close 2750 (time 1787143459.16 total 30.01 last 86.36)
 - open/close 3778 (time 1787143469.17 total 40.02 last 102.69)
 - open/close 4758 (time 1787143479.18 total 50.03 last 97.95)
 - open/close 5701 (time 1787143489.18 total 60.03 last 94.26)
 - open/close 6585 (time 1787143499.20 total 70.05 last 88.25)
 - open/close 7455 (time 1787143509.20 total 80.05 last 86.98)
 - open/close 8403 (time 1787143519.21 total 90.06 last 94.74)
 - open/close 9246 (time 1787143529.22 total 100.07 last 84.22)
total: 10000 open/close in 107.94 seconds: 92.65 ops/second
 - unlinked 0 (time 1787143542 ; total 0 ; last 0)
total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second
 - open/close 537 (time 1787143627.15 total 10.01 last 53.63)
 - open/close 1352 (time 1787143637.15 total 20.02 last 81.46)
 - open/close 2348 (time 1787143647.15 total 30.02 last 99.59)
 - open/close 3180 (time 1787143657.16 total 40.03 last 83.11)
 - open/close 4125 (time 1787143667.17 total 50.04 last 94.42)
 - open/close 4981 (time 1787143677.18 total 60.04 last 85.56)
 - open/close 5849 (time 1787143687.18 total 70.04 last 86.79)
 - open/close 6944 (time 1787143697.18 total 80.05 last 109.45)
 - open/close 7840 (time 1787143707.20 total 90.06 last 89.48)
 - open/close 8725 (time 1787143717.20 total 100.07 last 88.45)
 - open/close 9581 (time 1787143727.21 total 110.07 last 85.55)
total: 10000 open/close in 114.07 seconds: 87.66 ops/second
 - unlinked 0 (time 1787143735 ; total 0 ; last 0)
total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second
 - open/close 928 (time 1787143821.80 total 10.00 last 92.79)
 - open/close 1815 (time 1787143831.82 total 20.02 last 88.58)
 - open/close 2660 (time 1787143841.82 total 30.02 last 84.46)
 - open/close 3512 (time 1787143851.83 total 40.03 last 85.11)
 - open/close 4321 (time 1787143861.85 total 50.04 last 80.78)
 - open/close 5263 (time 1787143871.85 total 60.05 last 94.17)
 - open/close 6059 (time 1787143881.85 total 70.05 last 79.59)
 - open/close 6950 (time 1787143891.86 total 80.06 last 89.00)
 - open/close 7894 (time 1787143901.86 total 90.06 last 94.40)
 - open/close 8693 (time 1787143911.87 total 100.07 last 79.85)
 - open/close 9540 (time 1787143921.87 total 110.07 last 84.70)
total: 10000 open/close in 116.24 seconds: 86.03 ops/second
 - unlinked 0 (time 1787143933 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 603 (time 1787144026.04 total 10.00 last 60.27)
 - open/close 1363 (time 1787144036.04 total 20.01 last 75.96)
 - open/close 2268 (time 1787144046.05 total 30.01 last 90.47)
 - open/close 2901 (time 1787144056.06 total 40.03 last 63.22)
 - open/close 3700 (time 1787144066.07 total 50.03 last 79.85)
 - open/close 4597 (time 1787144076.07 total 60.04 last 89.65)
 - open/close 5409 (time 1787144086.08 total 70.05 last 81.11)
 - open/close 6179 (time 1787144096.09 total 80.05 last 76.96)
 - open/close 7130 (time 1787144106.09 total 90.05 last 95.10)
 - open/close 7938 (time 1787144116.10 total 100.06 last 80.71)
 - open/close 8784 (time 1787144126.10 total 110.07 last 84.57)
 - open/close 9570 (time 1787144136.11 total 120.08 last 78.51)
total: 10000 open/close in 125.69 seconds: 79.56 ops/second
 - unlinked 0 (time 1787144146 ; total 0 ; last 0)
total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second
 - open/close 885 (time 1787144236.43 total 10.00 last 88.46)
 - open/close 1763 (time 1787144246.43 total 20.01 last 87.80)
 - open/close 2505 (time 1787144256.44 total 30.01 last 74.14)
 - open/close 3182 (time 1787144266.45 total 40.02 last 67.65)
 - open/close 4005 (time 1787144276.46 total 50.03 last 82.24)
 - open/close 4906 (time 1787144286.46 total 60.03 last 90.04)
 - open/close 5762 (time 1787144296.46 total 70.04 last 85.60)
 - open/close 6655 (time 1787144306.47 total 80.04 last 89.24)
 - open/close 7483 (time 1787144316.48 total 90.05 last 82.74)
 - open/close 8408 (time 1787144326.48 total 100.05 last 92.47)
 - open/close 9342 (time 1787144336.49 total 110.06 last 93.33)
total: 10000 open/close in 117.19 seconds: 85.33 ops/second
 - unlinked 0 (time 1787144347 ; total 0 ; last 0)
total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second
 - open/close 976 (time 1787144430.19 total 10.00 last 97.55)
 - open/close 1886 (time 1787144440.20 total 20.01 last 90.92)
 - open/close 2764 (time 1787144450.20 total 30.02 last 87.77)
 - open/close 3573 (time 1787144460.21 total 40.03 last 80.83)
 - open/close 4477 (time 1787144470.22 total 50.03 last 90.36)
 - open/close 5465 (time 1787144480.22 total 60.04 last 98.74)
 - open/close 6395 (time 1787144490.22 total 70.04 last 92.98)
 - open/close 7302 (time 1787144500.23 total 80.04 last 90.68)
 - open/close 8101 (time 1787144510.23 total 90.05 last 79.86)
 - open/close 9103 (time 1787144520.23 total 100.05 last 100.17)
total: 10000 open/close in 108.36 seconds: 92.29 ops/second
 - unlinked 0 (time 1787144532 ; total 0 ; last 0)
total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second
 - open/close 965 (time 1787144614.10 total 10.00 last 96.50)
 - open/close 1864 (time 1787144624.10 total 20.00 last 89.88)
 - open/close 2717 (time 1787144634.11 total 30.01 last 85.24)
 - open/close 3622 (time 1787144644.11 total 40.01 last 90.49)
 - open/close 4542 (time 1787144654.11 total 50.02 last 91.96)
 - open/close 5484 (time 1787144664.12 total 60.02 last 94.14)
 - open/close 6281 (time 1787144674.12 total 70.02 last 79.68)
 - open/close 7225 (time 1787144684.12 total 80.03 last 94.36)
 - open/close 8203 (time 1787144694.13 total 90.04 last 97.71)
 - open/close 9072 (time 1787144704.14 total 100.05 last 86.81)
 - open/close 9929 (time 1787144714.16 total 110.06 last 85.57)
total: 10000 open/close in 111.08 seconds: 90.03 ops/second
 - unlinked 0 (time 1787144720 ; total 0 ; last 0)
total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second
 - open/close 933 (time 1787144808.96 total 10.01 last 93.23)
 - open/close 1952 (time 1787144818.97 total 20.02 last 101.81)
 - open/close 2941 (time 1787144828.97 total 30.02 last 98.84)
 - open/close 3964 (time 1787144838.98 total 40.03 last 102.19)
 - open/close 4972 (time 1787144848.99 total 50.04 last 100.69)
 - open/close 5907 (time 1787144858.99 total 60.04 last 93.50)
 - open/close 6908 (time 1787144869.00 total 70.04 last 100.09)
 - open/close 7859 (time 1787144879.00 total 80.04 last 95.10)
 - open/close 8813 (time 1787144889.00 total 90.05 last 95.33)
 - open/close 9740 (time 1787144899.01 total 100.06 last 92.66)
total: 10000 open/close in 102.56 seconds: 97.51 ops/second
 - unlinked 0 (time 1787144905 ; total 0 ; last 0)
total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg115-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg115-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg115-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.115@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.115@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg115-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all
pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1
Started lustre-OST0000
oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg115-server: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg115-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg115-client.virtnet:  -o user_xattr,flock 192.168.201.115@tcp:/lustre /mnt/lustre
On OST0, 9878 used inodes rc=0
pdsh@oleg115-client: oleg115-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg115-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg115-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg115-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.115@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.115@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.115@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.201.115@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:mgsnode=192.168.201.115@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg115-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all
pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1
Started lustre-OST0000
oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0
oleg115-server: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg115-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg115-client.virtnet:  -o user_xattr,flock 192.168.201.115@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg115-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg115-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg115-server
stop mds service on oleg115-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg115-server

unloading modules via unload_modules_local on: 'oleg115-server'
oleg115-server: oleg115-server.virtnet: executing unload_modules_local
oleg115-server: 
modules unloaded.
pdsh@oleg115-client: oleg115-client: ssh exited with exit code 2
pdsh@oleg115-client: oleg115-server: ssh exited with exit code 2
pdsh@oleg115-client: oleg115-client: ssh exited with exit code 2
pdsh@oleg115-client: oleg115-server: ssh exited with exit code 2