== conf-sanity test 69: replace an OST with the same index ========================================================== 17:25:14 (1781817914)
start mds service on oleg144-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
mdt/mdt options: 'mdt_enable_flr_ec=1'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg144-server'
oleg144-server: oleg144-server.virtnet: executing load_modules_local
oleg144-server: Loading modules from /home/green/git/lustre-release/lustre
oleg144-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy
oleg144-server: detected 4 online CPUs by sysfs
oleg144-server: MODOPTS_LIBCFS=
oleg144-server: Force libcfs to create 2 CPU partitions
oleg144-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg144-server: quota/lquota options: 'hash_lqs_cur_bits=3'
oleg144-server: mdt/mdt options: 'mdt_enable_flr_ec=1'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all
pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
start ost1 service on oleg144-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all
pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1
Started lustre-OST0000
oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg144-client.virtnet:  -o user_xattr,flock 192.168.201.144@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11904 inodes available. Want 99872. rc=0
 - open/close 2549 (time 1781817989.23 total 10.00 last 254.80)
 - open/close 4954 (time 1781817999.23 total 20.01 last 240.47)
 - open/close 7213 (time 1781818009.23 total 30.01 last 225.83)
 - open/close 9630 (time 1781818019.23 total 40.01 last 241.68)
total: 10000 open/close in 41.30 seconds: 242.12 ops/second
 - unlinked 0 (time 1781818022 ; total 0 ; last 0)
total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second
 - open/close 2485 (time 1781818060.35 total 10.00 last 248.45)
 - open/close 5061 (time 1781818070.36 total 20.00 last 257.53)
 - open/close 7003 (time 1781818080.36 total 30.01 last 194.14)
 - open/close 9882 (time 1781818090.36 total 40.01 last 287.90)
total: 10000 open/close in 40.49 seconds: 246.98 ops/second
 - unlinked 0 (time 1781818092 ; total 0 ; last 0)
total: 10000 unlinks in 28 seconds: 357.142853 unlinks/second
 - open/close 2570 (time 1781818133.02 total 10.00 last 256.95)
 - open/close 5040 (time 1781818143.02 total 20.00 last 246.99)
 - open/close 8068 (time 1781818153.03 total 30.01 last 302.66)
total: 10000 open/close in 37.78 seconds: 264.70 ops/second
 - unlinked 0 (time 1781818162 ; total 0 ; last 0)
total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second
 - open/close 2892 (time 1781818197.16 total 10.00 last 289.08)
 - open/close 5599 (time 1781818207.16 total 20.01 last 270.64)
 - open/close 8594 (time 1781818217.16 total 30.01 last 299.35)
total: 10000 open/close in 34.87 seconds: 286.78 ops/second
 - unlinked 0 (time 1781818224 ; total 0 ; last 0)
total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second
 - open/close 2909 (time 1781818258.75 total 10.00 last 290.85)
 - open/close 5685 (time 1781818268.75 total 20.00 last 277.58)
 - open/close 8126 (time 1781818278.75 total 30.00 last 244.09)
total: 10000 open/close in 37.57 seconds: 266.20 ops/second
 - unlinked 0 (time 1781818288 ; total 0 ; last 0)
total: 10000 unlinks in 113 seconds: 88.495575 unlinks/second
 - open/close 532 (time 1781818417.62 total 10.02 last 53.10)
 - open/close 1086 (time 1781818427.63 total 20.02 last 55.38)
 - open/close 1665 (time 1781818437.65 total 30.05 last 57.76)
 - open/close 2283 (time 1781818447.67 total 40.06 last 61.70)
 - open/close 2846 (time 1781818457.69 total 50.08 last 56.18)
 - open/close 3371 (time 1781818467.70 total 60.09 last 52.46)
 - open/close 3946 (time 1781818477.70 total 70.09 last 57.50)
 - open/close 4539 (time 1781818487.70 total 80.09 last 59.29)
 - open/close 5146 (time 1781818497.70 total 90.10 last 60.68)
 - open/close 5660 (time 1781818507.71 total 100.11 last 51.36)
 - open/close 6270 (time 1781818517.73 total 110.12 last 60.90)
 - open/close 6869 (time 1781818527.73 total 120.13 last 59.87)
 - open/close 7477 (time 1781818537.74 total 130.14 last 60.75)
 - open/close 8004 (time 1781818547.75 total 140.15 last 52.63)
 - open/close 8543 (time 1781818557.77 total 150.16 last 53.84)
 - open/close 9153 (time 1781818567.83 total 160.23 last 60.59)
 - open/close 9659 (time 1781818577.85 total 170.25 last 50.50)
total: 10000 open/close in 177.88 seconds: 56.22 ops/second
 - unlinked 0 (time 1781818592 ; total 0 ; last 0)
total: 10000 unlinks in 213 seconds: 46.948357 unlinks/second
 - open/close 283 (time 1781818825.17 total 10.00 last 28.30)
 - open/close 566 (time 1781818835.18 total 20.01 last 28.27)
 - open/close 859 (time 1781818845.20 total 30.03 last 29.24)
 - open/close 1156 (time 1781818855.21 total 40.04 last 29.68)
 - open/close 1585 (time 1781818865.23 total 50.06 last 42.81)
 - open/close 1971 (time 1781818875.25 total 60.08 last 38.53)
 - open/close 2275 (time 1781818885.30 total 70.13 last 30.25)
 - open/close 2694 (time 1781818895.30 total 80.13 last 41.88)
 - open/close 3021 (time 1781818905.32 total 90.15 last 32.65)
 - open/close 3341 (time 1781818915.33 total 100.16 last 31.95)
 - open/close 3704 (time 1781818925.37 total 110.20 last 36.17)
 - open/close 4090 (time 1781818935.38 total 120.21 last 38.56)
 - open/close 4432 (time 1781818945.39 total 130.22 last 34.16)
 - open/close 4786 (time 1781818955.40 total 140.23 last 35.37)
 - open/close 5109 (time 1781818965.41 total 150.24 last 32.25)
 - open/close 5464 (time 1781818975.43 total 160.26 last 35.43)
 - open/close 5842 (time 1781818985.43 total 170.26 last 37.79)
 - open/close 6145 (time 1781818995.45 total 180.28 last 30.25)
 - open/close 6462 (time 1781819005.46 total 190.29 last 31.68)
 - open/close 6799 (time 1781819015.48 total 200.31 last 33.63)
 - open/close 7049 (time 1781819025.51 total 210.34 last 24.93)
 - open/close 7349 (time 1781819035.53 total 220.36 last 29.93)
 - open/close 7669 (time 1781819045.54 total 230.38 last 31.95)
 - open/close 7969 (time 1781819055.57 total 240.40 last 29.92)
 - open/close 8343 (time 1781819065.58 total 250.41 last 37.37)
 - open/close 8641 (time 1781819075.60 total 260.43 last 29.74)
 - open/close 8910 (time 1781819085.61 total 270.44 last 26.87)
 - open/close 9189 (time 1781819095.62 total 280.45 last 27.87)
 - open/close 9527 (time 1781819105.65 total 290.48 last 33.71)
 - open/close 9818 (time 1781819115.70 total 300.53 last 28.95)
total: 10000 open/close in 306.42 seconds: 32.63 ops/second
 - unlinked 0 (time 1781819135 ; total 0 ; last 0)
total: 10000 unlinks in 218 seconds: 45.871559 unlinks/second
 - open/close 261 (time 1781819374.92 total 10.03 last 26.02)
 - open/close 557 (time 1781819384.92 total 20.04 last 29.58)
 - open/close 964 (time 1781819394.92 total 30.04 last 40.70)
 - open/close 1287 (time 1781819404.96 total 40.07 last 32.19)
 - open/close 1599 (time 1781819414.97 total 50.08 last 31.17)
 - open/close 1980 (time 1781819424.99 total 60.10 last 38.03)
 - open/close 2353 (time 1781819435.01 total 70.12 last 37.23)
 - open/close 2699 (time 1781819445.03 total 80.14 last 34.53)
 - open/close 3065 (time 1781819455.05 total 90.16 last 36.54)
 - open/close 3381 (time 1781819465.06 total 100.17 last 31.56)
 - open/close 3744 (time 1781819475.09 total 110.20 last 36.20)
 - open/close 4057 (time 1781819485.10 total 120.22 last 31.24)
 - open/close 4348 (time 1781819495.12 total 130.23 last 29.07)
 - open/close 4722 (time 1781819505.12 total 140.24 last 37.38)
 - open/close 5104 (time 1781819515.13 total 150.24 last 38.19)
 - open/close 5463 (time 1781819525.13 total 160.24 last 35.90)
 - open/close 5842 (time 1781819535.13 total 170.24 last 37.88)
 - open/close 6180 (time 1781819545.14 total 180.25 last 33.78)
 - open/close 6612 (time 1781819555.14 total 190.26 last 43.16)
 - open/close 7000 (time 1781819565.18 total 200.29 last 38.66)
 - open/close 7352 (time 1781819575.20 total 210.32 last 35.12)
 - open/close 7701 (time 1781819585.21 total 220.33 last 34.86)
 - open/close 8089 (time 1781819595.23 total 230.35 last 38.73)
 - open/close 8423 (time 1781819605.24 total 240.35 last 33.39)
 - open/close 8778 (time 1781819615.25 total 250.36 last 35.46)
 - open/close 9132 (time 1781819625.26 total 260.38 last 35.34)
 - open/close 9409 (time 1781819635.27 total 270.38 last 27.69)
 - open/close 9692 (time 1781819645.30 total 280.41 last 28.23)
total: 10000 open/close in 288.97 seconds: 34.61 ops/second
 - unlinked 0 (time 1781819662 ; total 0 ; last 0)
total: 10000 unlinks in 167 seconds: 59.880241 unlinks/second
 - open/close 413 (time 1781819844.74 total 10.02 last 41.21)
 - open/close 806 (time 1781819854.75 total 20.03 last 39.29)
 - open/close 1237 (time 1781819864.75 total 30.03 last 43.08)
 - open/close 1640 (time 1781819874.76 total 40.03 last 40.28)
 - open/close 2034 (time 1781819884.76 total 50.04 last 39.39)
 - open/close 2433 (time 1781819894.76 total 60.04 last 39.89)
 - open/close 2830 (time 1781819904.79 total 70.07 last 39.58)
 - open/close 3286 (time 1781819914.81 total 80.09 last 45.51)
 - open/close 3731 (time 1781819924.81 total 90.09 last 44.49)
 - open/close 4151 (time 1781819934.89 total 100.16 last 41.69)
 - open/close 4593 (time 1781819944.89 total 110.17 last 44.18)
 - open/close 5066 (time 1781819954.90 total 120.18 last 47.24)
 - open/close 5496 (time 1781819964.90 total 130.18 last 43.00)
 - open/close 5880 (time 1781819974.91 total 140.18 last 38.39)
 - open/close 6331 (time 1781819984.91 total 150.19 last 45.07)
 - open/close 6811 (time 1781819994.92 total 160.19 last 47.98)
 - open/close 7232 (time 1781820004.94 total 170.22 last 42.00)
 - open/close 7619 (time 1781820014.95 total 180.22 last 38.67)
 - open/close 8063 (time 1781820024.96 total 190.24 last 44.35)
 - open/close 8500 (time 1781820034.99 total 200.26 last 43.58)
 - open/close 8835 (time 1781820045.04 total 210.32 last 33.32)
 - open/close 9214 (time 1781820055.05 total 220.32 last 37.87)
 - open/close 9563 (time 1781820065.07 total 230.35 last 34.81)
 - open/close 9981 (time 1781820075.10 total 240.37 last 41.70)
total: 10000 open/close in 240.88 seconds: 41.51 ops/second
 - unlinked 0 (time 1781820080 ; total 0 ; last 0)
total: 10000 unlinks in 137 seconds: 72.992699 unlinks/second
 - open/close 472 (time 1781820233.44 total 10.01 last 47.16)
 - open/close 1063 (time 1781820243.45 total 20.02 last 59.05)
 - open/close 1705 (time 1781820253.46 total 30.03 last 64.13)
 - open/close 2388 (time 1781820263.46 total 40.03 last 68.26)
 - open/close 3048 (time 1781820273.49 total 50.06 last 65.83)
 - open/close 3588 (time 1781820283.49 total 60.06 last 53.98)
 - open/close 3979 (time 1781820293.52 total 70.09 last 38.98)
 - open/close 4470 (time 1781820303.55 total 80.12 last 48.94)
 - open/close 4882 (time 1781820313.57 total 90.14 last 41.14)
 - open/close 5327 (time 1781820323.60 total 100.17 last 44.36)
 - open/close 5654 (time 1781820333.62 total 110.19 last 32.62)
 - open/close 6004 (time 1781820343.65 total 120.21 last 34.92)
 - open/close 6351 (time 1781820353.65 total 130.22 last 34.69)
 - open/close 6714 (time 1781820363.66 total 140.23 last 36.27)
 - open/close 7159 (time 1781820373.68 total 150.25 last 44.41)
 - open/close 7633 (time 1781820383.70 total 160.27 last 47.29)
 - open/close 8056 (time 1781820393.72 total 170.29 last 42.19)
 - open/close 8409 (time 1781820403.73 total 180.30 last 35.27)
 - open/close 8798 (time 1781820413.77 total 190.34 last 38.77)
 - open/close 9160 (time 1781820423.83 total 200.40 last 35.96)
 - open/close 9551 (time 1781820433.84 total 210.40 last 39.09)
 - open/close 9964 (time 1781820443.85 total 220.42 last 41.24)
total: 10000 open/close in 221.20 seconds: 45.21 ops/second
 - unlinked 0 (time 1781820449 ; total 0 ; last 0)
total: 10000 unlinks in 151 seconds: 66.225166 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg144-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg144-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg144-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.144@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.144@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg144-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all
pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1
Started lustre-OST0000
oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg144-server: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg144-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg144-client.virtnet:  -o user_xattr,flock 192.168.201.144@tcp:/lustre /mnt/lustre
On OST0, 20326 used inodes rc=0
pdsh@oleg144-client: oleg144-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg144-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg144-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg144-server

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.201.144@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.201.144@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.144@tcp

   Permanent disk data:
Device:     lustre-ost1/ost1
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.201.144@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.201.144@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg144-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all
pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1
Started lustre-OST0000
oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg144-server: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg144-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg144-client.virtnet:  -o user_xattr,flock 192.168.201.144@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg144-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg144-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg144-server
stop mds service on oleg144-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg144-server

unloading modules via unload_modules_local on: 'oleg144-server'
oleg144-server: oleg144-server.virtnet: executing unload_modules_local
oleg144-server: 
modules unloaded.
pdsh@oleg144-client: oleg144-client: ssh exited with exit code 2
pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2
pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2
pdsh@oleg144-client: oleg144-client: ssh exited with exit code 2