== conf-sanity test 69: replace an OST with the same index ========================================================== 15:55:03 (1768251303)
start mds service on oleg435-server
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
Loading modules from /home/green/git/lustre-release/lustre
detected 4 online CPUs by sysfs
MODOPTS_LIBCFS=
Force libcfs to create 2 CPU partitions
ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
quota/lquota options: 'hash_lqs_cur_bits=3'
ln: failed to create symbolic link '/sbin/.libs': Read-only file system
loading modules on: 'oleg435-server'
oleg435-server: oleg435-server.virtnet: executing load_modules_local
oleg435-server: Loading modules from /home/green/git/lustre-release/lustre
oleg435-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy
oleg435-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy
oleg435-server: detected 4 online CPUs by sysfs
oleg435-server: MODOPTS_LIBCFS=
oleg435-server: Force libcfs to create 2 CPU partitions
oleg435-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1'
oleg435-server: quota/lquota options: 'hash_lqs_cur_bits=3'
Start mds1: mount -t lustre -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all
pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1
Started lustre-MDT0000
start mds service on oleg435-server
Start mds2: mount -t lustre -o localrecov  lustre-mdt2/mdt2 /mnt/lustre-mds2
oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all
pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1
Started lustre-MDT0001
oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
start ost1 service on oleg435-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all
pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1
Started lustre-OST0000
oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
mount lustre  on /mnt/lustre.....
Starting client: oleg435-client.virtnet:  -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre
seq.cli-lustre-OST0000-super.width=0x1ffffff
On OST0, 11840 inodes available. Want 99872. rc=0
 - open/close 804 (time 1768251422.72 total 10.00 last 80.36)
 - open/close 1575 (time 1768251432.72 total 20.01 last 77.09)
 - open/close 2238 (time 1768251442.73 total 30.01 last 66.28)
 - open/close 2841 (time 1768251452.73 total 40.01 last 60.27)
 - open/close 3517 (time 1768251462.73 total 50.02 last 67.59)
 - open/close 4332 (time 1768251472.74 total 60.02 last 81.43)
 - open/close 5152 (time 1768251482.75 total 70.03 last 81.97)
 - open/close 5892 (time 1768251492.75 total 80.03 last 73.96)
 - open/close 6662 (time 1768251502.75 total 90.04 last 76.98)
 - open/close 7508 (time 1768251512.76 total 100.04 last 84.54)
 - open/close 8304 (time 1768251522.77 total 110.05 last 79.54)
 - open/close 9145 (time 1768251532.77 total 120.05 last 84.07)
 - open/close 9849 (time 1768251542.78 total 130.06 last 70.34)
total: 10000 open/close in 132.17 seconds: 75.66 ops/second
 - unlinked 0 (time 1768251549 ; total 0 ; last 0)
total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second
 - open/close 802 (time 1768251636.61 total 10.01 last 80.10)
 - open/close 1615 (time 1768251646.61 total 20.01 last 81.30)
 - open/close 2400 (time 1768251656.62 total 30.02 last 78.44)
 - open/close 3268 (time 1768251666.62 total 40.02 last 86.78)
 - open/close 4064 (time 1768251676.64 total 50.04 last 79.50)
 - open/close 4905 (time 1768251686.65 total 60.05 last 83.99)
 - open/close 5665 (time 1768251696.66 total 70.06 last 75.92)
 - open/close 6393 (time 1768251706.67 total 80.07 last 72.73)
 - open/close 7168 (time 1768251716.69 total 90.09 last 77.32)
 - open/close 7854 (time 1768251726.69 total 100.09 last 68.59)
 - open/close 8632 (time 1768251736.71 total 110.11 last 77.65)
 - open/close 9467 (time 1768251746.72 total 120.12 last 83.47)
total: 10000 open/close in 127.81 seconds: 78.24 ops/second
 - unlinked 0 (time 1768251760 ; total 0 ; last 0)
total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second
 - open/close 661 (time 1768251858.83 total 10.01 last 66.06)
 - open/close 1385 (time 1768251868.84 total 20.01 last 72.34)
 - open/close 2247 (time 1768251878.84 total 30.02 last 86.18)
 - open/close 3096 (time 1768251888.85 total 40.02 last 84.84)
 - open/close 3759 (time 1768251898.86 total 50.03 last 66.27)
 - open/close 4418 (time 1768251908.86 total 60.04 last 65.85)
 - open/close 4940 (time 1768251918.87 total 70.04 last 52.19)
 - open/close 5668 (time 1768251928.88 total 80.05 last 72.74)
 - open/close 6351 (time 1768251938.89 total 90.06 last 68.19)
 - open/close 7190 (time 1768251948.90 total 100.07 last 83.87)
 - open/close 7897 (time 1768251958.90 total 110.08 last 70.65)
 - open/close 8568 (time 1768251968.91 total 120.08 last 67.08)
 - open/close 9221 (time 1768251978.92 total 130.09 last 65.22)
 - open/close 9963 (time 1768251988.93 total 140.10 last 74.13)
total: 10000 open/close in 140.56 seconds: 71.14 ops/second
 - unlinked 0 (time 1768251994 ; total 0 ; last 0)
total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second
 - open/close 837 (time 1768252087.92 total 10.00 last 83.68)
 - open/close 1619 (time 1768252097.92 total 20.00 last 78.20)
 - open/close 2335 (time 1768252107.93 total 30.01 last 71.56)
 - open/close 3157 (time 1768252117.94 total 40.02 last 82.12)
 - open/close 3866 (time 1768252127.95 total 50.03 last 70.79)
 - open/close 4689 (time 1768252137.96 total 60.04 last 82.23)
 - open/close 5348 (time 1768252147.97 total 70.05 last 65.86)
 - open/close 6019 (time 1768252157.97 total 80.05 last 67.09)
 - open/close 6726 (time 1768252167.97 total 90.05 last 70.69)
 - open/close 7486 (time 1768252177.99 total 100.07 last 75.86)
 - open/close 8351 (time 1768252187.99 total 110.08 last 86.45)
 - open/close 9057 (time 1768252198.00 total 120.09 last 70.52)
 - open/close 9866 (time 1768252208.01 total 130.10 last 80.83)
total: 10000 open/close in 131.53 seconds: 76.03 ops/second
 - unlinked 0 (time 1768252213 ; total 0 ; last 0)
total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second
 - open/close 694 (time 1768252304.88 total 10.01 last 69.36)
 - open/close 1374 (time 1768252314.89 total 20.01 last 67.97)
 - open/close 2051 (time 1768252324.89 total 30.02 last 67.65)
 - open/close 2768 (time 1768252334.90 total 40.02 last 71.65)
 - open/close 3280 (time 1768252344.92 total 50.04 last 51.13)
 - open/close 3875 (time 1768252354.92 total 60.04 last 59.47)
 - open/close 4635 (time 1768252364.93 total 70.05 last 75.95)
 - open/close 5545 (time 1768252374.93 total 80.05 last 90.97)
 - open/close 6426 (time 1768252384.94 total 90.06 last 88.05)
 - open/close 7314 (time 1768252394.94 total 100.06 last 88.74)
 - open/close 8180 (time 1768252404.96 total 110.08 last 86.49)
 - open/close 9082 (time 1768252414.96 total 120.08 last 90.18)
 - open/close 9774 (time 1768252424.96 total 130.08 last 69.16)
total: 10000 open/close in 133.43 seconds: 74.95 ops/second
 - unlinked 0 (time 1768252433 ; total 0 ; last 0)
total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second
 - open/close 809 (time 1768252529.03 total 10.00 last 80.89)
 - open/close 1693 (time 1768252539.03 total 20.01 last 88.35)
 - open/close 2531 (time 1768252549.03 total 30.01 last 83.80)
 - open/close 3521 (time 1768252559.03 total 40.01 last 98.99)
 - open/close 4361 (time 1768252569.04 total 50.01 last 83.95)
 - open/close 5266 (time 1768252579.04 total 60.01 last 90.49)
 - open/close 6160 (time 1768252589.05 total 70.02 last 89.35)
 - open/close 6920 (time 1768252599.05 total 80.02 last 75.97)
 - open/close 7578 (time 1768252609.07 total 90.04 last 65.67)
 - open/close 8480 (time 1768252619.07 total 100.04 last 90.20)
 - open/close 9262 (time 1768252629.08 total 110.05 last 78.11)
total: 10000 open/close in 118.49 seconds: 84.39 ops/second
 - unlinked 0 (time 1768252642 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 841 (time 1768252741.15 total 10.01 last 84.03)
 - open/close 1411 (time 1768252751.16 total 20.02 last 56.91)
 - open/close 2221 (time 1768252761.18 total 30.04 last 80.85)
 - open/close 2904 (time 1768252771.19 total 40.05 last 68.22)
 - open/close 3607 (time 1768252781.20 total 50.06 last 70.25)
 - open/close 4278 (time 1768252791.22 total 60.08 last 66.95)
 - open/close 5078 (time 1768252801.24 total 70.10 last 79.88)
 - open/close 5811 (time 1768252811.24 total 80.10 last 73.30)
 - open/close 6369 (time 1768252821.26 total 90.12 last 55.67)
 - open/close 7052 (time 1768252831.27 total 100.13 last 68.23)
 - open/close 7880 (time 1768252841.30 total 110.16 last 82.58)
 - open/close 8534 (time 1768252851.32 total 120.18 last 65.29)
 - open/close 9220 (time 1768252861.33 total 130.19 last 68.54)
 - open/close 9829 (time 1768252871.34 total 140.20 last 60.82)
total: 10000 open/close in 143.10 seconds: 69.88 ops/second
 - unlinked 0 (time 1768252879 ; total 0 ; last 0)
total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second
 - open/close 702 (time 1768252980.79 total 10.00 last 70.18)
 - open/close 1371 (time 1768252990.80 total 20.01 last 66.84)
 - open/close 1946 (time 1768253000.80 total 30.01 last 57.49)
 - open/close 2613 (time 1768253010.81 total 40.02 last 66.67)
 - open/close 3107 (time 1768253020.82 total 50.03 last 49.36)
 - open/close 3752 (time 1768253030.83 total 60.04 last 64.44)
 - open/close 4373 (time 1768253040.84 total 70.05 last 62.03)
 - open/close 5054 (time 1768253050.85 total 80.06 last 68.02)
 - open/close 5803 (time 1768253060.86 total 90.07 last 74.81)
 - open/close 6484 (time 1768253070.87 total 100.08 last 68.05)
 - open/close 7182 (time 1768253080.89 total 110.10 last 69.63)
 - open/close 7832 (time 1768253090.90 total 120.11 last 64.98)
 - open/close 8427 (time 1768253100.90 total 130.11 last 59.48)
 - open/close 9063 (time 1768253110.90 total 140.11 last 63.57)
 - open/close 9853 (time 1768253120.92 total 150.13 last 78.91)
total: 10000 open/close in 152.20 seconds: 65.70 ops/second
 - unlinked 0 (time 1768253127 ; total 0 ; last 0)
total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second
 - open/close 651 (time 1768253239.42 total 10.01 last 65.04)
 - open/close 1280 (time 1768253249.43 total 20.02 last 62.85)
 - open/close 1997 (time 1768253259.44 total 30.03 last 71.64)
 - open/close 2623 (time 1768253269.45 total 40.04 last 62.51)
 - open/close 3419 (time 1768253279.46 total 50.05 last 79.52)
 - open/close 4091 (time 1768253289.46 total 60.05 last 67.19)
 - open/close 4741 (time 1768253299.47 total 70.06 last 64.96)
 - open/close 5469 (time 1768253309.47 total 80.06 last 72.80)
 - open/close 6251 (time 1768253319.47 total 90.07 last 78.14)
 - open/close 6980 (time 1768253329.48 total 100.07 last 72.84)
 - open/close 7720 (time 1768253339.48 total 110.07 last 74.00)
 - open/close 8485 (time 1768253349.49 total 120.08 last 76.42)
 - open/close 9317 (time 1768253359.50 total 130.09 last 83.11)
 - open/close 9973 (time 1768253369.52 total 140.12 last 65.45)
total: 10000 open/close in 140.58 seconds: 71.13 ops/second
 - unlinked 0 (time 1768253374 ; total 0 ; last 0)
total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second
 - open/close 684 (time 1768253471.56 total 10.01 last 68.30)
 - open/close 1300 (time 1768253481.57 total 20.03 last 61.52)
 - open/close 2131 (time 1768253491.59 total 30.04 last 83.00)
 - open/close 2803 (time 1768253501.63 total 40.08 last 66.92)
 - open/close 3499 (time 1768253511.66 total 50.11 last 69.39)
 - open/close 4159 (time 1768253521.67 total 60.13 last 65.90)
 - open/close 4920 (time 1768253531.68 total 70.13 last 76.07)
 - open/close 5769 (time 1768253541.68 total 80.13 last 84.87)
 - open/close 6510 (time 1768253551.69 total 90.15 last 74.00)
 - open/close 7203 (time 1768253561.69 total 100.15 last 69.30)
 - open/close 7808 (time 1768253571.70 total 110.15 last 60.46)
 - open/close 8479 (time 1768253581.71 total 120.16 last 67.03)
 - open/close 9194 (time 1768253591.73 total 130.18 last 71.37)
 - open/close 9947 (time 1768253601.75 total 140.20 last 75.14)
total: 10000 open/close in 141.05 seconds: 70.90 ops/second
 - unlinked 0 (time 1768253607 ; total 0 ; last 0)
total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second
rm: missing operand
Try 'rm --help' for more information.
umount lustre on /mnt/lustre.....
Stopping client oleg435-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg435-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.135@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.135@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg435-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all
pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1
Started lustre-OST0000
oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg435-server: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg435-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg435-server: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg435-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg435-client.virtnet:  -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre
On OST0, 20519 used inodes rc=0
pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22
umount lustre on /mnt/lustre.....
Stopping client oleg435-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg435-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server

   Permanent disk data:
Target:     lustre:OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: mgsnode=192.168.204.135@tcp sys.timeout=20 autodegrade=on
mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde
mkfs_cmd = zfs create -o canmount=off  -o quota=409600000 lustre-ost1/ost1
  xattr=sa
  dnodesize=auto
  recordsize=1M
  compression=off
Writing lustre-ost1/ost1 properties
  lustre:mgsnode=192.168.204.135@tcp
  lustre:sys.timeout=20
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=98
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre:OST0000
checking for existing Lustre data: found

   Read previous values:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x62
              (OST first_time update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.204.135@tcp

   Permanent disk data:
Target:     lustre-OST0000
Index:      0
Lustre FS:  lustre
Mount type: zfs
Flags:      0x42
              (OST update )
Persistent mount opts: 
Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.204.135@tcp autodegrade=on
  lustre:svname=lustre-OST0000
Writing lustre-ost1/ost1 properties
  lustre:autodegrade=on
  lustre:sys.timeout=20
  lustre:mgsnode=192.168.204.135@tcp
  lustre:autodegrade=on
  lustre:version=1
  lustre:flags=66
  lustre:index=0
  lustre:fsname=lustre
  lustre:svname=lustre-OST0000
start ost1 service on oleg435-server
Start ost1: mount -t lustre -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all
pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1
Started lustre-OST0000
oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
oleg435-server: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg435-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec
oleg435-server: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50
oleg435-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec
mount lustre  on /mnt/lustre.....
Starting client: oleg435-client.virtnet:  -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre
umount lustre on /mnt/lustre.....
Stopping client oleg435-client.virtnet /mnt/lustre (opts:)
stop ost1 service on oleg435-server
Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server
stop mds service on oleg435-server
Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server
stop mds service on oleg435-server
Stopping /mnt/lustre-mds2 (opts:-f) on oleg435-server
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
/home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
unloading modules via unload_modules_local on: 'oleg435-server'
oleg435-server: oleg435-server.virtnet: executing unload_modules_local
oleg435-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy
oleg435-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy
modules unloaded.
pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2
pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2
pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2
pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2