== conf-sanity test 69: replace an OST with the same index ========================================================== 14:57:29 (1782241049) start mds service on oleg457-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing load_modules_local oleg457-server: Loading modules from /home/green/git/lustre-release/lustre oleg457-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg457-server: detected 4 online CPUs by sysfs oleg457-server: MODOPTS_LIBCFS= oleg457-server: Force libcfs to create 2 CPU partitions oleg457-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg457-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg457-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg457-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 617 (time 1782241130.55 total 10.01 last 61.64) - open/close 1193 (time 1782241140.57 total 20.03 last 57.46) - open/close 1811 (time 1782241150.57 total 30.03 last 61.79) - open/close 2479 (time 1782241160.59 total 40.05 last 66.67) - open/close 3050 (time 1782241170.60 total 50.06 last 57.05) - open/close 3716 (time 1782241180.61 total 60.07 last 66.56) - open/close 4323 (time 1782241190.62 total 70.08 last 60.62) - open/close 4979 (time 1782241200.62 total 80.08 last 65.60) - open/close 5587 (time 1782241210.63 total 90.09 last 60.77) - open/close 6202 (time 1782241220.64 total 100.10 last 61.44) - open/close 6829 (time 1782241230.64 total 110.11 last 62.64) - open/close 7408 (time 1782241240.65 total 120.12 last 57.84) - open/close 8033 (time 1782241250.66 total 130.12 last 62.47) - open/close 8634 (time 1782241260.72 total 140.18 last 59.73) - open/close 9226 (time 1782241270.73 total 150.19 last 59.16) - open/close 9852 (time 1782241280.73 total 160.19 last 62.56) total: 10000 open/close in 162.48 seconds: 61.55 ops/second - unlinked 0 (time 1782241287 ; total 0 ; last 0) total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second - open/close 615 (time 1782241405.04 total 10.00 last 61.49) - open/close 1265 (time 1782241415.05 total 20.01 last 64.94) - open/close 1907 (time 1782241425.05 total 30.02 last 64.16) - open/close 2562 (time 1782241435.07 total 40.04 last 65.39) - open/close 3209 (time 1782241445.08 total 50.05 last 64.63) - open/close 3832 (time 1782241455.10 total 60.06 last 62.18) - open/close 4444 (time 1782241465.10 total 70.07 last 61.18) - open/close 5053 (time 1782241475.12 total 80.09 last 60.79) - open/close 5663 (time 1782241485.13 total 90.10 last 60.91) - open/close 6285 (time 1782241495.14 total 100.11 last 62.15) - open/close 6941 (time 1782241505.14 total 110.11 last 65.59) - open/close 7583 (time 1782241515.15 total 120.12 last 64.16) - open/close 8211 (time 1782241525.17 total 130.14 last 62.65) - open/close 8791 (time 1782241535.19 total 140.15 last 57.93) - open/close 9235 (time 1782241545.21 total 150.17 last 44.31) - open/close 9650 (time 1782241555.25 total 160.21 last 41.33) total: 10000 open/close in 166.47 seconds: 60.07 ops/second - unlinked 0 (time 1782241566 ; total 0 ; last 0) total: 10000 unlinks in 107 seconds: 93.457947 unlinks/second - open/close 638 (time 1782241687.80 total 10.02 last 63.68) - open/close 1187 (time 1782241697.81 total 20.03 last 54.84) - open/close 1744 (time 1782241707.84 total 30.06 last 55.56) - open/close 2335 (time 1782241717.86 total 40.08 last 58.95) - open/close 2960 (time 1782241727.87 total 50.08 last 62.49) - open/close 3561 (time 1782241737.87 total 60.09 last 60.08) - open/close 4146 (time 1782241747.87 total 70.09 last 58.48) - open/close 4721 (time 1782241757.88 total 80.10 last 57.46) - open/close 5319 (time 1782241767.89 total 90.10 last 59.75) - open/close 5822 (time 1782241777.90 total 100.11 last 50.26) - open/close 6304 (time 1782241787.90 total 110.12 last 48.16) - open/close 6826 (time 1782241797.91 total 120.12 last 52.18) - open/close 7339 (time 1782241807.93 total 130.15 last 51.19) - open/close 7907 (time 1782241817.95 total 140.17 last 56.67) - open/close 8510 (time 1782241827.96 total 150.18 last 60.25) - open/close 9098 (time 1782241837.97 total 160.19 last 58.74) - open/close 9719 (time 1782241848.00 total 170.21 last 61.94) total: 10000 open/close in 174.97 seconds: 57.15 ops/second - unlinked 0 (time 1782241857 ; total 1 ; last 1) total: 10000 unlinks in 109 seconds: 91.743118 unlinks/second - open/close 616 (time 1782241979.75 total 10.00 last 61.60) - open/close 1216 (time 1782241989.75 total 20.00 last 59.99) - open/close 1837 (time 1782241999.76 total 30.01 last 62.02) - open/close 2480 (time 1782242009.77 total 40.02 last 64.24) - open/close 3075 (time 1782242019.78 total 50.03 last 59.44) - open/close 3646 (time 1782242029.79 total 60.04 last 57.06) - open/close 4173 (time 1782242039.81 total 70.06 last 52.60) - open/close 4743 (time 1782242049.81 total 80.06 last 57.00) - open/close 5275 (time 1782242059.81 total 90.06 last 53.18) - open/close 5817 (time 1782242069.82 total 100.07 last 54.16) - open/close 6341 (time 1782242079.83 total 110.08 last 52.35) - open/close 6930 (time 1782242089.84 total 120.09 last 58.86) - open/close 7463 (time 1782242099.84 total 130.09 last 53.27) - open/close 8126 (time 1782242109.84 total 140.09 last 66.29) - open/close 8795 (time 1782242119.86 total 150.11 last 66.82) - open/close 9377 (time 1782242129.88 total 160.13 last 58.07) - open/close 9923 (time 1782242139.89 total 170.14 last 54.53) total: 10000 open/close in 171.47 seconds: 58.32 ops/second - unlinked 0 (time 1782242145 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 614 (time 1782242259.22 total 10.01 last 61.32) - open/close 1225 (time 1782242269.23 total 20.02 last 61.03) - open/close 1843 (time 1782242279.24 total 30.03 last 61.78) - open/close 2499 (time 1782242289.24 total 40.03 last 65.56) - open/close 3168 (time 1782242299.26 total 50.05 last 66.80) - open/close 3819 (time 1782242309.27 total 60.06 last 65.03) - open/close 4502 (time 1782242319.27 total 70.07 last 68.25) - open/close 5133 (time 1782242329.30 total 80.09 last 62.95) - open/close 5779 (time 1782242339.30 total 90.09 last 64.57) - open/close 6463 (time 1782242349.32 total 100.11 last 68.32) - open/close 7128 (time 1782242359.32 total 110.11 last 66.47) - open/close 7783 (time 1782242369.32 total 120.11 last 65.49) - open/close 8416 (time 1782242379.33 total 130.12 last 63.25) - open/close 9014 (time 1782242389.34 total 140.13 last 59.74) - open/close 9596 (time 1782242399.36 total 150.15 last 58.08) total: 10000 open/close in 156.45 seconds: 63.92 ops/second - unlinked 0 (time 1782242409 ; total 0 ; last 0) total: 10000 unlinks in 94 seconds: 106.382980 unlinks/second - open/close 664 (time 1782242517.60 total 10.01 last 66.34) - open/close 1312 (time 1782242527.62 total 20.03 last 64.68) - open/close 2025 (time 1782242537.63 total 30.04 last 71.24) - open/close 2618 (time 1782242547.63 total 40.04 last 59.30) - open/close 3249 (time 1782242557.63 total 50.04 last 63.06) - open/close 3896 (time 1782242567.64 total 60.05 last 64.66) - open/close 4506 (time 1782242577.65 total 70.06 last 60.94) - open/close 5128 (time 1782242587.65 total 80.06 last 62.16) - open/close 5721 (time 1782242597.66 total 90.07 last 59.25) - open/close 6343 (time 1782242607.67 total 100.08 last 62.15) - open/close 7008 (time 1782242617.67 total 110.08 last 66.49) - open/close 7647 (time 1782242627.71 total 120.12 last 63.67) - open/close 8278 (time 1782242637.71 total 130.12 last 63.09) - open/close 8905 (time 1782242647.71 total 140.12 last 62.69) - open/close 9500 (time 1782242657.73 total 150.13 last 59.43) total: 10000 open/close in 157.23 seconds: 63.60 ops/second - unlinked 0 (time 1782242668 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 610 (time 1782242782.01 total 10.03 last 60.83) - open/close 1174 (time 1782242792.02 total 20.03 last 56.39) - open/close 1807 (time 1782242802.03 total 30.04 last 63.23) - open/close 2463 (time 1782242812.04 total 40.05 last 65.53) - open/close 3139 (time 1782242822.05 total 50.06 last 67.53) - open/close 3775 (time 1782242832.06 total 60.08 last 63.51) - open/close 4371 (time 1782242842.07 total 70.08 last 59.57) - open/close 5020 (time 1782242852.08 total 80.09 last 64.83) - open/close 5634 (time 1782242862.12 total 90.14 last 61.14) - open/close 6190 (time 1782242872.13 total 100.15 last 55.54) - open/close 6855 (time 1782242882.17 total 110.18 last 66.27) - open/close 7512 (time 1782242892.20 total 120.21 last 65.51) - open/close 8196 (time 1782242902.20 total 130.21 last 68.37) - open/close 8832 (time 1782242912.21 total 140.22 last 63.56) - open/close 9415 (time 1782242922.22 total 150.24 last 58.21) total: 10000 open/close in 160.08 seconds: 62.47 ops/second - unlinked 0 (time 1782242937 ; total 0 ; last 0) total: 10000 unlinks in 100 seconds: 100.000000 unlinks/second - open/close 652 (time 1782243052.61 total 10.01 last 65.14) - open/close 1249 (time 1782243062.61 total 20.01 last 59.69) - open/close 1882 (time 1782243072.62 total 30.02 last 63.23) - open/close 2553 (time 1782243082.62 total 40.02 last 67.10) - open/close 3169 (time 1782243092.64 total 50.04 last 61.51) - open/close 3857 (time 1782243102.65 total 60.05 last 68.69) - open/close 4493 (time 1782243112.66 total 70.06 last 63.56) - open/close 5105 (time 1782243122.68 total 80.08 last 61.05) - open/close 5721 (time 1782243132.69 total 90.09 last 61.55) - open/close 6416 (time 1782243142.69 total 100.09 last 69.50) - open/close 7092 (time 1782243152.71 total 110.11 last 67.51) - open/close 7631 (time 1782243162.72 total 120.12 last 53.83) - open/close 8153 (time 1782243172.73 total 130.13 last 52.15) - open/close 8781 (time 1782243182.74 total 140.14 last 62.73) - open/close 9457 (time 1782243192.74 total 150.14 last 67.59) total: 10000 open/close in 158.91 seconds: 62.93 ops/second - unlinked 0 (time 1782243206 ; total 0 ; last 0) total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second - open/close 686 (time 1782243323.56 total 10.01 last 68.52) - open/close 1349 (time 1782243333.57 total 20.03 last 66.19) - open/close 1969 (time 1782243343.57 total 30.03 last 61.99) - open/close 2570 (time 1782243353.58 total 40.03 last 60.07) - open/close 3195 (time 1782243363.58 total 50.04 last 62.46) - open/close 3782 (time 1782243373.59 total 60.04 last 58.68) - open/close 4453 (time 1782243383.59 total 70.05 last 67.08) - open/close 5104 (time 1782243393.59 total 80.05 last 65.09) - open/close 5789 (time 1782243403.60 total 90.06 last 68.43) - open/close 6402 (time 1782243413.62 total 100.07 last 61.21) - open/close 7061 (time 1782243423.63 total 110.08 last 65.83) - open/close 7716 (time 1782243433.63 total 120.09 last 65.47) - open/close 8347 (time 1782243443.64 total 130.10 last 63.05) - open/close 8960 (time 1782243453.65 total 140.11 last 61.22) - open/close 9575 (time 1782243463.66 total 150.12 last 61.44) total: 10000 open/close in 158.78 seconds: 62.98 ops/second - unlinked 0 (time 1782243477 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 650 (time 1782243593.47 total 10.02 last 64.90) - open/close 1301 (time 1782243603.48 total 20.03 last 65.03) - open/close 1921 (time 1782243613.49 total 30.03 last 61.98) - open/close 2569 (time 1782243623.49 total 40.03 last 64.78) - open/close 3140 (time 1782243633.50 total 50.04 last 57.06) - open/close 3782 (time 1782243643.50 total 60.04 last 64.19) - open/close 4404 (time 1782243653.50 total 70.05 last 62.18) - open/close 4974 (time 1782243663.51 total 80.05 last 56.97) - open/close 5581 (time 1782243673.54 total 90.08 last 60.52) - open/close 6169 (time 1782243683.56 total 100.10 last 58.69) - open/close 6749 (time 1782243693.58 total 110.12 last 57.90) - open/close 7304 (time 1782243703.58 total 120.12 last 55.46) - open/close 7983 (time 1782243713.58 total 130.12 last 67.89) - open/close 8637 (time 1782243723.59 total 140.13 last 65.38) - open/close 9276 (time 1782243733.61 total 150.15 last 63.74) - open/close 9875 (time 1782243743.64 total 160.18 last 59.74) total: 10000 open/close in 162.53 seconds: 61.53 ops/second - unlinked 0 (time 1782243750 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg457-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg457-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.157@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.157@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg457-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg457-server: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg457-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre On OST0, 20326 used inodes rc=0 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg457-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg457-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.157@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.157@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.204.157@tcp sys.timeout=20 autodegrade=on Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.157@tcp sys.timeout=20 autodegrade=on autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.157@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg457-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg457-server: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg457-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg457-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg457-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server stop mds service on oleg457-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server unloading modules via unload_modules_local on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing unload_modules_local oleg457-server: modules unloaded. pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-client: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-client: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2