== conf-sanity test 69: replace an OST with the same index ========================================================== 00:20:11 (1781238011) start mds service on oleg239-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg239-server' oleg239-server: oleg239-server.virtnet: executing load_modules_local oleg239-server: Loading modules from /home/green/git/lustre-release/lustre oleg239-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg239-server: detected 4 online CPUs by sysfs oleg239-server: MODOPTS_LIBCFS= oleg239-server: Force libcfs to create 2 CPU partitions oleg239-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg239-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg239-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg239-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0000 oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11904 inodes available. Want 99872. rc=0 - open/close 745 (time 1781238097.12 total 10.00 last 74.46) - open/close 1616 (time 1781238107.13 total 20.01 last 87.03) - open/close 2512 (time 1781238117.14 total 30.02 last 89.49) - open/close 3311 (time 1781238127.14 total 40.03 last 79.88) - open/close 4138 (time 1781238137.18 total 50.07 last 82.37) - open/close 4891 (time 1781238147.19 total 60.07 last 75.26) - open/close 5679 (time 1781238157.20 total 70.08 last 78.74) - open/close 6510 (time 1781238167.20 total 80.08 last 83.05) - open/close 7357 (time 1781238177.21 total 90.09 last 84.66) - open/close 8222 (time 1781238187.22 total 100.10 last 86.42) - open/close 9050 (time 1781238197.22 total 110.10 last 82.76) - open/close 9810 (time 1781238207.22 total 120.11 last 75.98) total: 10000 open/close in 122.26 seconds: 81.79 ops/second - unlinked 0 (time 1781238213 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 815 (time 1781238306.70 total 10.01 last 81.45) - open/close 1476 (time 1781238316.71 total 20.02 last 66.03) - open/close 2419 (time 1781238326.71 total 30.02 last 94.27) - open/close 3219 (time 1781238336.73 total 40.03 last 79.88) - open/close 3915 (time 1781238346.73 total 50.04 last 69.55) - open/close 4695 (time 1781238356.74 total 60.05 last 77.93) - open/close 5416 (time 1781238366.75 total 70.06 last 72.01) - open/close 6198 (time 1781238376.76 total 80.07 last 78.13) - open/close 6944 (time 1781238386.77 total 90.07 last 74.58) - open/close 7716 (time 1781238396.77 total 100.08 last 77.14) - open/close 8577 (time 1781238406.79 total 110.10 last 85.99) - open/close 9286 (time 1781238416.80 total 120.10 last 70.84) - open/close 9949 (time 1781238426.80 total 130.11 last 66.26) total: 10000 open/close in 130.73 seconds: 76.49 ops/second - unlinked 0 (time 1781238432 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 814 (time 1781238528.64 total 10.01 last 81.31) - open/close 1446 (time 1781238538.65 total 20.02 last 63.15) - open/close 2285 (time 1781238548.65 total 30.02 last 83.87) - open/close 3132 (time 1781238558.66 total 40.03 last 84.62) - open/close 3969 (time 1781238568.67 total 50.05 last 83.59) - open/close 4833 (time 1781238578.68 total 60.05 last 86.35) - open/close 5653 (time 1781238588.69 total 70.06 last 81.95) - open/close 6502 (time 1781238598.69 total 80.06 last 84.85) - open/close 7323 (time 1781238608.70 total 90.07 last 82.04) - open/close 8051 (time 1781238618.71 total 100.08 last 72.74) - open/close 8846 (time 1781238628.71 total 110.09 last 79.46) - open/close 9675 (time 1781238638.72 total 120.09 last 82.83) total: 10000 open/close in 124.11 seconds: 80.58 ops/second - unlinked 0 (time 1781238646 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 690 (time 1781238739.48 total 10.01 last 68.94) - open/close 1452 (time 1781238749.49 total 20.01 last 76.17) - open/close 2219 (time 1781238759.49 total 30.02 last 76.66) - open/close 3092 (time 1781238769.50 total 40.03 last 87.22) - open/close 3861 (time 1781238779.50 total 50.03 last 76.89) - open/close 4555 (time 1781238789.52 total 60.04 last 69.32) - open/close 5353 (time 1781238799.54 total 70.06 last 79.61) - open/close 6198 (time 1781238809.58 total 80.10 last 84.18) - open/close 6985 (time 1781238819.58 total 90.10 last 78.68) - open/close 7772 (time 1781238829.58 total 100.11 last 78.66) - open/close 8516 (time 1781238839.59 total 110.12 last 74.33) - open/close 9252 (time 1781238849.60 total 120.13 last 73.53) - open/close 9952 (time 1781238859.61 total 130.14 last 69.93) total: 10000 open/close in 130.69 seconds: 76.52 ops/second - unlinked 0 (time 1781238864 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 724 (time 1781238961.95 total 10.00 last 72.37) - open/close 1518 (time 1781238971.96 total 20.01 last 79.32) - open/close 2199 (time 1781238981.97 total 30.03 last 68.01) - open/close 2954 (time 1781238991.98 total 40.04 last 75.42) - open/close 3604 (time 1781239001.99 total 50.04 last 64.96) - open/close 4227 (time 1781239011.99 total 60.04 last 62.30) - open/close 4868 (time 1781239021.99 total 70.05 last 64.06) - open/close 5442 (time 1781239032.00 total 80.06 last 57.36) - open/close 6106 (time 1781239042.01 total 90.06 last 66.36) - open/close 6741 (time 1781239052.01 total 100.07 last 63.46) - open/close 7516 (time 1781239062.03 total 110.09 last 77.33) - open/close 8270 (time 1781239072.04 total 120.10 last 75.37) - open/close 8950 (time 1781239082.04 total 130.10 last 67.99) - open/close 9635 (time 1781239092.06 total 140.12 last 68.37) total: 10000 open/close in 144.52 seconds: 69.19 ops/second - unlinked 0 (time 1781239100 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 822 (time 1781239202.18 total 10.01 last 82.11) - open/close 1497 (time 1781239212.20 total 20.03 last 67.39) - open/close 2186 (time 1781239222.20 total 30.03 last 68.86) - open/close 2901 (time 1781239232.24 total 40.07 last 71.23) - open/close 3335 (time 1781239242.24 total 50.08 last 43.38) - open/close 3987 (time 1781239252.25 total 60.09 last 65.13) - open/close 4685 (time 1781239262.28 total 70.11 last 69.65) - open/close 5425 (time 1781239272.28 total 80.11 last 73.97) - open/close 6208 (time 1781239282.29 total 90.13 last 78.20) - open/close 6903 (time 1781239292.30 total 100.13 last 69.43) - open/close 7605 (time 1781239302.30 total 110.14 last 70.20) - open/close 8457 (time 1781239312.31 total 120.14 last 85.17) - open/close 9227 (time 1781239322.32 total 130.16 last 76.86) total: 10000 open/close in 139.64 seconds: 71.61 ops/second - unlinked 0 (time 1781239336 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 665 (time 1781239433.62 total 10.01 last 66.41) - open/close 1336 (time 1781239443.63 total 20.02 last 67.04) - open/close 2050 (time 1781239453.64 total 30.03 last 71.35) - open/close 2797 (time 1781239463.65 total 40.04 last 74.61) - open/close 3354 (time 1781239473.67 total 50.06 last 55.59) - open/close 4094 (time 1781239483.68 total 60.07 last 73.96) - open/close 4785 (time 1781239493.68 total 70.07 last 69.09) - open/close 5406 (time 1781239503.68 total 80.07 last 62.07) - open/close 6117 (time 1781239513.70 total 90.09 last 70.97) - open/close 6790 (time 1781239523.70 total 100.09 last 67.28) - open/close 7466 (time 1781239533.71 total 110.10 last 67.57) - open/close 8253 (time 1781239543.71 total 120.10 last 78.67) - open/close 9011 (time 1781239553.71 total 130.11 last 75.79) - open/close 9755 (time 1781239563.73 total 140.12 last 74.27) total: 10000 open/close in 143.16 seconds: 69.85 ops/second - unlinked 0 (time 1781239571 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 753 (time 1781239671.77 total 10.01 last 75.25) - open/close 1564 (time 1781239681.77 total 20.01 last 81.05) - open/close 2308 (time 1781239691.78 total 30.02 last 74.35) - open/close 3094 (time 1781239701.79 total 40.03 last 78.48) - open/close 3888 (time 1781239711.80 total 50.04 last 79.33) - open/close 4537 (time 1781239721.81 total 60.05 last 64.88) - open/close 5143 (time 1781239731.82 total 70.06 last 60.52) - open/close 5807 (time 1781239741.83 total 80.07 last 66.36) - open/close 6529 (time 1781239751.83 total 90.07 last 72.18) - open/close 7227 (time 1781239761.83 total 100.07 last 69.78) - open/close 7970 (time 1781239771.85 total 110.09 last 74.19) - open/close 8682 (time 1781239781.85 total 120.09 last 71.15) - open/close 9504 (time 1781239791.86 total 130.10 last 82.16) total: 10000 open/close in 135.69 seconds: 73.70 ops/second - unlinked 0 (time 1781239802 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 807 (time 1781239901.99 total 10.00 last 80.67) - open/close 1576 (time 1781239911.99 total 20.01 last 76.85) - open/close 2315 (time 1781239922.00 total 30.01 last 73.88) - open/close 3018 (time 1781239932.00 total 40.02 last 70.25) - open/close 3797 (time 1781239942.01 total 50.02 last 77.87) - open/close 4522 (time 1781239952.03 total 60.05 last 72.34) - open/close 5184 (time 1781239962.03 total 70.05 last 66.20) - open/close 5902 (time 1781239972.03 total 80.05 last 71.78) - open/close 6739 (time 1781239982.03 total 90.05 last 83.67) - open/close 7440 (time 1781239992.06 total 100.08 last 69.93) - open/close 8203 (time 1781240002.06 total 110.08 last 76.27) - open/close 8836 (time 1781240012.07 total 120.09 last 63.23) - open/close 9408 (time 1781240022.08 total 130.10 last 57.18) total: 10000 open/close in 138.05 seconds: 72.44 ops/second - unlinked 0 (time 1781240035 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 821 (time 1781240137.05 total 10.01 last 82.06) - open/close 1564 (time 1781240147.07 total 20.02 last 74.18) - open/close 2144 (time 1781240157.09 total 30.05 last 57.86) - open/close 2872 (time 1781240167.09 total 40.05 last 72.76) - open/close 3632 (time 1781240177.10 total 50.06 last 75.96) - open/close 4448 (time 1781240187.12 total 60.08 last 81.44) - open/close 5158 (time 1781240197.13 total 70.08 last 70.96) - open/close 5817 (time 1781240207.13 total 80.09 last 65.86) - open/close 6636 (time 1781240217.13 total 90.09 last 81.86) - open/close 7328 (time 1781240227.14 total 100.09 last 69.20) - open/close 8103 (time 1781240237.14 total 110.09 last 77.49) - open/close 8882 (time 1781240247.15 total 120.10 last 77.84) - open/close 9684 (time 1781240257.15 total 130.10 last 80.17) total: 10000 open/close in 134.26 seconds: 74.49 ops/second - unlinked 0 (time 1781240266 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg239-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg239-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg239-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.139@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.139@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg239-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0000 oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg239-server: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg239-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre On OST0, 19062 used inodes rc=0 pdsh@oleg239-client: oleg239-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg239-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg239-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg239-server Permanent disk data: Device: lustre-ost1/ost1 Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.139@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.139@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre:OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.139@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.139@tcp autodegrade=on lustre:svname=lustre-OST0000 Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.202.139@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg239-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0000 oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg239-server: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg239-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg239-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg239-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg239-server stop mds service on oleg239-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg239-server unloading modules via unload_modules_local on: 'oleg239-server' oleg239-server: oleg239-server.virtnet: executing unload_modules_local oleg239-server: modules unloaded. pdsh@oleg239-client: oleg239-client: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-server: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-client: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-server: ssh exited with exit code 2