== conf-sanity test 69: replace an OST with the same index ========================================================== 12:20:12 (1781281212) start mds service on oleg607-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg607-server' oleg607-server: oleg607-server.virtnet: executing load_modules_local oleg607-server: Loading modules from /home/green/git/lustre-release/lustre oleg607-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg607-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg607-server: detected 4 online CPUs by sysfs oleg607-server: Force libcfs to create 2 CPU partitions oleg607-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg607-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg607-server: oleg607-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg607-client: oleg607-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg607-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg607-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg607-server: oleg607-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg607-client: oleg607-server: ssh exited with exit code 1 Started lustre-OST0000 oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg607-client.virtnet: -o user_xattr,flock oleg607-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 635 (time 1781281290.38 total 10.00 last 63.50) - open/close 1240 (time 1781281300.39 total 20.00 last 60.48) - open/close 1893 (time 1781281310.39 total 30.01 last 65.26) - open/close 2566 (time 1781281320.41 total 40.03 last 67.18) - open/close 3223 (time 1781281330.42 total 50.03 last 65.66) - open/close 3910 (time 1781281340.42 total 60.04 last 68.68) - open/close 4570 (time 1781281350.42 total 70.04 last 65.98) - open/close 5205 (time 1781281360.43 total 80.05 last 63.43) - open/close 5846 (time 1781281370.45 total 90.06 last 64.02) - open/close 6453 (time 1781281380.45 total 100.07 last 60.65) - open/close 7069 (time 1781281390.46 total 110.08 last 61.58) - open/close 7719 (time 1781281400.47 total 120.08 last 64.94) - open/close 8368 (time 1781281410.48 total 130.10 last 64.82) - open/close 9056 (time 1781281420.48 total 140.10 last 68.79) - open/close 9703 (time 1781281430.49 total 150.11 last 64.63) total: 10000 open/close in 154.61 seconds: 64.68 ops/second - unlinked 0 (time 1781281439 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 769 (time 1781281540.83 total 10.02 last 76.78) - open/close 1492 (time 1781281550.83 total 20.02 last 72.28) - open/close 2197 (time 1781281560.83 total 30.02 last 70.48) - open/close 2882 (time 1781281570.84 total 40.02 last 68.48) - open/close 3669 (time 1781281580.84 total 50.03 last 78.67) - open/close 4405 (time 1781281590.84 total 60.03 last 73.60) - open/close 5124 (time 1781281600.84 total 70.03 last 71.90) - open/close 5741 (time 1781281610.85 total 80.04 last 61.63) - open/close 6414 (time 1781281620.86 total 90.04 last 67.28) - open/close 7001 (time 1781281630.87 total 100.05 last 58.63) - open/close 7643 (time 1781281640.87 total 110.06 last 64.15) - open/close 8299 (time 1781281650.89 total 120.07 last 65.52) - open/close 8959 (time 1781281660.89 total 130.08 last 65.98) - open/close 9628 (time 1781281670.90 total 140.09 last 66.83) total: 10000 open/close in 145.20 seconds: 68.87 ops/second - unlinked 0 (time 1781281679 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 686 (time 1781281775.15 total 10.02 last 68.47) - open/close 1400 (time 1781281785.15 total 20.02 last 71.38) - open/close 2131 (time 1781281795.15 total 30.02 last 73.10) - open/close 2885 (time 1781281805.15 total 40.02 last 75.39) - open/close 3653 (time 1781281815.18 total 50.05 last 76.60) - open/close 4427 (time 1781281825.19 total 60.06 last 77.34) - open/close 5229 (time 1781281835.20 total 70.07 last 80.13) - open/close 6027 (time 1781281845.20 total 80.07 last 79.78) - open/close 6729 (time 1781281855.20 total 90.07 last 70.18) - open/close 7488 (time 1781281865.20 total 100.07 last 75.89) - open/close 8243 (time 1781281875.22 total 110.09 last 75.35) - open/close 8954 (time 1781281885.23 total 120.10 last 71.06) - open/close 9719 (time 1781281895.24 total 130.11 last 76.41) total: 10000 open/close in 133.71 seconds: 74.79 ops/second - unlinked 0 (time 1781281901 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 709 (time 1781281981.31 total 10.01 last 70.84) - open/close 1580 (time 1781281991.31 total 20.01 last 87.05) - open/close 2421 (time 1781282001.32 total 30.02 last 84.07) - open/close 3240 (time 1781282011.32 total 40.02 last 81.88) - open/close 4028 (time 1781282021.32 total 50.02 last 78.76) - open/close 4853 (time 1781282031.32 total 60.02 last 82.50) - open/close 5574 (time 1781282041.33 total 70.03 last 72.05) - open/close 6423 (time 1781282051.34 total 80.04 last 84.85) - open/close 7139 (time 1781282061.34 total 90.05 last 71.54) - open/close 7976 (time 1781282071.35 total 100.06 last 83.62) - open/close 8767 (time 1781282081.37 total 110.07 last 79.00) - open/close 9512 (time 1781282091.37 total 120.07 last 74.48) total: 10000 open/close in 126.15 seconds: 79.27 ops/second - unlinked 0 (time 1781282100 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 711 (time 1781282182.93 total 10.00 last 71.08) - open/close 1366 (time 1781282192.94 total 20.01 last 65.47) - open/close 2035 (time 1781282202.95 total 30.02 last 66.84) - open/close 2882 (time 1781282212.96 total 40.03 last 84.58) - open/close 3656 (time 1781282222.97 total 50.04 last 77.35) - open/close 4455 (time 1781282232.98 total 60.05 last 79.84) - open/close 5174 (time 1781282242.98 total 70.05 last 71.89) - open/close 5929 (time 1781282252.98 total 80.05 last 75.49) - open/close 6758 (time 1781282262.99 total 90.06 last 82.79) - open/close 7529 (time 1781282273.00 total 100.07 last 77.05) - open/close 8312 (time 1781282283.00 total 110.07 last 78.28) - open/close 9067 (time 1781282293.00 total 120.07 last 75.49) - open/close 9810 (time 1781282303.01 total 130.08 last 74.27) total: 10000 open/close in 132.56 seconds: 75.44 ops/second - unlinked 0 (time 1781282309 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 803 (time 1781282404.63 total 10.01 last 80.25) - open/close 1563 (time 1781282414.64 total 20.01 last 75.94) - open/close 2315 (time 1781282424.64 total 30.02 last 75.18) - open/close 3004 (time 1781282434.65 total 40.03 last 68.81) - open/close 3709 (time 1781282444.65 total 50.03 last 70.48) - open/close 4394 (time 1781282454.66 total 60.04 last 68.43) - open/close 5072 (time 1781282464.67 total 70.05 last 67.73) - open/close 5709 (time 1781282474.68 total 80.05 last 63.68) - open/close 6397 (time 1781282484.69 total 90.07 last 68.72) - open/close 7064 (time 1781282494.70 total 100.08 last 66.63) - open/close 7747 (time 1781282504.71 total 110.09 last 68.22) - open/close 8339 (time 1781282514.72 total 120.10 last 59.15) - open/close 8960 (time 1781282524.73 total 130.11 last 62.01) - open/close 9619 (time 1781282534.74 total 140.12 last 65.86) total: 10000 open/close in 145.69 seconds: 68.64 ops/second - unlinked 0 (time 1781282544 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 623 (time 1781282647.12 total 10.01 last 62.23) - open/close 1296 (time 1781282657.12 total 20.02 last 67.26) - open/close 1953 (time 1781282667.14 total 30.03 last 65.62) - open/close 2629 (time 1781282677.14 total 40.03 last 67.57) - open/close 3315 (time 1781282687.15 total 50.04 last 68.54) - open/close 3993 (time 1781282697.16 total 60.05 last 67.73) - open/close 4608 (time 1781282707.17 total 70.06 last 61.45) - open/close 5318 (time 1781282717.18 total 80.07 last 70.94) - open/close 6025 (time 1781282727.18 total 90.08 last 70.65) - open/close 6719 (time 1781282737.19 total 100.09 last 69.34) - open/close 7460 (time 1781282747.20 total 110.09 last 74.07) - open/close 8161 (time 1781282757.21 total 120.11 last 69.99) - open/close 8838 (time 1781282767.22 total 130.12 last 67.63) - open/close 9535 (time 1781282777.23 total 140.12 last 69.64) total: 10000 open/close in 146.26 seconds: 68.37 ops/second - unlinked 0 (time 1781282786 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 720 (time 1781282883.33 total 10.01 last 71.96) - open/close 1391 (time 1781282893.34 total 20.02 last 67.03) - open/close 2114 (time 1781282903.34 total 30.02 last 72.25) - open/close 2836 (time 1781282913.35 total 40.03 last 72.16) - open/close 3544 (time 1781282923.36 total 50.04 last 70.74) - open/close 4225 (time 1781282933.37 total 60.05 last 68.03) - open/close 4937 (time 1781282943.37 total 70.05 last 71.19) - open/close 5631 (time 1781282953.38 total 80.06 last 69.33) - open/close 6370 (time 1781282963.38 total 90.06 last 73.89) - open/close 7150 (time 1781282973.39 total 100.06 last 77.97) - open/close 7953 (time 1781282983.39 total 110.07 last 80.26) - open/close 8750 (time 1781282993.40 total 120.07 last 79.67) - open/close 9626 (time 1781283003.40 total 130.07 last 87.60) total: 10000 open/close in 134.83 seconds: 74.17 ops/second - unlinked 0 (time 1781283012 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 873 (time 1781283102.75 total 10.00 last 87.29) - open/close 1803 (time 1781283112.76 total 20.01 last 92.94) - open/close 2631 (time 1781283122.76 total 30.01 last 82.78) - open/close 3432 (time 1781283132.77 total 40.02 last 80.06) - open/close 4190 (time 1781283142.78 total 50.02 last 75.75) - open/close 5010 (time 1781283152.78 total 60.03 last 81.98) - open/close 5706 (time 1781283162.78 total 70.03 last 69.57) - open/close 6481 (time 1781283172.79 total 80.03 last 77.47) - open/close 7206 (time 1781283182.80 total 90.04 last 72.44) - open/close 7925 (time 1781283192.81 total 100.05 last 71.82) - open/close 8591 (time 1781283202.81 total 110.06 last 66.55) - open/close 9277 (time 1781283212.82 total 120.07 last 68.55) total: 10000 open/close in 129.89 seconds: 76.99 ops/second - unlinked 0 (time 1781283226 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 749 (time 1781283316.16 total 10.02 last 74.75) - open/close 1477 (time 1781283326.17 total 20.02 last 72.78) - open/close 2232 (time 1781283336.18 total 30.03 last 75.43) - open/close 2941 (time 1781283346.18 total 40.04 last 70.84) - open/close 3629 (time 1781283356.19 total 50.05 last 68.74) - open/close 4336 (time 1781283366.20 total 60.06 last 70.64) - open/close 5064 (time 1781283376.21 total 70.07 last 72.71) - open/close 5839 (time 1781283386.21 total 80.07 last 77.49) - open/close 6631 (time 1781283396.22 total 90.08 last 79.14) - open/close 7373 (time 1781283406.22 total 100.08 last 74.20) - open/close 8136 (time 1781283416.23 total 110.08 last 76.28) - open/close 8917 (time 1781283426.23 total 120.08 last 78.08) - open/close 9632 (time 1781283436.23 total 130.09 last 71.45) total: 10000 open/close in 135.12 seconds: 74.01 ops/second - unlinked 0 (time 1781283445 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg607-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg607-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg607-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.206.107@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.206.107@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg607-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg607-server: oleg607-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg607-client: oleg607-server: ssh exited with exit code 1 Started lustre-OST0000 oleg607-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg607-server: oleg607-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg607-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg607-client.virtnet: -o user_xattr,flock oleg607-server@tcp:/lustre /mnt/lustre On OST0, 10357 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg607-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg607-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg607-server stop mds service on oleg607-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg607-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg607-server' oleg607-server: oleg607-server.virtnet: executing unload_modules_local oleg607-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg607-server: LNET ready to unload oleg607-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg607-client: oleg607-client: ssh exited with exit code 2 pdsh@oleg607-client: oleg607-server: ssh exited with exit code 2 pdsh@oleg607-client: oleg607-server: ssh exited with exit code 2 pdsh@oleg607-client: oleg607-client: ssh exited with exit code 2