== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:36:10 (1787661370) start mds service on oleg346-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: Force libcfs to create 2 CPU partitions oleg346-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg346-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg346-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg346-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 125 (time 1787661459.71 total 10.00 last 12.50) - open/close 364 (time 1787661469.73 total 20.02 last 23.85) - open/close 514 (time 1787661480.52 total 30.82 last 13.90) - open/close 673 (time 1787661490.54 total 40.83 last 15.88) - open/close 743 (time 1787661501.52 total 51.82 last 6.37) - open/close 927 (time 1787661511.64 total 61.94 last 18.19) - open/close 1171 (time 1787661522.70 total 73.00 last 22.05) - open/close 1332 (time 1787661536.19 total 86.49 last 11.93) - open/close 1563 (time 1787661546.38 total 96.68 last 22.68) - open/close 1806 (time 1787661556.77 total 107.06 last 23.40) - open/close 2032 (time 1787661567.19 total 117.49 last 21.67) - open/close 2259 (time 1787661577.48 total 127.78 last 22.06) - open/close 2475 (time 1787661587.48 total 137.78 last 21.60) - open/close 2736 (time 1787661597.50 total 147.80 last 26.06) - open/close 2847 (time 1787661609.03 total 159.32 last 9.63) - open/close 3087 (time 1787661619.81 total 170.11 last 22.26) - open/close 3265 (time 1787661629.84 total 180.13 last 17.75) - open/close 3389 (time 1787661639.97 total 190.26 last 12.24) - open/close 3476 (time 1787661650.06 total 200.36 last 8.61) - open/close 3632 (time 1787661660.55 total 210.85 last 14.87) - open/close 3757 (time 1787661670.56 total 220.86 last 12.49) - open/close 3957 (time 1787661680.58 total 230.88 last 19.96) - open/close 4179 (time 1787661690.82 total 241.12 last 21.68) - open/close 4404 (time 1787661700.83 total 251.13 last 22.48) - open/close 4611 (time 1787661710.83 total 261.13 last 20.70) - open/close 4827 (time 1787661720.97 total 271.26 last 21.31) - open/close 5064 (time 1787661731.35 total 281.65 last 22.82) - open/close 5248 (time 1787661741.36 total 291.66 last 18.38) - open/close 5443 (time 1787661751.49 total 301.79 last 19.26) - open/close 5652 (time 1787661761.66 total 311.96 last 20.54) - open/close 5872 (time 1787661771.86 total 322.16 last 21.58) - open/close 6136 (time 1787661782.24 total 332.54 last 25.43) - open/close 6200 (time 1787661792.54 total 342.84 last 6.21) - open/close 6420 (time 1787661802.55 total 352.85 last 21.98) - open/close 6589 (time 1787661813.20 total 363.50 last 15.87) - open/close 6796 (time 1787661823.21 total 373.51 last 20.68) - open/close 7030 (time 1787661834.25 total 384.54 last 21.20) - open/close 7268 (time 1787661844.26 total 394.56 last 23.77) - open/close 7442 (time 1787661854.81 total 405.11 last 16.49) - open/close 7633 (time 1787661864.84 total 415.14 last 19.03) - open/close 7876 (time 1787661875.77 total 426.07 last 22.24) - open/close 8094 (time 1787661886.46 total 436.76 last 20.39) - open/close 8359 (time 1787661896.49 total 446.79 last 26.44) - open/close 8576 (time 1787661907.42 total 457.72 last 19.85) - open/close 8685 (time 1787661917.44 total 467.73 last 10.88) - open/close 8921 (time 1787661927.44 total 477.74 last 23.59) - open/close 9142 (time 1787661937.44 total 487.74 last 22.10) - open/close 9454 (time 1787661947.46 total 497.75 last 31.17) - open/close 9744 (time 1787661957.47 total 507.77 last 28.96) - open/close 9932 (time 1787661967.82 total 518.12 last 18.16) - open/close 10000 (time 1787661971.54 total 521.84 last 18.29) - open/close 10123 (time 1787661981.77 total 532.06 last 12.03) - open/close 10296 (time 1787661991.78 total 542.07 last 17.28) - open/close 10500 (time 1787662001.79 total 552.09 last 20.37) - open/close 10723 (time 1787662011.79 total 562.09 last 22.29) - open/close 10928 (time 1787662021.81 total 572.11 last 20.46) - open/close 11149 (time 1787662032.07 total 582.37 last 21.53) - open/close 11372 (time 1787662042.08 total 592.37 last 22.29) - open/close 11592 (time 1787662052.99 total 603.29 last 20.15) - open/close 11786 (time 1787662065.54 total 615.84 last 15.46) - open/close 11985 (time 1787662075.58 total 625.88 last 19.82) - open/close 12114 (time 1787662086.59 total 636.89 last 11.72) - open/close 12356 (time 1787662097.62 total 647.92 last 21.94) - open/close 12594 (time 1787662107.64 total 657.94 last 23.76) - open/close 12763 (time 1787662117.65 total 667.94 last 16.89) - open/close 12966 (time 1787662127.72 total 678.01 last 20.16) - open/close 13106 (time 1787662138.39 total 688.69 last 13.11) - open/close 13262 (time 1787662148.40 total 698.70 last 15.58) - open/close 13431 (time 1787662158.41 total 708.70 last 16.89) - open/close 13652 (time 1787662168.42 total 718.71 last 22.08) - open/close 13862 (time 1787662179.41 total 729.71 last 19.10) - open/close 14117 (time 1787662190.44 total 740.73 last 23.13) - open/close 14361 (time 1787662200.96 total 751.26 last 23.18) - open/close 14630 (time 1787662210.97 total 761.27 last 26.88) - open/close 14830 (time 1787662220.98 total 771.28 last 19.98) - open/close 15078 (time 1787662231.46 total 781.76 last 23.67) - open/close 15315 (time 1787662241.48 total 791.77 last 23.66) - open/close 15524 (time 1787662252.50 total 802.79 last 18.96) - open/close 15727 (time 1787662262.98 total 813.28 last 19.37) - open/close 15807 (time 1787662273.88 total 824.17 last 7.34) - open/close 15905 (time 1787662284.49 total 834.79 last 9.23) - open/close 16073 (time 1787662294.50 total 844.79 last 16.79) - open/close 16263 (time 1787662304.50 total 854.80 last 18.99) - open/close 16482 (time 1787662314.89 total 865.18 last 21.09) - open/close 16673 (time 1787662324.89 total 875.19 last 19.10) - open/close 16767 (time 1787662335.21 total 885.51 last 9.11) - open/close 16938 (time 1787662346.25 total 896.55 last 15.49) - open/close 17075 (time 1787662356.56 total 906.86 last 13.29) - open/close 17313 (time 1787662366.57 total 916.87 last 23.77) - open/close 17567 (time 1787662376.58 total 926.87 last 25.38) - open/close 18146 (time 1787662386.59 total 936.89 last 57.81) - open/close 18765 (time 1787662396.61 total 946.91 last 61.80) - open/close 19388 (time 1787662406.62 total 956.92 last 62.24) - open/close 20000 (time 1787662415.27 total 965.57 last 70.70) - open/close 20666 (time 1787662425.28 total 975.58 last 66.55) - open/close 21382 (time 1787662435.30 total 985.60 last 71.49) - open/close 22031 (time 1787662445.31 total 995.61 last 64.82) - open/close 22675 (time 1787662455.33 total 1005.62 last 64.30) - open/close 23201 (time 1787662465.33 total 1015.62 last 52.60) - open/close 23644 (time 1787662475.33 total 1025.63 last 44.27) - open/close 24314 (time 1787662485.34 total 1035.64 last 66.95) - open/close 25006 (time 1787662495.34 total 1045.64 last 69.17) - open/close 25672 (time 1787662505.34 total 1055.64 last 66.60) - open/close 26366 (time 1787662515.35 total 1065.64 last 69.38) - open/close 26986 (time 1787662525.35 total 1075.65 last 61.97) - open/close 27629 (time 1787662535.36 total 1085.66 last 64.24) - open/close 28241 (time 1787662545.37 total 1095.67 last 61.15) - open/close 28907 (time 1787662555.38 total 1105.68 last 66.52) - open/close 29484 (time 1787662565.42 total 1115.71 last 57.50) - open/close 30000 (time 1787662575.04 total 1125.33 last 53.64) - open/close 30593 (time 1787662585.04 total 1135.34 last 59.25) - open/close 31275 (time 1787662595.05 total 1145.35 last 68.13) - open/close 31967 (time 1787662605.06 total 1155.35 last 69.19) - open/close 32664 (time 1787662615.06 total 1165.36 last 69.68) - open/close 33325 (time 1787662625.09 total 1175.39 last 65.88) - open/close 34020 (time 1787662635.10 total 1185.40 last 69.43) - open/close 34681 (time 1787662645.11 total 1195.40 last 66.08) - open/close 35313 (time 1787662655.11 total 1205.41 last 63.15) - open/close 35988 (time 1787662665.12 total 1215.41 last 67.48) - open/close 36663 (time 1787662675.13 total 1225.43 last 67.39) - open/close 37264 (time 1787662685.14 total 1235.44 last 60.05) - open/close 37895 (time 1787662695.15 total 1245.44 last 63.05) - open/close 38591 (time 1787662705.15 total 1255.44 last 69.60) - open/close 39235 (time 1787662715.16 total 1265.46 last 64.32) - open/close 39838 (time 1787662725.17 total 1275.47 last 60.22) - open/close 40000 (time 1787662727.81 total 1278.10 last 61.53) - open/close 40689 (time 1787662737.82 total 1288.12 last 68.81) - open/close 41319 (time 1787662747.83 total 1298.12 last 62.96) - open/close 41982 (time 1787662757.83 total 1308.12 last 66.29) - open/close 42595 (time 1787662767.83 total 1318.13 last 61.27) - open/close 43221 (time 1787662777.84 total 1328.13 last 62.57) - open/close 43873 (time 1787662787.86 total 1338.15 last 65.08) - open/close 44509 (time 1787662797.86 total 1348.16 last 63.57) - open/close 45151 (time 1787662807.87 total 1358.17 last 64.15) - open/close 45818 (time 1787662817.88 total 1368.17 last 66.66) - open/close 46393 (time 1787662827.89 total 1378.19 last 57.41) - open/close 47036 (time 1787662837.89 total 1388.19 last 64.28) - open/close 47664 (time 1787662847.90 total 1398.20 last 62.76) - open/close 48309 (time 1787662857.90 total 1408.20 last 64.49) - open/close 48933 (time 1787662867.92 total 1418.22 last 62.30) - open/close 49575 (time 1787662877.93 total 1428.22 last 64.15) total: 50000 open/close in 1434.56 seconds: 34.85 ops/second - unlinked 0 (time 1787662889 ; total 0 ; last 0) - unlinked 10000 (time 1787663055 ; total 166 ; last 166) - unlinked 20000 (time 1787663217 ; total 328 ; last 162) - unlinked 30000 (time 1787663379 ; total 490 ; last 162) - unlinked 40000 (time 1787663544 ; total 655 ; last 165) total: 50000 unlinks in 821 seconds: 60.901340 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server unloading modules via unload_modules_local on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing unload_modules_local oleg346-server: modules unloaded. pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2