== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:20:06 (1787606406) start mds service on oleg340-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg340-server: detected 4 online CPUs by sysfs oleg340-server: MODOPTS_LIBCFS= oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg340-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg340-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg340-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0000 oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 85 (time 1787606492.95 total 10.14 last 8.39) - open/close 311 (time 1787606502.96 total 20.15 last 22.57) - open/close 504 (time 1787606512.99 total 30.18 last 19.25) - open/close 748 (time 1787606522.99 total 40.18 last 24.39) - open/close 975 (time 1787606533.92 total 51.11 last 20.77) - open/close 1268 (time 1787606543.93 total 61.12 last 29.28) - open/close 1366 (time 1787606558.20 total 75.38 last 6.87) - open/close 1607 (time 1787606568.20 total 85.38 last 24.10) - open/close 1858 (time 1787606579.16 total 96.35 last 22.89) - open/close 2064 (time 1787606589.40 total 106.59 last 20.11) - open/close 2264 (time 1787606600.34 total 117.53 last 18.28) - open/close 2443 (time 1787606610.83 total 128.02 last 17.07) - open/close 2680 (time 1787606621.85 total 139.04 last 21.50) - open/close 2869 (time 1787606631.87 total 149.06 last 18.87) - open/close 3054 (time 1787606641.88 total 159.07 last 18.48) - open/close 3288 (time 1787606651.90 total 169.09 last 23.35) - open/close 3530 (time 1787606661.92 total 179.10 last 24.16) - open/close 3779 (time 1787606671.96 total 189.15 last 24.78) - open/close 4004 (time 1787606682.79 total 199.98 last 20.79) - open/close 4227 (time 1787606692.80 total 209.99 last 22.27) - open/close 4397 (time 1787606702.81 total 220.00 last 16.99) - open/close 4531 (time 1787606713.32 total 230.51 last 12.75) - open/close 4801 (time 1787606723.33 total 240.52 last 26.98) - open/close 4923 (time 1787606733.37 total 250.55 last 12.16) - open/close 5025 (time 1787606745.24 total 262.43 last 8.59) - open/close 5157 (time 1787606757.78 total 274.97 last 10.52) - open/close 5297 (time 1787606767.78 total 284.97 last 14.00) - open/close 5520 (time 1787606778.49 total 295.67 last 20.84) - open/close 5632 (time 1787606788.49 total 305.68 last 11.19) - open/close 5740 (time 1787606798.69 total 315.88 last 10.59) - open/close 5823 (time 1787606809.06 total 326.25 last 8.00) - open/close 6004 (time 1787606819.82 total 337.00 last 16.84) - open/close 6230 (time 1787606830.25 total 347.44 last 21.66) - open/close 6393 (time 1787606840.26 total 357.45 last 16.29) - open/close 6614 (time 1787606850.52 total 367.71 last 21.53) - open/close 6800 (time 1787606860.53 total 377.72 last 18.59) - open/close 6928 (time 1787606870.56 total 387.75 last 12.76) - open/close 6979 (time 1787606880.98 total 398.16 last 4.90) - open/close 7010 (time 1787606891.11 total 408.29 last 3.06) - open/close 7153 (time 1787606901.13 total 418.32 last 14.26) - open/close 7307 (time 1787606911.89 total 429.08 last 14.31) - open/close 7545 (time 1787606921.89 total 439.08 last 23.79) - open/close 7764 (time 1787606932.24 total 449.43 last 21.17) - open/close 7905 (time 1787606942.30 total 459.49 last 14.01) - open/close 8071 (time 1787606953.00 total 470.18 last 15.52) - open/close 8193 (time 1787606963.11 total 480.30 last 12.06) - open/close 8322 (time 1787606973.60 total 490.79 last 12.30) - open/close 8422 (time 1787606984.53 total 501.72 last 9.15) - open/close 8605 (time 1787606994.77 total 511.96 last 17.88) - open/close 8803 (time 1787607004.79 total 521.97 last 19.77) - open/close 9043 (time 1787607014.79 total 531.98 last 23.99) - open/close 9278 (time 1787607024.81 total 541.99 last 23.46) - open/close 9446 (time 1787607034.82 total 552.01 last 16.78) - open/close 9670 (time 1787607044.83 total 562.01 last 22.39) - open/close 9826 (time 1787607055.46 total 572.65 last 14.67) - open/close 10000 (time 1787607065.27 total 582.46 last 17.74) - open/close 10161 (time 1787607077.65 total 594.84 last 13.01) - open/close 10387 (time 1787607090.58 total 607.77 last 17.47) - open/close 10579 (time 1787607101.92 total 619.10 last 16.94) - open/close 10695 (time 1787607113.90 total 631.09 last 9.68) - open/close 10880 (time 1787607124.33 total 641.51 last 17.74) - open/close 11002 (time 1787607135.39 total 652.58 last 11.02) - open/close 11173 (time 1787607145.40 total 662.58 last 17.09) - open/close 11408 (time 1787607156.28 total 673.46 last 21.60) - open/close 11620 (time 1787607166.32 total 683.51 last 21.10) - open/close 11695 (time 1787607176.33 total 693.52 last 7.49) - open/close 11915 (time 1787607187.15 total 704.34 last 20.32) - open/close 12123 (time 1787607197.17 total 714.36 last 20.77) - open/close 12261 (time 1787607208.16 total 725.35 last 12.55) - open/close 12330 (time 1787607218.17 total 735.35 last 6.90) - open/close 12511 (time 1787607228.82 total 746.00 last 17.00) - open/close 12718 (time 1787607240.30 total 757.48 last 18.03) - open/close 12949 (time 1787607250.78 total 767.97 last 22.02) - open/close 13166 (time 1787607261.01 total 778.20 last 21.22) - open/close 13337 (time 1787607271.02 total 788.21 last 17.08) - open/close 13568 (time 1787607281.04 total 798.23 last 23.05) - open/close 13771 (time 1787607291.93 total 809.11 last 18.65) - open/close 13958 (time 1787607302.88 total 820.07 last 17.07) - open/close 14193 (time 1787607313.65 total 830.83 last 21.84) - open/close 14394 (time 1787607323.84 total 841.02 last 19.72) - open/close 14627 (time 1787607334.35 total 851.54 last 22.16) - open/close 14831 (time 1787607344.42 total 861.61 last 20.27) - open/close 15035 (time 1787607354.44 total 871.63 last 20.35) - open/close 15195 (time 1787607364.46 total 881.65 last 15.97) - open/close 15351 (time 1787607377.33 total 894.51 last 12.13) - open/close 15457 (time 1787607387.75 total 904.93 last 10.17) - open/close 15662 (time 1787607397.90 total 915.09 last 20.19) - open/close 15876 (time 1787607407.91 total 925.10 last 21.38) - open/close 16061 (time 1787607417.93 total 935.12 last 18.46) - open/close 16289 (time 1787607428.38 total 945.57 last 21.81) - open/close 16482 (time 1787607438.38 total 955.57 last 19.29) - open/close 16631 (time 1787607449.06 total 966.25 last 13.96) - open/close 16772 (time 1787607459.62 total 976.80 last 13.36) - open/close 16943 (time 1787607469.63 total 986.82 last 17.07) - open/close 17088 (time 1787607479.65 total 996.84 last 14.47) - open/close 17324 (time 1787607489.67 total 1006.85 last 23.57) - open/close 17866 (time 1787607499.67 total 1016.86 last 54.17) - open/close 18576 (time 1787607509.68 total 1026.87 last 70.95) - open/close 19271 (time 1787607519.69 total 1036.88 last 69.42) - open/close 19903 (time 1787607529.71 total 1046.89 last 63.10) - open/close 20604 (time 1787607539.71 total 1056.90 last 70.05) - open/close 21328 (time 1787607549.71 total 1066.90 last 72.39) - open/close 21971 (time 1787607559.71 total 1076.90 last 64.29) - open/close 22548 (time 1787607569.73 total 1086.92 last 57.61) - open/close 23166 (time 1787607579.74 total 1096.93 last 61.74) - open/close 23854 (time 1787607589.74 total 1106.93 last 68.77) - open/close 24543 (time 1787607599.75 total 1116.93 last 68.88) - open/close 25234 (time 1787607609.75 total 1126.94 last 69.08) - open/close 25927 (time 1787607619.75 total 1136.94 last 69.27) - open/close 26552 (time 1787607629.76 total 1146.95 last 62.44) - open/close 27177 (time 1787607639.77 total 1156.96 last 62.45) - open/close 27748 (time 1787607649.78 total 1166.97 last 57.06) - open/close 28314 (time 1787607659.79 total 1176.98 last 56.54) - open/close 28964 (time 1787607669.81 total 1186.99 last 64.90) - open/close 29629 (time 1787607679.81 total 1197.00 last 66.47) - open/close 30000 (time 1787607686.01 total 1203.20 last 59.85) - open/close 30619 (time 1787607696.02 total 1213.21 last 61.85) - open/close 31249 (time 1787607706.03 total 1223.22 last 62.94) - open/close 31884 (time 1787607716.03 total 1233.22 last 63.49) - open/close 32548 (time 1787607726.05 total 1243.23 last 66.30) - open/close 33185 (time 1787607736.05 total 1253.24 last 63.66) - open/close 33884 (time 1787607746.05 total 1263.24 last 69.89) - open/close 34539 (time 1787607756.06 total 1273.25 last 65.45) - open/close 35193 (time 1787607766.06 total 1283.25 last 65.39) - open/close 35869 (time 1787607776.07 total 1293.26 last 67.53) - open/close 36532 (time 1787607786.09 total 1303.28 last 66.19) - open/close 37204 (time 1787607796.10 total 1313.28 last 67.16) - open/close 37828 (time 1787607806.10 total 1323.28 last 62.39) - open/close 38486 (time 1787607816.11 total 1333.30 last 65.72) - open/close 39137 (time 1787607826.11 total 1343.30 last 65.09) - open/close 39798 (time 1787607836.14 total 1353.32 last 65.93) - open/close 40000 (time 1787607839.48 total 1356.66 last 60.48) - open/close 40616 (time 1787607849.49 total 1366.68 last 61.52) - open/close 41226 (time 1787607859.49 total 1376.68 last 60.97) - open/close 41754 (time 1787607869.50 total 1386.69 last 52.76) - open/close 42307 (time 1787607879.53 total 1396.72 last 55.14) - open/close 42873 (time 1787607889.54 total 1406.73 last 56.57) - open/close 43446 (time 1787607899.55 total 1416.74 last 57.23) - open/close 44104 (time 1787607909.55 total 1426.74 last 65.78) - open/close 44682 (time 1787607919.56 total 1436.74 last 57.79) - open/close 45303 (time 1787607929.56 total 1446.74 last 62.09) - open/close 45872 (time 1787607939.57 total 1456.75 last 56.85) - open/close 46491 (time 1787607949.57 total 1466.76 last 61.86) - open/close 47159 (time 1787607959.58 total 1476.77 last 66.75) - open/close 47836 (time 1787607969.59 total 1486.78 last 67.65) - open/close 48483 (time 1787607979.60 total 1496.79 last 64.63) - open/close 49064 (time 1787607989.60 total 1506.79 last 58.08) - open/close 49662 (time 1787607999.61 total 1516.80 last 59.76) total: 50000 open/close in 1521.97 seconds: 32.85 ops/second - unlinked 0 (time 1787608008 ; total 0 ; last 0) - unlinked 10000 (time 1787608176 ; total 168 ; last 168) - unlinked 20000 (time 1787608335 ; total 327 ; last 159) - unlinked 30000 (time 1787608498 ; total 490 ; last 163) - unlinked 40000 (time 1787608661 ; total 653 ; last 163) total: 50000 unlinks in 815 seconds: 61.349693 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg340-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg340-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg340-server unloading modules via unload_modules_local on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing unload_modules_local oleg340-server: modules unloaded. pdsh@oleg340-client: oleg340-client: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-client: ssh exited with exit code 2