== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:34:06 (1786404846) start mds service on oleg101-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing load_modules_local oleg101-server: Loading modules from /home/green/git/lustre-release/lustre oleg101-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg101-server: detected 4 online CPUs by sysfs oleg101-server: MODOPTS_LIBCFS= oleg101-server: Force libcfs to create 2 CPU partitions oleg101-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg101-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg101-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg101-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-OST0000 oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg101-client.virtnet: -o user_xattr,flock 192.168.201.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 205 (time 1786404939.82 total 11.67 last 17.57) - open/close 386 (time 1786404950.95 total 22.79 last 16.27) - open/close 491 (time 1786404960.96 total 32.81 last 10.48) - open/close 652 (time 1786404971.41 total 43.26 last 15.40) - open/close 872 (time 1786404981.45 total 53.30 last 21.93) - open/close 1076 (time 1786404991.52 total 63.37 last 20.25) - open/close 1276 (time 1786405002.46 total 74.31 last 18.29) - open/close 1420 (time 1786405014.24 total 86.08 last 12.23) - open/close 1531 (time 1786405024.24 total 96.09 last 11.09) - open/close 1683 (time 1786405034.59 total 106.44 last 14.68) - open/close 1786 (time 1786405044.79 total 116.64 last 10.10) - open/close 1917 (time 1786405054.82 total 126.67 last 13.06) - open/close 2055 (time 1786405065.44 total 137.29 last 13.00) - open/close 2142 (time 1786405076.25 total 148.10 last 8.04) - open/close 2279 (time 1786405086.27 total 158.12 last 13.68) - open/close 2469 (time 1786405096.29 total 168.14 last 18.97) - open/close 2577 (time 1786405106.29 total 178.14 last 10.79) - open/close 2639 (time 1786405116.31 total 188.16 last 6.19) - open/close 2771 (time 1786405127.34 total 199.19 last 11.97) - open/close 2976 (time 1786405140.49 total 212.34 last 15.59) - open/close 3071 (time 1786405150.70 total 222.55 last 9.31) - open/close 3180 (time 1786405160.70 total 232.55 last 10.90) - open/close 3353 (time 1786405171.95 total 243.80 last 15.37) - open/close 3430 (time 1786405182.52 total 254.36 last 7.29) - open/close 3555 (time 1786405193.57 total 265.42 last 11.31) - open/close 3696 (time 1786405203.59 total 275.44 last 14.06) - open/close 3766 (time 1786405213.60 total 285.45 last 7.00) - open/close 3852 (time 1786405224.32 total 296.17 last 8.02) - open/close 3959 (time 1786405234.33 total 306.18 last 10.69) - open/close 4013 (time 1786405244.39 total 316.24 last 5.37) - open/close 4110 (time 1786405254.40 total 326.24 last 9.70) - open/close 4277 (time 1786405264.42 total 336.27 last 16.66) - open/close 4469 (time 1786405274.43 total 346.28 last 19.18) - open/close 4686 (time 1786405284.44 total 356.29 last 21.68) - open/close 4862 (time 1786405294.44 total 366.29 last 17.59) - open/close 4982 (time 1786405304.86 total 376.71 last 11.52) - open/close 5081 (time 1786405315.76 total 387.61 last 9.08) - open/close 5209 (time 1786405326.83 total 398.68 last 11.56) - open/close 5376 (time 1786405336.84 total 408.69 last 16.69) - open/close 5608 (time 1786405347.64 total 419.48 last 21.49) - open/close 5749 (time 1786405357.66 total 429.51 last 14.07) - open/close 5957 (time 1786405369.33 total 441.18 last 17.82) - open/close 6061 (time 1786405382.26 total 454.11 last 8.04) - open/close 6186 (time 1786405392.67 total 464.52 last 12.01) - open/close 6274 (time 1786405402.97 total 474.82 last 8.54) - open/close 6389 (time 1786405412.98 total 484.83 last 11.49) - open/close 6630 (time 1786405423.00 total 494.85 last 24.05) - open/close 6856 (time 1786405434.52 total 506.37 last 19.62) - open/close 7035 (time 1786405444.53 total 516.38 last 17.88) - open/close 7156 (time 1786405455.43 total 527.28 last 11.10) - open/close 7244 (time 1786405466.65 total 538.50 last 7.84) - open/close 7395 (time 1786405477.48 total 549.33 last 13.94) - open/close 7624 (time 1786405487.49 total 559.34 last 22.88) - open/close 7746 (time 1786405498.02 total 569.86 last 11.59) - open/close 7969 (time 1786405508.03 total 579.88 last 22.26) - open/close 8177 (time 1786405518.79 total 590.63 last 19.34) - open/close 8393 (time 1786405528.82 total 600.67 last 21.53) - open/close 8596 (time 1786405538.82 total 610.67 last 20.30) - open/close 8802 (time 1786405550.05 total 621.90 last 18.34) - open/close 8948 (time 1786405562.59 total 634.44 last 11.64) - open/close 9076 (time 1786405573.31 total 645.15 last 11.95) - open/close 9145 (time 1786405583.32 total 655.16 last 6.89) - open/close 9391 (time 1786405596.27 total 668.11 last 19.00) - open/close 9492 (time 1786405606.76 total 678.61 last 9.62) - open/close 9687 (time 1786405616.77 total 688.62 last 19.48) - open/close 9874 (time 1786405626.85 total 698.70 last 18.55) - open/close 10000 (time 1786405633.93 total 705.78 last 17.81) - open/close 10213 (time 1786405644.98 total 716.83 last 19.28) - open/close 10361 (time 1786405655.52 total 727.37 last 14.04) - open/close 10497 (time 1786405665.83 total 737.68 last 13.19) - open/close 10682 (time 1786405675.84 total 747.68 last 18.49) - open/close 10785 (time 1786405686.65 total 758.50 last 9.52) - open/close 10915 (time 1786405698.35 total 770.20 last 11.12) - open/close 11061 (time 1786405708.91 total 780.76 last 13.82) - open/close 11165 (time 1786405719.86 total 791.71 last 9.50) - open/close 11254 (time 1786405731.68 total 803.53 last 7.53) - open/close 11374 (time 1786405741.93 total 813.77 last 11.71) - open/close 11476 (time 1786405752.61 total 824.46 last 9.55) - open/close 11602 (time 1786405765.50 total 837.35 last 9.77) - open/close 11764 (time 1786405776.82 total 848.67 last 14.32) - open/close 11934 (time 1786405787.33 total 859.18 last 16.17) - open/close 12072 (time 1786405800.31 total 872.16 last 10.63) - open/close 12238 (time 1786405810.56 total 882.41 last 16.20) - open/close 12404 (time 1786405820.70 total 892.55 last 16.36) - open/close 12596 (time 1786405830.76 total 902.61 last 19.09) - open/close 12799 (time 1786405841.08 total 912.93 last 19.67) - open/close 12937 (time 1786405851.82 total 923.67 last 12.85) - open/close 13086 (time 1786405861.82 total 933.67 last 14.90) - open/close 13132 (time 1786405872.60 total 944.45 last 4.27) - open/close 13236 (time 1786405883.25 total 955.10 last 9.76) - open/close 13416 (time 1786405894.24 total 966.09 last 16.37) - open/close 13584 (time 1786405905.18 total 977.03 last 15.35) - open/close 13685 (time 1786405915.20 total 987.05 last 10.08) - open/close 13841 (time 1786405925.23 total 997.08 last 15.56) - open/close 14093 (time 1786405935.78 total 1007.63 last 23.89) - open/close 14343 (time 1786405946.02 total 1017.87 last 24.40) - open/close 14483 (time 1786405956.76 total 1028.61 last 13.04) - open/close 14689 (time 1786405967.54 total 1039.39 last 19.11) - open/close 14844 (time 1786405979.21 total 1051.06 last 13.29) - open/close 15015 (time 1786405989.21 total 1061.06 last 17.09) - open/close 15230 (time 1786405999.98 total 1071.83 last 19.97) - open/close 15472 (time 1786406010.94 total 1082.79 last 22.07) - open/close 15823 (time 1786406021.00 total 1092.85 last 34.90) - open/close 16572 (time 1786406031.01 total 1102.85 last 74.84) - open/close 17207 (time 1786406041.01 total 1112.86 last 63.49) - open/close 17935 (time 1786406051.01 total 1122.86 last 72.75) - open/close 18643 (time 1786406061.02 total 1132.87 last 70.76) - open/close 19368 (time 1786406071.03 total 1142.87 last 72.46) - open/close 19965 (time 1786406081.03 total 1152.88 last 59.65) - open/close 20574 (time 1786406091.04 total 1162.88 last 60.89) - open/close 21270 (time 1786406101.04 total 1172.89 last 69.58) - open/close 21941 (time 1786406111.04 total 1182.89 last 67.09) - open/close 22590 (time 1786406121.04 total 1192.89 last 64.88) - open/close 23367 (time 1786406131.05 total 1202.89 last 77.67) - open/close 24057 (time 1786406141.05 total 1212.90 last 68.94) - open/close 24788 (time 1786406151.05 total 1222.90 last 73.10) - open/close 25600 (time 1786406161.06 total 1232.91 last 81.14) - open/close 26313 (time 1786406171.06 total 1242.91 last 71.28) - open/close 26977 (time 1786406181.07 total 1252.92 last 66.37) - open/close 27593 (time 1786406191.08 total 1262.93 last 61.52) - open/close 28145 (time 1786406201.10 total 1272.95 last 55.11) - open/close 28623 (time 1786406211.11 total 1282.96 last 47.74) - open/close 29342 (time 1786406221.11 total 1292.96 last 71.88) - open/close 29874 (time 1786406231.13 total 1302.98 last 53.11) - open/close 30000 (time 1786406233.72 total 1305.57 last 48.53) - open/close 30777 (time 1786406243.73 total 1315.58 last 77.66) - open/close 31452 (time 1786406253.75 total 1325.60 last 67.38) - open/close 32118 (time 1786406263.75 total 1335.60 last 66.60) - open/close 32832 (time 1786406273.77 total 1345.62 last 71.25) - open/close 33536 (time 1786406283.79 total 1355.64 last 70.24) - open/close 34253 (time 1786406293.79 total 1365.64 last 71.68) - open/close 34921 (time 1786406303.80 total 1375.65 last 66.76) - open/close 35506 (time 1786406313.81 total 1385.66 last 58.46) - open/close 36251 (time 1786406323.82 total 1395.67 last 74.37) - open/close 36927 (time 1786406333.83 total 1405.68 last 67.54) - open/close 37544 (time 1786406343.83 total 1415.68 last 61.70) - open/close 38320 (time 1786406353.86 total 1425.71 last 77.42) - open/close 38977 (time 1786406363.87 total 1435.72 last 65.63) - open/close 39558 (time 1786406373.87 total 1445.72 last 58.09) - open/close 40000 (time 1786406380.96 total 1452.81 last 62.33) - open/close 40519 (time 1786406390.96 total 1462.81 last 51.89) - open/close 41262 (time 1786406400.97 total 1472.81 last 74.27) - open/close 41882 (time 1786406410.97 total 1482.82 last 61.98) - open/close 42492 (time 1786406420.97 total 1492.82 last 60.97) - open/close 43101 (time 1786406430.98 total 1502.82 last 60.89) - open/close 43814 (time 1786406441.04 total 1512.89 last 70.82) - open/close 44288 (time 1786406451.07 total 1522.92 last 47.26) - open/close 44965 (time 1786406461.08 total 1532.93 last 67.64) - open/close 45525 (time 1786406471.09 total 1542.94 last 55.95) - open/close 46139 (time 1786406481.10 total 1552.95 last 61.36) - open/close 46744 (time 1786406491.12 total 1562.97 last 60.38) - open/close 47391 (time 1786406501.12 total 1572.97 last 64.69) - open/close 47980 (time 1786406511.13 total 1582.98 last 58.82) - open/close 48496 (time 1786406521.16 total 1593.00 last 51.49) - open/close 48954 (time 1786406531.17 total 1603.02 last 45.74) - open/close 49539 (time 1786406541.18 total 1613.03 last 58.44) total: 50000 open/close in 1620.52 seconds: 30.85 ops/second - unlinked 0 (time 1786406553 ; total 0 ; last 0) - unlinked 10000 (time 1786406731 ; total 178 ; last 178) - unlinked 20000 (time 1786406915 ; total 362 ; last 184) - unlinked 30000 (time 1786407082 ; total 529 ; last 167) - unlinked 40000 (time 1786407230 ; total 677 ; last 148) total: 50000 unlinks in 822 seconds: 60.827251 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg101-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg101-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg101-server stop mds service on oleg101-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg101-server unloading modules via unload_modules_local on: 'oleg101-server' oleg101-server: oleg101-server.virtnet: executing unload_modules_local oleg101-server: modules unloaded. pdsh@oleg101-client: oleg101-client: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-client: ssh exited with exit code 2 pdsh@oleg101-client: oleg101-server: ssh exited with exit code 2