== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:53:12 (1709787192) start mds service on oleg452-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg452-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0000 oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg452-client.virtnet: -o user_xattr,flock oleg452-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 210 (time 1709787227.39 total 10.40 last 20.19) - open/close 384 (time 1709787238.24 total 21.25 last 16.04) - open/close 605 (time 1709787249.14 total 32.15 last 20.27) - open/close 732 (time 1709787259.82 total 42.83 last 11.89) - open/close 896 (time 1709787269.82 total 52.83 last 16.40) - open/close 995 (time 1709787280.26 total 63.27 last 9.48) - open/close 1204 (time 1709787293.27 total 76.28 last 16.07) - open/close 1325 (time 1709787303.96 total 86.97 last 11.32) - open/close 1483 (time 1709787314.26 total 97.26 last 15.35) - open/close 1620 (time 1709787325.05 total 108.06 last 12.69) - open/close 1778 (time 1709787335.29 total 118.30 last 15.43) - open/close 1945 (time 1709787347.73 total 130.74 last 13.42) - open/close 2162 (time 1709787358.56 total 141.57 last 20.04) - open/close 2410 (time 1709787369.25 total 152.26 last 23.20) - open/close 2751 (time 1709787379.80 total 162.81 last 32.30) - open/close 2889 (time 1709787390.10 total 173.10 last 13.41) - open/close 3025 (time 1709787400.10 total 183.11 last 13.59) - open/close 3190 (time 1709787410.68 total 193.69 last 15.59) - open/close 3361 (time 1709787421.49 total 204.49 last 15.83) - open/close 3616 (time 1709787431.86 total 214.87 last 24.58) - open/close 3790 (time 1709787441.86 total 224.87 last 17.40) - open/close 4020 (time 1709787451.86 total 234.87 last 23.00) - open/close 4146 (time 1709787462.29 total 245.30 last 12.08) - open/close 4336 (time 1709787472.31 total 255.31 last 18.97) - open/close 4510 (time 1709787483.32 total 266.32 last 15.80) - open/close 4695 (time 1709787493.83 total 276.83 last 17.60) - open/close 4877 (time 1709787504.09 total 287.10 last 17.74) - open/close 4992 (time 1709787514.09 total 297.10 last 11.49) - open/close 5149 (time 1709787524.10 total 307.11 last 15.69) - open/close 5267 (time 1709787534.10 total 317.11 last 11.80) - open/close 5547 (time 1709787544.11 total 327.12 last 27.98) - open/close 5688 (time 1709787555.37 total 338.38 last 12.52) - open/close 5814 (time 1709787566.44 total 349.45 last 11.38) - open/close 6110 (time 1709787576.73 total 359.73 last 28.79) - open/close 6422 (time 1709787587.53 total 370.53 last 28.89) - open/close 6728 (time 1709787597.60 total 380.61 last 30.37) - open/close 6960 (time 1709787607.61 total 390.62 last 23.18) - open/close 7166 (time 1709787618.39 total 401.40 last 19.12) - open/close 7422 (time 1709787628.95 total 411.96 last 24.24) - open/close 7599 (time 1709787639.37 total 422.38 last 16.98) - open/close 7712 (time 1709787650.10 total 433.11 last 10.53) - open/close 7939 (time 1709787660.11 total 443.12 last 22.68) - open/close 8223 (time 1709787670.13 total 453.14 last 28.33) - open/close 8379 (time 1709787681.69 total 464.70 last 13.49) - open/close 8555 (time 1709787692.20 total 475.21 last 16.76) - open/close 8849 (time 1709787702.21 total 485.22 last 29.36) - open/close 9029 (time 1709787712.23 total 495.24 last 17.97) - open/close 9245 (time 1709787723.19 total 506.20 last 19.70) - open/close 9443 (time 1709787733.20 total 516.21 last 19.77) - open/close 10000 (time 1709787742.03 total 525.04 last 63.10) - open/close 10732 (time 1709787752.03 total 535.04 last 73.20) - open/close 11337 (time 1709787762.05 total 545.06 last 60.38) - open/close 11900 (time 1709787772.07 total 555.08 last 56.21) - open/close 12400 (time 1709787782.08 total 565.08 last 49.96) - open/close 12896 (time 1709787792.08 total 575.09 last 49.59) - open/close 13473 (time 1709787802.09 total 585.10 last 57.62) - open/close 14078 (time 1709787812.11 total 595.12 last 60.40) - open/close 14625 (time 1709787822.12 total 605.13 last 54.61) - open/close 15146 (time 1709787832.12 total 615.13 last 52.10) - open/close 15698 (time 1709787842.14 total 625.14 last 55.14) - open/close 16271 (time 1709787852.14 total 635.15 last 57.30) - open/close 16876 (time 1709787862.15 total 645.16 last 60.42) - open/close 17467 (time 1709787872.16 total 655.17 last 59.05) - open/close 18069 (time 1709787882.17 total 665.18 last 60.13) - open/close 18627 (time 1709787892.18 total 675.18 last 55.76) - open/close 19200 (time 1709787902.18 total 685.19 last 57.30) - open/close 19689 (time 1709787912.18 total 695.19 last 48.89) - open/close 20000 (time 1709787918.10 total 701.10 last 52.56) - open/close 20504 (time 1709787928.12 total 711.12 last 50.30) - open/close 21038 (time 1709787938.12 total 721.13 last 53.39) - open/close 21540 (time 1709787948.12 total 731.13 last 50.17) - open/close 22088 (time 1709787958.12 total 741.13 last 54.79) - open/close 22568 (time 1709787968.13 total 751.14 last 47.98) - open/close 23029 (time 1709787978.13 total 761.14 last 46.09) - open/close 23543 (time 1709787988.14 total 771.14 last 51.38) - open/close 24075 (time 1709787998.15 total 781.16 last 53.10) - open/close 24428 (time 1709788008.16 total 791.17 last 35.28) - open/close 24790 (time 1709788018.18 total 801.19 last 36.13) - open/close 25214 (time 1709788028.19 total 811.20 last 42.35) - open/close 25703 (time 1709788038.20 total 821.21 last 48.84) - open/close 26173 (time 1709788048.22 total 831.22 last 46.95) - open/close 26606 (time 1709788058.23 total 841.24 last 43.22) - open/close 27052 (time 1709788068.25 total 851.26 last 44.54) - open/close 27508 (time 1709788078.27 total 861.28 last 45.49) - open/close 27946 (time 1709788088.28 total 871.29 last 43.76) - open/close 28369 (time 1709788098.29 total 881.30 last 42.24) - open/close 28798 (time 1709788108.30 total 891.31 last 42.86) - open/close 29216 (time 1709788118.33 total 901.33 last 41.71) - open/close 29640 (time 1709788128.34 total 911.35 last 42.33) - open/close 30000 (time 1709788138.12 total 921.13 last 36.80) - open/close 30374 (time 1709788148.13 total 931.14 last 37.37) - open/close 30721 (time 1709788158.19 total 941.20 last 34.50) - open/close 31037 (time 1709788168.22 total 951.23 last 31.51) - open/close 31392 (time 1709788178.23 total 961.23 last 35.47) - open/close 31791 (time 1709788188.23 total 971.24 last 39.88) - open/close 32118 (time 1709788198.23 total 981.24 last 32.70) - open/close 32467 (time 1709788208.25 total 991.26 last 34.83) - open/close 32858 (time 1709788218.26 total 1001.27 last 39.06) - open/close 33220 (time 1709788228.27 total 1011.28 last 36.18) - open/close 33584 (time 1709788238.31 total 1021.32 last 36.25) - open/close 33940 (time 1709788248.33 total 1031.34 last 35.54) - open/close 34320 (time 1709788258.35 total 1041.36 last 37.92) - open/close 34702 (time 1709788268.36 total 1051.37 last 38.16) - open/close 35047 (time 1709788278.37 total 1061.38 last 34.46) - open/close 35406 (time 1709788288.37 total 1071.38 last 35.89) - open/close 35784 (time 1709788298.39 total 1081.40 last 37.73) - open/close 36118 (time 1709788308.40 total 1091.41 last 33.37) - open/close 36456 (time 1709788318.42 total 1101.43 last 33.74) - open/close 36813 (time 1709788328.43 total 1111.44 last 35.66) - open/close 37167 (time 1709788338.43 total 1121.44 last 35.40) - open/close 37515 (time 1709788348.45 total 1131.46 last 34.74) - open/close 37838 (time 1709788358.45 total 1141.46 last 32.29) - open/close 38185 (time 1709788368.47 total 1151.48 last 34.64) - open/close 38539 (time 1709788378.48 total 1161.49 last 35.37) - open/close 38882 (time 1709788388.48 total 1171.49 last 34.27) - open/close 39235 (time 1709788398.52 total 1181.52 last 35.19) - open/close 39577 (time 1709788408.54 total 1191.54 last 34.13) - open/close 39886 (time 1709788418.55 total 1201.56 last 30.87) - open/close 40000 (time 1709788422.47 total 1205.47 last 29.09) - open/close 40301 (time 1709788432.47 total 1215.48 last 30.08) - open/close 40603 (time 1709788442.50 total 1225.51 last 30.11) - open/close 40912 (time 1709788452.52 total 1235.53 last 30.85) - open/close 41209 (time 1709788462.53 total 1245.54 last 29.67) - open/close 41529 (time 1709788472.55 total 1255.56 last 31.94) - open/close 41821 (time 1709788482.56 total 1265.57 last 29.17) - open/close 42109 (time 1709788492.57 total 1275.58 last 28.77) - open/close 42408 (time 1709788502.58 total 1285.59 last 29.88) - open/close 42738 (time 1709788512.59 total 1295.60 last 32.96) - open/close 43041 (time 1709788522.61 total 1305.62 last 30.23) - open/close 43323 (time 1709788532.64 total 1315.65 last 28.13) - open/close 43635 (time 1709788542.68 total 1325.69 last 31.08) - open/close 43949 (time 1709788552.70 total 1335.71 last 31.34) - open/close 44225 (time 1709788562.72 total 1345.72 last 27.55) - open/close 44521 (time 1709788572.74 total 1355.75 last 29.53) - open/close 44812 (time 1709788582.76 total 1365.77 last 29.03) - open/close 45109 (time 1709788592.79 total 1375.80 last 29.63) - open/close 45419 (time 1709788602.79 total 1385.80 last 30.99) - open/close 45710 (time 1709788612.81 total 1395.81 last 29.05) - open/close 45999 (time 1709788622.81 total 1405.82 last 28.89) - open/close 46300 (time 1709788632.82 total 1415.83 last 30.06) - open/close 46594 (time 1709788642.84 total 1425.85 last 29.34) - open/close 46868 (time 1709788652.85 total 1435.86 last 27.37) - open/close 47152 (time 1709788662.88 total 1445.89 last 28.32) - open/close 47381 (time 1709788672.91 total 1455.92 last 22.84) - open/close 47559 (time 1709788682.93 total 1465.94 last 17.76) - open/close 47749 (time 1709788692.95 total 1475.95 last 18.97) - open/close 47962 (time 1709788702.97 total 1485.98 last 21.24) - open/close 48225 (time 1709788712.97 total 1495.98 last 26.30) - open/close 48471 (time 1709788723.00 total 1506.01 last 24.53) - open/close 48736 (time 1709788733.05 total 1516.05 last 26.38) - open/close 48986 (time 1709788743.05 total 1526.06 last 24.99) - open/close 49220 (time 1709788753.06 total 1536.07 last 23.38) - open/close 49489 (time 1709788763.09 total 1546.10 last 26.81) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1553.80 seconds: 31.98 ops/second - unlinked 0 (time 1709788772 ; total 0 ; last 0) - unlinked 10000 (time 1709789197 ; total 425 ; last 425) - unlinked 20000 (time 1709789542 ; total 770 ; last 345) - unlinked 30000 (time 1709789770 ; total 998 ; last 228) - unlinked 40000 (time 1709789870 ; total 1098 ; last 100) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1138 seconds: 43.671352 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg452-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg452-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg452-server stop mds service on oleg452-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg452-server unloading modules on: 'oleg452-server' oleg452-server: oleg452-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg452-client: oleg452-client: ssh exited with exit code 2 pdsh@oleg452-client: oleg452-server: ssh exited with exit code 2 pdsh@oleg452-client: oleg452-client: ssh exited with exit code 2 pdsh@oleg452-client: oleg452-server: ssh exited with exit code 2