== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:49:34 (1709786974) start mds service on oleg429-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg429-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg429-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg429-server: oleg429-server.virtnet: executing set_default_debug -1 all pdsh@oleg429-client: oleg429-server: ssh exited with exit code 1 Started lustre-OST0000 oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg429-client.virtnet: -o user_xattr,flock oleg429-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 142 (time 1709787020.26 total 10.50 last 13.52) - open/close 491 (time 1709787030.30 total 20.54 last 34.76) - open/close 623 (time 1709787040.30 total 30.55 last 13.19) - open/close 710 (time 1709787050.30 total 40.55 last 8.70) - open/close 964 (time 1709787060.31 total 50.55 last 25.40) - open/close 1243 (time 1709787071.26 total 61.50 last 25.47) - open/close 1441 (time 1709787081.88 total 72.12 last 18.64) - open/close 1734 (time 1709787092.83 total 83.07 last 26.77) - open/close 2052 (time 1709787102.83 total 93.07 last 31.78) - open/close 2313 (time 1709787112.88 total 103.13 last 25.97) - open/close 2652 (time 1709787122.89 total 113.13 last 33.88) - open/close 2926 (time 1709787133.55 total 123.79 last 25.71) - open/close 3179 (time 1709787144.15 total 134.39 last 23.86) - open/close 3408 (time 1709787155.10 total 145.34 last 20.91) - open/close 3589 (time 1709787165.12 total 155.36 last 18.07) - open/close 3942 (time 1709787175.13 total 165.37 last 35.26) - open/close 4236 (time 1709787185.77 total 176.02 last 27.61) - open/close 4508 (time 1709787195.89 total 186.13 last 26.89) - open/close 4768 (time 1709787206.43 total 196.67 last 24.66) - open/close 4976 (time 1709787217.16 total 207.40 last 19.39) - open/close 5226 (time 1709787227.19 total 217.44 last 24.90) - open/close 5474 (time 1709787237.79 total 228.03 last 23.41) - open/close 5677 (time 1709787248.10 total 238.34 last 19.69) - open/close 5910 (time 1709787258.10 total 248.35 last 23.29) - open/close 6132 (time 1709787268.98 total 259.22 last 20.41) - open/close 6293 (time 1709787278.99 total 269.23 last 16.09) - open/close 6412 (time 1709787289.35 total 279.60 last 11.48) - open/close 6553 (time 1709787300.21 total 290.45 last 12.99) - open/close 6757 (time 1709787310.49 total 300.73 last 19.85) - open/close 6970 (time 1709787321.23 total 311.47 last 19.83) - open/close 7133 (time 1709787331.23 total 321.47 last 16.29) - open/close 7295 (time 1709787341.56 total 331.80 last 15.68) - open/close 7471 (time 1709787351.56 total 341.80 last 17.60) - open/close 7690 (time 1709787361.56 total 351.81 last 21.90) - open/close 7828 (time 1709787372.19 total 362.43 last 12.99) - open/close 7932 (time 1709787382.58 total 372.82 last 10.01) - open/close 8207 (time 1709787393.39 total 383.64 last 25.42) - open/close 8376 (time 1709787403.40 total 393.64 last 16.89) - open/close 8576 (time 1709787413.46 total 403.70 last 19.89) - open/close 8867 (time 1709787424.47 total 414.71 last 26.43) - open/close 9056 (time 1709787434.79 total 425.03 last 18.30) - open/close 9242 (time 1709787444.81 total 435.05 last 18.57) - open/close 9407 (time 1709787455.07 total 445.31 last 16.08) - open/close 9591 (time 1709787465.07 total 455.31 last 18.40) - open/close 9712 (time 1709787477.68 total 467.92 last 9.60) - open/close 9881 (time 1709787488.07 total 478.31 last 16.26) - open/close 10000 (time 1709787497.90 total 488.15 last 12.10) - open/close 10256 (time 1709787507.90 total 498.15 last 25.60) - open/close 10342 (time 1709787517.94 total 508.19 last 8.57) - open/close 10464 (time 1709787528.22 total 518.46 last 11.88) - open/close 10610 (time 1709787538.69 total 528.93 last 13.94) - open/close 10915 (time 1709787548.69 total 538.93 last 30.50) - open/close 11461 (time 1709787558.69 total 548.94 last 54.59) - open/close 12061 (time 1709787568.71 total 558.95 last 59.92) - open/close 12615 (time 1709787578.72 total 568.96 last 55.34) - open/close 13193 (time 1709787588.73 total 578.97 last 57.73) - open/close 13769 (time 1709787598.75 total 588.99 last 57.48) - open/close 14335 (time 1709787608.76 total 599.00 last 56.56) - open/close 14847 (time 1709787618.76 total 609.00 last 51.19) - open/close 15356 (time 1709787628.76 total 619.00 last 50.89) - open/close 15846 (time 1709787638.77 total 629.01 last 48.96) - open/close 16338 (time 1709787648.80 total 639.04 last 49.07) - open/close 16832 (time 1709787658.81 total 649.05 last 49.34) - open/close 17316 (time 1709787668.82 total 659.06 last 48.33) - open/close 17735 (time 1709787678.86 total 669.10 last 41.76) - open/close 18119 (time 1709787688.88 total 679.12 last 38.32) - open/close 18516 (time 1709787698.91 total 689.15 last 39.59) - open/close 18909 (time 1709787708.91 total 699.16 last 39.27) - open/close 19278 (time 1709787718.93 total 709.17 last 36.84) - open/close 19696 (time 1709787728.93 total 719.17 last 41.79) - open/close 20000 (time 1709787736.19 total 726.43 last 41.90) - open/close 20460 (time 1709787746.20 total 736.44 last 45.94) - open/close 20865 (time 1709787756.25 total 746.49 last 40.31) - open/close 21253 (time 1709787766.27 total 756.51 last 38.72) - open/close 21600 (time 1709787776.27 total 766.52 last 34.67) - open/close 21939 (time 1709787786.28 total 776.53 last 33.87) - open/close 22281 (time 1709787796.29 total 786.53 last 34.18) - open/close 22653 (time 1709787806.29 total 796.54 last 37.19) - open/close 23014 (time 1709787816.34 total 806.58 last 35.93) - open/close 23347 (time 1709787826.37 total 816.61 last 33.21) - open/close 23695 (time 1709787836.37 total 826.61 last 34.80) - open/close 24066 (time 1709787846.38 total 836.63 last 37.04) - open/close 24456 (time 1709787856.38 total 846.63 last 39.00) - open/close 24875 (time 1709787866.42 total 856.66 last 41.76) - open/close 25275 (time 1709787876.46 total 866.70 last 39.83) - open/close 25677 (time 1709787886.47 total 876.71 last 40.16) - open/close 26058 (time 1709787896.49 total 886.74 last 38.02) - open/close 26420 (time 1709787906.52 total 896.76 last 36.12) - open/close 26763 (time 1709787916.53 total 906.77 last 34.26) - open/close 27114 (time 1709787926.55 total 916.79 last 35.02) - open/close 27477 (time 1709787936.56 total 926.80 last 36.27) - open/close 27832 (time 1709787946.59 total 936.83 last 35.40) - open/close 28180 (time 1709787956.60 total 946.85 last 34.74) - open/close 28522 (time 1709787966.62 total 956.87 last 34.13) - open/close 28844 (time 1709787976.63 total 966.88 last 32.17) - open/close 29180 (time 1709787986.66 total 976.90 last 33.51) - open/close 29526 (time 1709787996.67 total 986.91 last 34.57) - open/close 29873 (time 1709788006.69 total 996.93 last 34.64) - open/close 30000 (time 1709788010.35 total 1000.59 last 34.66) - open/close 30340 (time 1709788020.36 total 1010.61 last 33.96) - open/close 30694 (time 1709788030.37 total 1020.62 last 35.36) - open/close 31022 (time 1709788040.38 total 1030.62 last 32.78) - open/close 31307 (time 1709788050.38 total 1040.62 last 28.50) - open/close 31606 (time 1709788060.39 total 1050.63 last 29.88) - open/close 31910 (time 1709788070.40 total 1060.64 last 30.38) - open/close 32230 (time 1709788080.40 total 1070.64 last 31.99) - open/close 32543 (time 1709788090.41 total 1080.65 last 31.27) - open/close 32866 (time 1709788100.44 total 1090.68 last 32.20) - open/close 33187 (time 1709788110.47 total 1100.71 last 32.03) - open/close 33503 (time 1709788120.49 total 1110.73 last 31.52) - open/close 33812 (time 1709788130.51 total 1120.75 last 30.83) - open/close 34106 (time 1709788140.54 total 1130.79 last 29.30) - open/close 34413 (time 1709788150.57 total 1140.81 last 30.63) - open/close 34693 (time 1709788160.58 total 1150.82 last 27.97) - open/close 34968 (time 1709788170.63 total 1160.88 last 27.34) - open/close 35253 (time 1709788180.64 total 1170.88 last 28.48) - open/close 35558 (time 1709788190.66 total 1180.90 last 30.44) - open/close 35837 (time 1709788200.68 total 1190.92 last 27.86) - open/close 36119 (time 1709788210.68 total 1200.92 last 28.18) - open/close 36389 (time 1709788220.73 total 1210.97 last 26.88) - open/close 36665 (time 1709788230.75 total 1220.99 last 27.53) - open/close 36940 (time 1709788240.77 total 1231.01 last 27.45) - open/close 37195 (time 1709788250.78 total 1241.02 last 25.48) - open/close 37490 (time 1709788260.80 total 1251.04 last 29.44) - open/close 37769 (time 1709788270.81 total 1261.05 last 27.86) - open/close 38031 (time 1709788280.84 total 1271.08 last 26.12) - open/close 38300 (time 1709788290.87 total 1281.11 last 26.82) - open/close 38578 (time 1709788300.88 total 1291.12 last 27.78) - open/close 38850 (time 1709788310.89 total 1301.14 last 27.15) - open/close 39116 (time 1709788320.91 total 1311.15 last 26.55) - open/close 39388 (time 1709788330.93 total 1321.18 last 27.14) - open/close 39642 (time 1709788340.97 total 1331.21 last 25.31) - open/close 39898 (time 1709788350.98 total 1341.22 last 25.57) - open/close 40000 (time 1709788354.83 total 1345.07 last 26.53) - open/close 40249 (time 1709788364.86 total 1355.10 last 24.82) - open/close 40496 (time 1709788374.88 total 1365.12 last 24.65) - open/close 40749 (time 1709788384.91 total 1375.15 last 25.24) - open/close 40993 (time 1709788394.92 total 1385.16 last 24.37) - open/close 41243 (time 1709788404.94 total 1395.19 last 24.94) - open/close 41498 (time 1709788414.96 total 1405.20 last 25.46) - open/close 41749 (time 1709788424.99 total 1415.23 last 25.03) - open/close 42005 (time 1709788435.02 total 1425.26 last 25.52) - open/close 42247 (time 1709788445.02 total 1435.27 last 24.19) - open/close 42493 (time 1709788455.03 total 1445.27 last 24.59) - open/close 42743 (time 1709788465.03 total 1455.28 last 24.98) - open/close 42994 (time 1709788475.04 total 1465.28 last 25.08) - open/close 43237 (time 1709788485.05 total 1475.29 last 24.28) - open/close 43489 (time 1709788495.07 total 1485.31 last 25.14) - open/close 43738 (time 1709788505.11 total 1495.35 last 24.81) - open/close 43994 (time 1709788515.13 total 1505.37 last 25.55) - open/close 44224 (time 1709788525.15 total 1515.40 last 22.94) - open/close 44466 (time 1709788535.18 total 1525.43 last 24.13) - open/close 44707 (time 1709788545.19 total 1535.43 last 24.08) - open/close 44947 (time 1709788555.22 total 1545.46 last 23.94) - open/close 45187 (time 1709788565.24 total 1555.48 last 23.94) - open/close 45428 (time 1709788575.27 total 1565.51 last 24.03) - open/close 45667 (time 1709788585.28 total 1575.52 last 23.88) - open/close 45913 (time 1709788595.28 total 1585.53 last 24.59) - open/close 46160 (time 1709788605.31 total 1595.55 last 24.64) - open/close 46400 (time 1709788615.32 total 1605.56 last 23.98) - open/close 46648 (time 1709788625.33 total 1615.57 last 24.77) - open/close 46901 (time 1709788635.33 total 1625.58 last 25.29) - open/close 47153 (time 1709788645.35 total 1635.59 last 25.17) - open/close 47402 (time 1709788655.39 total 1645.63 last 24.79) - open/close 47647 (time 1709788665.40 total 1655.64 last 24.48) - open/close 47895 (time 1709788675.44 total 1665.68 last 24.71) - open/close 48144 (time 1709788685.47 total 1675.71 last 24.83) - open/close 48398 (time 1709788695.49 total 1685.73 last 25.35) - open/close 48643 (time 1709788705.53 total 1695.77 last 24.40) - open/close 48879 (time 1709788715.53 total 1705.77 last 23.59) - open/close 49107 (time 1709788725.55 total 1715.79 last 22.75) - open/close 49338 (time 1709788735.59 total 1725.83 last 23.00) - open/close 49576 (time 1709788745.62 total 1735.86 last 23.74) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1736.86 seconds: 28.56 ops/second - unlinked 0 (time 1709788748 ; total 0 ; last 0) - unlinked 10000 (time 1709789241 ; total 493 ; last 493) - unlinked 20000 (time 1709789654 ; total 906 ; last 413) - unlinked 30000 (time 1709789953 ; total 1205 ; last 299) - unlinked 40000 (time 1709790091 ; total 1343 ; last 138) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1400 seconds: 35.428570 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg429-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg429-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg429-server stop mds service on oleg429-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg429-server stop mds service on oleg429-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg429-server unloading modules on: 'oleg429-server' oleg429-server: oleg429-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg429-client: oleg429-client: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-server: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-client: ssh exited with exit code 2 pdsh@oleg429-client: oleg429-server: ssh exited with exit code 2