== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:50:38 (1709787038) start mds service on oleg325-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg325-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg325-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg325-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg325-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-OST0000 oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg325-client.virtnet: -o user_xattr,flock oleg325-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 180 (time 1709787083.12 total 10.36 last 17.37) - open/close 328 (time 1709787093.13 total 20.38 last 14.78) - open/close 502 (time 1709787114.01 total 41.26 last 8.33) - open/close 635 (time 1709787124.97 total 52.22 last 12.13) - open/close 830 (time 1709787135.83 total 63.08 last 17.96) - open/close 1051 (time 1709787146.64 total 73.89 last 20.45) - open/close 1211 (time 1709787156.66 total 83.91 last 15.97) - open/close 1410 (time 1709787167.35 total 94.60 last 18.61) - open/close 1567 (time 1709787177.78 total 105.02 last 15.06) - open/close 1769 (time 1709787188.64 total 115.89 last 18.59) - open/close 1913 (time 1709787198.75 total 125.99 last 14.25) - open/close 2155 (time 1709787208.75 total 136.00 last 24.19) - open/close 2341 (time 1709787219.22 total 146.47 last 17.77) - open/close 2583 (time 1709787229.23 total 156.48 last 24.17) - open/close 2756 (time 1709787239.25 total 166.49 last 17.28) - open/close 3020 (time 1709787250.06 total 177.30 last 24.42) - open/close 3214 (time 1709787260.77 total 188.01 last 18.11) - open/close 3353 (time 1709787270.85 total 198.10 last 13.78) - open/close 3441 (time 1709787281.76 total 209.00 last 8.07) - open/close 3609 (time 1709787291.77 total 219.01 last 16.79) - open/close 3725 (time 1709787301.89 total 229.13 last 11.46) - open/close 3838 (time 1709787311.89 total 239.14 last 11.29) - open/close 4015 (time 1709787321.98 total 249.23 last 17.55) - open/close 4247 (time 1709787331.99 total 259.23 last 23.19) - open/close 4432 (time 1709787342.80 total 270.05 last 17.10) - open/close 4603 (time 1709787352.80 total 280.05 last 17.10) - open/close 4796 (time 1709787363.24 total 290.48 last 18.50) - open/close 5039 (time 1709787373.25 total 300.50 last 24.27) - open/close 5189 (time 1709787384.10 total 311.35 last 13.82) - open/close 5375 (time 1709787394.11 total 321.35 last 18.60) - open/close 5564 (time 1709787404.62 total 331.87 last 17.98) - open/close 5738 (time 1709787415.19 total 342.44 last 16.45) - open/close 5875 (time 1709787425.41 total 352.66 last 13.41) - open/close 6137 (time 1709787436.32 total 363.56 last 24.02) - open/close 6233 (time 1709787446.32 total 373.57 last 9.60) - open/close 6386 (time 1709787457.16 total 384.41 last 14.11) - open/close 6567 (time 1709787467.82 total 395.07 last 16.98) - open/close 6716 (time 1709787478.13 total 405.37 last 14.47) - open/close 6923 (time 1709787488.14 total 415.39 last 20.67) - open/close 7058 (time 1709787498.65 total 425.90 last 12.84) - open/close 7194 (time 1709787508.99 total 436.24 last 13.15) - open/close 7324 (time 1709787519.04 total 446.29 last 12.94) - open/close 7481 (time 1709787529.05 total 456.30 last 15.68) - open/close 7594 (time 1709787539.87 total 467.12 last 10.45) - open/close 7769 (time 1709787550.18 total 477.43 last 16.97) - open/close 7902 (time 1709787561.15 total 488.39 last 12.13) - open/close 8052 (time 1709787571.61 total 498.86 last 14.33) - open/close 8237 (time 1709787582.29 total 509.54 last 17.32) - open/close 8383 (time 1709787592.30 total 519.54 last 14.59) - open/close 8575 (time 1709787602.65 total 529.89 last 18.55) - open/close 8577 (time 1709787613.73 total 540.98 last 0.18) - open/close 8727 (time 1709787623.81 total 551.05 last 14.88) - open/close 8871 (time 1709787633.84 total 561.09 last 14.35) - open/close 9002 (time 1709787643.88 total 571.12 last 13.06) - open/close 9127 (time 1709787653.89 total 581.14 last 12.48) - open/close 9303 (time 1709787663.91 total 591.15 last 17.58) - open/close 9470 (time 1709787673.93 total 601.17 last 16.67) - open/close 9626 (time 1709787683.93 total 611.17 last 15.60) - open/close 9818 (time 1709787693.93 total 621.18 last 19.19) - open/close 9997 (time 1709787703.94 total 631.18 last 17.89) - open/close 10336 (time 1709787713.94 total 641.19 last 33.88) - open/close 10790 (time 1709787723.94 total 651.19 last 45.40) - open/close 11265 (time 1709787733.96 total 661.21 last 47.40) - open/close 11670 (time 1709787743.99 total 671.23 last 40.40) - open/close 12044 (time 1709787753.99 total 681.24 last 37.39) - open/close 12473 (time 1709787764.01 total 691.26 last 42.81) - open/close 12940 (time 1709787774.02 total 701.27 last 46.64) - open/close 13373 (time 1709787784.03 total 711.28 last 43.27) - open/close 13792 (time 1709787794.07 total 721.31 last 41.76) - open/close 14204 (time 1709787804.07 total 731.31 last 41.19) - open/close 14649 (time 1709787814.08 total 741.32 last 44.47) - open/close 15050 (time 1709787824.10 total 751.35 last 39.98) - open/close 15453 (time 1709787834.11 total 761.35 last 40.29) - open/close 15846 (time 1709787844.11 total 771.36 last 39.27) - open/close 16217 (time 1709787854.13 total 781.37 last 37.06) - open/close 16579 (time 1709787864.14 total 791.39 last 36.14) - open/close 16916 (time 1709787874.17 total 801.41 last 33.61) - open/close 17279 (time 1709787884.19 total 811.43 last 36.24) - open/close 17630 (time 1709787894.19 total 821.44 last 35.07) - open/close 18000 (time 1709787904.21 total 831.46 last 36.95) - open/close 18354 (time 1709787914.23 total 841.47 last 35.34) - open/close 18696 (time 1709787924.24 total 851.49 last 34.14) - open/close 19006 (time 1709787934.27 total 861.51 last 30.93) - open/close 19316 (time 1709787944.30 total 871.54 last 30.90) - open/close 19636 (time 1709787954.32 total 881.56 last 31.94) - open/close 19973 (time 1709787964.35 total 891.60 last 33.58) - open/close 20309 (time 1709787974.36 total 901.60 last 33.58) - open/close 20535 (time 1709787992.02 total 919.26 last 12.80) - open/close 20895 (time 1709788002.03 total 929.28 last 35.95) - open/close 21257 (time 1709788012.07 total 939.31 last 36.08) - open/close 21600 (time 1709788022.09 total 949.34 last 34.22) - open/close 21854 (time 1709788041.36 total 968.60 last 13.18) - open/close 22132 (time 1709788051.37 total 978.61 last 27.78) - open/close 22455 (time 1709788061.37 total 988.62 last 32.28) - open/close 22781 (time 1709788071.39 total 998.64 last 32.54) - open/close 23186 (time 1709788081.39 total 1008.64 last 40.49) - open/close 23597 (time 1709788091.42 total 1018.66 last 41.00) - open/close 24014 (time 1709788101.43 total 1028.68 last 41.64) - open/close 24426 (time 1709788111.45 total 1038.70 last 41.12) - open/close 24829 (time 1709788121.45 total 1048.70 last 40.28) - open/close 25225 (time 1709788131.48 total 1058.72 last 39.52) - open/close 25643 (time 1709788141.49 total 1068.73 last 41.76) - open/close 26012 (time 1709788151.51 total 1078.75 last 36.83) - open/close 26391 (time 1709788161.51 total 1088.76 last 37.87) - open/close 26769 (time 1709788171.54 total 1098.79 last 37.70) - open/close 27136 (time 1709788181.55 total 1108.79 last 36.68) - open/close 27505 (time 1709788191.57 total 1118.81 last 36.82) - open/close 27849 (time 1709788201.58 total 1128.82 last 34.37) - open/close 28196 (time 1709788211.58 total 1138.83 last 34.68) - open/close 28569 (time 1709788221.60 total 1148.84 last 37.24) - open/close 28925 (time 1709788231.62 total 1158.87 last 35.51) - open/close 29261 (time 1709788241.63 total 1168.88 last 33.58) - open/close 29572 (time 1709788251.63 total 1178.88 last 31.09) - open/close 29892 (time 1709788261.64 total 1188.89 last 31.96) - open/close 30000 (time 1709788265.01 total 1192.25 last 32.12) - open/close 30315 (time 1709788275.03 total 1202.28 last 31.42) - open/close 30620 (time 1709788285.07 total 1212.32 last 30.39) - open/close 30917 (time 1709788295.10 total 1222.34 last 29.62) - open/close 31226 (time 1709788305.10 total 1232.35 last 30.89) - open/close 31538 (time 1709788315.13 total 1242.38 last 31.10) - open/close 31852 (time 1709788325.14 total 1252.39 last 31.36) - open/close 32178 (time 1709788335.15 total 1262.40 last 32.57) - open/close 32495 (time 1709788345.17 total 1272.41 last 31.66) - open/close 32824 (time 1709788355.19 total 1282.43 last 32.83) - open/close 33125 (time 1709788365.19 total 1292.43 last 30.10) - open/close 33422 (time 1709788375.20 total 1302.45 last 29.65) - open/close 33726 (time 1709788385.21 total 1312.46 last 30.38) - open/close 34018 (time 1709788395.22 total 1322.46 last 29.18) - open/close 34317 (time 1709788405.24 total 1332.49 last 29.83) - open/close 34613 (time 1709788415.25 total 1342.50 last 29.57) - open/close 34924 (time 1709788425.28 total 1352.53 last 31.02) - open/close 35225 (time 1709788435.29 total 1362.54 last 30.07) - open/close 35509 (time 1709788445.32 total 1372.56 last 28.32) - open/close 35798 (time 1709788455.34 total 1382.59 last 28.83) - open/close 36079 (time 1709788465.34 total 1392.59 last 28.09) - open/close 36373 (time 1709788475.36 total 1402.61 last 29.35) - open/close 36656 (time 1709788485.38 total 1412.62 last 28.25) - open/close 36913 (time 1709788495.38 total 1422.62 last 25.70) - open/close 37190 (time 1709788505.40 total 1432.64 last 27.64) - open/close 37462 (time 1709788515.41 total 1442.66 last 27.16) - open/close 37736 (time 1709788525.41 total 1452.66 last 27.40) - open/close 38011 (time 1709788535.43 total 1462.67 last 27.46) - open/close 38286 (time 1709788545.44 total 1472.69 last 27.46) - open/close 38556 (time 1709788555.45 total 1482.70 last 26.97) - open/close 38828 (time 1709788565.49 total 1492.74 last 27.10) - open/close 39120 (time 1709788575.52 total 1502.77 last 29.11) - open/close 39398 (time 1709788585.54 total 1512.78 last 27.76) - open/close 39675 (time 1709788595.56 total 1522.81 last 27.63) - open/close 39965 (time 1709788605.61 total 1532.86 last 28.86) - open/close 40246 (time 1709788615.62 total 1542.87 last 28.07) - open/close 40533 (time 1709788625.63 total 1552.88 last 28.67) - open/close 40807 (time 1709788635.64 total 1562.89 last 27.37) - open/close 41070 (time 1709788645.65 total 1572.90 last 26.28) - open/close 41316 (time 1709788655.70 total 1582.95 last 24.48) - open/close 41566 (time 1709788665.74 total 1592.98 last 24.91) - open/close 41819 (time 1709788675.74 total 1602.99 last 25.28) - open/close 42074 (time 1709788685.75 total 1613.00 last 25.48) - open/close 42328 (time 1709788695.78 total 1623.03 last 25.32) - open/close 42591 (time 1709788705.81 total 1633.06 last 26.22) - open/close 42845 (time 1709788715.84 total 1643.09 last 25.34) - open/close 43115 (time 1709788725.86 total 1653.11 last 26.94) - open/close 43389 (time 1709788735.90 total 1663.15 last 27.28) - open/close 43669 (time 1709788745.94 total 1673.18 last 27.91) - open/close 43926 (time 1709788755.94 total 1683.19 last 25.68) - open/close 44171 (time 1709788765.95 total 1693.19 last 24.49) - open/close 44409 (time 1709788776.00 total 1703.24 last 23.68) - open/close 44652 (time 1709788786.04 total 1713.29 last 24.20) - open/close 44882 (time 1709788796.06 total 1723.30 last 22.96) - open/close 45119 (time 1709788806.10 total 1733.34 last 23.61) - open/close 45354 (time 1709788816.14 total 1743.39 last 23.40) - open/close 45574 (time 1709788826.18 total 1753.43 last 21.91) - open/close 45808 (time 1709788836.19 total 1763.43 last 23.39) - open/close 46037 (time 1709788846.20 total 1773.45 last 22.86) - open/close 46265 (time 1709788856.22 total 1783.47 last 22.76) - open/close 46480 (time 1709788866.26 total 1793.51 last 21.42) - open/close 46694 (time 1709788876.27 total 1803.51 last 21.38) - open/close 46921 (time 1709788886.29 total 1813.53 last 22.66) - open/close 47157 (time 1709788896.31 total 1823.56 last 23.54) - open/close 47380 (time 1709788906.34 total 1833.59 last 22.24) - open/close 47612 (time 1709788916.34 total 1843.59 last 23.20) - open/close 47844 (time 1709788926.36 total 1853.60 last 23.17) - open/close 48061 (time 1709788936.38 total 1863.62 last 21.66) - open/close 48275 (time 1709788946.40 total 1873.65 last 21.35) - open/close 48495 (time 1709788956.41 total 1883.66 last 21.97) - open/close 48711 (time 1709788966.45 total 1893.69 last 21.53) - open/close 48938 (time 1709788976.46 total 1903.71 last 22.66) - open/close 49160 (time 1709788986.47 total 1913.72 last 22.18) - open/close 49375 (time 1709788996.49 total 1923.73 last 21.46) - open/close 49591 (time 1709789006.50 total 1933.75 last 21.57) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1934.16 seconds: 25.64 ops/second - unlinked 0 (time 1709789008 ; total 0 ; last 0) - unlinked 10000 (time 1709789532 ; total 524 ; last 524) - unlinked 20000 (time 1709789960 ; total 952 ; last 428) - unlinked 30000 (time 1709790251 ; total 1243 ; last 291) - unlinked 40000 (time 1709790389 ; total 1381 ; last 138) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1414 seconds: 35.077793 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg325-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg325-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg325-server stop mds service on oleg325-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg325-server stop mds service on oleg325-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg325-server unloading modules on: 'oleg325-server' oleg325-server: oleg325-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg325-client: oleg325-client: ssh exited with exit code 2 pdsh@oleg325-client: oleg325-server: ssh exited with exit code 2 pdsh@oleg325-client: oleg325-client: ssh exited with exit code 2 pdsh@oleg325-client: oleg325-server: ssh exited with exit code 2