== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:55:03 (1709787303) start mds service on oleg345-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg345-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg345-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg345-server: oleg345-server.virtnet: executing set_default_debug -1 all pdsh@oleg345-client: oleg345-server: ssh exited with exit code 1 Started lustre-OST0000 oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg345-client.virtnet: -o user_xattr,flock oleg345-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 192 (time 1709787358.60 total 10.01 last 19.17) - open/close 373 (time 1709787369.01 total 20.42 last 17.39) - open/close 544 (time 1709787379.95 total 31.37 last 15.63) - open/close 810 (time 1709787390.70 total 42.12 last 24.75) - open/close 940 (time 1709787400.71 total 52.12 last 12.99) - open/close 1072 (time 1709787411.20 total 62.61 last 12.58) - open/close 1277 (time 1709787421.45 total 72.86 last 20.00) - open/close 1494 (time 1709787431.45 total 82.87 last 21.69) - open/close 1681 (time 1709787441.81 total 93.22 last 18.06) - open/close 1856 (time 1709787452.51 total 103.93 last 16.35) - open/close 2050 (time 1709787462.96 total 114.38 last 18.57) - open/close 2241 (time 1709787473.60 total 125.02 last 17.95) - open/close 2415 (time 1709787483.61 total 135.03 last 17.38) - open/close 2536 (time 1709787493.99 total 145.41 last 11.66) - open/close 2744 (time 1709787504.65 total 156.06 last 19.52) - open/close 2956 (time 1709787515.20 total 166.61 last 20.10) - open/close 3174 (time 1709787526.16 total 177.58 last 19.88) - open/close 3321 (time 1709787536.94 total 188.35 last 13.64) - open/close 3432 (time 1709787546.98 total 198.39 last 11.05) - open/close 3634 (time 1709787557.81 total 209.23 last 18.65) - open/close 3855 (time 1709787567.83 total 219.25 last 22.05) - open/close 4060 (time 1709787577.84 total 229.25 last 20.49) - open/close 4187 (time 1709787588.72 total 240.13 last 11.67) - open/close 4413 (time 1709787608.66 total 260.07 last 11.34) - open/close 4566 (time 1709787619.68 total 271.10 last 13.88) - open/close 4709 (time 1709787629.75 total 281.16 last 14.21) - open/close 4823 (time 1709787640.26 total 291.68 last 10.84) - open/close 5004 (time 1709787651.00 total 302.42 last 16.86) - open/close 5205 (time 1709787661.69 total 313.11 last 18.80) - open/close 5324 (time 1709787671.71 total 323.12 last 11.88) - open/close 5452 (time 1709787682.19 total 333.61 last 12.21) - open/close 5676 (time 1709787692.23 total 343.65 last 22.30) - open/close 5849 (time 1709787702.26 total 353.67 last 17.26) - open/close 6028 (time 1709787712.30 total 363.72 last 17.82) - open/close 6184 (time 1709787723.03 total 374.44 last 14.54) - open/close 6391 (time 1709787733.19 total 384.61 last 20.37) - open/close 6670 (time 1709787743.19 total 394.61 last 27.90) - open/close 6889 (time 1709787753.77 total 405.19 last 20.70) - open/close 7083 (time 1709787764.69 total 416.10 last 17.77) - open/close 7221 (time 1709787774.90 total 426.31 last 13.51) - open/close 7465 (time 1709787785.27 total 436.69 last 23.52) - open/close 7639 (time 1709787795.98 total 447.39 last 16.25) - open/close 7810 (time 1709787806.46 total 457.87 last 16.32) - open/close 7915 (time 1709787816.77 total 468.18 last 10.18) - open/close 8136 (time 1709787826.79 total 478.21 last 22.05) - open/close 8418 (time 1709787836.80 total 488.21 last 28.18) - open/close 8523 (time 1709787847.78 total 499.19 last 9.57) - open/close 8577 (time 1709787857.92 total 509.34 last 5.32) - open/close 8678 (time 1709787868.34 total 519.75 last 9.70) - open/close 8793 (time 1709787878.88 total 530.29 last 10.91) - open/close 8998 (time 1709787889.10 total 540.52 last 20.04) - open/close 9183 (time 1709787899.11 total 550.53 last 18.48) - open/close 9335 (time 1709787911.26 total 562.68 last 12.51) - open/close 9442 (time 1709787921.66 total 573.07 last 10.29) - open/close 9577 (time 1709787932.53 total 583.94 last 12.42) - open/close 9720 (time 1709787942.83 total 594.25 last 13.87) - open/close 9851 (time 1709787952.92 total 604.33 last 12.99) - open/close 10000 (time 1709787960.04 total 611.46 last 20.90) - open/close 10486 (time 1709787970.06 total 621.47 last 48.53) - open/close 10976 (time 1709787980.06 total 631.48 last 48.97) - open/close 11527 (time 1709787990.07 total 641.48 last 55.08) - open/close 12061 (time 1709788000.08 total 651.49 last 53.33) - open/close 12530 (time 1709788010.10 total 661.52 last 46.80) - open/close 12561 (time 1709788021.83 total 673.24 last 2.64) - open/close 13034 (time 1709788031.83 total 683.24 last 47.30) - open/close 13507 (time 1709788041.84 total 693.25 last 47.25) - open/close 13939 (time 1709788051.86 total 703.28 last 43.09) - open/close 14357 (time 1709788061.87 total 713.28 last 41.79) - open/close 14780 (time 1709788071.88 total 723.30 last 42.23) - open/close 15322 (time 1709788081.89 total 733.31 last 54.15) - open/close 15849 (time 1709788091.90 total 743.32 last 52.65) - open/close 16374 (time 1709788101.91 total 753.32 last 52.46) - open/close 16894 (time 1709788111.92 total 763.33 last 51.95) - open/close 17409 (time 1709788121.93 total 773.35 last 51.43) - open/close 17915 (time 1709788131.94 total 783.35 last 50.58) - open/close 18422 (time 1709788141.94 total 793.36 last 50.67) - open/close 18896 (time 1709788151.94 total 803.36 last 47.39) - open/close 19368 (time 1709788161.96 total 813.37 last 47.14) - open/close 19838 (time 1709788171.97 total 823.39 last 46.92) - open/close 20000 (time 1709788175.57 total 826.98 last 45.06) - open/close 20452 (time 1709788185.58 total 837.00 last 45.13) - open/close 20888 (time 1709788195.60 total 847.02 last 43.51) - open/close 21310 (time 1709788205.62 total 857.04 last 42.11) - open/close 21731 (time 1709788215.64 total 867.05 last 42.04) - open/close 22164 (time 1709788225.64 total 877.06 last 43.27) - open/close 22564 (time 1709788235.65 total 887.06 last 40.00) - open/close 22972 (time 1709788245.66 total 897.08 last 40.74) - open/close 23342 (time 1709788255.67 total 907.09 last 36.95) - open/close 23740 (time 1709788265.68 total 917.09 last 39.77) - open/close 24122 (time 1709788275.70 total 927.11 last 38.13) - open/close 24491 (time 1709788285.71 total 937.12 last 36.86) - open/close 24844 (time 1709788295.73 total 947.14 last 35.23) - open/close 25202 (time 1709788305.73 total 957.15 last 35.78) - open/close 25566 (time 1709788315.73 total 967.15 last 36.40) - open/close 25922 (time 1709788325.74 total 977.16 last 35.57) - open/close 26298 (time 1709788335.75 total 987.17 last 37.57) - open/close 26672 (time 1709788345.77 total 997.19 last 37.32) - open/close 27048 (time 1709788355.78 total 1007.19 last 37.58) - open/close 27411 (time 1709788365.80 total 1017.21 last 36.23) - open/close 27760 (time 1709788375.82 total 1027.24 last 34.80) - open/close 28102 (time 1709788385.84 total 1037.26 last 34.14) - open/close 28447 (time 1709788395.86 total 1047.27 last 34.45) - open/close 28802 (time 1709788405.86 total 1057.28 last 35.48) - open/close 29137 (time 1709788415.86 total 1067.28 last 33.50) - open/close 29489 (time 1709788425.88 total 1077.30 last 35.12) - open/close 29822 (time 1709788435.89 total 1087.31 last 33.26) - open/close 30000 (time 1709788441.32 total 1092.73 last 32.83) - open/close 30319 (time 1709788451.34 total 1102.75 last 31.83) - open/close 30644 (time 1709788461.35 total 1112.76 last 32.46) - open/close 30950 (time 1709788471.36 total 1122.77 last 30.57) - open/close 31277 (time 1709788481.38 total 1132.79 last 32.64) - open/close 31568 (time 1709788491.39 total 1142.81 last 29.06) - open/close 31872 (time 1709788501.42 total 1152.83 last 30.33) - open/close 32187 (time 1709788511.43 total 1162.85 last 31.45) - open/close 32499 (time 1709788521.45 total 1172.86 last 31.15) - open/close 32799 (time 1709788531.46 total 1182.88 last 29.96) - open/close 33113 (time 1709788541.48 total 1192.89 last 31.35) - open/close 33430 (time 1709788551.49 total 1202.91 last 31.65) - open/close 33734 (time 1709788561.50 total 1212.92 last 30.36) - open/close 34053 (time 1709788571.52 total 1222.94 last 31.85) - open/close 34361 (time 1709788581.55 total 1232.96 last 30.73) - open/close 34664 (time 1709788591.55 total 1242.96 last 30.30) - open/close 34972 (time 1709788601.57 total 1252.98 last 30.74) - open/close 35285 (time 1709788611.59 total 1263.00 last 31.23) - open/close 35593 (time 1709788621.61 total 1273.02 last 30.75) - open/close 35897 (time 1709788631.62 total 1283.04 last 30.35) - open/close 36178 (time 1709788641.63 total 1293.05 last 28.08) - open/close 36424 (time 1709788651.65 total 1303.07 last 24.55) - open/close 36659 (time 1709788661.67 total 1313.08 last 23.46) - open/close 36916 (time 1709788671.68 total 1323.10 last 25.66) - open/close 37189 (time 1709788681.69 total 1333.10 last 27.29) - open/close 37467 (time 1709788691.72 total 1343.14 last 27.71) - open/close 37742 (time 1709788701.72 total 1353.14 last 27.49) - open/close 38012 (time 1709788711.75 total 1363.16 last 26.94) - open/close 38283 (time 1709788721.76 total 1373.17 last 27.07) - open/close 38581 (time 1709788731.77 total 1383.19 last 29.77) - open/close 38885 (time 1709788741.80 total 1393.22 last 30.30) - open/close 39179 (time 1709788751.81 total 1403.23 last 29.37) - open/close 39450 (time 1709788761.85 total 1413.26 last 27.01) - open/close 39706 (time 1709788771.87 total 1423.29 last 25.53) - open/close 39969 (time 1709788781.90 total 1433.31 last 26.23) - open/close 40220 (time 1709788791.91 total 1443.32 last 25.07) - open/close 40469 (time 1709788801.93 total 1453.34 last 24.86) - open/close 40724 (time 1709788811.98 total 1463.40 last 25.36) - open/close 40967 (time 1709788822.01 total 1473.43 last 24.23) - open/close 41210 (time 1709788832.03 total 1483.45 last 24.26) - open/close 41472 (time 1709788842.06 total 1493.47 last 26.13) - open/close 41726 (time 1709788852.07 total 1503.49 last 25.36) - open/close 41966 (time 1709788862.11 total 1513.52 last 23.92) - open/close 42207 (time 1709788872.12 total 1523.54 last 24.07) - open/close 42451 (time 1709788882.14 total 1533.55 last 24.37) - open/close 42717 (time 1709788892.14 total 1543.56 last 26.59) - open/close 42973 (time 1709788902.17 total 1553.59 last 25.52) - open/close 43231 (time 1709788912.19 total 1563.60 last 25.76) - open/close 43492 (time 1709788922.19 total 1573.61 last 26.10) - open/close 43748 (time 1709788932.21 total 1583.62 last 25.55) - open/close 43991 (time 1709788942.24 total 1593.65 last 24.23) - open/close 44238 (time 1709788952.24 total 1603.65 last 24.69) - open/close 44480 (time 1709788962.25 total 1613.66 last 24.18) - open/close 44733 (time 1709788972.26 total 1623.68 last 25.26) - open/close 44994 (time 1709788982.27 total 1633.69 last 26.08) - open/close 45242 (time 1709788992.27 total 1643.69 last 24.79) - open/close 45483 (time 1709789002.28 total 1653.70 last 24.08) - open/close 45733 (time 1709789012.30 total 1663.71 last 24.96) - open/close 45968 (time 1709789022.31 total 1673.73 last 23.47) - open/close 46205 (time 1709789032.32 total 1683.74 last 23.67) - open/close 46434 (time 1709789042.33 total 1693.74 last 22.89) - open/close 46681 (time 1709789052.34 total 1703.76 last 24.66) - open/close 46931 (time 1709789062.38 total 1713.80 last 24.90) - open/close 47162 (time 1709789072.41 total 1723.82 last 23.04) - open/close 47384 (time 1709789082.44 total 1733.86 last 22.13) - open/close 47609 (time 1709789092.47 total 1743.88 last 22.45) - open/close 47845 (time 1709789102.47 total 1753.88 last 23.59) - open/close 48083 (time 1709789112.47 total 1763.89 last 23.79) - open/close 48316 (time 1709789122.49 total 1773.91 last 23.25) - open/close 48539 (time 1709789132.51 total 1783.93 last 22.26) - open/close 48762 (time 1709789142.52 total 1793.93 last 22.28) - open/close 48992 (time 1709789152.54 total 1803.96 last 22.94) - open/close 49224 (time 1709789162.56 total 1813.98 last 23.16) - open/close 49467 (time 1709789172.57 total 1823.98 last 24.29) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1829.44 seconds: 27.11 ops/second - unlinked 0 (time 1709789179 ; total 0 ; last 0) - unlinked 10000 (time 1709789658 ; total 479 ; last 479) - unlinked 20000 (time 1709790018 ; total 839 ; last 360) - unlinked 30000 (time 1709790252 ; total 1073 ; last 234) - unlinked 40000 (time 1709790348 ; total 1169 ; last 96) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1207 seconds: 41.093620 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg345-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg345-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg345-server stop mds service on oleg345-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg345-server stop mds service on oleg345-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg345-server unloading modules on: 'oleg345-server' oleg345-server: oleg345-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg345-client: oleg345-client: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-server: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-client: ssh exited with exit code 2 pdsh@oleg345-client: oleg345-server: ssh exited with exit code 2