== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:52:57 (1709787177) start mds service on oleg256-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg256-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg256-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg256-client.virtnet: -o user_xattr,flock oleg256-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 275 (time 1709787229.68 total 10.87 last 25.31) - open/close 493 (time 1709787239.82 total 21.01 last 21.49) - open/close 646 (time 1709787250.06 total 31.25 last 14.95) - open/close 815 (time 1709787260.07 total 41.26 last 16.88) - open/close 1063 (time 1709787270.48 total 51.67 last 23.82) - open/close 1215 (time 1709787280.69 total 61.89 last 14.88) - open/close 1411 (time 1709787291.48 total 72.67 last 18.17) - open/close 1555 (time 1709787302.37 total 83.56 last 13.22) - open/close 1767 (time 1709787312.79 total 93.98 last 20.34) - open/close 2039 (time 1709787323.59 total 104.78 last 25.19) - open/close 2354 (time 1709787334.88 total 116.07 last 27.90) - open/close 2553 (time 1709787345.10 total 126.29 last 19.47) - open/close 2706 (time 1709787355.67 total 136.86 last 14.48) - open/close 3096 (time 1709787365.69 total 146.88 last 38.92) - open/close 3306 (time 1709787375.69 total 156.88 last 20.99) - open/close 3530 (time 1709787386.68 total 167.87 last 20.38) - open/close 3750 (time 1709787396.77 total 177.97 last 21.80) - open/close 3940 (time 1709787406.83 total 188.02 last 18.90) - open/close 4159 (time 1709787417.45 total 198.64 last 20.61) - open/close 4255 (time 1709787428.34 total 209.53 last 8.82) - open/close 4442 (time 1709787438.64 total 219.83 last 18.14) - open/close 4574 (time 1709787449.20 total 230.39 last 12.50) - open/close 4773 (time 1709787459.96 total 241.16 last 18.49) - open/close 4914 (time 1709787470.87 total 252.06 last 12.93) - open/close 5101 (time 1709787481.67 total 262.86 last 17.32) - open/close 5278 (time 1709787491.72 total 272.91 last 17.61) - open/close 5395 (time 1709787501.73 total 282.92 last 11.69) - open/close 5593 (time 1709787511.80 total 292.99 last 19.66) - open/close 5744 (time 1709787522.44 total 303.63 last 14.19) - open/close 5870 (time 1709787533.33 total 314.52 last 11.57) - open/close 5992 (time 1709787543.35 total 324.54 last 12.18) - open/close 6187 (time 1709787553.35 total 334.54 last 19.50) - open/close 6320 (time 1709787563.36 total 344.55 last 13.28) - open/close 6417 (time 1709787574.41 total 355.60 last 8.78) - open/close 6576 (time 1709787584.70 total 365.89 last 15.46) - open/close 6767 (time 1709787594.71 total 375.90 last 19.07) - open/close 6894 (time 1709787605.51 total 386.70 last 11.76) - open/close 7170 (time 1709787615.52 total 396.71 last 27.57) - open/close 7332 (time 1709787625.53 total 406.72 last 16.19) - open/close 7520 (time 1709787635.77 total 416.96 last 18.35) - open/close 7657 (time 1709787646.18 total 427.38 last 13.15) - open/close 7858 (time 1709787656.19 total 437.38 last 20.09) - open/close 8017 (time 1709787666.55 total 447.74 last 15.35) - open/close 8104 (time 1709787677.55 total 458.74 last 7.91) - open/close 8289 (time 1709787688.02 total 469.21 last 17.67) - open/close 8401 (time 1709787698.03 total 479.22 last 11.18) - open/close 8567 (time 1709787708.51 total 489.70 last 15.84) - open/close 8761 (time 1709787718.51 total 499.70 last 19.40) - open/close 8933 (time 1709787728.76 total 509.95 last 16.79) - open/close 9104 (time 1709787739.24 total 520.43 last 16.31) - open/close 9240 (time 1709787749.78 total 530.98 last 12.90) - open/close 9373 (time 1709787759.93 total 541.12 last 13.11) - open/close 9531 (time 1709787770.04 total 551.23 last 15.63) - open/close 9583 (time 1709787780.35 total 561.54 last 5.05) - open/close 9908 (time 1709787790.36 total 571.55 last 32.45) - open/close 10000 (time 1709787799.31 total 580.50 last 10.28) - open/close 10191 (time 1709787810.00 total 591.19 last 17.87) - open/close 10619 (time 1709787820.01 total 601.20 last 42.74) - open/close 11120 (time 1709787839.90 total 621.09 last 25.19) - open/close 11332 (time 1709787855.70 total 636.89 last 13.42) - open/close 11808 (time 1709787865.72 total 646.91 last 47.50) - open/close 12357 (time 1709787875.73 total 656.92 last 54.85) - open/close 12861 (time 1709787885.74 total 666.93 last 50.36) - open/close 13318 (time 1709787895.74 total 676.93 last 45.69) - open/close 13780 (time 1709787905.76 total 686.95 last 46.11) - open/close 14261 (time 1709787915.77 total 696.96 last 48.02) - open/close 14703 (time 1709787925.80 total 706.99 last 44.08) - open/close 15162 (time 1709787935.81 total 717.00 last 45.85) - open/close 15648 (time 1709787945.82 total 727.01 last 48.53) - open/close 16185 (time 1709787955.84 total 737.03 last 53.63) - open/close 16711 (time 1709787965.85 total 747.04 last 52.55) - open/close 17200 (time 1709787975.86 total 757.06 last 48.82) - open/close 17689 (time 1709787985.88 total 767.07 last 48.82) - open/close 18180 (time 1709787995.89 total 777.08 last 49.05) - open/close 18759 (time 1709788005.90 total 787.09 last 57.87) - open/close 19327 (time 1709788015.90 total 797.09 last 56.76) - open/close 19893 (time 1709788025.92 total 807.11 last 56.51) - open/close 20000 (time 1709788027.95 total 809.14 last 52.70) - open/close 20588 (time 1709788037.95 total 819.14 last 58.78) - open/close 21161 (time 1709788047.95 total 829.14 last 57.30) - open/close 21729 (time 1709788057.96 total 839.15 last 56.75) - open/close 22250 (time 1709788067.97 total 849.16 last 52.07) - open/close 22777 (time 1709788077.97 total 859.17 last 52.67) - open/close 23344 (time 1709788087.98 total 869.18 last 56.64) - open/close 23843 (time 1709788097.99 total 879.18 last 49.87) - open/close 24347 (time 1709788108.00 total 889.19 last 50.36) - open/close 24851 (time 1709788118.01 total 899.21 last 50.32) - open/close 25322 (time 1709788128.03 total 909.22 last 47.02) - open/close 25828 (time 1709788138.04 total 919.23 last 50.55) - open/close 26313 (time 1709788148.06 total 929.25 last 48.41) - open/close 26796 (time 1709788158.08 total 939.27 last 48.22) - open/close 27285 (time 1709788168.09 total 949.28 last 48.84) - open/close 27760 (time 1709788178.11 total 959.30 last 47.41) - open/close 28222 (time 1709788188.13 total 969.32 last 46.10) - open/close 28703 (time 1709788198.14 total 979.33 last 48.04) - open/close 29156 (time 1709788208.15 total 989.34 last 45.29) - open/close 29625 (time 1709788218.16 total 999.36 last 46.81) - open/close 30000 (time 1709788226.41 total 1007.60 last 45.48) - open/close 30445 (time 1709788236.43 total 1017.62 last 44.42) - open/close 30891 (time 1709788246.43 total 1027.62 last 44.57) - open/close 31327 (time 1709788256.46 total 1037.65 last 43.49) - open/close 31745 (time 1709788266.47 total 1047.67 last 41.74) - open/close 32174 (time 1709788276.48 total 1057.67 last 42.86) - open/close 32568 (time 1709788286.49 total 1067.68 last 39.37) - open/close 32964 (time 1709788296.49 total 1077.69 last 39.59) - open/close 33367 (time 1709788306.50 total 1087.69 last 40.29) - open/close 33773 (time 1709788316.52 total 1097.71 last 40.52) - open/close 34157 (time 1709788326.52 total 1107.71 last 38.40) - open/close 34541 (time 1709788336.53 total 1117.72 last 38.37) - open/close 34927 (time 1709788346.55 total 1127.74 last 38.50) - open/close 35295 (time 1709788356.56 total 1137.75 last 36.76) - open/close 35672 (time 1709788366.56 total 1147.76 last 37.69) - open/close 36033 (time 1709788376.57 total 1157.76 last 36.08) - open/close 36392 (time 1709788386.59 total 1167.78 last 35.82) - open/close 36750 (time 1709788396.59 total 1177.78 last 35.79) - open/close 37128 (time 1709788406.61 total 1187.80 last 37.73) - open/close 37496 (time 1709788416.64 total 1197.83 last 36.71) - open/close 37854 (time 1709788426.64 total 1207.83 last 35.79) - open/close 38205 (time 1709788436.66 total 1217.85 last 35.05) - open/close 38546 (time 1709788446.67 total 1227.86 last 34.06) - open/close 38896 (time 1709788456.68 total 1237.87 last 34.95) - open/close 39234 (time 1709788466.70 total 1247.90 last 33.72) - open/close 39585 (time 1709788476.73 total 1257.93 last 35.00) - open/close 39943 (time 1709788486.75 total 1267.94 last 35.75) - open/close 40281 (time 1709788496.76 total 1277.95 last 33.75) - open/close 40620 (time 1709788506.77 total 1287.96 last 33.88) - open/close 40969 (time 1709788516.77 total 1297.96 last 34.89) - open/close 41307 (time 1709788526.79 total 1307.98 last 33.73) - open/close 41635 (time 1709788536.79 total 1317.98 last 32.80) - open/close 41965 (time 1709788546.80 total 1327.99 last 32.97) - open/close 42298 (time 1709788556.81 total 1338.00 last 33.26) - open/close 42642 (time 1709788566.84 total 1348.03 last 34.32) - open/close 42974 (time 1709788576.84 total 1358.03 last 33.18) - open/close 43311 (time 1709788586.87 total 1368.06 last 33.62) - open/close 43658 (time 1709788596.88 total 1378.07 last 34.67) - open/close 43993 (time 1709788606.88 total 1388.07 last 33.48) - open/close 44325 (time 1709788616.90 total 1398.09 last 33.14) - open/close 44659 (time 1709788626.91 total 1408.10 last 33.37) - open/close 44970 (time 1709788636.91 total 1418.10 last 31.10) - open/close 45274 (time 1709788646.93 total 1428.12 last 30.34) - open/close 45590 (time 1709788656.96 total 1438.15 last 31.50) - open/close 45893 (time 1709788666.97 total 1448.16 last 30.26) - open/close 46202 (time 1709788676.98 total 1458.17 last 30.87) - open/close 46505 (time 1709788687.01 total 1468.20 last 30.22) - open/close 46810 (time 1709788697.01 total 1478.20 last 30.49) - open/close 47110 (time 1709788707.04 total 1488.23 last 29.91) - open/close 47410 (time 1709788717.06 total 1498.26 last 29.93) - open/close 47719 (time 1709788727.08 total 1508.27 last 30.86) - open/close 48015 (time 1709788737.08 total 1518.27 last 29.58) - open/close 48302 (time 1709788747.09 total 1528.28 last 28.67) - open/close 48584 (time 1709788757.10 total 1538.29 last 28.19) - open/close 48872 (time 1709788767.12 total 1548.31 last 28.72) - open/close 49155 (time 1709788777.13 total 1558.32 last 28.29) - open/close 49433 (time 1709788787.13 total 1568.32 last 27.79) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1574.26 seconds: 31.51 ops/second - unlinked 0 (time 1709788794 ; total 0 ; last 0) - unlinked 10000 (time 1709789243 ; total 449 ; last 449) - unlinked 20000 (time 1709789614 ; total 820 ; last 371) - unlinked 30000 (time 1709789870 ; total 1076 ; last 256) - unlinked 40000 (time 1709789993 ; total 1199 ; last 123) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1235 seconds: 40.161942 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg256-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg256-server unloading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2