== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:54:26 (1709787266) start mds service on oleg443-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg443-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0000 oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg443-client.virtnet: -o user_xattr,flock oleg443-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 135 (time 1709787303.83 total 10.13 last 13.33) - open/close 307 (time 1709787314.43 total 20.73 last 16.21) - open/close 502 (time 1709787324.97 total 31.27 last 18.51) - open/close 704 (time 1709787334.99 total 41.29 last 20.16) - open/close 909 (time 1709787344.99 total 51.29 last 20.49) - open/close 1065 (time 1709787355.02 total 61.32 last 15.55) - open/close 1258 (time 1709787365.03 total 71.33 last 19.28) - open/close 1458 (time 1709787377.38 total 83.68 last 16.19) - open/close 1619 (time 1709787387.86 total 94.16 last 15.37) - open/close 1858 (time 1709787397.90 total 104.20 last 23.80) - open/close 2046 (time 1709787408.55 total 114.85 last 17.65) - open/close 2213 (time 1709787418.70 total 125.00 last 16.45) - open/close 2424 (time 1709787429.45 total 135.75 last 19.62) - open/close 2618 (time 1709787439.97 total 146.27 last 18.44) - open/close 2762 (time 1709787450.56 total 156.86 last 13.60) - open/close 2882 (time 1709787460.58 total 166.88 last 11.98) - open/close 3027 (time 1709787471.42 total 177.72 last 13.37) - open/close 3261 (time 1709787482.36 total 188.66 last 21.39) - open/close 3424 (time 1709787492.36 total 198.67 last 16.29) - open/close 3580 (time 1709787503.14 total 209.44 last 14.48) - open/close 3786 (time 1709787513.31 total 219.61 last 20.25) - open/close 3972 (time 1709787524.03 total 230.33 last 17.34) - open/close 4034 (time 1709787539.11 total 245.41 last 4.11) - open/close 4257 (time 1709787549.13 total 255.43 last 22.26) - open/close 4421 (time 1709787559.13 total 265.43 last 16.39) - open/close 4653 (time 1709787569.65 total 275.95 last 22.07) - open/close 4786 (time 1709787579.65 total 285.95 last 13.29) - open/close 4961 (time 1709787590.32 total 296.62 last 16.40) - open/close 5254 (time 1709787600.46 total 306.76 last 28.90) - open/close 5458 (time 1709787611.39 total 317.69 last 18.67) - open/close 5594 (time 1709787621.83 total 328.13 last 13.03) - open/close 5785 (time 1709787631.84 total 338.14 last 19.08) - open/close 6003 (time 1709787642.36 total 348.66 last 20.72) - open/close 6298 (time 1709787652.95 total 359.25 last 27.86) - open/close 6495 (time 1709787662.96 total 369.26 last 19.67) - open/close 6671 (time 1709787673.92 total 380.22 last 16.06) - open/close 6759 (time 1709787684.80 total 391.10 last 8.09) - open/close 6872 (time 1709787694.94 total 401.24 last 11.15) - open/close 7073 (time 1709787704.97 total 411.27 last 20.04) - open/close 7240 (time 1709787714.97 total 421.27 last 16.70) - open/close 7349 (time 1709787724.98 total 431.28 last 10.89) - open/close 7508 (time 1709787735.49 total 441.79 last 15.13) - open/close 7717 (time 1709787746.46 total 452.76 last 19.05) - open/close 7957 (time 1709787756.62 total 462.92 last 23.63) - open/close 8128 (time 1709787767.06 total 473.36 last 16.39) - open/close 8307 (time 1709787777.07 total 483.37 last 17.87) - open/close 8655 (time 1709787787.80 total 494.10 last 32.45) - open/close 8759 (time 1709787798.64 total 504.94 last 9.59) - open/close 8951 (time 1709787809.04 total 515.34 last 18.46) - open/close 9214 (time 1709787819.04 total 525.34 last 26.29) - open/close 9404 (time 1709787829.05 total 535.35 last 18.99) - open/close 10000 (time 1709787838.54 total 544.84 last 62.79) - open/close 10633 (time 1709787848.55 total 554.85 last 63.24) - open/close 11237 (time 1709787858.57 total 564.87 last 60.32) - open/close 11882 (time 1709787868.57 total 574.87 last 64.44) - open/close 12520 (time 1709787878.59 total 584.89 last 63.70) - open/close 13156 (time 1709787888.60 total 594.90 last 63.52) - open/close 13746 (time 1709787898.61 total 604.91 last 58.95) - open/close 14246 (time 1709787908.62 total 614.92 last 49.97) - open/close 14753 (time 1709787918.62 total 624.92 last 50.66) - open/close 15256 (time 1709787928.64 total 634.94 last 50.22) - open/close 15775 (time 1709787938.65 total 644.95 last 51.86) - open/close 16266 (time 1709787948.65 total 654.95 last 49.09) - open/close 16804 (time 1709787958.65 total 664.95 last 53.77) - open/close 17284 (time 1709787968.65 total 674.95 last 48.00) - open/close 17762 (time 1709787978.66 total 684.96 last 47.76) - open/close 18240 (time 1709787988.67 total 694.97 last 47.77) - open/close 18757 (time 1709787998.69 total 705.00 last 51.56) - open/close 19150 (time 1709788008.71 total 715.02 last 39.22) - open/close 19539 (time 1709788018.74 total 725.04 last 38.82) - open/close 19964 (time 1709788028.75 total 735.05 last 42.43) - open/close 20414 (time 1709788038.76 total 745.06 last 44.97) - open/close 20864 (time 1709788048.76 total 755.06 last 45.00) - open/close 21263 (time 1709788058.78 total 765.08 last 39.82) - open/close 21690 (time 1709788068.80 total 775.10 last 42.63) - open/close 22134 (time 1709788078.81 total 785.11 last 44.35) - open/close 22576 (time 1709788088.81 total 795.11 last 44.19) - open/close 22979 (time 1709788098.81 total 805.11 last 40.29) - open/close 23376 (time 1709788108.82 total 815.12 last 39.68) - open/close 23781 (time 1709788118.82 total 825.12 last 40.48) - open/close 24178 (time 1709788128.83 total 835.13 last 39.68) - open/close 24539 (time 1709788138.84 total 845.14 last 36.06) - open/close 24903 (time 1709788148.91 total 855.21 last 36.15) - open/close 25206 (time 1709788158.92 total 865.22 last 30.27) - open/close 25520 (time 1709788168.94 total 875.24 last 31.33) - open/close 25884 (time 1709788178.95 total 885.26 last 36.35) - open/close 26269 (time 1709788188.96 total 895.26 last 38.48) - open/close 26600 (time 1709788198.99 total 905.29 last 32.99) - open/close 26955 (time 1709788209.01 total 915.31 last 35.45) - open/close 27329 (time 1709788219.02 total 925.32 last 37.35) - open/close 27683 (time 1709788229.03 total 935.33 last 35.38) - open/close 28012 (time 1709788239.05 total 945.35 last 32.82) - open/close 28331 (time 1709788249.06 total 955.36 last 31.87) - open/close 28726 (time 1709788259.07 total 965.37 last 39.44) - open/close 29088 (time 1709788269.10 total 975.40 last 36.09) - open/close 29422 (time 1709788279.11 total 985.41 last 33.38) - open/close 29760 (time 1709788289.13 total 995.43 last 33.72) - open/close 30000 (time 1709788295.98 total 1002.28 last 35.06) - open/close 30340 (time 1709788306.01 total 1012.31 last 33.90) - open/close 30664 (time 1709788316.04 total 1022.34 last 32.30) - open/close 30996 (time 1709788326.05 total 1032.35 last 33.19) - open/close 31351 (time 1709788336.06 total 1042.36 last 35.44) - open/close 31673 (time 1709788346.08 total 1052.38 last 32.13) - open/close 31994 (time 1709788356.11 total 1062.41 last 32.01) - open/close 32321 (time 1709788366.13 total 1072.43 last 32.65) - open/close 32671 (time 1709788376.15 total 1082.45 last 34.91) - open/close 33002 (time 1709788386.16 total 1092.46 last 33.08) - open/close 33344 (time 1709788396.17 total 1102.47 last 34.15) - open/close 33669 (time 1709788406.18 total 1112.48 last 32.48) - open/close 33974 (time 1709788416.18 total 1122.48 last 30.50) - open/close 34252 (time 1709788426.21 total 1132.51 last 27.72) - open/close 34537 (time 1709788436.21 total 1142.51 last 28.49) - open/close 34819 (time 1709788446.24 total 1152.54 last 28.14) - open/close 35101 (time 1709788456.28 total 1162.58 last 28.08) - open/close 35381 (time 1709788466.28 total 1172.58 last 27.99) - open/close 35684 (time 1709788476.30 total 1182.60 last 30.26) - open/close 35946 (time 1709788486.30 total 1192.60 last 26.19) - open/close 36237 (time 1709788496.32 total 1202.63 last 29.03) - open/close 36530 (time 1709788506.33 total 1212.63 last 29.29) - open/close 36840 (time 1709788516.33 total 1222.63 last 30.99) - open/close 37100 (time 1709788526.34 total 1232.64 last 25.99) - open/close 37390 (time 1709788536.34 total 1242.64 last 28.98) - open/close 37697 (time 1709788546.35 total 1252.65 last 30.67) - open/close 37983 (time 1709788556.36 total 1262.66 last 28.57) - open/close 38261 (time 1709788566.39 total 1272.69 last 27.74) - open/close 38552 (time 1709788576.40 total 1282.70 last 29.07) - open/close 38823 (time 1709788586.41 total 1292.71 last 27.07) - open/close 39134 (time 1709788596.41 total 1302.71 last 31.10) - open/close 39437 (time 1709788606.45 total 1312.75 last 30.18) - open/close 39727 (time 1709788616.45 total 1322.76 last 28.98) - open/close 40000 (time 1709788625.75 total 1332.05 last 29.38) - open/close 40295 (time 1709788635.78 total 1342.08 last 29.42) - open/close 40589 (time 1709788645.80 total 1352.10 last 29.34) - open/close 40867 (time 1709788655.80 total 1362.10 last 27.78) - open/close 41149 (time 1709788665.81 total 1372.11 last 28.17) - open/close 41366 (time 1709788675.86 total 1382.16 last 21.60) - open/close 41566 (time 1709788685.86 total 1392.16 last 20.00) - open/close 41789 (time 1709788695.88 total 1402.18 last 22.26) - open/close 42039 (time 1709788705.89 total 1412.19 last 24.96) - open/close 42289 (time 1709788715.92 total 1422.22 last 24.94) - open/close 42539 (time 1709788725.94 total 1432.24 last 24.95) - open/close 42794 (time 1709788735.94 total 1442.24 last 25.49) - open/close 43028 (time 1709788745.99 total 1452.29 last 23.29) - open/close 43263 (time 1709788756.01 total 1462.31 last 23.45) - open/close 43537 (time 1709788766.01 total 1472.31 last 27.40) - open/close 43809 (time 1709788776.02 total 1482.32 last 27.18) - open/close 44040 (time 1709788786.06 total 1492.37 last 22.99) - open/close 44286 (time 1709788796.09 total 1502.39 last 24.53) - open/close 44529 (time 1709788806.12 total 1512.42 last 24.23) - open/close 44790 (time 1709788816.15 total 1522.45 last 26.03) - open/close 45042 (time 1709788826.18 total 1532.48 last 25.11) - open/close 45293 (time 1709788836.19 total 1542.49 last 25.08) - open/close 45526 (time 1709788846.23 total 1552.53 last 23.21) - open/close 45755 (time 1709788856.26 total 1562.56 last 22.83) - open/close 46024 (time 1709788866.27 total 1572.57 last 26.87) - open/close 46296 (time 1709788876.30 total 1582.60 last 27.12) - open/close 46549 (time 1709788886.33 total 1592.63 last 25.23) - open/close 46804 (time 1709788896.34 total 1602.64 last 25.48) - open/close 47051 (time 1709788906.34 total 1612.64 last 24.70) - open/close 47302 (time 1709788916.34 total 1622.64 last 25.09) - open/close 47530 (time 1709788926.35 total 1632.65 last 22.78) - open/close 47779 (time 1709788936.37 total 1642.67 last 24.85) - open/close 48020 (time 1709788946.37 total 1652.67 last 24.09) - open/close 48255 (time 1709788956.37 total 1662.67 last 23.49) - open/close 48492 (time 1709788966.41 total 1672.71 last 23.61) - open/close 48733 (time 1709788976.41 total 1682.71 last 24.10) - open/close 48955 (time 1709788986.43 total 1692.73 last 22.16) - open/close 49192 (time 1709788996.44 total 1702.74 last 23.69) - open/close 49426 (time 1709789006.44 total 1712.74 last 23.38) - open/close 49646 (time 1709789016.48 total 1722.78 last 21.93) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1725.09 seconds: 28.81 ops/second - unlinked 0 (time 1709789020 ; total 0 ; last 0) - unlinked 10000 (time 1709789506 ; total 486 ; last 486) - unlinked 20000 (time 1709789860 ; total 840 ; last 354) - unlinked 30000 (time 1709790035 ; total 1015 ; last 175) - unlinked 40000 (time 1709790121 ; total 1101 ; last 86) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1141 seconds: 43.556530 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg443-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg443-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg443-server stop mds service on oleg443-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg443-server unloading modules on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg443-client: oleg443-client: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-client: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2