== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:53:15 (1709787195) start mds service on oleg316-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg316-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg316-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg316-client.virtnet: -o user_xattr,flock oleg316-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 206 (time 1709787247.43 total 10.86 last 18.98) - open/close 382 (time 1709787257.67 total 21.09 last 17.19) - open/close 629 (time 1709787268.26 total 31.68 last 23.33) - open/close 909 (time 1709787278.26 total 41.68 last 27.99) - open/close 1113 (time 1709787288.74 total 52.16 last 19.47) - open/close 1329 (time 1709787299.52 total 62.94 last 20.04) - open/close 1557 (time 1709787309.66 total 73.08 last 22.49) - open/close 1751 (time 1709787319.66 total 83.08 last 19.39) - open/close 2036 (time 1709787329.76 total 93.18 last 28.22) - open/close 2242 (time 1709787340.56 total 103.98 last 19.08) - open/close 2531 (time 1709787350.69 total 114.11 last 28.54) - open/close 2714 (time 1709787360.75 total 124.18 last 18.18) - open/close 2908 (time 1709787370.76 total 134.18 last 19.39) - open/close 3067 (time 1709787381.47 total 144.89 last 14.85) - open/close 3230 (time 1709787392.27 total 155.69 last 15.09) - open/close 3348 (time 1709787402.35 total 165.77 last 11.70) - open/close 3489 (time 1709787412.36 total 175.79 last 14.08) - open/close 3686 (time 1709787422.93 total 186.35 last 18.64) - open/close 3835 (time 1709787433.30 total 196.72 last 14.38) - open/close 3963 (time 1709787443.37 total 206.80 last 12.70) - open/close 4219 (time 1709787453.84 total 217.26 last 24.47) - open/close 4369 (time 1709787464.39 total 227.82 last 14.21) - open/close 4555 (time 1709787474.99 total 238.41 last 17.55) - open/close 4729 (time 1709787485.92 total 249.34 last 15.92) - open/close 4980 (time 1709787495.93 total 259.35 last 25.08) - open/close 5164 (time 1709787506.38 total 269.80 last 17.61) - open/close 5303 (time 1709787516.39 total 279.81 last 13.88) - open/close 5506 (time 1709787526.39 total 289.82 last 20.30) - open/close 5665 (time 1709787536.44 total 299.86 last 15.83) - open/close 5888 (time 1709787552.27 total 315.69 last 14.09) - open/close 6069 (time 1709787562.27 total 325.70 last 18.09) - open/close 6278 (time 1709787572.27 total 335.70 last 20.90) - open/close 6450 (time 1709787582.39 total 345.81 last 17.00) - open/close 6598 (time 1709787592.40 total 355.82 last 14.79) - open/close 6783 (time 1709787603.06 total 366.48 last 17.35) - open/close 6977 (time 1709787613.07 total 376.49 last 19.38) - open/close 7159 (time 1709787623.33 total 386.75 last 17.74) - open/close 7340 (time 1709787633.34 total 396.76 last 18.08) - open/close 7564 (time 1709787643.58 total 407.00 last 21.88) - open/close 7688 (time 1709787653.59 total 417.01 last 12.39) - open/close 7803 (time 1709787663.84 total 427.26 last 11.22) - open/close 8062 (time 1709787673.86 total 437.28 last 25.84) - open/close 8271 (time 1709787683.87 total 447.29 last 20.88) - open/close 8445 (time 1709787694.08 total 457.50 last 17.05) - open/close 8651 (time 1709787704.73 total 468.15 last 19.34) - open/close 8841 (time 1709787714.73 total 478.16 last 19.00) - open/close 8985 (time 1709787724.98 total 488.40 last 14.05) - open/close 9161 (time 1709787734.99 total 498.41 last 17.59) - open/close 9413 (time 1709787745.01 total 508.43 last 25.14) - open/close 9597 (time 1709787755.03 total 518.45 last 18.36) - open/close 9784 (time 1709787765.34 total 528.76 last 18.15) - open/close 9958 (time 1709787775.57 total 538.99 last 17.01) - open/close 10000 (time 1709787779.61 total 543.04 last 10.38) - open/close 10100 (time 1709787789.91 total 553.33 last 9.72) - open/close 10290 (time 1709787800.01 total 563.43 last 18.81) - open/close 10435 (time 1709787810.03 total 573.45 last 14.48) - open/close 10580 (time 1709787820.24 total 583.66 last 14.20) - open/close 10695 (time 1709787830.85 total 594.28 last 10.83) - open/close 10906 (time 1709787841.17 total 604.60 last 20.44) - open/close 11057 (time 1709787851.18 total 614.60 last 15.09) - open/close 11598 (time 1709787861.20 total 624.62 last 53.99) - open/close 12154 (time 1709787871.20 total 634.63 last 55.59) - open/close 12668 (time 1709787881.22 total 644.64 last 51.31) - open/close 13183 (time 1709787891.23 total 654.65 last 51.45) - open/close 13714 (time 1709787901.24 total 664.66 last 53.07) - open/close 14257 (time 1709787911.25 total 674.67 last 54.26) - open/close 14752 (time 1709787921.26 total 684.69 last 49.42) - open/close 15222 (time 1709787931.30 total 694.72 last 46.84) - open/close 15667 (time 1709787941.30 total 704.72 last 44.50) - open/close 16106 (time 1709787951.30 total 714.73 last 43.88) - open/close 16245 (time 1709787965.30 total 728.73 last 9.93) - open/close 16685 (time 1709787975.33 total 738.75 last 43.89) - open/close 17138 (time 1709787985.34 total 748.76 last 45.26) - open/close 17601 (time 1709787995.37 total 758.79 last 46.16) - open/close 18051 (time 1709788005.38 total 768.80 last 44.96) - open/close 18497 (time 1709788015.39 total 778.82 last 44.52) - open/close 18926 (time 1709788025.45 total 788.87 last 42.67) - open/close 19317 (time 1709788035.48 total 798.90 last 38.99) - open/close 19719 (time 1709788045.48 total 808.91 last 40.18) - open/close 20000 (time 1709788052.76 total 816.18 last 38.61) - open/close 20372 (time 1709788062.76 total 826.19 last 37.18) - open/close 20780 (time 1709788072.77 total 836.20 last 40.76) - open/close 21237 (time 1709788082.77 total 846.20 last 45.70) - open/close 21686 (time 1709788092.78 total 856.20 last 44.89) - open/close 22127 (time 1709788102.80 total 866.22 last 43.99) - open/close 22563 (time 1709788112.80 total 876.22 last 43.60) - open/close 23012 (time 1709788122.81 total 886.23 last 44.86) - open/close 23454 (time 1709788132.83 total 896.25 last 44.11) - open/close 23900 (time 1709788142.85 total 906.27 last 44.51) - open/close 24310 (time 1709788152.86 total 916.28 last 40.96) - open/close 24691 (time 1709788173.08 total 936.50 last 18.84) - open/close 25076 (time 1709788183.08 total 946.51 last 38.48) - open/close 25484 (time 1709788193.10 total 956.53 last 40.72) - open/close 25853 (time 1709788203.11 total 966.53 last 36.88) - open/close 26252 (time 1709788213.12 total 976.55 last 39.83) - open/close 26629 (time 1709788223.15 total 986.57 last 37.62) - open/close 27015 (time 1709788233.16 total 996.59 last 38.54) - open/close 27394 (time 1709788243.17 total 1006.59 last 37.88) - open/close 27749 (time 1709788253.17 total 1016.59 last 35.49) - open/close 28097 (time 1709788263.18 total 1026.60 last 34.76) - open/close 28441 (time 1709788273.19 total 1036.61 last 34.38) - open/close 28782 (time 1709788283.20 total 1046.62 last 34.07) - open/close 29117 (time 1709788293.20 total 1056.62 last 33.48) - open/close 29446 (time 1709788303.22 total 1066.64 last 32.84) - open/close 29768 (time 1709788313.24 total 1076.66 last 32.13) - open/close 30000 (time 1709788320.37 total 1083.79 last 32.54) - open/close 30330 (time 1709788330.38 total 1093.80 last 32.97) - open/close 30668 (time 1709788340.39 total 1103.81 last 33.76) - open/close 31009 (time 1709788350.41 total 1113.83 last 34.04) - open/close 31363 (time 1709788360.44 total 1123.86 last 35.31) - open/close 31686 (time 1709788370.44 total 1133.86 last 32.30) - open/close 32027 (time 1709788380.44 total 1143.86 last 34.09) - open/close 32358 (time 1709788390.44 total 1153.87 last 33.09) - open/close 32687 (time 1709788400.45 total 1163.87 last 32.89) - open/close 33011 (time 1709788410.45 total 1173.87 last 32.40) - open/close 33339 (time 1709788420.46 total 1183.88 last 32.77) - open/close 33671 (time 1709788430.50 total 1193.92 last 33.07) - open/close 34019 (time 1709788440.52 total 1203.94 last 34.72) - open/close 34336 (time 1709788450.54 total 1213.96 last 31.64) - open/close 34666 (time 1709788460.55 total 1223.98 last 32.96) - open/close 34980 (time 1709788470.58 total 1234.01 last 31.31) - open/close 35308 (time 1709788480.61 total 1244.03 last 32.73) - open/close 35593 (time 1709788490.64 total 1254.06 last 28.42) - open/close 35886 (time 1709788500.66 total 1264.08 last 29.24) - open/close 36190 (time 1709788510.68 total 1274.10 last 30.34) - open/close 36503 (time 1709788520.70 total 1284.13 last 31.21) - open/close 36798 (time 1709788530.71 total 1294.13 last 29.48) - open/close 37112 (time 1709788540.74 total 1304.16 last 31.31) - open/close 37419 (time 1709788550.74 total 1314.17 last 30.69) - open/close 37726 (time 1709788560.78 total 1324.20 last 30.60) - open/close 38030 (time 1709788570.81 total 1334.23 last 30.31) - open/close 38333 (time 1709788580.83 total 1344.25 last 30.23) - open/close 38632 (time 1709788590.84 total 1354.26 last 29.88) - open/close 38928 (time 1709788600.85 total 1364.27 last 29.56) - open/close 39227 (time 1709788610.85 total 1374.28 last 29.89) - open/close 39522 (time 1709788620.85 total 1384.28 last 29.50) - open/close 39809 (time 1709788630.86 total 1394.28 last 28.68) - open/close 40000 (time 1709788637.72 total 1401.14 last 27.85) - open/close 40229 (time 1709788647.82 total 1411.24 last 22.67) - open/close 40424 (time 1709788657.83 total 1421.25 last 19.48) - open/close 40634 (time 1709788667.83 total 1431.26 last 20.99) - open/close 40894 (time 1709788677.84 total 1441.26 last 25.99) - open/close 41135 (time 1709788687.85 total 1451.27 last 24.08) - open/close 41404 (time 1709788697.85 total 1461.27 last 26.89) - open/close 41665 (time 1709788707.89 total 1471.31 last 26.01) - open/close 41923 (time 1709788717.90 total 1481.32 last 25.77) - open/close 42199 (time 1709788727.92 total 1491.34 last 27.54) - open/close 42481 (time 1709788737.92 total 1501.34 last 28.19) - open/close 42764 (time 1709788747.95 total 1511.37 last 28.23) - open/close 43038 (time 1709788757.96 total 1521.39 last 27.35) - open/close 43297 (time 1709788767.97 total 1531.40 last 25.88) - open/close 43540 (time 1709788777.99 total 1541.41 last 24.26) - open/close 43778 (time 1709788788.03 total 1551.45 last 23.72) - open/close 44017 (time 1709788798.03 total 1561.45 last 23.89) - open/close 44257 (time 1709788808.04 total 1571.46 last 23.99) - open/close 44490 (time 1709788818.05 total 1581.47 last 23.28) - open/close 44715 (time 1709788828.05 total 1591.47 last 22.49) - open/close 44970 (time 1709788838.06 total 1601.48 last 25.48) - open/close 45221 (time 1709788848.09 total 1611.51 last 25.03) - open/close 45451 (time 1709788858.14 total 1621.56 last 22.88) - open/close 45678 (time 1709788868.14 total 1631.56 last 22.70) - open/close 45913 (time 1709788878.17 total 1641.59 last 23.44) - open/close 46149 (time 1709788888.19 total 1651.61 last 23.54) - open/close 46397 (time 1709788898.22 total 1661.65 last 24.72) - open/close 46634 (time 1709788908.25 total 1671.67 last 23.64) - open/close 46881 (time 1709788918.30 total 1681.72 last 24.58) - open/close 47129 (time 1709788928.33 total 1691.75 last 24.72) - open/close 47361 (time 1709788938.35 total 1701.77 last 23.16) - open/close 47591 (time 1709788948.36 total 1711.78 last 22.98) - open/close 47828 (time 1709788958.37 total 1721.79 last 23.67) - open/close 48073 (time 1709788968.40 total 1731.83 last 24.42) - open/close 48326 (time 1709788978.41 total 1741.84 last 25.27) - open/close 48578 (time 1709788988.44 total 1751.86 last 25.14) - open/close 48815 (time 1709788998.48 total 1761.90 last 23.60) - open/close 49057 (time 1709789008.50 total 1771.93 last 24.14) - open/close 49293 (time 1709789018.54 total 1781.96 last 23.53) - open/close 49531 (time 1709789028.57 total 1791.99 last 23.72) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1794.82 seconds: 27.64 ops/second - unlinked 0 (time 1709789032 ; total 0 ; last 0) - unlinked 10000 (time 1709789503 ; total 471 ; last 471) - unlinked 20000 (time 1709789864 ; total 832 ; last 361) - unlinked 30000 (time 1709790082 ; total 1050 ; last 218) - unlinked 40000 (time 1709790198 ; total 1166 ; last 116) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1216 seconds: 40.789474 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg316-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg316-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg316-server stop mds service on oleg316-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg316-server stop mds service on oleg316-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg316-server unloading modules on: 'oleg316-server' oleg316-server: oleg316-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg316-client: oleg316-client: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-server: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-client: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-server: ssh exited with exit code 2