== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:45:32 (1709786732) start mds service on oleg351-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg351-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg351-client.virtnet: -o user_xattr,flock oleg351-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 310 (time 1709786759.67 total 10.94 last 28.35) - open/close 596 (time 1709786769.98 total 21.24 last 27.75) - open/close 913 (time 1709786780.68 total 31.95 last 29.61) - open/close 1221 (time 1709786790.87 total 42.13 last 30.24) - open/close 1460 (time 1709786801.77 total 53.03 last 21.93) - open/close 1693 (time 1709786812.21 total 63.47 last 22.32) - open/close 1968 (time 1709786823.02 total 74.29 last 25.42) - open/close 2306 (time 1709786833.94 total 85.20 last 30.96) - open/close 2576 (time 1709786843.95 total 95.21 last 26.99) - open/close 3029 (time 1709786853.95 total 105.21 last 45.28) - open/close 3284 (time 1709786864.91 total 116.17 last 23.27) - open/close 3617 (time 1709786875.48 total 126.74 last 31.50) - open/close 3849 (time 1709786886.29 total 137.55 last 21.46) - open/close 4116 (time 1709786896.34 total 147.61 last 26.55) - open/close 4348 (time 1709786906.52 total 157.78 last 22.80) - open/close 4512 (time 1709786917.44 total 168.71 last 15.02) - open/close 4676 (time 1709786927.70 total 178.96 last 16.00) - open/close 4787 (time 1709786937.70 total 188.96 last 11.09) - open/close 4921 (time 1709786948.11 total 199.37 last 12.87) - open/close 5101 (time 1709786958.12 total 209.38 last 17.99) - open/close 5349 (time 1709786968.12 total 219.38 last 24.79) - open/close 5581 (time 1709786978.15 total 229.41 last 23.13) - open/close 5767 (time 1709786988.65 total 239.91 last 17.72) - open/close 5936 (time 1709786998.67 total 249.94 last 16.85) - open/close 6051 (time 1709787009.45 total 260.71 last 10.67) - open/close 6310 (time 1709787019.93 total 271.19 last 24.72) - open/close 6381 (time 1709787030.08 total 281.34 last 6.99) - open/close 6504 (time 1709787040.36 total 291.63 last 11.96) - open/close 6753 (time 1709787051.13 total 302.40 last 23.12) - open/close 6994 (time 1709787061.14 total 312.41 last 24.08) - open/close 7225 (time 1709787071.94 total 323.20 last 21.40) - open/close 7381 (time 1709787090.13 total 341.39 last 8.58) - open/close 7621 (time 1709787100.14 total 351.40 last 23.98) - open/close 7686 (time 1709787110.58 total 361.84 last 6.22) - open/close 7899 (time 1709787120.58 total 371.84 last 21.30) - open/close 8122 (time 1709787131.24 total 382.50 last 20.92) - open/close 8328 (time 1709787141.75 total 393.01 last 19.60) - open/close 8485 (time 1709787154.36 total 405.62 last 12.45) - open/close 8639 (time 1709787165.22 total 416.49 last 14.18) - open/close 8810 (time 1709787175.51 total 426.77 last 16.62) - open/close 9007 (time 1709787185.52 total 436.78 last 19.69) - open/close 9145 (time 1709787195.58 total 446.84 last 13.72) - open/close 9541 (time 1709787205.59 total 456.86 last 39.53) - open/close 10000 (time 1709787214.07 total 465.33 last 54.17) - open/close 10502 (time 1709787224.09 total 475.35 last 50.11) - open/close 11091 (time 1709787234.09 total 485.36 last 58.85) - open/close 11631 (time 1709787244.12 total 495.38 last 53.85) - open/close 12193 (time 1709787254.15 total 505.41 last 56.04) - open/close 12293 (time 1709787267.11 total 518.38 last 7.71) - open/close 12531 (time 1709787283.15 total 534.41 last 14.84) - open/close 13066 (time 1709787293.15 total 544.42 last 53.46) - open/close 13553 (time 1709787303.17 total 554.43 last 48.63) - open/close 14064 (time 1709787313.17 total 564.43 last 51.09) - open/close 14551 (time 1709787323.19 total 574.45 last 48.63) - open/close 15038 (time 1709787333.23 total 584.49 last 48.49) - open/close 15540 (time 1709787343.23 total 594.49 last 50.19) - open/close 16009 (time 1709787353.27 total 604.53 last 46.73) - open/close 16488 (time 1709787363.27 total 614.53 last 47.88) - open/close 16894 (time 1709787373.29 total 624.55 last 40.53) - open/close 17281 (time 1709787383.30 total 634.56 last 38.66) - open/close 17715 (time 1709787393.30 total 644.57 last 43.37) - open/close 18091 (time 1709787403.31 total 654.57 last 37.57) - open/close 18466 (time 1709787413.34 total 664.60 last 37.40) - open/close 18882 (time 1709787423.34 total 674.60 last 41.59) - open/close 19270 (time 1709787433.34 total 684.61 last 38.79) - open/close 19669 (time 1709787443.35 total 694.61 last 39.88) - open/close 20000 (time 1709787452.05 total 703.31 last 38.06) - open/close 20361 (time 1709787462.06 total 713.32 last 36.05) - open/close 20608 (time 1709787472.07 total 723.33 last 24.69) - open/close 20944 (time 1709787482.08 total 733.35 last 33.54) - open/close 21296 (time 1709787492.12 total 743.38 last 35.08) - open/close 21560 (time 1709787502.14 total 753.40 last 26.34) - open/close 21893 (time 1709787512.16 total 763.42 last 33.25) - open/close 22213 (time 1709787522.16 total 773.43 last 31.98) - open/close 22524 (time 1709787532.18 total 783.44 last 31.04) - open/close 22838 (time 1709787542.20 total 793.47 last 31.33) - open/close 23171 (time 1709787552.24 total 803.50 last 33.19) - open/close 23467 (time 1709787562.26 total 813.53 last 29.52) - open/close 23737 (time 1709787572.32 total 823.58 last 26.85) - open/close 24007 (time 1709787582.34 total 833.61 last 26.94) - open/close 24311 (time 1709787592.38 total 843.64 last 30.29) - open/close 24555 (time 1709787602.45 total 853.71 last 24.24) - open/close 24838 (time 1709787612.51 total 863.77 last 28.14) - open/close 25108 (time 1709787622.51 total 873.77 last 27.00) - open/close 25395 (time 1709787632.52 total 883.78 last 28.68) - open/close 25688 (time 1709787642.53 total 893.79 last 29.25) - open/close 25950 (time 1709787652.56 total 903.82 last 26.14) - open/close 25974 (time 1709787664.60 total 915.87 last 1.99) - open/close 26233 (time 1709787674.61 total 925.88 last 25.87) - open/close 26481 (time 1709787684.64 total 935.91 last 24.73) - open/close 26739 (time 1709787694.70 total 945.96 last 25.65) - open/close 26954 (time 1709787704.71 total 955.97 last 21.48) - open/close 27187 (time 1709787714.72 total 965.99 last 23.27) - open/close 27480 (time 1709787724.75 total 976.01 last 29.22) - open/close 27755 (time 1709787734.76 total 986.02 last 27.48) - open/close 28008 (time 1709787744.79 total 996.05 last 25.23) - open/close 28253 (time 1709787754.82 total 1006.08 last 24.42) - open/close 28522 (time 1709787764.83 total 1016.09 last 26.87) - open/close 28785 (time 1709787774.84 total 1026.10 last 26.28) - open/close 28799 (time 1709787786.50 total 1037.76 last 1.20) - open/close 29073 (time 1709787796.51 total 1047.78 last 27.36) - open/close 29361 (time 1709787806.52 total 1057.78 last 28.78) - open/close 29657 (time 1709787816.56 total 1067.82 last 29.49) - open/close 29927 (time 1709787826.58 total 1077.85 last 26.93) - open/close 30000 (time 1709787828.94 total 1080.21 last 30.94) - open/close 30292 (time 1709787838.97 total 1090.23 last 29.13) - open/close 30588 (time 1709787848.99 total 1100.25 last 29.54) - open/close 30899 (time 1709787859.00 total 1110.27 last 31.06) - open/close 31174 (time 1709787869.02 total 1120.29 last 27.45) - open/close 31437 (time 1709787879.07 total 1130.33 last 26.17) - open/close 31622 (time 1709787896.47 total 1147.73 last 10.63) - open/close 31906 (time 1709787906.48 total 1157.74 last 28.37) - open/close 32184 (time 1709787916.50 total 1167.76 last 27.75) - open/close 32440 (time 1709787926.50 total 1177.76 last 25.59) - open/close 32705 (time 1709787936.51 total 1187.77 last 26.47) - open/close 32957 (time 1709787946.54 total 1197.80 last 25.14) - open/close 33231 (time 1709787956.54 total 1207.80 last 27.39) - open/close 33508 (time 1709787966.60 total 1217.86 last 27.54) - open/close 33779 (time 1709787976.60 total 1227.87 last 27.08) - open/close 34050 (time 1709787986.61 total 1237.87 last 27.09) - open/close 34339 (time 1709787996.61 total 1247.87 last 28.89) - open/close 34610 (time 1709788006.63 total 1257.89 last 27.04) - open/close 34884 (time 1709788016.63 total 1267.90 last 27.39) - open/close 35141 (time 1709788026.67 total 1277.94 last 25.60) - open/close 35388 (time 1709788036.71 total 1287.97 last 24.61) - open/close 35648 (time 1709788046.73 total 1298.00 last 25.94) - open/close 35900 (time 1709788056.77 total 1308.04 last 25.10) - open/close 36148 (time 1709788066.80 total 1318.07 last 24.72) - open/close 36441 (time 1709788076.81 total 1328.07 last 29.29) - open/close 36780 (time 1709788086.81 total 1338.07 last 33.89) - open/close 37093 (time 1709788096.87 total 1348.13 last 31.12) - open/close 37404 (time 1709788106.89 total 1358.15 last 31.05) - open/close 37720 (time 1709788116.91 total 1368.18 last 31.51) - open/close 38038 (time 1709788126.93 total 1378.19 last 31.75) - open/close 38352 (time 1709788136.94 total 1388.21 last 31.36) - open/close 38665 (time 1709788146.95 total 1398.21 last 31.29) - open/close 38967 (time 1709788156.97 total 1408.23 last 30.13) - open/close 39271 (time 1709788166.99 total 1418.25 last 30.36) - open/close 39572 (time 1709788176.99 total 1428.25 last 30.10) - open/close 39871 (time 1709788187.00 total 1438.27 last 29.84) - open/close 40000 (time 1709788191.54 total 1442.80 last 28.44) - open/close 40292 (time 1709788201.55 total 1452.81 last 29.17) - open/close 40586 (time 1709788211.56 total 1462.82 last 29.38) - open/close 40872 (time 1709788221.57 total 1472.84 last 28.55) - open/close 41165 (time 1709788231.58 total 1482.84 last 29.29) - open/close 41455 (time 1709788241.58 total 1492.84 last 28.99) - open/close 41741 (time 1709788251.59 total 1502.85 last 28.58) - open/close 42012 (time 1709788261.61 total 1512.87 last 27.05) - open/close 42291 (time 1709788271.63 total 1522.90 last 27.82) - open/close 42576 (time 1709788281.67 total 1532.93 last 28.41) - open/close 42842 (time 1709788291.70 total 1542.96 last 26.51) - open/close 43109 (time 1709788301.73 total 1552.99 last 26.62) - open/close 43374 (time 1709788311.75 total 1563.01 last 26.45) - open/close 43643 (time 1709788321.76 total 1573.02 last 26.86) - open/close 43924 (time 1709788331.77 total 1583.04 last 28.06) - open/close 44196 (time 1709788341.79 total 1593.06 last 27.15) - open/close 44465 (time 1709788351.82 total 1603.08 last 26.83) - open/close 44729 (time 1709788361.87 total 1613.13 last 26.27) - open/close 44988 (time 1709788371.90 total 1623.17 last 25.82) - open/close 45254 (time 1709788381.93 total 1633.20 last 26.52) - open/close 45514 (time 1709788391.94 total 1643.20 last 25.98) - open/close 45781 (time 1709788401.94 total 1653.21 last 26.69) - open/close 46039 (time 1709788411.96 total 1663.22 last 25.77) - open/close 46302 (time 1709788421.96 total 1673.23 last 26.28) - open/close 46559 (time 1709788432.00 total 1683.26 last 25.61) - open/close 46838 (time 1709788442.02 total 1693.28 last 27.85) - open/close 47097 (time 1709788452.02 total 1703.29 last 25.88) - open/close 47358 (time 1709788462.04 total 1713.31 last 26.05) - open/close 47613 (time 1709788472.05 total 1723.32 last 25.48) - open/close 47873 (time 1709788482.11 total 1733.38 last 25.84) - open/close 48113 (time 1709788492.13 total 1743.40 last 23.95) - open/close 48348 (time 1709788502.16 total 1753.42 last 23.44) - open/close 48604 (time 1709788512.19 total 1763.45 last 25.53) - open/close 48862 (time 1709788522.22 total 1773.49 last 25.71) - open/close 49109 (time 1709788532.23 total 1783.49 last 24.69) - open/close 49360 (time 1709788542.26 total 1793.52 last 25.03) - open/close 49609 (time 1709788552.27 total 1803.53 last 24.87) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1807.09 seconds: 27.50 ops/second - unlinked 0 (time 1709788557 ; total 0 ; last 0) - unlinked 10000 (time 1709789018 ; total 461 ; last 461) - unlinked 20000 (time 1709789393 ; total 836 ; last 375) - unlinked 30000 (time 1709789670 ; total 1113 ; last 277) - unlinked 40000 (time 1709789829 ; total 1272 ; last 159) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1326 seconds: 37.479637 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg351-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server unloading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2