== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:46:39 (1709786799) start mds service on oleg339-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg339-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock oleg339-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 200 (time 1709786832.05 total 11.45 last 17.47) - open/close 501 (time 1709786842.08 total 21.47 last 30.02) - open/close 723 (time 1709786852.31 total 31.71 last 21.68) - open/close 1044 (time 1709786862.35 total 41.75 last 31.99) - open/close 1250 (time 1709786872.69 total 52.08 last 19.93) - open/close 1512 (time 1709786883.32 total 62.72 last 24.63) - open/close 1630 (time 1709786893.33 total 72.72 last 11.79) - open/close 1892 (time 1709786903.66 total 83.06 last 25.34) - open/close 2176 (time 1709786913.67 total 93.07 last 28.38) - open/close 2391 (time 1709786924.27 total 103.67 last 20.28) - open/close 2557 (time 1709786934.99 total 114.39 last 15.48) - open/close 2736 (time 1709786945.40 total 124.80 last 17.20) - open/close 2876 (time 1709786955.40 total 134.80 last 14.00) - open/close 3041 (time 1709786965.96 total 145.35 last 15.64) - open/close 3137 (time 1709786976.36 total 155.76 last 9.23) - open/close 3426 (time 1709786986.66 total 166.06 last 28.05) - open/close 3561 (time 1709786996.68 total 176.08 last 13.47) - open/close 3740 (time 1709787007.60 total 186.99 last 16.40) - open/close 3977 (time 1709787018.16 total 197.56 last 22.43) - open/close 4311 (time 1709787028.59 total 207.99 last 32.03) - open/close 4506 (time 1709787038.60 total 218.00 last 19.47) - open/close 4677 (time 1709787048.93 total 228.32 last 16.56) - open/close 4957 (time 1709787058.93 total 238.33 last 27.99) - open/close 5166 (time 1709787069.74 total 249.13 last 19.34) - open/close 5429 (time 1709787079.84 total 259.24 last 26.02) - open/close 5569 (time 1709787089.88 total 269.28 last 13.95) - open/close 5775 (time 1709787100.37 total 279.77 last 19.63) - open/close 5985 (time 1709787113.06 total 292.46 last 16.55) - open/close 6253 (time 1709787123.07 total 302.46 last 26.79) - open/close 6389 (time 1709787133.28 total 312.67 last 13.32) - open/close 6545 (time 1709787143.91 total 323.31 last 14.67) - open/close 6760 (time 1709787154.43 total 333.82 last 20.45) - open/close 6906 (time 1709787165.05 total 344.44 last 13.74) - open/close 6970 (time 1709787175.18 total 354.57 last 6.32) - open/close 7042 (time 1709787188.55 total 367.95 last 5.38) - open/close 7177 (time 1709787198.60 total 378.00 last 13.43) - open/close 7329 (time 1709787208.61 total 388.01 last 15.19) - open/close 7399 (time 1709787225.10 total 404.50 last 4.25) - open/close 7526 (time 1709787235.47 total 414.87 last 12.25) - open/close 7759 (time 1709787245.87 total 425.27 last 22.39) - open/close 7967 (time 1709787256.78 total 436.17 last 19.08) - open/close 8138 (time 1709787267.62 total 447.01 last 15.77) - open/close 8239 (time 1709787277.86 total 457.26 last 9.86) - open/close 8346 (time 1709787287.95 total 467.35 last 10.61) - open/close 8499 (time 1709787298.18 total 477.57 last 14.96) - open/close 8648 (time 1709787308.99 total 488.38 last 13.78) - open/close 8753 (time 1709787319.13 total 498.52 last 10.35) - open/close 8915 (time 1709787329.60 total 508.99 last 15.47) - open/close 9585 (time 1709787339.62 total 519.01 last 66.86) - open/close 10000 (time 1709787345.53 total 524.93 last 70.20) - open/close 10307 (time 1709787361.79 total 541.19 last 18.88) - open/close 10904 (time 1709787371.79 total 551.19 last 59.70) - open/close 11447 (time 1709787381.80 total 561.19 last 54.28) - open/close 11981 (time 1709787391.80 total 571.19 last 53.39) - open/close 12475 (time 1709787401.81 total 581.21 last 49.33) - open/close 12962 (time 1709787411.86 total 591.25 last 48.48) - open/close 13455 (time 1709787421.91 total 601.31 last 49.04) - open/close 13954 (time 1709787431.91 total 611.31 last 49.89) - open/close 14436 (time 1709787441.92 total 621.32 last 48.16) - open/close 14909 (time 1709787451.92 total 631.32 last 47.29) - open/close 15374 (time 1709787461.95 total 641.35 last 46.37) - open/close 15761 (time 1709787471.96 total 651.36 last 38.66) - open/close 16200 (time 1709787481.97 total 661.36 last 43.88) - open/close 16605 (time 1709787491.97 total 671.37 last 40.47) - open/close 16966 (time 1709787501.98 total 681.37 last 36.09) - open/close 17376 (time 1709787511.98 total 691.38 last 40.97) - open/close 17782 (time 1709787521.99 total 701.39 last 40.55) - open/close 18111 (time 1709787532.01 total 711.41 last 32.85) - open/close 18504 (time 1709787542.02 total 721.41 last 39.27) - open/close 18900 (time 1709787552.03 total 731.43 last 39.55) - open/close 19228 (time 1709787562.04 total 741.43 last 32.78) - open/close 19320 (time 1709787575.90 total 755.30 last 6.63) - open/close 19652 (time 1709787585.92 total 765.32 last 33.13) - open/close 19989 (time 1709787595.94 total 775.33 last 33.66) - open/close 20330 (time 1709787605.95 total 785.35 last 34.06) - open/close 20681 (time 1709787615.96 total 795.36 last 35.05) - open/close 20986 (time 1709787625.97 total 805.37 last 30.48) - open/close 21296 (time 1709787636.01 total 815.40 last 30.89) - open/close 21594 (time 1709787646.02 total 825.41 last 29.77) - open/close 21882 (time 1709787656.05 total 835.45 last 28.71) - open/close 22213 (time 1709787666.06 total 845.45 last 33.08) - open/close 22488 (time 1709787676.06 total 855.45 last 27.50) - open/close 22776 (time 1709787686.06 total 865.46 last 28.78) - open/close 23047 (time 1709787696.08 total 875.47 last 27.07) - open/close 23306 (time 1709787706.08 total 885.48 last 25.88) - open/close 23614 (time 1709787716.09 total 895.49 last 30.77) - open/close 23927 (time 1709787726.09 total 905.49 last 31.30) - open/close 24231 (time 1709787736.13 total 915.52 last 30.30) - open/close 24496 (time 1709787746.21 total 925.60 last 26.29) - open/close 24774 (time 1709787756.21 total 935.60 last 27.80) - open/close 25091 (time 1709787766.23 total 945.62 last 31.64) - open/close 25349 (time 1709787776.25 total 955.65 last 25.74) - open/close 25648 (time 1709787786.27 total 965.66 last 29.85) - open/close 25926 (time 1709787796.32 total 975.71 last 27.66) - open/close 26236 (time 1709787806.33 total 985.72 last 30.97) - open/close 26554 (time 1709787816.35 total 995.75 last 31.72) - open/close 26843 (time 1709787826.36 total 1005.75 last 28.88) - open/close 27186 (time 1709787836.36 total 1015.76 last 34.28) - open/close 27493 (time 1709787846.38 total 1025.78 last 30.64) - open/close 27496 (time 1709787857.50 total 1036.89 last 0.27) - open/close 27792 (time 1709787867.52 total 1046.92 last 29.53) - open/close 28078 (time 1709787877.53 total 1056.93 last 28.57) - open/close 28376 (time 1709787887.56 total 1066.95 last 29.74) - open/close 28663 (time 1709787897.56 total 1076.96 last 28.68) - open/close 28971 (time 1709787907.58 total 1086.98 last 30.74) - open/close 29260 (time 1709787917.63 total 1097.02 last 28.78) - open/close 29549 (time 1709787927.65 total 1107.05 last 28.83) - open/close 29815 (time 1709787937.65 total 1117.05 last 26.60) - open/close 30000 (time 1709787944.50 total 1123.90 last 27.00) - open/close 30287 (time 1709787954.52 total 1133.91 last 28.66) - open/close 30583 (time 1709787964.54 total 1143.94 last 29.52) - open/close 30862 (time 1709787974.57 total 1153.97 last 27.82) - open/close 31148 (time 1709787984.59 total 1163.98 last 28.55) - open/close 31456 (time 1709787994.60 total 1174.00 last 30.75) - open/close 31760 (time 1709788004.63 total 1184.02 last 30.33) - open/close 32054 (time 1709788014.63 total 1194.03 last 29.38) - open/close 32337 (time 1709788024.64 total 1204.03 last 28.30) - open/close 32598 (time 1709788034.64 total 1214.04 last 26.08) - open/close 32877 (time 1709788044.66 total 1224.05 last 27.86) - open/close 33138 (time 1709788054.68 total 1234.08 last 26.03) - open/close 33413 (time 1709788064.69 total 1244.09 last 27.48) - open/close 33707 (time 1709788074.70 total 1254.09 last 29.38) - open/close 34055 (time 1709788084.71 total 1264.10 last 34.76) - open/close 34395 (time 1709788094.71 total 1274.11 last 33.99) - open/close 34741 (time 1709788104.73 total 1284.12 last 34.54) - open/close 35076 (time 1709788114.77 total 1294.17 last 33.35) - open/close 35412 (time 1709788124.78 total 1304.18 last 33.56) - open/close 35748 (time 1709788134.81 total 1314.21 last 33.50) - open/close 36083 (time 1709788144.83 total 1324.22 last 33.44) - open/close 36398 (time 1709788154.84 total 1334.24 last 31.45) - open/close 36724 (time 1709788164.85 total 1344.24 last 32.59) - open/close 37040 (time 1709788174.85 total 1354.25 last 31.58) - open/close 37355 (time 1709788184.87 total 1364.27 last 31.43) - open/close 37666 (time 1709788194.89 total 1374.28 last 31.06) - open/close 37975 (time 1709788204.90 total 1384.30 last 30.85) - open/close 38290 (time 1709788214.93 total 1394.33 last 31.42) - open/close 38599 (time 1709788224.93 total 1404.33 last 30.90) - open/close 38900 (time 1709788234.96 total 1414.35 last 30.02) - open/close 39200 (time 1709788244.98 total 1424.37 last 29.95) - open/close 39493 (time 1709788254.99 total 1434.38 last 29.27) - open/close 39786 (time 1709788265.01 total 1444.40 last 29.24) - open/close 40000 (time 1709788272.64 total 1452.03 last 28.05) - open/close 40280 (time 1709788282.65 total 1462.04 last 27.97) - open/close 40543 (time 1709788292.65 total 1472.04 last 26.30) - open/close 40819 (time 1709788302.68 total 1482.07 last 27.53) - open/close 41088 (time 1709788312.69 total 1492.08 last 26.87) - open/close 41366 (time 1709788322.70 total 1502.09 last 27.77) - open/close 41660 (time 1709788332.70 total 1512.10 last 29.39) - open/close 41950 (time 1709788342.73 total 1522.13 last 28.92) - open/close 42235 (time 1709788352.77 total 1532.16 last 28.40) - open/close 42526 (time 1709788362.79 total 1542.19 last 29.03) - open/close 42797 (time 1709788372.82 total 1552.21 last 27.02) - open/close 43080 (time 1709788382.82 total 1562.22 last 28.29) - open/close 43355 (time 1709788392.86 total 1572.26 last 27.38) - open/close 43637 (time 1709788402.89 total 1582.29 last 28.13) - open/close 43905 (time 1709788412.90 total 1592.29 last 26.78) - open/close 44191 (time 1709788422.93 total 1602.32 last 28.51) - open/close 44462 (time 1709788432.95 total 1612.34 last 27.05) - open/close 44752 (time 1709788442.98 total 1622.37 last 28.91) - open/close 45022 (time 1709788452.98 total 1632.37 last 27.00) - open/close 45292 (time 1709788463.01 total 1642.41 last 26.91) - open/close 45565 (time 1709788473.02 total 1652.41 last 27.29) - open/close 45839 (time 1709788483.03 total 1662.43 last 27.36) - open/close 46087 (time 1709788493.05 total 1672.45 last 24.74) - open/close 46331 (time 1709788503.07 total 1682.46 last 24.37) - open/close 46598 (time 1709788513.09 total 1692.49 last 26.63) - open/close 46863 (time 1709788523.12 total 1702.52 last 26.43) - open/close 47121 (time 1709788533.16 total 1712.56 last 25.69) - open/close 47383 (time 1709788543.18 total 1722.57 last 26.16) - open/close 47639 (time 1709788553.18 total 1732.58 last 25.59) - open/close 47896 (time 1709788563.19 total 1742.59 last 25.66) - open/close 48158 (time 1709788573.21 total 1752.61 last 26.15) - open/close 48415 (time 1709788583.25 total 1762.64 last 25.61) - open/close 48671 (time 1709788593.26 total 1772.66 last 25.57) - open/close 48926 (time 1709788603.28 total 1782.68 last 25.44) - open/close 49184 (time 1709788613.31 total 1792.71 last 25.72) - open/close 49438 (time 1709788623.33 total 1802.72 last 25.37) - open/close 49687 (time 1709788633.36 total 1812.76 last 24.81) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1813.18 seconds: 27.41 ops/second - unlinked 0 (time 1709788635 ; total 0 ; last 0) - unlinked 10000 (time 1709789100 ; total 465 ; last 465) - unlinked 20000 (time 1709789474 ; total 839 ; last 374) - unlinked 30000 (time 1709789746 ; total 1111 ; last 272) - unlinked 40000 (time 1709789898 ; total 1263 ; last 152) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1313 seconds: 37.850723 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2