== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:49:25 (1709786965) start mds service on oleg418-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg418-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg418-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg418-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg418-server: oleg418-server.virtnet: executing set_default_debug -1 all pdsh@oleg418-client: oleg418-server: ssh exited with exit code 1 Started lustre-OST0000 oleg418-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg418-client.virtnet: -o user_xattr,flock oleg418-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 142 (time 1709787012.11 total 11.02 last 12.89) - open/close 287 (time 1709787022.45 total 21.36 last 14.02) - open/close 499 (time 1709787033.59 total 32.50 last 19.03) - open/close 709 (time 1709787043.61 total 42.52 last 20.97) - open/close 889 (time 1709787053.97 total 52.89 last 17.36) - open/close 1177 (time 1709787064.18 total 63.09 last 28.23) - open/close 1360 (time 1709787074.21 total 73.12 last 18.24) - open/close 1558 (time 1709787084.70 total 83.61 last 18.86) - open/close 1703 (time 1709787094.70 total 93.62 last 14.50) - open/close 1879 (time 1709787104.76 total 103.67 last 17.50) - open/close 2139 (time 1709787114.96 total 113.87 last 25.49) - open/close 2263 (time 1709787125.69 total 124.60 last 11.56) - open/close 2489 (time 1709787136.65 total 135.56 last 20.62) - open/close 2632 (time 1709787146.69 total 145.60 last 14.24) - open/close 2830 (time 1709787156.82 total 155.73 last 19.54) - open/close 3176 (time 1709787167.07 total 165.99 last 33.74) - open/close 3436 (time 1709787177.92 total 176.83 last 23.98) - open/close 3636 (time 1709787187.93 total 186.84 last 19.98) - open/close 3925 (time 1709787198.45 total 197.36 last 27.46) - open/close 4129 (time 1709787209.19 total 208.10 last 18.99) - open/close 4337 (time 1709787220.03 total 218.94 last 19.18) - open/close 4477 (time 1709787230.33 total 229.25 last 13.59) - open/close 4620 (time 1709787240.91 total 239.82 last 13.52) - open/close 4841 (time 1709787250.95 total 249.86 last 22.01) - open/close 5002 (time 1709787260.96 total 259.87 last 16.09) - open/close 5141 (time 1709787271.37 total 270.28 last 13.35) - open/close 5356 (time 1709787281.38 total 280.29 last 21.49) - open/close 5519 (time 1709787291.76 total 290.67 last 15.70) - open/close 5689 (time 1709787301.77 total 300.68 last 16.98) - open/close 5990 (time 1709787312.67 total 311.58 last 27.62) - open/close 6127 (time 1709787322.69 total 321.60 last 13.67) - open/close 6328 (time 1709787333.40 total 332.31 last 18.78) - open/close 6540 (time 1709787344.39 total 343.31 last 19.27) - open/close 6757 (time 1709787354.52 total 353.43 last 21.43) - open/close 6914 (time 1709787365.52 total 364.43 last 14.27) - open/close 7087 (time 1709787376.07 total 374.99 last 16.40) - open/close 7268 (time 1709787386.11 total 385.02 last 18.04) - open/close 7381 (time 1709787396.57 total 395.48 last 10.80) - open/close 7518 (time 1709787406.64 total 405.55 last 13.60) - open/close 7668 (time 1709787417.43 total 416.34 last 13.91) - open/close 7910 (time 1709787427.46 total 426.37 last 24.12) - open/close 8119 (time 1709787438.50 total 437.41 last 18.93) - open/close 8240 (time 1709787448.54 total 447.45 last 12.06) - open/close 8399 (time 1709787459.42 total 458.33 last 14.61) - open/close 8611 (time 1709787469.47 total 468.38 last 21.09) - open/close 8762 (time 1709787480.24 total 479.15 last 14.02) - open/close 8889 (time 1709787491.11 total 490.02 last 11.69) - open/close 9097 (time 1709787501.52 total 500.44 last 19.97) - open/close 9250 (time 1709787511.54 total 510.45 last 15.28) - open/close 9385 (time 1709787530.43 total 529.34 last 7.15) - open/close 9468 (time 1709787540.46 total 539.37 last 8.28) - open/close 9757 (time 1709787550.48 total 549.39 last 28.85) - open/close 9982 (time 1709787565.88 total 564.79 last 14.61) - open/close 10487 (time 1709787575.89 total 574.80 last 50.43) - open/close 11005 (time 1709787585.90 total 584.81 last 51.75) - open/close 11578 (time 1709787595.91 total 594.82 last 57.28) - open/close 12098 (time 1709787605.92 total 604.83 last 51.95) - open/close 12578 (time 1709787615.93 total 614.84 last 47.94) - open/close 13063 (time 1709787625.95 total 624.86 last 48.42) - open/close 13520 (time 1709787635.95 total 634.86 last 45.69) - open/close 13978 (time 1709787645.95 total 644.86 last 45.78) - open/close 14458 (time 1709787655.96 total 654.87 last 47.97) - open/close 14915 (time 1709787665.97 total 664.88 last 45.67) - open/close 15335 (time 1709787675.97 total 674.88 last 41.98) - open/close 15730 (time 1709787685.98 total 684.89 last 39.47) - open/close 16153 (time 1709787695.98 total 694.90 last 42.28) - open/close 16551 (time 1709787705.99 total 704.91 last 39.76) - open/close 16899 (time 1709787716.00 total 714.91 last 34.78) - open/close 17254 (time 1709787726.02 total 724.93 last 35.42) - open/close 17653 (time 1709787736.03 total 734.94 last 39.87) - open/close 18089 (time 1709787746.03 total 744.94 last 43.59) - open/close 18468 (time 1709787756.04 total 754.95 last 37.88) - open/close 18791 (time 1709787766.09 total 765.01 last 32.12) - open/close 19095 (time 1709787776.13 total 775.04 last 30.30) - open/close 19394 (time 1709787786.14 total 785.05 last 29.88) - open/close 19693 (time 1709787796.14 total 795.05 last 29.88) - open/close 20000 (time 1709787805.74 total 804.66 last 31.97) - open/close 20352 (time 1709787815.78 total 814.69 last 35.09) - open/close 20675 (time 1709787825.79 total 824.70 last 32.27) - open/close 21015 (time 1709787835.80 total 834.71 last 33.95) - open/close 21395 (time 1709787845.81 total 844.72 last 37.95) - open/close 21787 (time 1709787855.82 total 854.73 last 39.17) - open/close 22208 (time 1709787865.83 total 864.74 last 42.06) - open/close 22597 (time 1709787875.84 total 874.75 last 38.84) - open/close 23006 (time 1709787885.85 total 884.76 last 40.86) - open/close 23391 (time 1709787895.88 total 894.79 last 38.41) - open/close 23772 (time 1709787905.88 total 904.79 last 38.09) - open/close 24135 (time 1709787915.90 total 914.81 last 36.22) - open/close 24514 (time 1709787925.90 total 924.82 last 37.90) - open/close 24880 (time 1709787935.91 total 934.82 last 36.59) - open/close 25234 (time 1709787945.93 total 944.84 last 35.32) - open/close 25599 (time 1709787955.96 total 954.87 last 36.40) - open/close 25804 (time 1709787972.40 total 971.31 last 12.47) - open/close 26145 (time 1709787982.41 total 981.32 last 34.07) - open/close 26508 (time 1709787992.41 total 991.32 last 36.30) - open/close 26858 (time 1709788002.43 total 1001.35 last 34.92) - open/close 27163 (time 1709788012.45 total 1011.36 last 30.46) - open/close 27460 (time 1709788022.46 total 1021.37 last 29.65) - open/close 27802 (time 1709788032.46 total 1031.38 last 34.20) - open/close 28151 (time 1709788042.47 total 1041.38 last 34.88) - open/close 28485 (time 1709788052.50 total 1051.41 last 33.31) - open/close 28822 (time 1709788062.51 total 1061.42 last 33.66) - open/close 29154 (time 1709788072.53 total 1071.44 last 33.12) - open/close 29477 (time 1709788082.54 total 1081.45 last 32.27) - open/close 29819 (time 1709788092.56 total 1091.47 last 34.14) - open/close 30000 (time 1709788098.09 total 1097.00 last 32.75) - open/close 30323 (time 1709788108.11 total 1107.02 last 32.23) - open/close 30644 (time 1709788118.12 total 1117.03 last 32.06) - open/close 30970 (time 1709788128.14 total 1127.05 last 32.54) - open/close 31264 (time 1709788138.15 total 1137.06 last 29.38) - open/close 31565 (time 1709788148.18 total 1147.09 last 30.00) - open/close 31842 (time 1709788158.20 total 1157.11 last 27.64) - open/close 32121 (time 1709788168.20 total 1167.11 last 27.89) - open/close 32414 (time 1709788178.22 total 1177.13 last 29.25) - open/close 32722 (time 1709788188.26 total 1187.17 last 30.68) - open/close 32995 (time 1709788198.26 total 1197.17 last 27.30) - open/close 33268 (time 1709788208.26 total 1207.17 last 27.30) - open/close 33560 (time 1709788218.27 total 1217.18 last 29.17) - open/close 33848 (time 1709788228.31 total 1227.22 last 28.69) - open/close 34130 (time 1709788238.35 total 1237.26 last 28.10) - open/close 34396 (time 1709788248.36 total 1247.27 last 26.56) - open/close 34702 (time 1709788258.38 total 1257.29 last 30.55) - open/close 34991 (time 1709788268.39 total 1267.30 last 28.87) - open/close 35267 (time 1709788278.39 total 1277.30 last 27.60) - open/close 35553 (time 1709788288.42 total 1287.34 last 28.49) - open/close 35844 (time 1709788298.48 total 1297.39 last 28.94) - open/close 36108 (time 1709788308.50 total 1307.41 last 26.35) - open/close 36371 (time 1709788318.52 total 1317.43 last 26.25) - open/close 36651 (time 1709788328.52 total 1327.43 last 27.99) - open/close 36917 (time 1709788338.56 total 1337.47 last 26.50) - open/close 37165 (time 1709788348.56 total 1347.47 last 24.79) - open/close 37412 (time 1709788358.60 total 1357.51 last 24.62) - open/close 37658 (time 1709788368.60 total 1367.51 last 24.59) - open/close 37902 (time 1709788378.64 total 1377.55 last 24.31) - open/close 38145 (time 1709788388.65 total 1387.56 last 24.28) - open/close 38390 (time 1709788398.68 total 1397.59 last 24.43) - open/close 38631 (time 1709788408.70 total 1407.61 last 24.04) - open/close 38890 (time 1709788418.71 total 1417.63 last 25.87) - open/close 39131 (time 1709788428.75 total 1427.66 last 24.02) - open/close 39366 (time 1709788438.77 total 1437.68 last 23.45) - open/close 39612 (time 1709788448.79 total 1447.70 last 24.54) - open/close 39852 (time 1709788458.83 total 1457.74 last 23.92) - open/close 40000 (time 1709788464.98 total 1463.90 last 24.03) - open/close 40257 (time 1709788475.01 total 1473.92 last 25.64) - open/close 40493 (time 1709788485.03 total 1483.95 last 23.54) - open/close 40747 (time 1709788495.04 total 1493.95 last 25.38) - open/close 41007 (time 1709788505.07 total 1503.98 last 25.93) - open/close 41272 (time 1709788515.11 total 1514.02 last 26.39) - open/close 41511 (time 1709788525.15 total 1524.06 last 23.82) - open/close 41757 (time 1709788535.18 total 1534.09 last 24.53) - open/close 42014 (time 1709788545.21 total 1544.12 last 25.62) - open/close 42266 (time 1709788555.24 total 1554.15 last 25.11) - open/close 42499 (time 1709788565.25 total 1564.16 last 23.28) - open/close 42747 (time 1709788575.28 total 1574.19 last 24.74) - open/close 42987 (time 1709788585.31 total 1584.22 last 23.92) - open/close 43246 (time 1709788595.34 total 1594.25 last 25.81) - open/close 43506 (time 1709788605.36 total 1604.27 last 25.96) - open/close 43750 (time 1709788615.36 total 1614.27 last 24.40) - open/close 44000 (time 1709788625.39 total 1624.30 last 24.93) - open/close 44259 (time 1709788635.39 total 1634.30 last 25.89) - open/close 44521 (time 1709788645.41 total 1644.32 last 26.16) - open/close 44774 (time 1709788655.43 total 1654.34 last 25.25) - open/close 45022 (time 1709788665.44 total 1664.35 last 24.76) - open/close 45241 (time 1709788675.46 total 1674.37 last 21.86) - open/close 45460 (time 1709788685.46 total 1684.37 last 21.89) - open/close 45691 (time 1709788695.47 total 1694.39 last 23.07) - open/close 45928 (time 1709788705.49 total 1704.41 last 23.65) - open/close 46164 (time 1709788715.54 total 1714.45 last 23.50) - open/close 46392 (time 1709788725.55 total 1724.46 last 22.77) - open/close 46625 (time 1709788735.56 total 1734.47 last 23.28) - open/close 46849 (time 1709788745.57 total 1744.48 last 22.38) - open/close 47072 (time 1709788755.59 total 1754.50 last 22.25) - open/close 47307 (time 1709788765.61 total 1764.52 last 23.45) - open/close 47543 (time 1709788775.62 total 1774.53 last 23.59) - open/close 47765 (time 1709788785.63 total 1784.54 last 22.16) - open/close 47992 (time 1709788795.67 total 1794.58 last 22.61) - open/close 48213 (time 1709788805.71 total 1804.62 last 22.02) - open/close 48436 (time 1709788815.71 total 1814.63 last 22.28) - open/close 48654 (time 1709788825.75 total 1824.66 last 21.72) - open/close 48881 (time 1709788835.80 total 1834.71 last 22.60) - open/close 49095 (time 1709788845.83 total 1844.74 last 21.32) - open/close 49306 (time 1709788855.83 total 1854.75 last 21.10) - open/close 49534 (time 1709788865.85 total 1864.76 last 22.76) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1867.68 seconds: 26.56 ops/second - unlinked 0 (time 1709788869 ; total 0 ; last 0) - unlinked 10000 (time 1709789374 ; total 505 ; last 505) - unlinked 20000 (time 1709789780 ; total 911 ; last 406) - unlinked 30000 (time 1709790057 ; total 1188 ; last 277) - unlinked 40000 (time 1709790191 ; total 1322 ; last 134) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1358 seconds: 36.524300 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg418-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg418-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg418-server stop mds service on oleg418-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg418-server unloading modules on: 'oleg418-server' oleg418-server: oleg418-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-client: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-client: ssh exited with exit code 2 pdsh@oleg418-client: oleg418-server: ssh exited with exit code 2