== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:47:40 (1709786860) start mds service on oleg435-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg435-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 281 (time 1709786891.24 total 10.00 last 28.09) - open/close 540 (time 1709786901.25 total 20.02 last 25.87) - open/close 719 (time 1709786911.55 total 30.31 last 17.39) - open/close 939 (time 1709786921.95 total 40.72 last 21.15) - open/close 1139 (time 1709786932.91 total 51.68 last 18.25) - open/close 1349 (time 1709786943.66 total 62.42 last 19.55) - open/close 1518 (time 1709786954.46 total 73.23 last 15.64) - open/close 1643 (time 1709786965.13 total 83.89 last 11.73) - open/close 1804 (time 1709786975.41 total 94.18 last 15.65) - open/close 2031 (time 1709786985.42 total 104.19 last 22.68) - open/close 2262 (time 1709786996.40 total 115.17 last 21.03) - open/close 2443 (time 1709787006.74 total 125.51 last 17.50) - open/close 2674 (time 1709787016.75 total 135.51 last 23.09) - open/close 3062 (time 1709787027.31 total 146.08 last 36.73) - open/close 3171 (time 1709787037.70 total 156.46 last 10.49) - open/close 3335 (time 1709787048.18 total 166.95 last 15.65) - open/close 3555 (time 1709787058.18 total 176.95 last 22.00) - open/close 3692 (time 1709787068.19 total 186.96 last 13.69) - open/close 3940 (time 1709787079.72 total 198.48 last 21.51) - open/close 4333 (time 1709787090.59 total 209.35 last 36.16) - open/close 4460 (time 1709787101.11 total 219.88 last 12.07) - open/close 4687 (time 1709787112.11 total 230.87 last 20.64) - open/close 4908 (time 1709787122.67 total 241.44 last 20.93) - open/close 5122 (time 1709787132.69 total 251.45 last 21.37) - open/close 5351 (time 1709787143.22 total 261.99 last 21.73) - open/close 5546 (time 1709787153.23 total 272.00 last 19.48) - open/close 5752 (time 1709787163.59 total 282.35 last 19.89) - open/close 5974 (time 1709787174.34 total 293.11 last 20.64) - open/close 6249 (time 1709787184.44 total 303.20 last 27.24) - open/close 6476 (time 1709787194.45 total 313.21 last 22.68) - open/close 6658 (time 1709787204.49 total 323.26 last 18.12) - open/close 6850 (time 1709787214.50 total 333.27 last 19.18) - open/close 7086 (time 1709787224.50 total 343.27 last 23.60) - open/close 7249 (time 1709787234.51 total 353.27 last 16.30) - open/close 7534 (time 1709787244.51 total 363.28 last 28.48) - open/close 7666 (time 1709787254.79 total 373.55 last 12.84) - open/close 7923 (time 1709787265.25 total 384.02 last 24.57) - open/close 8002 (time 1709787276.78 total 395.55 last 6.85) - open/close 8222 (time 1709787286.80 total 405.56 last 21.96) - open/close 8311 (time 1709787296.80 total 415.57 last 8.90) - open/close 8440 (time 1709787307.71 total 426.47 last 11.83) - open/close 8586 (time 1709787319.02 total 437.78 last 12.91) - open/close 8749 (time 1709787331.38 total 450.14 last 13.19) - open/close 8923 (time 1709787341.39 total 460.15 last 17.38) - open/close 9076 (time 1709787352.11 total 470.87 last 14.27) - open/close 9220 (time 1709787362.81 total 481.58 last 13.45) - open/close 9426 (time 1709787372.83 total 491.60 last 20.56) - open/close 9563 (time 1709787383.86 total 502.63 last 12.42) - open/close 9720 (time 1709787394.02 total 512.79 last 15.45) - open/close 10000 (time 1709787400.71 total 519.48 last 41.87) - open/close 10593 (time 1709787410.72 total 529.48 last 59.27) - open/close 11141 (time 1709787420.72 total 539.48 last 54.79) - open/close 11661 (time 1709787430.73 total 549.50 last 51.93) - open/close 12190 (time 1709787440.75 total 559.51 last 52.83) - open/close 12703 (time 1709787450.75 total 569.51 last 51.29) - open/close 13188 (time 1709787460.75 total 579.51 last 48.49) - open/close 13663 (time 1709787470.77 total 589.53 last 47.40) - open/close 14191 (time 1709787480.77 total 599.54 last 52.78) - open/close 14633 (time 1709787490.78 total 609.54 last 44.18) - open/close 15093 (time 1709787500.79 total 619.55 last 45.98) - open/close 15547 (time 1709787510.79 total 629.55 last 45.39) - open/close 15982 (time 1709787520.79 total 639.55 last 43.48) - open/close 16383 (time 1709787530.80 total 649.56 last 40.06) - open/close 16804 (time 1709787540.81 total 659.57 last 42.07) - open/close 17223 (time 1709787550.83 total 669.59 last 41.81) - open/close 17650 (time 1709787560.85 total 679.61 last 42.63) - open/close 17801 (time 1709787575.08 total 693.84 last 10.61) - open/close 17942 (time 1709787589.39 total 708.16 last 9.85) - open/close 18420 (time 1709787599.41 total 718.18 last 47.71) - open/close 18874 (time 1709787609.45 total 728.21 last 45.23) - open/close 19319 (time 1709787619.47 total 738.23 last 44.42) - open/close 19734 (time 1709787629.48 total 748.24 last 41.46) - open/close 20000 (time 1709787636.17 total 754.94 last 39.73) - open/close 20412 (time 1709787646.19 total 764.96 last 41.12) - open/close 20852 (time 1709787656.21 total 774.98 last 43.91) - open/close 21255 (time 1709787666.21 total 784.98 last 40.29) - open/close 21631 (time 1709787676.24 total 795.01 last 37.48) - open/close 21986 (time 1709787686.26 total 805.02 last 35.45) - open/close 22322 (time 1709787696.26 total 815.03 last 33.59) - open/close 22684 (time 1709787706.27 total 825.04 last 36.16) - open/close 23022 (time 1709787716.30 total 835.07 last 33.69) - open/close 23383 (time 1709787726.32 total 845.08 last 36.04) - open/close 23775 (time 1709787736.33 total 855.09 last 39.17) - open/close 24191 (time 1709787746.33 total 865.10 last 41.58) - open/close 24548 (time 1709787756.33 total 875.10 last 35.70) - open/close 24875 (time 1709787766.35 total 885.11 last 32.65) - open/close 25188 (time 1709787776.36 total 895.13 last 31.25) - open/close 25422 (time 1709787795.22 total 913.99 last 12.41) - open/close 25757 (time 1709787805.24 total 924.00 last 33.45) - open/close 26082 (time 1709787815.24 total 934.01 last 32.49) - open/close 26382 (time 1709787825.26 total 944.03 last 29.94) - open/close 26715 (time 1709787835.27 total 954.04 last 33.27) - open/close 27062 (time 1709787845.30 total 964.06 last 34.61) - open/close 27405 (time 1709787855.30 total 974.07 last 34.29) - open/close 27795 (time 1709787865.32 total 984.08 last 38.94) - open/close 28145 (time 1709787875.33 total 994.10 last 34.95) - open/close 28521 (time 1709787885.36 total 1004.12 last 37.51) - open/close 28865 (time 1709787895.37 total 1014.13 last 34.37) - open/close 29206 (time 1709787905.37 total 1024.14 last 34.07) - open/close 29524 (time 1709787915.41 total 1034.17 last 31.70) - open/close 29843 (time 1709787925.43 total 1044.20 last 31.81) - open/close 30000 (time 1709787930.49 total 1049.25 last 31.07) - open/close 30334 (time 1709787940.49 total 1059.25 last 33.40) - open/close 30646 (time 1709787950.52 total 1069.28 last 31.11) - open/close 30957 (time 1709787960.53 total 1079.30 last 31.04) - open/close 31268 (time 1709787970.56 total 1089.32 last 31.03) - open/close 31570 (time 1709787980.59 total 1099.36 last 30.10) - open/close 31889 (time 1709787990.61 total 1109.37 last 31.84) - open/close 32218 (time 1709788000.62 total 1119.38 last 32.88) - open/close 32558 (time 1709788010.64 total 1129.40 last 33.94) - open/close 32875 (time 1709788020.65 total 1139.42 last 31.65) - open/close 33170 (time 1709788030.66 total 1149.43 last 29.47) - open/close 33462 (time 1709788040.67 total 1159.43 last 29.18) - open/close 33743 (time 1709788050.69 total 1169.45 last 28.05) - open/close 34034 (time 1709788060.70 total 1179.46 last 29.07) - open/close 34325 (time 1709788070.70 total 1189.47 last 29.08) - open/close 34628 (time 1709788080.72 total 1199.48 last 30.26) - open/close 34934 (time 1709788090.75 total 1209.51 last 30.51) - open/close 35224 (time 1709788100.78 total 1219.54 last 28.90) - open/close 35517 (time 1709788110.81 total 1229.58 last 29.20) - open/close 35814 (time 1709788120.82 total 1239.58 last 29.69) - open/close 36103 (time 1709788130.83 total 1249.59 last 28.88) - open/close 36375 (time 1709788140.85 total 1259.61 last 27.14) - open/close 36663 (time 1709788150.89 total 1269.65 last 28.69) - open/close 36925 (time 1709788160.91 total 1279.67 last 26.14) - open/close 37184 (time 1709788170.94 total 1289.71 last 25.81) - open/close 37454 (time 1709788180.98 total 1299.74 last 26.90) - open/close 37732 (time 1709788190.99 total 1309.75 last 27.78) - open/close 37994 (time 1709788201.01 total 1319.77 last 26.14) - open/close 38258 (time 1709788211.03 total 1329.80 last 26.34) - open/close 38520 (time 1709788221.07 total 1339.84 last 26.10) - open/close 38788 (time 1709788231.11 total 1349.87 last 26.70) - open/close 39029 (time 1709788241.14 total 1359.90 last 24.03) - open/close 39277 (time 1709788251.16 total 1369.93 last 24.73) - open/close 39559 (time 1709788261.19 total 1379.95 last 28.13) - open/close 39820 (time 1709788271.21 total 1389.98 last 26.03) - open/close 40000 (time 1709788278.23 total 1397.00 last 25.64) - open/close 40064 (time 1709788291.83 total 1410.60 last 4.71) - open/close 40329 (time 1709788301.88 total 1420.64 last 26.38) - open/close 40585 (time 1709788311.92 total 1430.68 last 25.50) - open/close 40837 (time 1709788321.95 total 1440.72 last 25.11) - open/close 41098 (time 1709788331.97 total 1450.74 last 26.05) - open/close 41343 (time 1709788341.98 total 1460.74 last 24.49) - open/close 41579 (time 1709788351.99 total 1470.75 last 23.58) - open/close 41819 (time 1709788362.01 total 1480.77 last 23.95) - open/close 42055 (time 1709788372.05 total 1490.82 last 23.50) - open/close 42298 (time 1709788382.07 total 1500.84 last 24.26) - open/close 42537 (time 1709788392.09 total 1510.86 last 23.85) - open/close 42781 (time 1709788402.15 total 1520.91 last 24.27) - open/close 43020 (time 1709788412.17 total 1530.93 last 23.84) - open/close 43261 (time 1709788422.20 total 1540.96 last 24.04) - open/close 43505 (time 1709788432.22 total 1550.98 last 24.35) - open/close 43748 (time 1709788442.24 total 1561.00 last 24.24) - open/close 43986 (time 1709788452.27 total 1571.03 last 23.74) - open/close 44224 (time 1709788462.28 total 1581.05 last 23.76) - open/close 44469 (time 1709788472.30 total 1591.06 last 24.46) - open/close 44701 (time 1709788482.32 total 1601.09 last 23.15) - open/close 44943 (time 1709788492.35 total 1611.12 last 24.12) - open/close 45188 (time 1709788502.37 total 1621.14 last 24.45) - open/close 45440 (time 1709788512.38 total 1631.14 last 25.18) - open/close 45664 (time 1709788522.39 total 1641.15 last 22.39) - open/close 45897 (time 1709788532.42 total 1651.18 last 23.23) - open/close 46131 (time 1709788542.46 total 1661.22 last 23.31) - open/close 46368 (time 1709788552.50 total 1671.26 last 23.59) - open/close 46601 (time 1709788562.54 total 1681.30 last 23.22) - open/close 46833 (time 1709788572.58 total 1691.34 last 23.10) - open/close 47068 (time 1709788582.59 total 1701.35 last 23.47) - open/close 47307 (time 1709788592.62 total 1711.38 last 23.82) - open/close 47549 (time 1709788602.66 total 1721.42 last 24.11) - open/close 47780 (time 1709788612.67 total 1731.44 last 23.07) - open/close 48019 (time 1709788622.71 total 1741.48 last 23.80) - open/close 48266 (time 1709788632.72 total 1751.48 last 24.69) - open/close 48513 (time 1709788642.72 total 1761.49 last 24.70) - open/close 48755 (time 1709788652.74 total 1771.51 last 24.15) - open/close 48992 (time 1709788662.77 total 1781.53 last 23.65) - open/close 49220 (time 1709788672.81 total 1791.57 last 22.71) - open/close 49467 (time 1709788682.85 total 1801.61 last 24.60) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1810.73 seconds: 27.45 ops/second - unlinked 0 (time 1709788693 ; total 0 ; last 0) - unlinked 10000 (time 1709789187 ; total 494 ; last 494) - unlinked 20000 (time 1709789607 ; total 914 ; last 420) - unlinked 30000 (time 1709789912 ; total 1219 ; last 305) - unlinked 40000 (time 1709790059 ; total 1366 ; last 147) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1423 seconds: 34.924809 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server unloading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2