== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:32:28 (1709785948) start mds service on oleg354-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg354-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg354-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0000 oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg354-client.virtnet: -o user_xattr,flock oleg354-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 760 (time 1709785977.86 total 10.81 last 70.34) - open/close 1445 (time 1709785988.72 total 21.66 last 63.11) - open/close 2118 (time 1709785999.55 total 32.49 last 62.15) - open/close 2737 (time 1709786010.32 total 43.26 last 57.48) - open/close 3133 (time 1709786020.73 total 53.68 last 38.01) - open/close 3476 (time 1709786030.91 total 63.86 last 33.70) - open/close 3838 (time 1709786041.18 total 74.13 last 35.25) - open/close 4185 (time 1709786051.55 total 84.49 last 33.48) - open/close 4455 (time 1709786061.62 total 94.56 last 26.80) - open/close 4756 (time 1709786071.91 total 104.85 last 29.25) - open/close 5070 (time 1709786082.24 total 115.18 last 30.40) - open/close 5490 (time 1709786093.01 total 125.95 last 38.99) - open/close 5945 (time 1709786103.03 total 135.97 last 45.41) - open/close 6437 (time 1709786113.36 total 146.30 last 47.64) - open/close 6754 (time 1709786123.95 total 156.89 last 29.93) - open/close 7038 (time 1709786135.50 total 168.44 last 24.58) - open/close 7287 (time 1709786145.84 total 178.79 last 24.08) - open/close 7644 (time 1709786155.84 total 188.79 last 35.70) - open/close 8012 (time 1709786166.98 total 199.92 last 33.06) - open/close 8288 (time 1709786177.63 total 210.58 last 25.89) - open/close 8741 (time 1709786188.39 total 221.33 last 42.11) - open/close 9128 (time 1709786198.74 total 231.68 last 37.40) - open/close 9438 (time 1709786208.77 total 241.72 last 30.89) - open/close 10000 (time 1709786217.80 total 250.74 last 62.30) - open/close 11380 (time 1709786227.80 total 260.74 last 137.96) - open/close 12668 (time 1709786237.80 total 270.74 last 128.77) - open/close 13809 (time 1709786247.81 total 280.75 last 114.04) - open/close 14750 (time 1709786257.81 total 290.75 last 94.04) - open/close 15724 (time 1709786267.81 total 300.76 last 97.38) - open/close 16733 (time 1709786277.81 total 310.76 last 100.90) - open/close 17649 (time 1709786287.82 total 320.76 last 91.55) - open/close 18252 (time 1709786297.82 total 330.77 last 60.28) - open/close 18761 (time 1709786307.83 total 340.77 last 50.86) - open/close 19382 (time 1709786317.85 total 350.79 last 62.01) - open/close 20000 (time 1709786326.91 total 359.85 last 68.17) - open/close 20632 (time 1709786336.92 total 369.86 last 63.19) - open/close 21023 (time 1709786357.05 total 390.00 last 19.42) - open/close 21763 (time 1709786367.06 total 400.00 last 73.93) - open/close 22484 (time 1709786377.06 total 410.01 last 72.09) - open/close 23173 (time 1709786387.07 total 420.01 last 68.86) - open/close 23749 (time 1709786397.08 total 430.02 last 57.57) - open/close 24350 (time 1709786407.09 total 440.03 last 60.01) - open/close 25025 (time 1709786417.10 total 450.05 last 67.41) - open/close 25648 (time 1709786427.11 total 460.06 last 62.24) - open/close 26316 (time 1709786437.12 total 470.06 last 66.78) - open/close 26889 (time 1709786447.13 total 480.07 last 57.25) - open/close 27544 (time 1709786457.14 total 490.08 last 65.43) - open/close 28207 (time 1709786467.15 total 500.09 last 66.20) - open/close 28809 (time 1709786477.16 total 510.10 last 60.16) - open/close 29499 (time 1709786487.16 total 520.11 last 68.95) - open/close 30000 (time 1709786495.11 total 528.05 last 63.08) - open/close 30675 (time 1709786505.11 total 538.05 last 67.50) - open/close 31178 (time 1709786515.13 total 548.07 last 50.21) - open/close 31646 (time 1709786525.14 total 558.08 last 46.74) - open/close 32060 (time 1709786535.16 total 568.10 last 41.31) - open/close 32386 (time 1709786545.17 total 578.11 last 32.58) - open/close 32648 (time 1709786555.19 total 588.13 last 26.15) - open/close 32918 (time 1709786565.19 total 598.13 last 27.00) - open/close 33190 (time 1709786575.20 total 608.14 last 27.18) - open/close 33412 (time 1709786594.61 total 627.56 last 11.43) - open/close 33676 (time 1709786604.67 total 637.61 last 26.25) - open/close 33917 (time 1709786614.74 total 647.68 last 23.94) - open/close 34156 (time 1709786624.75 total 657.69 last 23.86) - open/close 34408 (time 1709786634.76 total 667.70 last 25.19) - open/close 34675 (time 1709786644.77 total 677.71 last 26.66) - open/close 34982 (time 1709786654.79 total 687.73 last 30.65) - open/close 35324 (time 1709786664.80 total 697.74 last 34.16) - open/close 35688 (time 1709786674.80 total 707.75 last 36.37) - open/close 35979 (time 1709786684.84 total 717.78 last 28.99) - open/close 36077 (time 1709786698.86 total 731.81 last 6.99) - open/close 36480 (time 1709786708.87 total 741.81 last 40.29) - open/close 36830 (time 1709786718.87 total 751.81 last 35.00) - open/close 37142 (time 1709786728.89 total 761.84 last 31.12) - open/close 37487 (time 1709786738.91 total 771.85 last 34.46) - open/close 37802 (time 1709786748.91 total 781.85 last 31.50) - open/close 38147 (time 1709786758.92 total 791.86 last 34.45) - open/close 38590 (time 1709786768.94 total 801.88 last 44.21) - open/close 38971 (time 1709786778.96 total 811.90 last 38.04) - open/close 39387 (time 1709786788.97 total 821.91 last 41.55) - open/close 39722 (time 1709786798.98 total 831.92 last 33.46) - open/close 40000 (time 1709786807.57 total 840.51 last 32.38) - open/close 40294 (time 1709786817.58 total 850.53 last 29.35) - open/close 40591 (time 1709786827.60 total 860.54 last 29.65) - open/close 40977 (time 1709786837.62 total 870.56 last 38.52) - open/close 41311 (time 1709786847.64 total 880.58 last 33.34) - open/close 41688 (time 1709786857.66 total 890.60 last 37.64) - open/close 41985 (time 1709786867.68 total 900.62 last 29.62) - open/close 42331 (time 1709786877.70 total 910.64 last 34.54) - open/close 42652 (time 1709786887.71 total 920.65 last 32.07) - open/close 42969 (time 1709786897.74 total 930.68 last 31.61) - open/close 43130 (time 1709786915.66 total 948.61 last 8.98) - open/close 43355 (time 1709786925.68 total 958.63 last 22.46) - open/close 43553 (time 1709786935.72 total 968.66 last 19.73) - open/close 43761 (time 1709786945.73 total 978.67 last 20.77) - open/close 43958 (time 1709786955.77 total 988.71 last 19.63) - open/close 44166 (time 1709786965.79 total 998.73 last 20.76) - open/close 44391 (time 1709786975.81 total 1008.76 last 22.44) - open/close 44575 (time 1709786985.89 total 1018.83 last 18.26) - open/close 44776 (time 1709786995.90 total 1028.84 last 20.07) - open/close 44995 (time 1709787005.93 total 1038.87 last 21.84) - open/close 45252 (time 1709787015.97 total 1048.91 last 25.61) - open/close 45470 (time 1709787025.98 total 1058.92 last 21.77) - open/close 45682 (time 1709787036.00 total 1068.94 last 21.16) - open/close 45871 (time 1709787046.11 total 1079.05 last 18.70) - open/close 46116 (time 1709787056.16 total 1089.10 last 24.37) - open/close 46341 (time 1709787066.17 total 1099.11 last 22.48) - open/close 46555 (time 1709787076.18 total 1109.12 last 21.37) - open/close 46772 (time 1709787086.18 total 1119.12 last 21.70) - open/close 46976 (time 1709787096.23 total 1129.17 last 20.31) - open/close 47205 (time 1709787106.24 total 1139.18 last 22.86) - open/close 47400 (time 1709787116.30 total 1149.24 last 19.39) - open/close 47622 (time 1709787126.31 total 1159.25 last 22.17) - open/close 47812 (time 1709787136.37 total 1169.31 last 18.89) - open/close 48014 (time 1709787146.40 total 1179.34 last 20.13) - open/close 48209 (time 1709787156.43 total 1189.37 last 19.45) - open/close 48404 (time 1709787166.44 total 1199.38 last 19.47) - open/close 48593 (time 1709787176.48 total 1209.42 last 18.82) - open/close 48780 (time 1709787186.51 total 1219.45 last 18.65) - open/close 48970 (time 1709787196.54 total 1229.48 last 18.95) - open/close 49151 (time 1709787206.58 total 1239.52 last 18.03) - open/close 49333 (time 1709787216.58 total 1249.52 last 18.19) - open/close 49498 (time 1709787226.59 total 1259.53 last 16.49) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1265.27 seconds: 39.20 ops/second - unlinked 0 (time 1709787234 ; total 0 ; last 0) - unlinked 10000 (time 1709787795 ; total 561 ; last 561) - unlinked 20000 (time 1709788183 ; total 949 ; last 388) - unlinked 30000 (time 1709788448 ; total 1214 ; last 265) - unlinked 40000 (time 1709788608 ; total 1374 ; last 160) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1431 seconds: 34.661076 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg354-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg354-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg354-server stop mds service on oleg354-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg354-server stop mds service on oleg354-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg354-server unloading modules on: 'oleg354-server' oleg354-server: oleg354-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg354-client: oleg354-client: ssh exited with exit code 2 pdsh@oleg354-client: oleg354-server: ssh exited with exit code 2 pdsh@oleg354-client: oleg354-client: ssh exited with exit code 2 pdsh@oleg354-client: oleg354-server: ssh exited with exit code 2