== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:18:58 (1709785138) start mds service on oleg350-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg350-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock oleg350-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 695 (time 1709785164.35 total 10.35 last 67.18) - open/close 1368 (time 1709785174.86 total 20.85 last 64.06) - open/close 2152 (time 1709785185.71 total 31.70 last 72.25) - open/close 2929 (time 1709785196.56 total 42.56 last 71.60) - open/close 3599 (time 1709785207.30 total 53.29 last 62.41) - open/close 4223 (time 1709785217.97 total 63.97 last 58.44) - open/close 4813 (time 1709785228.68 total 74.67 last 55.11) - open/close 5412 (time 1709785239.36 total 85.35 last 56.11) - open/close 5978 (time 1709785249.36 total 95.35 last 56.60) - open/close 6349 (time 1709785259.76 total 105.75 last 35.66) - open/close 6828 (time 1709785270.60 total 116.59 last 44.20) - open/close 7019 (time 1709785281.48 total 127.47 last 17.55) - open/close 7365 (time 1709785292.00 total 137.99 last 32.91) - open/close 7662 (time 1709785302.28 total 148.27 last 28.88) - open/close 8196 (time 1709785312.28 total 158.27 last 53.40) - open/close 8665 (time 1709785322.34 total 168.33 last 46.61) - open/close 9165 (time 1709785332.63 total 178.63 last 48.58) - open/close 9628 (time 1709785342.67 total 188.66 last 46.14) - open/close 9947 (time 1709785353.28 total 199.27 last 30.06) - open/close 10430 (time 1709785364.72 total 210.72 last 42.21) - open/close 10775 (time 1709785376.56 total 222.56 last 29.13) - open/close 11019 (time 1709785386.95 total 232.94 last 23.50) - open/close 11431 (time 1709785397.43 total 243.42 last 39.30) - open/close 12550 (time 1709785407.43 total 253.43 last 111.85) - open/close 13938 (time 1709785417.44 total 263.43 last 138.74) - open/close 15263 (time 1709785427.44 total 273.43 last 132.49) - open/close 16546 (time 1709785437.44 total 283.44 last 128.24) - open/close 17739 (time 1709785447.45 total 293.44 last 119.23) - open/close 18913 (time 1709785457.45 total 303.45 last 117.36) - open/close 20000 (time 1709785467.26 total 313.25 last 110.84) - open/close 21031 (time 1709785477.27 total 323.26 last 103.02) - open/close 21978 (time 1709785487.27 total 333.27 last 94.65) - open/close 22872 (time 1709785497.28 total 343.28 last 89.30) - open/close 23718 (time 1709785507.28 total 353.28 last 84.60) - open/close 24542 (time 1709785517.29 total 363.29 last 82.32) - open/close 25448 (time 1709785527.30 total 373.29 last 90.55) - open/close 26274 (time 1709785537.31 total 383.31 last 82.49) - open/close 27077 (time 1709785547.32 total 393.31 last 80.26) - open/close 27836 (time 1709785557.32 total 403.31 last 75.88) - open/close 28614 (time 1709785567.33 total 413.32 last 77.76) - open/close 29409 (time 1709785577.34 total 423.33 last 79.42) - open/close 30000 (time 1709785584.96 total 430.95 last 77.56) - open/close 30748 (time 1709785594.96 total 440.96 last 74.74) - open/close 31455 (time 1709785604.98 total 450.97 last 70.62) - open/close 32144 (time 1709785614.99 total 460.98 last 68.82) - open/close 32787 (time 1709785624.99 total 470.98 last 64.29) - open/close 33463 (time 1709785635.00 total 480.99 last 67.54) - open/close 34045 (time 1709785645.01 total 491.00 last 58.13) - open/close 34648 (time 1709785655.02 total 501.01 last 60.25) - open/close 35223 (time 1709785665.03 total 511.02 last 57.47) - open/close 35870 (time 1709785675.04 total 521.03 last 64.62) - open/close 36443 (time 1709785685.04 total 531.03 last 57.30) - open/close 36931 (time 1709785695.04 total 541.03 last 48.80) - open/close 37457 (time 1709785705.05 total 551.04 last 52.56) - open/close 37960 (time 1709785715.06 total 561.05 last 50.23) - open/close 38524 (time 1709785725.07 total 571.06 last 56.37) - open/close 39087 (time 1709785735.08 total 581.07 last 56.20) - open/close 39646 (time 1709785745.09 total 591.08 last 55.86) - open/close 40000 (time 1709785751.32 total 597.31 last 56.82) - open/close 40544 (time 1709785761.32 total 607.31 last 54.40) - open/close 41086 (time 1709785771.33 total 617.33 last 54.13) - open/close 41622 (time 1709785781.34 total 627.33 last 53.55) - open/close 42132 (time 1709785791.35 total 637.35 last 50.94) - open/close 42664 (time 1709785801.35 total 647.35 last 53.20) - open/close 43166 (time 1709785811.37 total 657.36 last 50.11) - open/close 43689 (time 1709785821.37 total 667.37 last 52.29) - open/close 44161 (time 1709785831.38 total 677.38 last 47.15) - open/close 44693 (time 1709785841.39 total 687.38 last 53.19) - open/close 45204 (time 1709785851.40 total 697.39 last 51.06) - open/close 45713 (time 1709785861.40 total 707.39 last 50.87) - open/close 46214 (time 1709785871.41 total 717.41 last 50.04) - open/close 46740 (time 1709785881.42 total 727.41 last 52.60) - open/close 47201 (time 1709785891.42 total 737.41 last 46.09) - open/close 47679 (time 1709785901.43 total 747.42 last 47.75) - open/close 48149 (time 1709785911.43 total 757.43 last 46.97) - open/close 48620 (time 1709785921.44 total 767.43 last 47.09) - open/close 49081 (time 1709785931.45 total 777.44 last 46.03) - open/close 49544 (time 1709785941.45 total 787.45 last 46.29) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 790.67 seconds: 62.86 ops/second - unlinked 0 (time 1709785945 ; total 0 ; last 0) - unlinked 10000 (time 1709786210 ; total 265 ; last 265) - unlinked 20000 (time 1709786446 ; total 501 ; last 236) - unlinked 30000 (time 1709786643 ; total 698 ; last 197) - unlinked 40000 (time 1709786755 ; total 810 ; last 112) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 850 seconds: 58.468235 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server unloading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2