== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:18:23 (1709785103) start mds service on oleg207-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg207-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock oleg207-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 773 (time 1709785129.82 total 10.44 last 74.02) - open/close 1545 (time 1709785140.37 total 21.00 last 73.12) - open/close 2346 (time 1709785151.17 total 31.80 last 74.17) - open/close 3075 (time 1709785161.97 total 42.60 last 67.51) - open/close 3653 (time 1709785172.51 total 53.13 last 54.86) - open/close 4331 (time 1709785183.24 total 63.87 last 63.18) - open/close 4880 (time 1709785193.84 total 74.46 last 51.80) - open/close 5455 (time 1709785204.40 total 85.03 last 54.43) - open/close 6023 (time 1709785215.01 total 95.64 last 53.53) - open/close 6555 (time 1709785225.57 total 106.20 last 50.38) - open/close 7095 (time 1709785236.17 total 116.80 last 50.94) - open/close 7600 (time 1709785246.81 total 127.44 last 47.47) - open/close 8077 (time 1709785257.58 total 138.21 last 44.29) - open/close 8350 (time 1709785267.69 total 148.32 last 27.01) - open/close 8551 (time 1709785278.65 total 159.28 last 18.34) - open/close 8932 (time 1709785289.15 total 169.78 last 36.27) - open/close 9231 (time 1709785299.66 total 180.29 last 28.46) - open/close 9579 (time 1709785310.24 total 190.87 last 32.89) - open/close 9865 (time 1709785320.42 total 201.05 last 28.10) - open/close 10000 (time 1709785322.99 total 203.62 last 52.52) - open/close 10350 (time 1709785333.46 total 214.08 last 33.44) - open/close 10776 (time 1709785344.28 total 224.91 last 39.36) - open/close 11327 (time 1709785354.66 total 235.29 last 53.06) - open/close 13230 (time 1709785364.67 total 245.30 last 190.22) - open/close 15004 (time 1709785374.67 total 255.30 last 177.37) - open/close 16495 (time 1709785384.67 total 265.30 last 149.06) - open/close 17802 (time 1709785394.67 total 275.30 last 130.67) - open/close 19169 (time 1709785404.68 total 285.30 last 136.67) - open/close 20000 (time 1709785411.19 total 291.82 last 127.57) - open/close 21288 (time 1709785421.20 total 301.82 last 128.72) - open/close 22553 (time 1709785431.20 total 311.83 last 126.47) - open/close 23853 (time 1709785441.20 total 321.83 last 129.98) - open/close 25034 (time 1709785451.20 total 331.83 last 118.07) - open/close 26240 (time 1709785461.21 total 341.83 last 120.55) - open/close 27168 (time 1709785471.22 total 351.85 last 92.66) - open/close 28009 (time 1709785481.23 total 361.86 last 84.03) - open/close 28836 (time 1709785491.24 total 371.87 last 82.63) - open/close 29555 (time 1709785501.26 total 381.89 last 71.74) - open/close 30000 (time 1709785508.52 total 389.15 last 61.31) - open/close 30638 (time 1709785518.53 total 399.16 last 63.72) - open/close 31359 (time 1709785528.54 total 409.17 last 72.05) - open/close 32082 (time 1709785538.54 total 419.17 last 72.26) - open/close 32761 (time 1709785548.56 total 429.19 last 67.77) - open/close 33419 (time 1709785558.57 total 439.20 last 65.74) - open/close 34101 (time 1709785568.58 total 449.20 last 68.17) - open/close 34792 (time 1709785578.58 total 459.21 last 69.05) - open/close 35412 (time 1709785588.59 total 469.22 last 61.96) - open/close 36049 (time 1709785598.60 total 479.23 last 63.62) - open/close 36683 (time 1709785608.61 total 489.24 last 63.33) - open/close 37285 (time 1709785618.62 total 499.25 last 60.15) - open/close 37903 (time 1709785628.64 total 509.27 last 61.71) - open/close 38534 (time 1709785638.65 total 519.28 last 63.03) - open/close 39171 (time 1709785648.66 total 529.28 last 63.65) - open/close 39825 (time 1709785658.66 total 539.29 last 65.36) - open/close 40000 (time 1709785661.43 total 542.06 last 63.20) - open/close 40647 (time 1709785671.44 total 552.07 last 64.63) - open/close 41234 (time 1709785681.46 total 562.09 last 58.60) - open/close 41812 (time 1709785691.47 total 572.10 last 57.75) - open/close 42349 (time 1709785701.47 total 582.10 last 53.68) - open/close 42872 (time 1709785711.48 total 592.11 last 52.26) - open/close 43402 (time 1709785721.49 total 602.12 last 52.95) - open/close 43921 (time 1709785731.50 total 612.12 last 51.87) - open/close 44420 (time 1709785741.52 total 622.14 last 49.80) - open/close 44944 (time 1709785751.53 total 632.15 last 52.35) - open/close 45433 (time 1709785761.53 total 642.16 last 48.89) - open/close 45942 (time 1709785771.53 total 652.16 last 50.89) - open/close 46423 (time 1709785781.53 total 662.16 last 48.09) - open/close 46913 (time 1709785791.55 total 672.18 last 48.91) - open/close 47382 (time 1709785801.57 total 682.19 last 46.82) - open/close 47841 (time 1709785811.59 total 692.21 last 45.81) - open/close 48314 (time 1709785821.59 total 702.22 last 47.27) - open/close 48765 (time 1709785831.61 total 712.24 last 45.01) - open/close 49230 (time 1709785841.62 total 722.24 last 46.48) - open/close 49681 (time 1709785851.63 total 732.26 last 45.03) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 732.62 seconds: 67.84 ops/second - unlinked 0 (time 1709785852 ; total 0 ; last 0) - unlinked 10000 (time 1709786115 ; total 263 ; last 263) - unlinked 20000 (time 1709786333 ; total 481 ; last 218) - unlinked 30000 (time 1709786493 ; total 641 ; last 160) - unlinked 40000 (time 1709786584 ; total 732 ; last 91) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 779 seconds: 63.797176 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server unloading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2