== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:25:03 (1709785503) start mds service on oleg254-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg254-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg254-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 558 (time 1709785531.87 total 10.24 last 54.50) - open/close 1264 (time 1709785542.58 total 20.95 last 65.93) - open/close 2021 (time 1709785553.42 total 31.78 last 69.86) - open/close 2694 (time 1709785564.11 total 42.47 last 62.96) - open/close 3324 (time 1709785574.81 total 53.18 last 58.85) - open/close 3837 (time 1709785585.28 total 63.65 last 48.99) - open/close 4269 (time 1709785595.63 total 73.99 last 41.76) - open/close 4577 (time 1709785605.73 total 84.10 last 30.48) - open/close 4939 (time 1709785615.96 total 94.33 last 35.38) - open/close 5463 (time 1709785626.77 total 105.13 last 48.50) - open/close 5839 (time 1709785637.11 total 115.48 last 36.35) - open/close 6350 (time 1709785647.95 total 126.32 last 47.14) - open/close 6824 (time 1709785658.63 total 137.00 last 44.39) - open/close 7249 (time 1709785669.25 total 147.61 last 40.03) - open/close 7697 (time 1709785679.25 total 157.61 last 44.79) - open/close 8076 (time 1709785690.00 total 168.36 last 35.26) - open/close 8417 (time 1709785701.68 total 180.05 last 29.18) - open/close 8738 (time 1709785712.20 total 190.56 last 30.53) - open/close 8954 (time 1709785723.29 total 201.65 last 19.48) - open/close 9367 (time 1709785733.46 total 211.83 last 40.60) - open/close 9798 (time 1709785743.88 total 222.25 last 41.36) - open/close 10000 (time 1709785747.99 total 226.36 last 49.15) - open/close 10363 (time 1709785758.19 total 236.56 last 35.58) - open/close 10976 (time 1709785768.20 total 246.57 last 61.27) - open/close 12563 (time 1709785778.21 total 256.57 last 158.60) - open/close 13988 (time 1709785788.21 total 266.57 last 142.46) - open/close 15353 (time 1709785798.21 total 276.58 last 136.42) - open/close 16613 (time 1709785808.22 total 286.58 last 125.94) - open/close 17799 (time 1709785818.23 total 296.59 last 118.54) - open/close 18944 (time 1709785828.23 total 306.59 last 114.46) - open/close 19943 (time 1709785838.23 total 316.60 last 99.89) - open/close 20994 (time 1709785848.24 total 326.60 last 105.03) - open/close 21958 (time 1709785858.24 total 336.61 last 96.32) - open/close 22919 (time 1709785868.25 total 346.62 last 96.01) - open/close 23863 (time 1709785878.26 total 356.62 last 94.38) - open/close 24812 (time 1709785888.26 total 366.63 last 94.86) - open/close 25725 (time 1709785898.27 total 376.64 last 91.20) - open/close 26547 (time 1709785908.28 total 386.65 last 82.10) - open/close 27342 (time 1709785918.29 total 396.65 last 79.47) - open/close 28067 (time 1709785928.30 total 406.66 last 72.44) - open/close 28803 (time 1709785938.30 total 416.67 last 73.56) - open/close 29479 (time 1709785948.31 total 426.67 last 67.55) - open/close 30000 (time 1709785955.17 total 433.54 last 75.88) - open/close 30661 (time 1709785965.18 total 443.55 last 66.06) - open/close 31219 (time 1709785975.19 total 453.56 last 55.74) - open/close 31866 (time 1709785985.20 total 463.57 last 64.61) - open/close 32416 (time 1709785995.22 total 473.58 last 54.92) - open/close 32996 (time 1709786005.23 total 483.60 last 57.93) - open/close 33612 (time 1709786015.24 total 493.61 last 61.52) - open/close 34193 (time 1709786025.26 total 503.63 last 58.00) - open/close 34767 (time 1709786035.27 total 513.63 last 57.37) - open/close 35359 (time 1709786045.27 total 523.63 last 59.19) - open/close 35940 (time 1709786055.28 total 533.64 last 58.04) - open/close 36505 (time 1709786065.28 total 543.64 last 56.49) - open/close 37034 (time 1709786075.29 total 553.66 last 52.83) - open/close 37556 (time 1709786085.30 total 563.66 last 52.18) - open/close 38093 (time 1709786095.31 total 573.68 last 53.62) - open/close 38607 (time 1709786105.32 total 583.68 last 51.37) - open/close 39108 (time 1709786115.32 total 593.68 last 50.09) - open/close 39563 (time 1709786125.32 total 603.68 last 45.50) - open/close 40000 (time 1709786134.09 total 612.46 last 49.82) - open/close 40478 (time 1709786144.10 total 622.47 last 47.75) - open/close 40970 (time 1709786154.12 total 632.48 last 49.13) - open/close 41442 (time 1709786164.12 total 642.48 last 47.19) - open/close 41939 (time 1709786174.13 total 652.50 last 49.62) - open/close 42431 (time 1709786184.14 total 662.51 last 49.17) - open/close 42924 (time 1709786194.15 total 672.52 last 49.25) - open/close 43408 (time 1709786204.17 total 682.53 last 48.32) - open/close 43882 (time 1709786214.17 total 692.54 last 47.38) - open/close 44333 (time 1709786224.19 total 702.55 last 45.02) - open/close 44801 (time 1709786234.20 total 712.56 last 46.76) - open/close 45239 (time 1709786244.21 total 722.58 last 43.73) - open/close 45691 (time 1709786254.23 total 732.60 last 45.11) - open/close 46134 (time 1709786264.24 total 742.60 last 44.29) - open/close 46624 (time 1709786274.24 total 752.61 last 48.96) - open/close 47090 (time 1709786284.26 total 762.63 last 46.52) - open/close 47429 (time 1709786294.28 total 772.64 last 33.84) - open/close 47781 (time 1709786304.30 total 782.66 last 35.14) - open/close 48115 (time 1709786314.30 total 792.67 last 33.38) - open/close 48507 (time 1709786324.31 total 802.67 last 39.17) - open/close 48822 (time 1709786334.31 total 812.68 last 31.49) - open/close 49158 (time 1709786344.33 total 822.69 last 33.54) - open/close 49509 (time 1709786354.35 total 832.72 last 35.01) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 835.49 seconds: 59.37 ops/second - unlinked 0 (time 1709786358 ; total 0 ; last 0) - unlinked 10000 (time 1709786673 ; total 315 ; last 315) - unlinked 20000 (time 1709786980 ; total 622 ; last 307) - unlinked 30000 (time 1709787238 ; total 880 ; last 258) - unlinked 40000 (time 1709787406 ; total 1048 ; last 168) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1138 seconds: 43.585236 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg254-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg254-server unloading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2