== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:24:18 (1709785458) start mds service on oleg125-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg125-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg125-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg125-client.virtnet: -o user_xattr,flock oleg125-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 770 (time 1709785487.72 total 10.53 last 73.13) - open/close 1413 (time 1709785498.23 total 21.04 last 61.14) - open/close 2023 (time 1709785508.94 total 31.75 last 56.96) - open/close 2640 (time 1709785519.40 total 42.21 last 59.01) - open/close 3260 (time 1709785530.03 total 52.84 last 58.32) - open/close 3771 (time 1709785540.42 total 63.24 last 49.16) - open/close 4295 (time 1709785550.88 total 73.69 last 50.13) - open/close 4850 (time 1709785561.39 total 84.20 last 52.80) - open/close 5447 (time 1709785572.17 total 94.99 last 55.36) - open/close 5915 (time 1709785582.53 total 105.35 last 45.16) - open/close 6423 (time 1709785593.06 total 115.87 last 48.27) - open/close 6904 (time 1709785603.89 total 126.70 last 44.41) - open/close 7412 (time 1709785614.72 total 137.53 last 46.93) - open/close 7753 (time 1709785624.94 total 147.76 last 33.34) - open/close 8165 (time 1709785635.58 total 158.39 last 38.75) - open/close 8661 (time 1709785646.55 total 169.36 last 45.20) - open/close 9252 (time 1709785657.28 total 180.10 last 55.05) - open/close 9752 (time 1709785667.46 total 190.27 last 49.15) - open/close 10000 (time 1709785670.66 total 193.47 last 77.53) - open/close 10358 (time 1709785681.47 total 204.28 last 33.10) - open/close 11010 (time 1709785691.47 total 214.29 last 65.18) - open/close 11381 (time 1709785701.97 total 224.79 last 35.33) - open/close 11769 (time 1709785711.98 total 234.79 last 38.79) - open/close 12775 (time 1709785721.98 total 244.79 last 100.56) - open/close 14282 (time 1709785731.98 total 254.80 last 150.64) - open/close 15901 (time 1709785741.98 total 264.80 last 161.89) - open/close 17238 (time 1709785751.99 total 274.80 last 133.61) - open/close 18494 (time 1709785761.99 total 284.81 last 125.59) - open/close 19625 (time 1709785772.00 total 294.81 last 113.05) - open/close 20000 (time 1709785775.24 total 298.06 last 115.46) - open/close 21139 (time 1709785785.25 total 308.06 last 113.87) - open/close 22177 (time 1709785795.25 total 318.06 last 103.80) - open/close 23167 (time 1709785805.25 total 328.07 last 98.92) - open/close 24098 (time 1709785815.26 total 338.07 last 93.08) - open/close 24961 (time 1709785825.26 total 348.07 last 86.29) - open/close 25772 (time 1709785835.27 total 358.08 last 81.03) - open/close 26516 (time 1709785845.28 total 368.09 last 74.32) - open/close 27270 (time 1709785855.29 total 378.10 last 75.34) - open/close 27970 (time 1709785865.30 total 388.11 last 69.92) - open/close 28699 (time 1709785875.30 total 398.12 last 72.85) - open/close 29391 (time 1709785885.31 total 408.12 last 69.18) - open/close 30000 (time 1709785894.38 total 417.19 last 67.13) - open/close 30604 (time 1709785904.39 total 427.21 last 60.31) - open/close 31193 (time 1709785914.40 total 437.21 last 58.85) - open/close 31834 (time 1709785924.40 total 447.22 last 64.08) - open/close 32408 (time 1709785934.42 total 457.23 last 57.33) - open/close 33004 (time 1709785944.43 total 467.24 last 59.53) - open/close 33595 (time 1709785954.44 total 477.25 last 59.05) - open/close 34150 (time 1709785964.44 total 487.26 last 55.47) - open/close 34703 (time 1709785974.45 total 497.27 last 55.24) - open/close 35232 (time 1709785984.46 total 507.27 last 52.87) - open/close 35731 (time 1709785994.47 total 517.28 last 49.85) - open/close 36232 (time 1709786004.49 total 527.30 last 50.01) - open/close 36724 (time 1709786014.49 total 537.31 last 49.17) - open/close 37205 (time 1709786024.51 total 547.33 last 48.02) - open/close 37681 (time 1709786034.52 total 557.34 last 47.54) - open/close 38123 (time 1709786044.54 total 567.35 last 44.15) - open/close 38592 (time 1709786054.54 total 577.35 last 46.88) - open/close 39056 (time 1709786064.55 total 587.36 last 46.36) - open/close 39550 (time 1709786074.56 total 597.37 last 49.35) - open/close 39975 (time 1709786084.57 total 607.38 last 42.46) - open/close 40449 (time 1709786094.58 total 617.40 last 47.33) - open/close 40919 (time 1709786104.60 total 627.42 last 46.90) - open/close 41351 (time 1709786114.62 total 637.43 last 43.15) - open/close 41803 (time 1709786124.64 total 647.45 last 45.11) - open/close 42256 (time 1709786134.65 total 657.46 last 45.25) - open/close 42711 (time 1709786144.67 total 667.48 last 45.40) - open/close 43158 (time 1709786154.69 total 677.50 last 44.62) - open/close 43590 (time 1709786164.69 total 687.50 last 43.19) - open/close 44043 (time 1709786174.70 total 697.52 last 45.24) - open/close 44452 (time 1709786184.72 total 707.53 last 40.85) - open/close 44862 (time 1709786194.74 total 717.55 last 40.91) - open/close 45253 (time 1709786204.75 total 727.57 last 39.03) - open/close 45671 (time 1709786214.76 total 737.58 last 41.77) - open/close 46090 (time 1709786224.78 total 747.59 last 41.82) - open/close 46528 (time 1709786234.79 total 757.61 last 43.75) - open/close 46964 (time 1709786244.80 total 767.62 last 43.56) - open/close 47421 (time 1709786254.82 total 777.64 last 45.61) - open/close 47866 (time 1709786264.83 total 787.64 last 44.46) - open/close 48278 (time 1709786274.84 total 797.65 last 41.18) - open/close 48733 (time 1709786284.86 total 807.67 last 45.41) - open/close 49153 (time 1709786294.88 total 817.69 last 41.90) - open/close 49537 (time 1709786304.89 total 827.71 last 38.34) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 829.98 seconds: 59.76 ops/second - unlinked 0 (time 1709786308 ; total 0 ; last 0) - unlinked 10000 (time 1709786592 ; total 284 ; last 284) - unlinked 20000 (time 1709786887 ; total 579 ; last 295) - unlinked 30000 (time 1709787151 ; total 843 ; last 264) - unlinked 40000 (time 1709787324 ; total 1016 ; last 173) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1110 seconds: 44.684685 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg125-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg125-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg125-server unloading modules on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg125-client: oleg125-client: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-client: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2