== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:24:10 (1709785450) start mds service on oleg226-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg226-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg226-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg226-client.virtnet: -o user_xattr,flock oleg226-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 878 (time 1709785480.46 total 10.91 last 80.50) - open/close 1516 (time 1709785491.97 total 22.42 last 55.43) - open/close 2021 (time 1709785502.38 total 32.83 last 48.52) - open/close 2489 (time 1709785512.58 total 43.03 last 45.88) - open/close 2953 (time 1709785524.10 total 54.54 last 40.28) - open/close 3576 (time 1709785534.69 total 65.13 last 58.84) - open/close 4194 (time 1709785545.68 total 76.12 last 56.24) - open/close 4801 (time 1709785556.36 total 86.81 last 56.80) - open/close 5329 (time 1709785566.95 total 97.39 last 49.87) - open/close 5842 (time 1709785577.43 total 107.87 last 48.96) - open/close 6412 (time 1709785588.39 total 118.83 last 52.01) - open/close 6815 (time 1709785600.86 total 131.30 last 32.31) - open/close 7202 (time 1709785613.41 total 143.86 last 30.83) - open/close 7560 (time 1709785623.80 total 154.24 last 34.48) - open/close 7795 (time 1709785634.80 total 165.24 last 21.37) - open/close 8192 (time 1709785645.36 total 175.80 last 37.60) - open/close 8582 (time 1709785656.06 total 186.50 last 36.44) - open/close 8968 (time 1709785666.64 total 197.08 last 36.48) - open/close 9416 (time 1709785676.64 total 207.08 last 44.79) - open/close 9787 (time 1709785687.53 total 217.98 last 34.06) - open/close 10000 (time 1709785689.58 total 220.02 last 104.14) - open/close 10338 (time 1709785699.60 total 230.04 last 33.74) - open/close 10724 (time 1709785709.73 total 240.17 last 38.10) - open/close 12180 (time 1709785719.73 total 250.17 last 145.57) - open/close 13654 (time 1709785729.73 total 260.18 last 147.33) - open/close 14961 (time 1709785739.74 total 270.18 last 130.68) - open/close 16255 (time 1709785749.74 total 280.18 last 129.36) - open/close 17523 (time 1709785759.75 total 290.19 last 126.67) - open/close 18686 (time 1709785769.75 total 300.19 last 116.30) - open/close 19755 (time 1709785779.76 total 310.20 last 106.81) - open/close 20000 (time 1709785782.23 total 312.67 last 99.27) - open/close 21105 (time 1709785792.23 total 322.67 last 110.49) - open/close 22097 (time 1709785802.23 total 332.67 last 99.18) - open/close 23022 (time 1709785812.23 total 342.68 last 92.45) - open/close 23885 (time 1709785822.24 total 352.68 last 86.26) - open/close 24678 (time 1709785832.24 total 362.69 last 79.25) - open/close 25484 (time 1709785842.25 total 372.70 last 80.53) - open/close 26298 (time 1709785852.26 total 382.70 last 81.36) - open/close 27107 (time 1709785862.27 total 392.72 last 80.76) - open/close 27914 (time 1709785872.28 total 402.72 last 80.68) - open/close 28742 (time 1709785882.28 total 412.72 last 82.77) - open/close 29466 (time 1709785892.30 total 422.74 last 72.28) - open/close 30000 (time 1709785899.98 total 430.42 last 69.50) - open/close 30659 (time 1709785909.98 total 440.43 last 65.88) - open/close 31341 (time 1709785920.00 total 450.45 last 68.05) - open/close 32056 (time 1709785930.01 total 460.45 last 71.50) - open/close 32775 (time 1709785940.01 total 470.46 last 71.83) - open/close 33444 (time 1709785950.03 total 480.47 last 66.82) - open/close 34127 (time 1709785960.03 total 490.47 last 68.27) - open/close 34818 (time 1709785970.05 total 500.49 last 69.00) - open/close 35448 (time 1709785980.05 total 510.50 last 62.97) - open/close 36096 (time 1709785990.05 total 520.50 last 64.79) - open/close 36686 (time 1709786000.06 total 530.50 last 58.95) - open/close 37286 (time 1709786010.06 total 540.51 last 59.98) - open/close 37834 (time 1709786020.07 total 550.52 last 54.75) - open/close 38409 (time 1709786030.08 total 560.53 last 57.45) - open/close 38943 (time 1709786040.09 total 570.53 last 53.36) - open/close 39538 (time 1709786050.10 total 580.54 last 59.46) - open/close 40000 (time 1709786058.44 total 588.88 last 55.40) - open/close 40520 (time 1709786068.44 total 598.88 last 51.98) - open/close 41033 (time 1709786078.45 total 608.89 last 51.24) - open/close 41537 (time 1709786088.47 total 618.91 last 50.32) - open/close 41987 (time 1709786098.48 total 628.92 last 44.94) - open/close 42398 (time 1709786108.50 total 638.95 last 41.00) - open/close 42784 (time 1709786118.53 total 648.97 last 38.50) - open/close 43162 (time 1709786128.53 total 658.98 last 37.78) - open/close 43521 (time 1709786138.55 total 668.99 last 35.85) - open/close 43870 (time 1709786148.56 total 679.01 last 34.85) - open/close 44279 (time 1709786158.57 total 689.02 last 40.85) - open/close 44711 (time 1709786168.59 total 699.04 last 43.12) - open/close 45197 (time 1709786178.61 total 709.05 last 48.52) - open/close 45689 (time 1709786188.63 total 719.07 last 49.11) - open/close 46154 (time 1709786198.65 total 729.09 last 46.41) - open/close 46599 (time 1709786208.67 total 739.11 last 44.41) - open/close 46963 (time 1709786218.68 total 749.12 last 36.36) - open/close 47410 (time 1709786228.68 total 759.13 last 44.68) - open/close 47860 (time 1709786238.69 total 769.14 last 44.95) - open/close 48272 (time 1709786248.71 total 779.15 last 41.15) - open/close 48708 (time 1709786258.73 total 789.17 last 43.49) - open/close 49170 (time 1709786268.73 total 799.18 last 46.18) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 808.96 seconds: 61.31 ops/second - unlinked 0 (time 1709786279 ; total 0 ; last 0) - unlinked 10000 (time 1709786583 ; total 304 ; last 304) - unlinked 20000 (time 1709786900 ; total 621 ; last 317) - unlinked 30000 (time 1709787186 ; total 907 ; last 286) - unlinked 40000 (time 1709787348 ; total 1069 ; last 162) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1161 seconds: 42.721790 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg226-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg226-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg226-server stop mds service on oleg226-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg226-server stop mds service on oleg226-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg226-server unloading modules on: 'oleg226-server' oleg226-server: oleg226-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg226-client: oleg226-client: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-client: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 2