== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:28:05 (1709785685) start mds service on oleg249-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg249-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg249-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-OST0000 oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg249-client.virtnet: -o user_xattr,flock oleg249-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 624 (time 1709785715.69 total 10.35 last 60.28) - open/close 1122 (time 1709785725.80 total 20.46 last 49.25) - open/close 1835 (time 1709785736.44 total 31.10 last 67.04) - open/close 2355 (time 1709785746.87 total 41.53 last 49.84) - open/close 2958 (time 1709785757.42 total 52.08 last 57.19) - open/close 3638 (time 1709785768.21 total 62.88 last 62.97) - open/close 4252 (time 1709785779.16 total 73.82 last 56.09) - open/close 4792 (time 1709785790.04 total 84.70 last 49.64) - open/close 5299 (time 1709785800.70 total 95.36 last 47.56) - open/close 5793 (time 1709785811.43 total 106.09 last 46.02) - open/close 6283 (time 1709785822.40 total 117.06 last 44.69) - open/close 6626 (time 1709785832.85 total 127.52 last 32.80) - open/close 6963 (time 1709785843.21 total 137.87 last 32.54) - open/close 7330 (time 1709785853.79 total 148.45 last 34.68) - open/close 7849 (time 1709785864.14 total 158.81 last 50.13) - open/close 8272 (time 1709785874.94 total 169.60 last 39.19) - open/close 8563 (time 1709785885.21 total 179.87 last 28.32) - open/close 9112 (time 1709785895.89 total 190.55 last 51.42) - open/close 9516 (time 1709785905.89 total 200.56 last 40.38) - open/close 9954 (time 1709785916.15 total 210.82 last 42.69) - open/close 10301 (time 1709785926.23 total 220.89 last 34.43) - open/close 10648 (time 1709785937.09 total 231.75 last 31.96) - open/close 10826 (time 1709785949.19 total 243.85 last 14.72) - open/close 12619 (time 1709785959.19 total 253.85 last 179.22) - open/close 14214 (time 1709785969.19 total 263.85 last 159.48) - open/close 15573 (time 1709785979.19 total 273.86 last 135.89) - open/close 16828 (time 1709785989.19 total 283.86 last 125.48) - open/close 17849 (time 1709785999.20 total 293.86 last 102.08) - open/close 19058 (time 1709786009.20 total 303.86 last 120.86) - open/close 20000 (time 1709786017.62 total 312.29 last 111.84) - open/close 21081 (time 1709786027.63 total 322.29 last 108.06) - open/close 22086 (time 1709786037.63 total 332.29 last 100.45) - open/close 23125 (time 1709786047.64 total 342.30 last 103.86) - open/close 24060 (time 1709786057.64 total 352.30 last 93.44) - open/close 24971 (time 1709786067.65 total 362.31 last 91.00) - open/close 25769 (time 1709786077.66 total 372.32 last 79.72) - open/close 26591 (time 1709786087.67 total 382.33 last 82.17) - open/close 27378 (time 1709786097.67 total 392.33 last 78.67) - open/close 28126 (time 1709786107.67 total 402.34 last 74.78) - open/close 28900 (time 1709786117.68 total 412.34 last 77.38) - open/close 29656 (time 1709786127.69 total 422.35 last 75.51) - open/close 30000 (time 1709786133.07 total 427.73 last 63.89) - open/close 30708 (time 1709786143.08 total 437.74 last 70.77) - open/close 31420 (time 1709786153.08 total 447.74 last 71.17) - open/close 32050 (time 1709786163.09 total 457.75 last 62.94) - open/close 32726 (time 1709786173.10 total 467.76 last 67.52) - open/close 33357 (time 1709786183.10 total 477.76 last 63.10) - open/close 33980 (time 1709786193.11 total 487.78 last 62.22) - open/close 34641 (time 1709786203.13 total 497.79 last 66.03) - open/close 35213 (time 1709786213.14 total 507.80 last 57.13) - open/close 35763 (time 1709786223.16 total 517.82 last 54.90) - open/close 36341 (time 1709786233.17 total 527.83 last 57.72) - open/close 36906 (time 1709786243.17 total 537.84 last 56.48) - open/close 37414 (time 1709786253.18 total 547.85 last 50.74) - open/close 37988 (time 1709786263.19 total 557.85 last 57.36) - open/close 38594 (time 1709786273.20 total 567.87 last 60.52) - open/close 39151 (time 1709786283.22 total 577.88 last 55.62) - open/close 39548 (time 1709786293.25 total 587.92 last 39.56) - open/close 39952 (time 1709786303.27 total 597.93 last 40.34) - open/close 40377 (time 1709786313.27 total 607.93 last 42.49) - open/close 40824 (time 1709786323.28 total 617.94 last 44.68) - open/close 41207 (time 1709786333.29 total 627.95 last 38.25) - open/close 41614 (time 1709786343.30 total 637.96 last 40.67) - open/close 42042 (time 1709786353.31 total 647.97 last 42.73) - open/close 42421 (time 1709786363.33 total 657.99 last 37.84) - open/close 42849 (time 1709786373.34 total 668.00 last 42.76) - open/close 43259 (time 1709786383.35 total 678.01 last 40.96) - open/close 43576 (time 1709786393.37 total 688.04 last 31.62) - open/close 43945 (time 1709786403.38 total 698.05 last 36.86) - open/close 44377 (time 1709786413.40 total 708.06 last 43.13) - open/close 44826 (time 1709786423.40 total 718.07 last 44.89) - open/close 45234 (time 1709786433.41 total 728.07 last 40.79) - open/close 45637 (time 1709786443.42 total 738.09 last 40.23) - open/close 45993 (time 1709786453.44 total 748.10 last 35.55) - open/close 46432 (time 1709786463.44 total 758.10 last 43.89) - open/close 46906 (time 1709786473.44 total 768.11 last 47.37) - open/close 47301 (time 1709786483.47 total 778.13 last 39.40) - open/close 47722 (time 1709786493.48 total 788.15 last 42.04) - open/close 48088 (time 1709786503.49 total 798.15 last 36.59) - open/close 48451 (time 1709786513.50 total 808.16 last 36.25) - open/close 48818 (time 1709786523.52 total 818.18 last 36.63) - open/close 49189 (time 1709786533.54 total 828.20 last 37.05) - open/close 49562 (time 1709786543.54 total 838.20 last 37.28) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 839.37 seconds: 59.09 ops/second - unlinked 0 (time 1709786545 ; total 0 ; last 0) - unlinked 10000 (time 1709786952 ; total 407 ; last 407) - unlinked 20000 (time 1709787358 ; total 813 ; last 406) - unlinked 30000 (time 1709787676 ; total 1131 ; last 318) - unlinked 40000 (time 1709787856 ; total 1311 ; last 180) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1405 seconds: 35.302490 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg249-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg249-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg249-server stop mds service on oleg249-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg249-server stop mds service on oleg249-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg249-server unloading modules on: 'oleg249-server' oleg249-server: oleg249-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg249-client: oleg249-client: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-server: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-client: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-server: ssh exited with exit code 2