== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:19:05 (1709785145) start mds service on oleg340-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg340-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg340-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0000 oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 731 (time 1709785174.34 total 10.63 last 68.75) - open/close 1424 (time 1709785185.04 total 21.33 last 64.79) - open/close 2042 (time 1709785195.46 total 31.76 last 59.27) - open/close 2665 (time 1709785206.04 total 42.33 last 58.91) - open/close 3241 (time 1709785216.61 total 52.90 last 54.48) - open/close 3908 (time 1709785227.53 total 63.82 last 61.08) - open/close 4430 (time 1709785238.16 total 74.45 last 49.12) - open/close 4970 (time 1709785248.99 total 85.28 last 49.86) - open/close 5476 (time 1709785259.75 total 96.04 last 47.01) - open/close 5946 (time 1709785270.49 total 106.79 last 43.75) - open/close 6195 (time 1709785280.49 total 116.79 last 24.90) - open/close 6428 (time 1709785291.44 total 127.73 last 21.29) - open/close 6876 (time 1709785302.34 total 138.63 last 41.09) - open/close 7339 (time 1709785313.24 total 149.53 last 42.49) - open/close 7820 (time 1709785324.21 total 160.50 last 43.83) - open/close 8361 (time 1709785334.67 total 170.97 last 51.72) - open/close 8815 (time 1709785344.81 total 181.10 last 44.79) - open/close 9316 (time 1709785355.22 total 191.51 last 48.13) - open/close 9892 (time 1709785366.10 total 202.39 last 52.93) - open/close 10437 (time 1709785376.10 total 212.39 last 54.49) - open/close 10783 (time 1709785386.10 total 222.40 last 34.59) - open/close 10964 (time 1709785396.24 total 232.54 last 17.85) - open/close 11570 (time 1709785406.25 total 242.54 last 60.59) - open/close 13038 (time 1709785416.25 total 252.54 last 146.73) - open/close 14438 (time 1709785426.26 total 262.55 last 139.93) - open/close 15764 (time 1709785436.26 total 272.55 last 132.59) - open/close 17016 (time 1709785446.26 total 282.55 last 125.19) - open/close 18159 (time 1709785456.26 total 292.55 last 114.29) - open/close 19311 (time 1709785466.26 total 302.56 last 115.13) - open/close 20000 (time 1709785472.45 total 308.74 last 111.33) - open/close 21002 (time 1709785482.46 total 318.75 last 100.13) - open/close 21986 (time 1709785492.46 total 328.75 last 98.38) - open/close 22914 (time 1709785502.47 total 338.76 last 92.70) - open/close 23802 (time 1709785512.47 total 348.77 last 88.78) - open/close 24682 (time 1709785522.48 total 358.77 last 87.99) - open/close 25583 (time 1709785532.48 total 368.77 last 90.08) - open/close 26381 (time 1709785542.49 total 378.78 last 79.73) - open/close 27215 (time 1709785552.49 total 388.78 last 83.34) - open/close 28000 (time 1709785562.50 total 398.79 last 78.48) - open/close 28815 (time 1709785572.50 total 408.79 last 81.48) - open/close 29621 (time 1709785582.51 total 418.80 last 80.52) - open/close 30000 (time 1709785587.18 total 423.48 last 81.05) - open/close 30750 (time 1709785597.19 total 433.48 last 74.94) - open/close 31448 (time 1709785607.19 total 443.49 last 69.78) - open/close 32128 (time 1709785617.20 total 453.50 last 67.93) - open/close 32776 (time 1709785627.21 total 463.50 last 64.79) - open/close 33462 (time 1709785637.21 total 473.51 last 68.55) - open/close 34085 (time 1709785647.22 total 483.52 last 62.23) - open/close 34703 (time 1709785657.24 total 493.53 last 61.71) - open/close 35329 (time 1709785667.25 total 503.54 last 62.53) - open/close 35965 (time 1709785677.26 total 513.55 last 63.54) - open/close 36514 (time 1709785687.27 total 523.56 last 54.86) - open/close 37082 (time 1709785697.27 total 533.56 last 56.79) - open/close 37651 (time 1709785707.28 total 543.57 last 56.85) - open/close 38247 (time 1709785717.28 total 553.58 last 59.56) - open/close 38805 (time 1709785727.30 total 563.59 last 55.72) - open/close 39346 (time 1709785737.30 total 573.59 last 54.10) - open/close 39895 (time 1709785747.30 total 583.60 last 54.86) - open/close 40432 (time 1709785757.32 total 593.61 last 53.64) - open/close 40964 (time 1709785767.32 total 603.61 last 53.18) - open/close 41504 (time 1709785777.34 total 613.63 last 53.90) - open/close 42049 (time 1709785787.35 total 623.65 last 54.41) - open/close 42557 (time 1709785797.36 total 633.66 last 50.75) - open/close 43061 (time 1709785807.37 total 643.66 last 50.35) - open/close 43567 (time 1709785817.37 total 653.67 last 50.60) - open/close 44064 (time 1709785827.38 total 663.68 last 49.65) - open/close 44564 (time 1709785837.39 total 673.68 last 49.96) - open/close 45051 (time 1709785847.39 total 683.68 last 48.69) - open/close 45540 (time 1709785857.40 total 693.69 last 48.88) - open/close 46040 (time 1709785867.41 total 703.70 last 49.94) - open/close 46536 (time 1709785877.42 total 713.71 last 49.56) - open/close 47047 (time 1709785887.44 total 723.73 last 51.01) - open/close 47521 (time 1709785897.44 total 733.74 last 47.36) - open/close 47991 (time 1709785907.46 total 743.75 last 46.94) - open/close 48454 (time 1709785917.47 total 753.76 last 46.25) - open/close 48896 (time 1709785927.49 total 763.78 last 44.12) - open/close 49352 (time 1709785937.49 total 773.79 last 45.56) - open/close 49803 (time 1709785947.51 total 783.80 last 45.03) total: 50000 open/close in 788.31 seconds: 63.43 ops/second - unlinked 0 (time 1709785952 ; total 0 ; last 0) - unlinked 10000 (time 1709786227 ; total 275 ; last 275) - unlinked 20000 (time 1709786476 ; total 524 ; last 249) - unlinked 30000 (time 1709786722 ; total 770 ; last 246) - unlinked 40000 (time 1709786826 ; total 874 ; last 104) total: 50000 unlinks in 926 seconds: 53.995682 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg340-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg340-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg340-server unloading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg340-client: oleg340-client: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-client: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2