== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:21:01 (1709785261) start mds service on oleg307-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg307-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg307-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Started lustre-OST0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg307-client.virtnet: -o user_xattr,flock oleg307-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 246 (time 1709785290.15 total 10.64 last 23.12) - open/close 756 (time 1709785300.35 total 20.84 last 49.99) - open/close 1520 (time 1709785311.17 total 31.66 last 70.61) - open/close 2177 (time 1709785321.86 total 42.35 last 61.46) - open/close 2854 (time 1709785332.80 total 53.29 last 61.91) - open/close 3471 (time 1709785343.57 total 64.06 last 57.28) - open/close 4034 (time 1709785354.35 total 74.84 last 52.22) - open/close 4594 (time 1709785365.16 total 85.65 last 51.80) - open/close 5167 (time 1709785375.16 total 95.65 last 57.28) - open/close 5512 (time 1709785385.55 total 106.05 last 33.19) - open/close 5954 (time 1709785396.41 total 116.90 last 40.72) - open/close 6166 (time 1709785406.41 total 126.90 last 21.20) - open/close 6711 (time 1709785417.24 total 137.74 last 50.31) - open/close 7182 (time 1709785429.56 total 150.05 last 38.24) - open/close 7671 (time 1709785440.06 total 160.56 last 46.55) - open/close 8085 (time 1709785450.17 total 170.66 last 40.96) - open/close 8636 (time 1709785460.17 total 180.67 last 55.09) - open/close 9103 (time 1709785470.81 total 191.30 last 43.90) - open/close 9456 (time 1709785480.84 total 201.33 last 35.19) - open/close 9875 (time 1709785491.48 total 211.97 last 39.38) - open/close 10000 (time 1709785497.35 total 217.84 last 21.30) - open/close 10204 (time 1709785507.78 total 228.27 last 19.56) - open/close 10440 (time 1709785518.39 total 238.88 last 22.23) - open/close 11085 (time 1709785528.39 total 248.89 last 64.50) - open/close 12554 (time 1709785538.40 total 258.89 last 146.83) - open/close 13818 (time 1709785548.40 total 268.89 last 126.35) - open/close 15090 (time 1709785558.41 total 278.90 last 127.17) - open/close 16363 (time 1709785568.41 total 288.90 last 127.24) - open/close 17657 (time 1709785578.41 total 298.90 last 129.40) - open/close 18898 (time 1709785588.42 total 308.91 last 124.00) - open/close 20000 (time 1709785598.07 total 318.57 last 114.12) - open/close 21074 (time 1709785608.08 total 328.57 last 107.37) - open/close 22049 (time 1709785618.08 total 338.57 last 97.45) - open/close 22976 (time 1709785628.09 total 348.58 last 92.67) - open/close 23901 (time 1709785638.10 total 358.59 last 92.40) - open/close 24768 (time 1709785648.10 total 368.59 last 86.68) - open/close 25620 (time 1709785658.11 total 378.60 last 85.09) - open/close 26495 (time 1709785668.11 total 388.60 last 87.48) - open/close 27366 (time 1709785678.12 total 398.61 last 87.02) - open/close 28096 (time 1709785688.13 total 408.62 last 72.98) - open/close 28851 (time 1709785698.14 total 418.63 last 75.42) - open/close 29475 (time 1709785708.15 total 428.64 last 62.31) - open/close 30000 (time 1709785716.13 total 436.62 last 65.84) - open/close 30720 (time 1709785726.14 total 446.63 last 71.92) - open/close 31442 (time 1709785736.15 total 456.64 last 72.12) - open/close 32098 (time 1709785746.16 total 466.65 last 65.54) - open/close 32768 (time 1709785756.17 total 476.66 last 66.91) - open/close 33409 (time 1709785766.17 total 486.66 last 64.08) - open/close 34082 (time 1709785776.17 total 496.67 last 67.28) - open/close 34787 (time 1709785786.18 total 506.67 last 70.45) - open/close 35421 (time 1709785796.18 total 516.68 last 63.39) - open/close 36027 (time 1709785806.19 total 526.68 last 60.55) - open/close 36624 (time 1709785816.19 total 536.68 last 59.70) - open/close 37206 (time 1709785826.20 total 546.69 last 58.19) - open/close 37825 (time 1709785836.21 total 556.70 last 61.83) - open/close 38437 (time 1709785846.22 total 566.71 last 61.13) - open/close 39019 (time 1709785856.22 total 576.71 last 58.20) - open/close 39623 (time 1709785866.23 total 586.72 last 60.32) - open/close 40000 (time 1709785872.91 total 593.40 last 56.46) - open/close 40604 (time 1709785882.92 total 603.41 last 60.36) - open/close 41188 (time 1709785892.92 total 613.42 last 58.35) - open/close 41724 (time 1709785902.93 total 623.42 last 53.55) - open/close 42203 (time 1709785912.95 total 633.44 last 47.83) - open/close 42723 (time 1709785922.95 total 643.44 last 51.99) - open/close 43213 (time 1709785932.96 total 653.45 last 48.96) - open/close 43730 (time 1709785942.96 total 663.46 last 51.67) - open/close 44230 (time 1709785952.97 total 673.46 last 49.99) - open/close 44742 (time 1709785962.97 total 683.47 last 51.15) - open/close 45244 (time 1709785972.99 total 693.48 last 50.13) - open/close 45733 (time 1709785983.00 total 703.49 last 48.87) - open/close 46212 (time 1709785993.01 total 713.51 last 47.81) - open/close 46698 (time 1709786003.02 total 723.52 last 48.55) - open/close 47182 (time 1709786013.03 total 733.52 last 48.38) - open/close 47648 (time 1709786023.04 total 743.53 last 46.55) - open/close 48097 (time 1709786033.04 total 753.53 last 44.88) - open/close 48564 (time 1709786043.06 total 763.55 last 46.62) - open/close 49009 (time 1709786053.07 total 773.56 last 44.47) - open/close 49459 (time 1709786063.07 total 783.56 last 44.99) - open/close 49898 (time 1709786073.07 total 793.56 last 43.89) total: 50000 open/close in 795.84 seconds: 62.83 ops/second - unlinked 0 (time 1709786076 ; total 0 ; last 0) - unlinked 10000 (time 1709786359 ; total 283 ; last 283) - unlinked 20000 (time 1709786617 ; total 541 ; last 258) - unlinked 30000 (time 1709786822 ; total 746 ; last 205) - unlinked 40000 (time 1709786922 ; total 846 ; last 100) total: 50000 unlinks in 939 seconds: 53.248135 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg307-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg307-server unloading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2