== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:27:24 (1746142044) start mds service on oleg153-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg153-server' oleg153-server: oleg153-server.virtnet: executing load_modules_local oleg153-server: Loading modules from /home/green/git/lustre-release/lustre oleg153-server: detected 4 online CPUs by sysfs oleg153-server: MODOPTS_LIBCFS= oleg153-server: Force libcfs to create 2 CPU partitions oleg153-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg153-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg153-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg153-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Started lustre-OST0000 oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 144 (time 1746142093.96 total 10.02 last 14.37) - open/close 326 (time 1746142104.76 total 20.82 last 16.86) - open/close 487 (time 1746142114.76 total 30.82 last 16.10) - open/close 611 (time 1746142124.77 total 40.83 last 12.39) - open/close 735 (time 1746142135.11 total 51.17 last 11.99) - open/close 883 (time 1746142145.15 total 61.21 last 14.74) - open/close 1063 (time 1746142155.16 total 71.22 last 17.99) - open/close 1228 (time 1746142165.38 total 81.44 last 16.15) - open/close 1419 (time 1746142175.63 total 91.69 last 18.62) - open/close 1616 (time 1746142186.55 total 102.62 last 18.04) - open/close 1772 (time 1746142196.64 total 112.70 last 15.46) - open/close 1849 (time 1746142206.84 total 122.91 last 7.55) - open/close 1959 (time 1746142217.28 total 133.34 last 10.55) - open/close 2102 (time 1746142227.33 total 143.39 last 14.22) - open/close 2230 (time 1746142238.04 total 154.10 last 11.96) - open/close 2338 (time 1746142248.45 total 164.51 last 10.37) - open/close 2506 (time 1746142258.66 total 174.72 last 16.45) - open/close 2635 (time 1746142268.73 total 184.79 last 12.81) - open/close 2835 (time 1746142279.28 total 195.34 last 18.96) - open/close 3048 (time 1746142289.56 total 205.62 last 20.72) - open/close 3183 (time 1746142299.66 total 215.72 last 13.37) - open/close 3358 (time 1746142309.88 total 225.94 last 17.12) - open/close 3526 (time 1746142320.67 total 236.73 last 15.57) - open/close 3705 (time 1746142330.88 total 246.94 last 17.54) - open/close 3938 (time 1746142341.47 total 257.53 last 22.00) - open/close 4084 (time 1746142354.44 total 270.50 last 11.26) - open/close 4322 (time 1746142364.82 total 280.88 last 22.92) - open/close 4490 (time 1746142374.94 total 291.00 last 16.60) - open/close 4638 (time 1746142385.12 total 301.18 last 14.55) - open/close 4783 (time 1746142395.12 total 311.18 last 14.49) - open/close 4863 (time 1746142405.37 total 321.43 last 7.81) - open/close 4982 (time 1746142416.05 total 332.11 last 11.14) - open/close 5131 (time 1746142426.42 total 342.48 last 14.37) - open/close 5336 (time 1746142436.43 total 352.49 last 20.49) - open/close 5489 (time 1746142446.65 total 362.71 last 14.97) - open/close 5685 (time 1746142456.82 total 372.88 last 19.26) - open/close 5930 (time 1746142467.17 total 383.23 last 23.68) - open/close 6128 (time 1746142477.40 total 393.46 last 19.36) - open/close 6272 (time 1746142487.40 total 403.46 last 14.39) - open/close 6376 (time 1746142498.08 total 414.14 last 9.74) - open/close 6586 (time 1746142508.49 total 424.55 last 20.18) - open/close 6720 (time 1746142518.65 total 434.71 last 13.19) - open/close 6782 (time 1746142528.65 total 444.71 last 6.20) - open/close 6953 (time 1746142538.65 total 454.72 last 17.10) - open/close 7083 (time 1746142549.58 total 465.64 last 11.90) - open/close 7194 (time 1746142560.12 total 476.18 last 10.53) - open/close 7399 (time 1746142570.19 total 486.25 last 20.35) - open/close 7547 (time 1746142580.51 total 496.57 last 14.34) - open/close 7688 (time 1746142590.67 total 506.73 last 13.87) - open/close 7838 (time 1746142600.95 total 517.01 last 14.59) - open/close 7960 (time 1746142610.95 total 527.01 last 12.20) - open/close 8141 (time 1746142621.58 total 537.64 last 17.03) - open/close 8393 (time 1746142632.39 total 548.45 last 23.32) - open/close 8563 (time 1746142642.55 total 558.61 last 16.72) - open/close 8695 (time 1746142652.56 total 568.62 last 13.19) - open/close 8822 (time 1746142663.26 total 579.32 last 11.86) - open/close 8952 (time 1746142674.10 total 590.16 last 12.00) - open/close 9112 (time 1746142684.10 total 600.16 last 16.00) - open/close 10000 (time 1746142693.18 total 609.24 last 97.81) - open/close 10930 (time 1746142703.19 total 619.25 last 92.88) - open/close 11865 (time 1746142713.20 total 629.26 last 93.45) - open/close 12743 (time 1746142723.21 total 639.27 last 87.71) - open/close 13600 (time 1746142733.21 total 649.27 last 85.69) - open/close 14654 (time 1746142743.21 total 659.27 last 105.38) - open/close 15784 (time 1746142753.21 total 669.27 last 112.98) - open/close 16692 (time 1746142763.23 total 679.29 last 90.66) - open/close 17651 (time 1746142773.24 total 689.30 last 95.82) - open/close 18508 (time 1746142783.24 total 699.30 last 85.68) - open/close 19395 (time 1746142793.24 total 709.31 last 88.66) - open/close 20000 (time 1746142799.59 total 715.65 last 95.38) - open/close 20913 (time 1746142809.59 total 725.65 last 91.30) - open/close 21790 (time 1746142819.59 total 735.65 last 87.68) - open/close 22565 (time 1746142829.59 total 745.65 last 77.50) - open/close 23423 (time 1746142839.59 total 755.65 last 85.78) - open/close 24327 (time 1746142849.60 total 765.66 last 90.37) - open/close 25195 (time 1746142859.62 total 775.68 last 86.64) - open/close 26025 (time 1746142869.62 total 785.68 last 82.96) - open/close 26766 (time 1746142879.63 total 795.69 last 74.06) - open/close 27568 (time 1746142889.64 total 805.70 last 80.08) - open/close 28534 (time 1746142899.64 total 815.71 last 96.57) - open/close 29381 (time 1746142909.65 total 825.71 last 84.66) - open/close 30000 (time 1746142916.28 total 832.34 last 93.31) - open/close 30864 (time 1746142926.28 total 842.35 last 86.38) - open/close 31727 (time 1746142936.29 total 852.35 last 86.26) - open/close 32590 (time 1746142946.30 total 862.36 last 86.18) - open/close 33408 (time 1746142956.31 total 872.37 last 81.79) - open/close 34307 (time 1746142966.31 total 882.37 last 89.85) - open/close 35197 (time 1746142976.32 total 892.38 last 88.94) - open/close 36165 (time 1746142986.33 total 902.39 last 96.71) - open/close 37046 (time 1746142996.33 total 912.39 last 88.09) - open/close 37992 (time 1746143006.33 total 922.39 last 94.57) - open/close 38915 (time 1746143016.34 total 932.40 last 92.21) - open/close 39992 (time 1746143026.34 total 942.41 last 107.67) - open/close 40849 (time 1746143036.35 total 952.41 last 85.68) - open/close 41714 (time 1746143046.36 total 962.42 last 86.36) - open/close 42670 (time 1746143056.37 total 972.43 last 95.55) - open/close 43616 (time 1746143066.38 total 982.44 last 94.48) - open/close 44658 (time 1746143076.39 total 992.45 last 104.14) - open/close 45688 (time 1746143086.39 total 1002.45 last 102.97) - open/close 46576 (time 1746143096.40 total 1012.46 last 88.67) - open/close 47416 (time 1746143106.42 total 1022.48 last 83.89) - open/close 48288 (time 1746143116.42 total 1032.48 last 87.16) - open/close 49136 (time 1746143126.44 total 1042.50 last 84.65) - open/close 49975 (time 1746143136.44 total 1052.50 last 83.90) total: 50000 open/close in 1052.71 seconds: 47.50 ops/second - unlinked 0 (time 1746143138 ; total 0 ; last 0) - unlinked 10000 (time 1746143232 ; total 94 ; last 94) - unlinked 20000 (time 1746143287 ; total 149 ; last 55) - unlinked 30000 (time 1746143355 ; total 217 ; last 68) - unlinked 40000 (time 1746143417 ; total 279 ; last 62) total: 50000 unlinks in 338 seconds: 147.929001 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg153-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg153-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg153-server stop mds service on oleg153-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg153-server unloading modules via unload_modules_local on: 'oleg153-server' oleg153-server: oleg153-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg153-client: oleg153-client: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-server: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-client: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-server: ssh exited with exit code 2