== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:33:26 (1746149606) start mds service on oleg118-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing load_modules_local oleg118-server: Loading modules from /home/green/git/lustre-release/lustre oleg118-server: detected 4 online CPUs by sysfs oleg118-server: MODOPTS_LIBCFS= oleg118-server: Force libcfs to create 2 CPU partitions oleg118-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg118-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg118-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 189 (time 1746149668.54 total 10.01 last 18.88) - open/close 368 (time 1746149678.55 total 20.02 last 17.88) - open/close 544 (time 1746149688.55 total 30.02 last 17.60) - open/close 753 (time 1746149698.85 total 40.31 last 20.30) - open/close 957 (time 1746149708.86 total 50.33 last 20.37) - open/close 1175 (time 1746149718.88 total 60.35 last 21.76) - open/close 1377 (time 1746149729.15 total 70.62 last 19.67) - open/close 1568 (time 1746149739.45 total 80.92 last 18.53) - open/close 1779 (time 1746149750.07 total 91.54 last 19.86) - open/close 1954 (time 1746149760.09 total 101.55 last 17.48) - open/close 2147 (time 1746149771.07 total 112.54 last 17.57) - open/close 2332 (time 1746149781.08 total 122.55 last 18.49) - open/close 2506 (time 1746149791.10 total 132.57 last 17.36) - open/close 2687 (time 1746149801.84 total 143.31 last 16.85) - open/close 2875 (time 1746149811.84 total 153.31 last 18.79) - open/close 3063 (time 1746149821.86 total 163.33 last 18.77) - open/close 3303 (time 1746149831.87 total 173.34 last 23.97) - open/close 3505 (time 1746149841.87 total 183.34 last 20.19) - open/close 3701 (time 1746149852.71 total 194.18 last 18.08) - open/close 3999 (time 1746149863.13 total 204.60 last 28.62) - open/close 4139 (time 1746149873.13 total 214.60 last 13.99) - open/close 4343 (time 1746149883.43 total 224.90 last 19.80) - open/close 4567 (time 1746149894.23 total 235.70 last 20.75) - open/close 4747 (time 1746149904.24 total 245.71 last 17.98) - open/close 4896 (time 1746149914.54 total 256.01 last 14.47) - open/close 5109 (time 1746149924.54 total 266.01 last 21.30) - open/close 5324 (time 1746149935.00 total 276.47 last 20.55) - open/close 5524 (time 1746149945.52 total 286.99 last 19.02) - open/close 5715 (time 1746149955.54 total 297.01 last 19.05) - open/close 5923 (time 1746149965.55 total 307.02 last 20.79) - open/close 6115 (time 1746149975.56 total 317.02 last 19.18) - open/close 6302 (time 1746149985.56 total 327.03 last 18.69) - open/close 6510 (time 1746149996.27 total 337.74 last 19.43) - open/close 6754 (time 1746150006.27 total 347.74 last 24.39) - open/close 6945 (time 1746150016.44 total 357.91 last 18.78) - open/close 7101 (time 1746150026.46 total 367.93 last 15.56) - open/close 7308 (time 1746150036.46 total 377.93 last 20.70) - open/close 7487 (time 1746150046.47 total 387.93 last 17.89) - open/close 7689 (time 1746150056.48 total 397.94 last 20.18) - open/close 7884 (time 1746150066.50 total 407.97 last 19.45) - open/close 8096 (time 1746150076.87 total 418.34 last 20.45) - open/close 8303 (time 1746150087.78 total 429.25 last 18.97) - open/close 8478 (time 1746150098.32 total 439.79 last 16.60) - open/close 8674 (time 1746150109.25 total 450.72 last 17.92) - open/close 8858 (time 1746150119.26 total 460.73 last 18.39) - open/close 9061 (time 1746150130.03 total 471.50 last 18.86) - open/close 9273 (time 1746150140.74 total 482.21 last 19.79) - open/close 9482 (time 1746150151.51 total 492.98 last 19.40) - open/close 9646 (time 1746150162.15 total 503.62 last 15.42) - open/close 9843 (time 1746150172.45 total 513.92 last 19.12) - open/close 10000 (time 1746150179.95 total 521.42 last 20.95) - open/close 10169 (time 1746150189.97 total 531.44 last 16.86) - open/close 10391 (time 1746150200.71 total 542.18 last 20.68) - open/close 10591 (time 1746150211.27 total 552.74 last 18.94) - open/close 10729 (time 1746150221.33 total 562.80 last 13.72) - open/close 10879 (time 1746150231.70 total 573.17 last 14.46) - open/close 11033 (time 1746150242.24 total 583.71 last 14.62) - open/close 11216 (time 1746150253.15 total 594.62 last 16.77) - open/close 11395 (time 1746150264.09 total 605.56 last 16.36) - open/close 11589 (time 1746150274.77 total 616.23 last 18.18) - open/close 11802 (time 1746150284.77 total 626.24 last 21.28) - open/close 11973 (time 1746150294.99 total 636.46 last 16.74) - open/close 12211 (time 1746150305.72 total 647.19 last 22.17) - open/close 12485 (time 1746150315.74 total 657.21 last 27.37) - open/close 12669 (time 1746150326.03 total 667.50 last 17.88) - open/close 12848 (time 1746150336.17 total 677.64 last 17.64) - open/close 13068 (time 1746150347.13 total 688.60 last 20.09) - open/close 13273 (time 1746150357.60 total 699.07 last 19.58) - open/close 13465 (time 1746150368.34 total 709.81 last 17.87) - open/close 13637 (time 1746150379.00 total 720.47 last 16.14) - open/close 13787 (time 1746150389.01 total 730.47 last 14.99) - open/close 14662 (time 1746150399.01 total 740.48 last 87.45) - open/close 15502 (time 1746150409.02 total 750.49 last 83.90) - open/close 16381 (time 1746150419.02 total 760.49 last 87.90) - open/close 17254 (time 1746150429.02 total 770.49 last 87.29) - open/close 18202 (time 1746150439.02 total 780.49 last 94.79) - open/close 18886 (time 1746150449.05 total 790.52 last 68.23) - open/close 19665 (time 1746150459.06 total 800.53 last 77.80) - open/close 20000 (time 1746150462.50 total 803.97 last 97.48) - open/close 20798 (time 1746150472.50 total 813.97 last 79.77) - open/close 21516 (time 1746150482.51 total 823.98 last 71.75) - open/close 22249 (time 1746150492.51 total 833.98 last 73.30) - open/close 23088 (time 1746150502.51 total 843.98 last 83.87) - open/close 23912 (time 1746150512.52 total 853.99 last 82.37) - open/close 24736 (time 1746150522.53 total 864.00 last 82.28) - open/close 25608 (time 1746150532.55 total 874.01 last 87.08) - open/close 26521 (time 1746150542.55 total 884.02 last 91.28) - open/close 27502 (time 1746150552.56 total 894.02 last 98.02) - open/close 28432 (time 1746150562.56 total 904.03 last 92.96) - open/close 29371 (time 1746150572.57 total 914.03 last 93.84) - open/close 30000 (time 1746150580.47 total 921.94 last 79.53) - open/close 30989 (time 1746150590.48 total 931.94 last 98.89) - open/close 31872 (time 1746150600.48 total 941.95 last 88.28) - open/close 32751 (time 1746150610.48 total 951.95 last 87.85) - open/close 33581 (time 1746150620.48 total 961.95 last 83.00) - open/close 34495 (time 1746150630.49 total 971.96 last 91.35) - open/close 35553 (time 1746150640.49 total 981.96 last 105.79) - open/close 36426 (time 1746150650.50 total 991.97 last 87.24) - open/close 37227 (time 1746150660.50 total 1001.97 last 80.09) - open/close 38008 (time 1746150670.51 total 1011.98 last 78.02) - open/close 38797 (time 1746150680.52 total 1021.99 last 78.79) - open/close 39603 (time 1746150690.53 total 1032.00 last 80.52) - open/close 40000 (time 1746150695.59 total 1037.06 last 78.47) - open/close 40678 (time 1746150705.61 total 1047.08 last 67.69) - open/close 41436 (time 1746150715.62 total 1057.09 last 75.70) - open/close 42206 (time 1746150725.63 total 1067.10 last 76.98) - open/close 43302 (time 1746150735.63 total 1077.10 last 109.52) - open/close 44044 (time 1746150745.65 total 1087.12 last 74.10) - open/close 44864 (time 1746150755.65 total 1097.12 last 81.99) - open/close 45738 (time 1746150765.66 total 1107.13 last 87.33) - open/close 46722 (time 1746150775.66 total 1117.13 last 98.33) - open/close 47644 (time 1746150785.67 total 1127.14 last 92.18) - open/close 48549 (time 1746150795.67 total 1137.14 last 90.48) - open/close 49478 (time 1746150805.68 total 1147.14 last 92.83) total: 50000 open/close in 1151.92 seconds: 43.41 ops/second - unlinked 0 (time 1746150812 ; total 0 ; last 0) - unlinked 10000 (time 1746150920 ; total 108 ; last 108) - unlinked 20000 (time 1746151001 ; total 189 ; last 81) - unlinked 30000 (time 1746151069 ; total 257 ; last 68) - unlinked 40000 (time 1746151144 ; total 332 ; last 75) total: 50000 unlinks in 416 seconds: 120.192307 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg118-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server unloading modules via unload_modules_local on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2