== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:35:32 (1746146132) start mds service on oleg231-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing load_modules_local oleg231-server: Loading modules from /home/green/git/lustre-release/lustre oleg231-server: detected 4 online CPUs by sysfs oleg231-server: MODOPTS_LIBCFS= oleg231-server: Force libcfs to create 2 CPU partitions oleg231-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg231-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg231-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0000 oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 210 (time 1746146172.86 total 10.22 last 20.55) - open/close 418 (time 1746146183.69 total 21.05 last 19.20) - open/close 629 (time 1746146194.64 total 32.00 last 19.27) - open/close 825 (time 1746146204.94 total 42.31 last 19.02) - open/close 1108 (time 1746146215.08 total 52.44 last 27.92) - open/close 1356 (time 1746146225.39 total 62.75 last 24.05) - open/close 1554 (time 1746146235.55 total 72.92 last 19.48) - open/close 1849 (time 1746146246.43 total 83.79 last 27.12) - open/close 2035 (time 1746146257.05 total 94.41 last 17.52) - open/close 2202 (time 1746146267.55 total 104.92 last 15.89) - open/close 2366 (time 1746146277.76 total 115.12 last 16.07) - open/close 2519 (time 1746146288.04 total 125.41 last 14.88) - open/close 2729 (time 1746146298.89 total 136.25 last 19.36) - open/close 2885 (time 1746146309.11 total 146.47 last 15.26) - open/close 3130 (time 1746146320.10 total 157.46 last 22.29) - open/close 3296 (time 1746146330.46 total 167.82 last 16.03) - open/close 3491 (time 1746146341.14 total 178.50 last 18.26) - open/close 3621 (time 1746146351.24 total 188.61 last 12.86) - open/close 3754 (time 1746146361.35 total 198.72 last 13.16) - open/close 3986 (time 1746146372.35 total 209.71 last 21.10) - open/close 4213 (time 1746146383.36 total 220.72 last 20.63) - open/close 4442 (time 1746146393.64 total 231.00 last 22.27) - open/close 4729 (time 1746146404.45 total 241.81 last 26.54) - open/close 4939 (time 1746146415.33 total 252.70 last 19.30) - open/close 5047 (time 1746146425.48 total 262.84 last 10.64) - open/close 5323 (time 1746146436.01 total 273.38 last 26.21) - open/close 5569 (time 1746146446.08 total 283.45 last 24.43) - open/close 5887 (time 1746146456.79 total 294.15 last 29.71) - open/close 6111 (time 1746146466.79 total 304.16 last 22.38) - open/close 6331 (time 1746146477.00 total 314.36 last 21.56) - open/close 6572 (time 1746146487.00 total 324.36 last 24.10) - open/close 6827 (time 1746146497.22 total 334.59 last 24.94) - open/close 7060 (time 1746146509.57 total 346.93 last 18.88) - open/close 7333 (time 1746146520.58 total 357.94 last 24.79) - open/close 7575 (time 1746146530.84 total 368.20 last 23.58) - open/close 7822 (time 1746146541.45 total 378.81 last 23.29) - open/close 8053 (time 1746146552.40 total 389.76 last 21.10) - open/close 8290 (time 1746146562.76 total 400.12 last 22.87) - open/close 8449 (time 1746146572.77 total 410.14 last 15.88) - open/close 8632 (time 1746146583.57 total 420.93 last 16.96) - open/close 9340 (time 1746146593.57 total 430.93 last 70.76) - open/close 10000 (time 1746146598.95 total 436.31 last 122.72) - open/close 11254 (time 1746146608.95 total 446.31 last 125.39) - open/close 12400 (time 1746146618.95 total 456.31 last 114.59) - open/close 13496 (time 1746146628.96 total 466.32 last 109.55) - open/close 14657 (time 1746146638.97 total 476.33 last 115.99) - open/close 15891 (time 1746146648.97 total 486.33 last 123.32) - open/close 17112 (time 1746146658.98 total 496.34 last 122.03) - open/close 18260 (time 1746146668.98 total 506.34 last 114.77) - open/close 19443 (time 1746146678.98 total 516.35 last 118.25) - open/close 20000 (time 1746146683.53 total 520.90 last 122.40) - open/close 21267 (time 1746146693.55 total 530.91 last 126.56) - open/close 22467 (time 1746146703.55 total 540.91 last 119.98) - open/close 23539 (time 1746146713.57 total 550.93 last 106.98) - open/close 24840 (time 1746146723.59 total 560.95 last 129.89) - open/close 26182 (time 1746146733.59 total 570.95 last 134.19) - open/close 27939 (time 1746146743.59 total 580.95 last 175.66) - open/close 29571 (time 1746146753.59 total 590.95 last 163.18) - open/close 30000 (time 1746146756.41 total 593.77 last 152.18) - open/close 31447 (time 1746146766.42 total 603.78 last 144.60) - open/close 32946 (time 1746146776.42 total 613.78 last 149.86) - open/close 34325 (time 1746146786.42 total 623.78 last 137.88) - open/close 35583 (time 1746146796.42 total 633.78 last 125.76) - open/close 36838 (time 1746146806.43 total 643.79 last 125.47) - open/close 38036 (time 1746146816.43 total 653.79 last 119.72) - open/close 39298 (time 1746146826.43 total 663.79 last 126.19) - open/close 40000 (time 1746146831.75 total 669.11 last 132.04) - open/close 41495 (time 1746146841.76 total 679.12 last 149.37) - open/close 43152 (time 1746146851.76 total 689.13 last 165.61) - open/close 44510 (time 1746146861.77 total 699.13 last 135.77) - open/close 46199 (time 1746146871.77 total 709.13 last 168.82) - open/close 47782 (time 1746146881.79 total 719.15 last 157.99) - open/close 49483 (time 1746146891.80 total 729.16 last 169.99) total: 50000 open/close in 731.85 seconds: 68.32 ops/second - unlinked 0 (time 1746146895 ; total 0 ; last 0) - unlinked 10000 (time 1746146943 ; total 48 ; last 48) - unlinked 20000 (time 1746146989 ; total 94 ; last 46) - unlinked 30000 (time 1746147034 ; total 139 ; last 45) - unlinked 40000 (time 1746147082 ; total 187 ; last 48) total: 50000 unlinks in 232 seconds: 215.517242 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg231-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg231-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server stop mds service on oleg231-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg231-server unloading modules via unload_modules_local on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg231-client: oleg231-client: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-client: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2