== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:16:27 (1746072987) start mds service on oleg436-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg436-server' oleg436-server: oleg436-server.virtnet: executing load_modules_local oleg436-server: Loading modules from /home/green/git/lustre-release/lustre oleg436-server: detected 4 online CPUs by sysfs oleg436-server: MODOPTS_LIBCFS= oleg436-server: Force libcfs to create 2 CPU partitions oleg436-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg436-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg436-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Started lustre-OST0000 oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 130 (time 1746073034.39 total 12.82 last 10.14) - open/close 400 (time 1746073045.06 total 23.49 last 25.30) - open/close 655 (time 1746073055.36 total 33.79 last 24.76) - open/close 830 (time 1746073066.04 total 44.47 last 16.39) - open/close 1115 (time 1746073076.11 total 54.54 last 28.32) - open/close 1333 (time 1746073086.37 total 64.80 last 21.24) - open/close 1530 (time 1746073096.61 total 75.05 last 19.23) - open/close 1800 (time 1746073106.63 total 85.06 last 26.96) - open/close 2024 (time 1746073116.63 total 95.07 last 22.38) - open/close 2244 (time 1746073128.04 total 106.47 last 19.29) - open/close 2399 (time 1746073139.07 total 117.50 last 14.05) - open/close 2561 (time 1746073149.08 total 127.51 last 16.18) - open/close 2787 (time 1746073159.24 total 137.67 last 22.25) - open/close 2913 (time 1746073169.84 total 148.28 last 11.88) - open/close 3071 (time 1746073179.85 total 158.28 last 15.79) - open/close 3149 (time 1746073191.93 total 170.37 last 6.45) - open/close 3331 (time 1746073203.12 total 181.55 last 16.27) - open/close 3520 (time 1746073213.13 total 191.56 last 18.89) - open/close 3646 (time 1746073224.01 total 202.44 last 11.58) - open/close 3758 (time 1746073234.86 total 213.29 last 10.32) - open/close 3931 (time 1746073245.15 total 223.58 last 16.80) - open/close 4012 (time 1746073255.44 total 233.88 last 7.87) - open/close 4206 (time 1746073265.48 total 243.91 last 19.33) - open/close 4317 (time 1746073276.31 total 254.74 last 10.25) - open/close 4441 (time 1746073287.21 total 265.64 last 11.38) - open/close 4584 (time 1746073297.26 total 275.69 last 14.23) - open/close 4684 (time 1746073307.97 total 286.40 last 9.34) - open/close 4770 (time 1746073318.29 total 296.73 last 8.33) - open/close 4895 (time 1746073329.06 total 307.50 last 11.61) - open/close 4992 (time 1746073339.55 total 317.99 last 9.25) - open/close 5025 (time 1746073350.03 total 328.46 last 3.15) - open/close 5218 (time 1746073360.05 total 338.49 last 19.26) - open/close 5491 (time 1746073370.78 total 349.21 last 25.45) - open/close 5685 (time 1746073380.79 total 359.23 last 19.38) - open/close 5818 (time 1746073390.80 total 369.23 last 13.29) - open/close 6027 (time 1746073401.09 total 379.53 last 20.30) - open/close 6276 (time 1746073411.55 total 389.99 last 23.81) - open/close 6470 (time 1746073422.13 total 400.56 last 18.34) - open/close 6693 (time 1746073432.82 total 411.25 last 20.86) - open/close 6903 (time 1746073443.26 total 421.70 last 20.10) - open/close 7030 (time 1746073454.25 total 432.69 last 11.56) - open/close 7188 (time 1746073464.54 total 442.97 last 15.37) - open/close 7340 (time 1746073475.56 total 453.99 last 13.79) - open/close 7433 (time 1746073485.81 total 464.25 last 9.07) - open/close 7637 (time 1746073496.56 total 474.99 last 18.98) - open/close 7908 (time 1746073506.63 total 485.06 last 26.91) - open/close 8186 (time 1746073517.33 total 495.76 last 25.98) - open/close 8397 (time 1746073528.32 total 506.76 last 19.19) - open/close 9052 (time 1746073538.33 total 516.76 last 65.47) - open/close 10000 (time 1746073546.51 total 524.94 last 115.90) - open/close 11056 (time 1746073556.52 total 534.95 last 105.49) - open/close 12153 (time 1746073566.53 total 544.96 last 109.61) - open/close 13254 (time 1746073576.53 total 554.96 last 110.06) - open/close 14310 (time 1746073586.53 total 564.97 last 105.54) - open/close 15300 (time 1746073596.54 total 574.97 last 99.00) - open/close 16276 (time 1746073606.55 total 584.98 last 97.49) - open/close 17296 (time 1746073616.55 total 594.98 last 101.95) - open/close 18346 (time 1746073626.55 total 604.98 last 105.00) - open/close 19477 (time 1746073636.55 total 614.99 last 113.09) - open/close 20000 (time 1746073641.68 total 620.11 last 101.97) - open/close 21017 (time 1746073651.69 total 630.13 last 101.58) - open/close 21970 (time 1746073661.70 total 640.13 last 95.22) - open/close 22968 (time 1746073671.72 total 650.15 last 99.59) - open/close 24046 (time 1746073681.72 total 660.16 last 107.79) - open/close 25133 (time 1746073691.73 total 670.16 last 108.64) - open/close 26307 (time 1746073701.73 total 680.16 last 117.40) - open/close 27424 (time 1746073711.73 total 690.16 last 111.66) - open/close 28577 (time 1746073721.73 total 700.17 last 115.28) - open/close 29576 (time 1746073731.74 total 710.17 last 99.84) - open/close 30000 (time 1746073736.19 total 714.62 last 95.23) - open/close 31106 (time 1746073746.20 total 724.63 last 110.56) - open/close 32222 (time 1746073756.20 total 734.63 last 111.58) - open/close 33286 (time 1746073766.20 total 744.64 last 106.35) - open/close 34272 (time 1746073776.21 total 754.64 last 98.54) - open/close 35235 (time 1746073786.21 total 764.64 last 96.30) - open/close 36352 (time 1746073796.22 total 774.65 last 111.59) - open/close 37376 (time 1746073806.22 total 784.65 last 102.37) - open/close 38345 (time 1746073816.22 total 794.65 last 96.88) - open/close 39525 (time 1746073826.22 total 804.65 last 118.00) - open/close 40000 (time 1746073830.51 total 808.95 last 110.65) - open/close 41273 (time 1746073840.52 total 818.95 last 127.27) - open/close 42610 (time 1746073850.52 total 828.95 last 133.67) - open/close 43827 (time 1746073860.52 total 838.96 last 121.63) - open/close 45144 (time 1746073870.53 total 848.96 last 131.65) - open/close 46258 (time 1746073880.53 total 858.97 last 111.34) - open/close 47441 (time 1746073890.54 total 868.97 last 118.27) - open/close 48972 (time 1746073900.54 total 878.97 last 153.03) total: 50000 open/close in 887.88 seconds: 56.31 ops/second - unlinked 0 (time 1746073911 ; total 0 ; last 0) - unlinked 10000 (time 1746073969 ; total 58 ; last 58) - unlinked 20000 (time 1746074021 ; total 110 ; last 52) - unlinked 30000 (time 1746074063 ; total 152 ; last 42) - unlinked 40000 (time 1746074103 ; total 192 ; last 40) total: 50000 unlinks in 236 seconds: 211.864410 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg436-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg436-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg436-server stop mds service on oleg436-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg436-server unloading modules via unload_modules_local on: 'oleg436-server' oleg436-server: oleg436-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg436-client: oleg436-client: ssh exited with exit code 2 pdsh@oleg436-client: oleg436-server: ssh exited with exit code 2 pdsh@oleg436-client: oleg436-client: ssh exited with exit code 2 pdsh@oleg436-client: oleg436-server: ssh exited with exit code 2