== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:48:41 (1746146921) start mds service on oleg223-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing load_modules_local oleg223-server: Loading modules from /home/green/git/lustre-release/lustre oleg223-server: detected 4 online CPUs by sysfs oleg223-server: MODOPTS_LIBCFS= oleg223-server: Force libcfs to create 2 CPU partitions oleg223-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg223-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg223-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-OST0000 oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg223-client.virtnet: -o user_xattr,flock 192.168.202.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 187 (time 1746146967.25 total 10.28 last 18.19) - open/close 304 (time 1746146978.87 total 21.90 last 10.07) - open/close 599 (time 1746146988.94 total 31.97 last 29.29) - open/close 710 (time 1746147001.65 total 44.68 last 8.74) - open/close 908 (time 1746147012.27 total 55.30 last 18.64) - open/close 1186 (time 1746147022.76 total 65.79 last 26.50) - open/close 1387 (time 1746147032.90 total 75.92 last 19.84) - open/close 1544 (time 1746147043.71 total 86.74 last 14.51) - open/close 1679 (time 1746147054.46 total 97.49 last 12.57) - open/close 1911 (time 1746147065.04 total 108.07 last 21.92) - open/close 2176 (time 1746147076.00 total 119.03 last 24.17) - open/close 2400 (time 1746147086.22 total 129.25 last 21.92) - open/close 2533 (time 1746147096.83 total 139.86 last 12.54) - open/close 2752 (time 1746147107.43 total 150.46 last 20.66) - open/close 3085 (time 1746147117.96 total 160.98 last 31.64) - open/close 3198 (time 1746147130.46 total 173.49 last 9.03) - open/close 3386 (time 1746147140.47 total 183.49 last 18.80) - open/close 3536 (time 1746147151.47 total 194.50 last 13.63) - open/close 3784 (time 1746147163.97 total 207.00 last 19.84) - open/close 3979 (time 1746147174.32 total 217.35 last 18.85) - open/close 4177 (time 1746147184.47 total 227.50 last 19.51) - open/close 4352 (time 1746147194.47 total 237.50 last 17.49) - open/close 4577 (time 1746147204.48 total 247.51 last 22.47) - open/close 4780 (time 1746147214.48 total 257.51 last 20.30) - open/close 5028 (time 1746147224.62 total 267.65 last 24.46) - open/close 5201 (time 1746147235.53 total 278.56 last 15.85) - open/close 5378 (time 1746147245.85 total 288.88 last 17.16) - open/close 5504 (time 1746147256.34 total 299.37 last 12.01) - open/close 5713 (time 1746147267.01 total 310.04 last 19.60) - open/close 5882 (time 1746147277.13 total 320.16 last 16.70) - open/close 6011 (time 1746147287.13 total 330.16 last 12.90) - open/close 6234 (time 1746147298.10 total 341.13 last 20.33) - open/close 6392 (time 1746147309.05 total 352.07 last 14.43) - open/close 6696 (time 1746147319.28 total 362.31 last 29.70) - open/close 6981 (time 1746147329.29 total 372.31 last 28.48) - open/close 7215 (time 1746147341.58 total 384.61 last 19.03) - open/close 7382 (time 1746147351.68 total 394.71 last 16.53) - open/close 7491 (time 1746147362.10 total 405.13 last 10.46) - open/close 7619 (time 1746147372.76 total 415.79 last 12.01) - open/close 7755 (time 1746147382.76 total 425.79 last 13.60) - open/close 7959 (time 1746147393.04 total 436.07 last 19.84) - open/close 8032 (time 1746147404.18 total 447.21 last 6.55) - open/close 8155 (time 1746147414.79 total 457.82 last 11.60) - open/close 8406 (time 1746147424.81 total 467.84 last 25.04) - open/close 8667 (time 1746147434.81 total 477.84 last 26.09) - open/close 8789 (time 1746147445.34 total 488.37 last 11.59) - open/close 8892 (time 1746147455.70 total 498.73 last 9.94) - open/close 9767 (time 1746147465.71 total 508.74 last 87.41) - open/close 10973 (time 1746147475.72 total 518.75 last 120.48) - open/close 12071 (time 1746147485.74 total 528.76 last 109.67) - open/close 13230 (time 1746147495.74 total 538.77 last 115.89) - open/close 14809 (time 1746147505.74 total 548.77 last 157.83) - open/close 16076 (time 1746147515.74 total 558.77 last 126.67) - open/close 17468 (time 1746147525.75 total 568.77 last 139.17) - open/close 18636 (time 1746147535.75 total 578.78 last 116.79) - open/close 19797 (time 1746147545.75 total 588.78 last 116.03) - open/close 21136 (time 1746147555.77 total 598.80 last 133.71) - open/close 22315 (time 1746147565.77 total 608.80 last 117.88) - open/close 23705 (time 1746147575.77 total 618.80 last 138.97) - open/close 25106 (time 1746147585.78 total 628.80 last 140.04) - open/close 26399 (time 1746147595.78 total 638.81 last 129.19) - open/close 27877 (time 1746147605.79 total 648.82 last 147.67) - open/close 29263 (time 1746147615.80 total 658.83 last 138.55) - open/close 30000 (time 1746147620.90 total 663.93 last 144.40) - open/close 31105 (time 1746147630.91 total 673.94 last 110.35) - open/close 32114 (time 1746147640.92 total 683.95 last 100.79) - open/close 33225 (time 1746147650.93 total 693.96 last 111.06) - open/close 34332 (time 1746147660.93 total 703.96 last 110.63) - open/close 35527 (time 1746147670.94 total 713.97 last 119.43) - open/close 36635 (time 1746147680.94 total 723.97 last 110.80) - open/close 37804 (time 1746147690.95 total 733.98 last 116.78) - open/close 38993 (time 1746147700.95 total 743.98 last 118.87) - open/close 40000 (time 1746147709.00 total 752.03 last 125.13) - open/close 41151 (time 1746147719.01 total 762.03 last 115.05) - open/close 42130 (time 1746147729.01 total 772.03 last 97.89) - open/close 43452 (time 1746147739.01 total 782.04 last 132.10) - open/close 44692 (time 1746147749.01 total 792.04 last 123.98) - open/close 45730 (time 1746147759.02 total 802.05 last 103.78) - open/close 46991 (time 1746147769.02 total 812.05 last 126.08) - open/close 48072 (time 1746147779.03 total 822.05 last 108.02) - open/close 49301 (time 1746147789.03 total 832.06 last 122.85) total: 50000 open/close in 838.61 seconds: 59.62 ops/second - unlinked 0 (time 1746147797 ; total 0 ; last 0) - unlinked 10000 (time 1746147855 ; total 58 ; last 58) - unlinked 20000 (time 1746147901 ; total 104 ; last 46) - unlinked 30000 (time 1746147943 ; total 146 ; last 42) - unlinked 40000 (time 1746147985 ; total 188 ; last 42) total: 50000 unlinks in 230 seconds: 217.391312 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg223-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server stop mds service on oleg223-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg223-server unloading modules via unload_modules_local on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg223-client: oleg223-client: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-client: ssh exited with exit code 2