== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:42:18 (1746142938) start mds service on oleg223-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing load_modules_local oleg223-server: Loading modules from /home/green/git/lustre-release/lustre oleg223-server: detected 4 online CPUs by sysfs oleg223-server: MODOPTS_LIBCFS= oleg223-server: Force libcfs to create 2 CPU partitions oleg223-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg223-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg223-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg223-server: oleg223-server.virtnet: executing set_default_debug -1 all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-OST0000 oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg223-client.virtnet: -o user_xattr,flock 192.168.202.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 201 (time 1746142983.96 total 10.47 last 19.20) - open/close 314 (time 1746142994.63 total 21.13 last 10.59) - open/close 570 (time 1746143005.43 total 31.93 last 23.70) - open/close 860 (time 1746143015.85 total 42.35 last 27.83) - open/close 956 (time 1746143026.12 total 52.62 last 9.35) - open/close 1080 (time 1746143036.67 total 63.18 last 11.75) - open/close 1297 (time 1746143046.88 total 73.38 last 21.27) - open/close 1473 (time 1746143058.22 total 84.73 last 15.51) - open/close 1618 (time 1746143068.23 total 94.74 last 14.48) - open/close 1722 (time 1746143078.75 total 105.25 last 9.89) - open/close 1850 (time 1746143089.13 total 115.63 last 12.33) - open/close 2043 (time 1746143099.77 total 126.27 last 18.15) - open/close 2159 (time 1746143110.47 total 136.98 last 10.84) - open/close 2415 (time 1746143120.60 total 147.10 last 25.28) - open/close 2544 (time 1746143131.52 total 158.03 last 11.81) - open/close 2700 (time 1746143142.36 total 168.87 last 14.39) - open/close 2914 (time 1746143152.59 total 179.09 last 20.93) - open/close 3106 (time 1746143163.63 total 190.14 last 17.38) - open/close 3284 (time 1746143173.68 total 200.19 last 17.72) - open/close 3438 (time 1746143184.08 total 210.59 last 14.81) - open/close 3517 (time 1746143194.30 total 220.81 last 7.73) - open/close 3716 (time 1746143204.60 total 231.11 last 19.32) - open/close 3794 (time 1746143214.65 total 241.15 last 7.77) - open/close 3960 (time 1746143224.65 total 251.16 last 16.59) - open/close 4100 (time 1746143235.55 total 262.05 last 12.85) - open/close 4306 (time 1746143246.04 total 272.55 last 19.63) - open/close 4439 (time 1746143256.83 total 283.34 last 12.32) - open/close 4547 (time 1746143267.22 total 293.72 last 10.40) - open/close 4733 (time 1746143277.89 total 304.40 last 17.43) - open/close 4815 (time 1746143288.26 total 314.76 last 7.91) - open/close 5009 (time 1746143298.56 total 325.07 last 18.82) - open/close 5140 (time 1746143309.45 total 335.95 last 12.03) - open/close 5280 (time 1746143320.31 total 346.82 last 12.89) - open/close 5381 (time 1746143330.93 total 357.44 last 9.51) - open/close 5545 (time 1746143341.36 total 367.87 last 15.73) - open/close 5692 (time 1746143351.40 total 377.90 last 14.65) - open/close 5820 (time 1746143362.12 total 388.63 last 11.93) - open/close 5957 (time 1746143372.87 total 399.37 last 12.75) - open/close 6101 (time 1746143382.87 total 409.38 last 14.39) - open/close 6226 (time 1746143392.88 total 419.38 last 12.49) - open/close 6355 (time 1746143403.88 total 430.39 last 11.72) - open/close 6503 (time 1746143413.89 total 440.40 last 14.79) - open/close 6620 (time 1746143423.90 total 450.41 last 11.69) - open/close 6771 (time 1746143433.96 total 460.47 last 15.01) - open/close 6877 (time 1746143444.54 total 471.04 last 10.02) - open/close 7059 (time 1746143454.62 total 481.13 last 18.04) - open/close 7157 (time 1746143464.69 total 491.19 last 9.74) - open/close 7364 (time 1746143474.77 total 501.27 last 20.53) - open/close 7552 (time 1746143485.20 total 511.70 last 18.03) - open/close 7646 (time 1746143495.48 total 521.99 last 9.14) - open/close 7868 (time 1746143506.71 total 533.21 last 19.77) - open/close 7985 (time 1746143517.25 total 543.75 last 11.10) - open/close 8541 (time 1746143527.25 total 553.75 last 55.60) - open/close 9732 (time 1746143537.25 total 563.76 last 119.04) - open/close 10000 (time 1746143539.56 total 566.06 last 116.40) - open/close 11070 (time 1746143549.56 total 576.06 last 106.98) - open/close 12143 (time 1746143559.56 total 586.06 last 107.28) - open/close 13216 (time 1746143569.57 total 596.07 last 107.23) - open/close 14288 (time 1746143579.57 total 606.07 last 107.16) - open/close 15351 (time 1746143589.57 total 616.08 last 106.28) - open/close 16407 (time 1746143599.57 total 626.08 last 105.58) - open/close 17467 (time 1746143609.58 total 636.08 last 105.96) - open/close 18643 (time 1746143619.58 total 646.09 last 117.55) - open/close 19746 (time 1746143629.59 total 656.09 last 110.22) - open/close 20000 (time 1746143631.81 total 658.32 last 114.27) - open/close 21169 (time 1746143641.82 total 668.32 last 116.83) - open/close 22198 (time 1746143651.83 total 678.34 last 102.76) - open/close 23215 (time 1746143661.83 total 688.34 last 101.69) - open/close 24130 (time 1746143671.84 total 698.35 last 91.41) - open/close 25096 (time 1746143681.85 total 708.36 last 96.48) - open/close 26047 (time 1746143691.86 total 718.37 last 95.04) - open/close 27045 (time 1746143701.87 total 728.37 last 99.74) - open/close 28051 (time 1746143711.87 total 738.37 last 100.59) - open/close 29182 (time 1746143721.88 total 748.38 last 112.96) - open/close 30000 (time 1746143730.11 total 756.61 last 99.41) - open/close 31081 (time 1746143740.11 total 766.62 last 108.03) - open/close 32211 (time 1746143750.12 total 776.62 last 112.98) - open/close 33322 (time 1746143760.12 total 786.62 last 111.08) - open/close 34607 (time 1746143770.12 total 796.63 last 128.44) - open/close 35909 (time 1746143780.13 total 806.63 last 130.16) - open/close 37151 (time 1746143790.13 total 816.63 last 124.14) - open/close 38433 (time 1746143800.13 total 826.64 last 128.16) - open/close 39805 (time 1746143810.13 total 836.64 last 137.18) - open/close 41109 (time 1746143820.14 total 846.64 last 130.37) - open/close 42473 (time 1746143830.16 total 856.66 last 136.14) - open/close 43661 (time 1746143840.17 total 866.68 last 118.61) - open/close 44762 (time 1746143850.18 total 876.69 last 110.01) - open/close 45858 (time 1746143860.19 total 886.69 last 109.53) - open/close 46925 (time 1746143870.20 total 896.70 last 106.60) - open/close 48166 (time 1746143880.20 total 906.70 last 124.08) - open/close 49524 (time 1746143890.20 total 916.71 last 135.74) total: 50000 open/close in 920.35 seconds: 54.33 ops/second - unlinked 0 (time 1746143895 ; total 0 ; last 0) - unlinked 10000 (time 1746143965 ; total 70 ; last 70) - unlinked 20000 (time 1746144012 ; total 117 ; last 47) - unlinked 30000 (time 1746144073 ; total 178 ; last 61) - unlinked 40000 (time 1746144129 ; total 234 ; last 56) total: 50000 unlinks in 282 seconds: 177.304962 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg223-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg223-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg223-server stop mds service on oleg223-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg223-server unloading modules via unload_modules_local on: 'oleg223-server' oleg223-server: oleg223-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg223-client: oleg223-client: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-client: ssh exited with exit code 2 pdsh@oleg223-client: oleg223-server: ssh exited with exit code 2