== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:23:39 (1761247419) start mds service on oleg635-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg635-server' oleg635-server: oleg635-server.virtnet: executing load_modules_local oleg635-server: Loading modules from /home/green/git/lustre-release/lustre oleg635-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg635-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg635-server: detected 4 online CPUs by sysfs oleg635-server: MODOPTS_LIBCFS= oleg635-server: Force libcfs to create 2 CPU partitions oleg635-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg635-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg635-server: oleg635-server.virtnet: executing set_default_debug -1 all pdsh@oleg635-client: oleg635-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg635-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg635-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg635-server: oleg635-server.virtnet: executing set_default_debug -1 all pdsh@oleg635-client: oleg635-server: ssh exited with exit code 1 Started lustre-OST0000 oleg635-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg635-client.virtnet: -o user_xattr,flock 192.168.206.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 182 (time 1761247488.57 total 10.37 last 17.56) - open/close 416 (time 1761247499.35 total 21.14 last 21.72) - open/close 687 (time 1761247509.35 total 31.14 last 27.09) - open/close 916 (time 1761247519.51 total 41.30 last 22.54) - open/close 1133 (time 1761247530.50 total 52.29 last 19.75) - open/close 1305 (time 1761247540.85 total 62.64 last 16.61) - open/close 1511 (time 1761247551.80 total 73.60 last 18.81) - open/close 1718 (time 1761247562.51 total 84.30 last 19.33) - open/close 1893 (time 1761247572.54 total 94.33 last 17.46) - open/close 2105 (time 1761247583.61 total 105.40 last 19.14) - open/close 2324 (time 1761247593.61 total 115.41 last 21.90) - open/close 2510 (time 1761247604.66 total 126.46 last 16.83) - open/close 2719 (time 1761247615.49 total 137.29 last 19.30) - open/close 2949 (time 1761247625.52 total 147.32 last 22.94) - open/close 3136 (time 1761247635.53 total 157.33 last 18.68) - open/close 3348 (time 1761247649.84 total 171.64 last 14.82) - open/close 3513 (time 1761247659.89 total 181.69 last 16.42) - open/close 3821 (time 1761247670.41 total 192.20 last 29.29) - open/close 4042 (time 1761247680.61 total 202.40 last 21.66) - open/close 4315 (time 1761247690.62 total 212.42 last 27.26) - open/close 4530 (time 1761247701.37 total 223.17 last 20.00) - open/close 4702 (time 1761247712.26 total 234.05 last 15.80) - open/close 4933 (time 1761247722.27 total 244.06 last 23.08) - open/close 5157 (time 1761247732.28 total 254.07 last 22.38) - open/close 5371 (time 1761247742.33 total 264.13 last 21.28) - open/close 5613 (time 1761247753.09 total 274.89 last 22.50) - open/close 5851 (time 1761247763.10 total 284.90 last 23.77) - open/close 6060 (time 1761247774.08 total 295.87 last 19.05) - open/close 6234 (time 1761247784.08 total 305.87 last 17.40) - open/close 6429 (time 1761247794.08 total 315.87 last 19.50) - open/close 6603 (time 1761247804.08 total 325.88 last 17.39) - open/close 6776 (time 1761247814.85 total 336.65 last 16.07) - open/close 7000 (time 1761247824.86 total 346.66 last 22.38) - open/close 7224 (time 1761247837.81 total 359.60 last 17.30) - open/close 7439 (time 1761247847.89 total 369.68 last 21.33) - open/close 7631 (time 1761247858.30 total 380.09 last 18.45) - open/close 7828 (time 1761247868.49 total 390.29 last 19.32) - open/close 8043 (time 1761247878.80 total 400.60 last 20.86) - open/close 8301 (time 1761247889.78 total 411.58 last 23.49) - open/close 8524 (time 1761247900.59 total 422.39 last 20.63) - open/close 8731 (time 1761247911.24 total 433.03 last 19.44) - open/close 8964 (time 1761247921.73 total 443.52 last 22.22) - open/close 9174 (time 1761247932.34 total 454.13 last 19.79) - open/close 9363 (time 1761247942.52 total 464.32 last 18.56) - open/close 9532 (time 1761247952.54 total 474.33 last 16.87) - open/close 9758 (time 1761247963.20 total 484.99 last 21.20) - open/close 9949 (time 1761247973.20 total 495.00 last 19.09) - open/close 10000 (time 1761247975.30 total 497.09 last 24.30) - open/close 10251 (time 1761247986.31 total 508.11 last 22.79) - open/close 10488 (time 1761247996.71 total 518.50 last 22.80) - open/close 10691 (time 1761248006.72 total 528.51 last 20.28) - open/close 10829 (time 1761248016.74 total 538.53 last 13.77) - open/close 11024 (time 1761248027.39 total 549.18 last 18.31) - open/close 11288 (time 1761248038.22 total 560.02 last 24.36) - open/close 11508 (time 1761248050.90 total 572.69 last 17.36) - open/close 11732 (time 1761248061.20 total 582.99 last 21.75) - open/close 11954 (time 1761248071.28 total 593.07 last 22.01) - open/close 12179 (time 1761248081.29 total 603.08 last 22.49) - open/close 12369 (time 1761248091.97 total 613.76 last 17.78) - open/close 12597 (time 1761248101.99 total 623.78 last 22.76) - open/close 12689 (time 1761248112.57 total 634.36 last 8.69) - open/close 12940 (time 1761248123.38 total 645.17 last 23.22) - open/close 13144 (time 1761248133.41 total 655.20 last 20.35) - open/close 13337 (time 1761248143.74 total 665.53 last 18.68) - open/close 13545 (time 1761248153.75 total 675.54 last 20.78) - open/close 13767 (time 1761248164.10 total 685.90 last 21.44) - open/close 13994 (time 1761248174.79 total 696.58 last 21.25) - open/close 14276 (time 1761248185.16 total 706.96 last 27.18) - open/close 14522 (time 1761248196.03 total 717.83 last 22.62) - open/close 14740 (time 1761248206.46 total 728.25 last 20.91) - open/close 14977 (time 1761248217.16 total 738.95 last 22.16) - open/close 15236 (time 1761248227.16 total 748.95 last 25.89) - open/close 15425 (time 1761248237.82 total 759.62 last 17.72) - open/close 15664 (time 1761248248.71 total 770.50 last 21.96) - open/close 15923 (time 1761248259.45 total 781.24 last 24.11) - open/close 16157 (time 1761248269.47 total 791.27 last 23.34) - open/close 16372 (time 1761248280.25 total 802.04 last 19.95) - open/close 16637 (time 1761248291.20 total 812.99 last 24.20) - open/close 16831 (time 1761248301.26 total 823.05 last 19.28) - open/close 17084 (time 1761248311.26 total 833.06 last 25.29) - open/close 17826 (time 1761248321.27 total 843.07 last 74.15) - open/close 18698 (time 1761248331.28 total 853.07 last 87.16) - open/close 19483 (time 1761248341.29 total 863.08 last 78.41) - open/close 20000 (time 1761248347.95 total 869.74 last 77.65) - open/close 20811 (time 1761248357.95 total 879.75 last 81.04) - open/close 21613 (time 1761248367.95 total 889.75 last 80.19) - open/close 22369 (time 1761248377.96 total 899.75 last 75.56) - open/close 23122 (time 1761248387.97 total 909.76 last 75.26) - open/close 23903 (time 1761248397.98 total 919.77 last 78.01) - open/close 24661 (time 1761248407.98 total 929.78 last 75.76) - open/close 25448 (time 1761248417.99 total 939.78 last 78.67) - open/close 26259 (time 1761248427.99 total 949.78 last 81.09) - open/close 26995 (time 1761248437.99 total 959.79 last 73.54) - open/close 27760 (time 1761248448.00 total 969.79 last 76.46) - open/close 28573 (time 1761248458.00 total 979.80 last 81.27) - open/close 29402 (time 1761248468.01 total 989.80 last 82.84) - open/close 30000 (time 1761248476.45 total 998.24 last 70.86) - open/close 30729 (time 1761248486.46 total 1008.25 last 72.83) - open/close 31454 (time 1761248496.46 total 1018.26 last 72.47) - open/close 32201 (time 1761248506.46 total 1028.26 last 74.70) - open/close 32948 (time 1761248516.48 total 1038.28 last 74.57) - open/close 33715 (time 1761248526.50 total 1048.30 last 76.55) - open/close 34536 (time 1761248536.52 total 1058.32 last 81.93) - open/close 35304 (time 1761248546.53 total 1068.32 last 76.78) - open/close 36006 (time 1761248556.53 total 1078.33 last 70.14) - open/close 36817 (time 1761248566.54 total 1088.34 last 81.02) - open/close 37510 (time 1761248576.55 total 1098.35 last 69.24) - open/close 38324 (time 1761248586.57 total 1108.36 last 81.25) - open/close 39106 (time 1761248596.57 total 1118.37 last 78.17) - open/close 39822 (time 1761248606.59 total 1128.38 last 71.51) - open/close 40000 (time 1761248609.42 total 1131.22 last 62.74) - open/close 40652 (time 1761248619.44 total 1141.23 last 65.12) - open/close 41230 (time 1761248629.45 total 1151.24 last 57.73) - open/close 41701 (time 1761248639.46 total 1161.25 last 47.05) - open/close 42314 (time 1761248649.47 total 1171.26 last 61.23) - open/close 42950 (time 1761248659.48 total 1181.27 last 63.53) - open/close 43594 (time 1761248669.48 total 1191.28 last 64.38) - open/close 44281 (time 1761248679.50 total 1201.29 last 68.59) - open/close 45051 (time 1761248689.53 total 1211.32 last 76.78) - open/close 45787 (time 1761248699.53 total 1221.33 last 73.57) - open/close 46579 (time 1761248709.54 total 1231.33 last 79.17) - open/close 47305 (time 1761248719.54 total 1241.34 last 72.54) - open/close 48053 (time 1761248729.55 total 1251.35 last 74.72) - open/close 48859 (time 1761248739.57 total 1261.36 last 80.50) - open/close 49615 (time 1761248749.57 total 1271.36 last 75.57) total: 50000 open/close in 1276.50 seconds: 39.17 ops/second - unlinked 0 (time 1761248758 ; total 0 ; last 0) - unlinked 10000 (time 1761248877 ; total 119 ; last 119) - unlinked 20000 (time 1761249005 ; total 247 ; last 128) - unlinked 30000 (time 1761249136 ; total 378 ; last 131) - unlinked 40000 (time 1761249264 ; total 506 ; last 128) total: 50000 unlinks in 635 seconds: 78.740158 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg635-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg635-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg635-server stop mds service on oleg635-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg635-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg635-server' oleg635-server: oleg635-server.virtnet: executing unload_modules_local oleg635-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg635-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg635-client: oleg635-server: ssh exited with exit code 2 pdsh@oleg635-client: oleg635-client: ssh exited with exit code 2 pdsh@oleg635-client: oleg635-client: ssh exited with exit code 2 pdsh@oleg635-client: oleg635-server: ssh exited with exit code 2