== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:49:10 (1761284950) start mds service on oleg312-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg312-server' oleg312-server: oleg312-server.virtnet: executing load_modules_local oleg312-server: Loading modules from /home/green/git/lustre-release/lustre oleg312-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg312-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg312-server: detected 4 online CPUs by sysfs oleg312-server: MODOPTS_LIBCFS= oleg312-server: Force libcfs to create 2 CPU partitions oleg312-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg312-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg312-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg312-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg312-server: oleg312-server.virtnet: executing set_default_debug -1 all pdsh@oleg312-client: oleg312-server: ssh exited with exit code 1 Started lustre-OST0000 oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg312-client.virtnet: -o user_xattr,flock 192.168.203.112@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 145 (time 1761285022.15 total 10.29 last 14.09) - open/close 393 (time 1761285032.16 total 20.30 last 24.78) - open/close 631 (time 1761285042.43 total 30.57 last 23.18) - open/close 875 (time 1761285053.39 total 41.54 last 22.25) - open/close 1087 (time 1761285063.87 total 52.01 last 20.24) - open/close 1308 (time 1761285074.64 total 62.78 last 20.53) - open/close 1480 (time 1761285084.70 total 72.84 last 17.09) - open/close 1692 (time 1761285094.70 total 82.85 last 21.19) - open/close 1892 (time 1761285104.78 total 92.92 last 19.85) - open/close 2160 (time 1761285114.79 total 102.93 last 26.77) - open/close 2360 (time 1761285125.06 total 113.21 last 19.47) - open/close 2542 (time 1761285135.37 total 123.52 last 17.65) - open/close 2779 (time 1761285145.38 total 133.53 last 23.69) - open/close 2968 (time 1761285155.38 total 143.53 last 18.89) - open/close 3151 (time 1761285166.37 total 154.52 last 16.65) - open/close 3388 (time 1761285176.40 total 164.55 last 23.63) - open/close 3456 (time 1761285187.40 total 175.55 last 6.18) - open/close 3638 (time 1761285198.31 total 186.45 last 16.69) - open/close 3870 (time 1761285208.52 total 196.66 last 22.72) - open/close 4093 (time 1761285218.52 total 206.67 last 22.29) - open/close 4316 (time 1761285228.54 total 216.68 last 22.26) - open/close 4522 (time 1761285238.98 total 227.12 last 19.73) - open/close 4751 (time 1761285248.99 total 237.14 last 22.86) - open/close 4981 (time 1761285259.00 total 247.15 last 22.98) - open/close 5186 (time 1761285269.14 total 257.29 last 20.22) - open/close 5416 (time 1761285279.15 total 267.29 last 22.98) - open/close 5663 (time 1761285289.53 total 277.68 last 23.78) - open/close 5887 (time 1761285299.64 total 287.79 last 22.16) - open/close 6135 (time 1761285309.65 total 297.80 last 24.77) - open/close 6301 (time 1761285320.52 total 308.67 last 15.27) - open/close 6537 (time 1761285331.34 total 319.48 last 21.82) - open/close 6763 (time 1761285342.09 total 330.24 last 21.01) - open/close 6948 (time 1761285352.10 total 340.25 last 18.49) - open/close 7158 (time 1761285363.01 total 351.16 last 19.25) - open/close 7385 (time 1761285373.02 total 361.17 last 22.67) - open/close 7598 (time 1761285383.03 total 371.17 last 21.30) - open/close 7774 (time 1761285393.10 total 381.25 last 17.46) - open/close 8009 (time 1761285404.10 total 392.25 last 21.37) - open/close 8199 (time 1761285414.10 total 402.25 last 18.99) - open/close 8439 (time 1761285424.11 total 412.26 last 23.98) - open/close 8579 (time 1761285434.64 total 422.78 last 13.30) - open/close 8754 (time 1761285444.64 total 432.79 last 17.49) - open/close 8926 (time 1761285454.65 total 442.79 last 17.19) - open/close 9104 (time 1761285464.65 total 452.80 last 17.79) - open/close 9348 (time 1761285475.52 total 463.66 last 22.45) - open/close 9588 (time 1761285486.47 total 474.61 last 21.92) - open/close 9775 (time 1761285496.49 total 484.63 last 18.66) - open/close 9892 (time 1761285506.49 total 494.63 last 11.70) - open/close 10000 (time 1761285510.74 total 498.89 last 25.39) - open/close 10129 (time 1761285520.97 total 509.11 last 12.62) - open/close 10242 (time 1761285532.03 total 520.17 last 10.22) - open/close 10362 (time 1761285542.07 total 530.22 last 11.94) - open/close 10533 (time 1761285552.09 total 540.23 last 17.08) - open/close 10683 (time 1761285562.77 total 550.91 last 14.05) - open/close 10908 (time 1761285573.01 total 561.15 last 21.97) - open/close 11119 (time 1761285583.68 total 571.83 last 19.77) - open/close 11295 (time 1761285593.68 total 581.83 last 17.60) - open/close 11506 (time 1761285604.29 total 592.44 last 19.89) - open/close 11780 (time 1761285614.73 total 602.88 last 26.24) - open/close 11998 (time 1761285624.78 total 612.92 last 21.71) - open/close 12245 (time 1761285635.01 total 623.15 last 24.14) - open/close 12426 (time 1761285645.12 total 633.27 last 17.89) - open/close 12661 (time 1761285655.13 total 643.27 last 23.49) - open/close 12890 (time 1761285665.86 total 654.01 last 21.33) - open/close 13117 (time 1761285676.81 total 664.96 last 20.73) - open/close 13365 (time 1761285687.75 total 675.90 last 22.68) - open/close 13586 (time 1761285698.56 total 686.71 last 20.44) - open/close 13761 (time 1761285708.82 total 696.97 last 17.06) - open/close 13960 (time 1761285719.18 total 707.32 last 19.22) - open/close 14182 (time 1761285729.18 total 717.33 last 22.18) - open/close 14409 (time 1761285740.23 total 728.37 last 20.55) - open/close 14625 (time 1761285750.28 total 738.42 last 21.50) - open/close 14856 (time 1761285760.28 total 748.43 last 23.09) - open/close 15032 (time 1761285771.28 total 759.43 last 16.00) - open/close 15213 (time 1761285781.32 total 769.47 last 18.03) - open/close 15436 (time 1761285791.33 total 779.48 last 22.27) - open/close 16157 (time 1761285801.34 total 789.49 last 72.05) - open/close 17006 (time 1761285811.35 total 799.50 last 84.79) - open/close 17829 (time 1761285821.36 total 809.50 last 82.26) - open/close 18628 (time 1761285831.37 total 819.51 last 79.84) - open/close 19491 (time 1761285841.38 total 829.52 last 86.22) - open/close 20000 (time 1761285847.37 total 835.51 last 84.97) - open/close 20812 (time 1761285857.38 total 845.52 last 81.09) - open/close 21670 (time 1761285867.39 total 855.54 last 85.69) - open/close 22499 (time 1761285877.40 total 865.55 last 82.83) - open/close 23337 (time 1761285887.40 total 875.55 last 83.77) - open/close 24114 (time 1761285897.41 total 885.55 last 77.67) - open/close 24905 (time 1761285907.43 total 895.58 last 78.91) - open/close 25682 (time 1761285917.43 total 905.58 last 77.70) - open/close 26535 (time 1761285927.44 total 915.58 last 85.25) - open/close 27278 (time 1761285937.45 total 925.59 last 74.22) - open/close 28050 (time 1761285947.46 total 935.60 last 77.12) - open/close 28856 (time 1761285957.46 total 945.61 last 80.58) - open/close 29624 (time 1761285967.46 total 955.61 last 76.77) - open/close 30000 (time 1761285972.33 total 960.47 last 77.34) - open/close 30822 (time 1761285982.33 total 970.48 last 82.14) - open/close 31682 (time 1761285992.34 total 980.48 last 85.95) - open/close 32503 (time 1761286002.35 total 990.49 last 82.02) - open/close 33284 (time 1761286012.35 total 1000.50 last 78.05) - open/close 34043 (time 1761286022.36 total 1010.51 last 75.83) - open/close 34852 (time 1761286032.37 total 1020.52 last 80.84) - open/close 35664 (time 1761286042.38 total 1030.53 last 81.12) - open/close 36465 (time 1761286052.39 total 1040.54 last 80.03) - open/close 37308 (time 1761286062.39 total 1050.54 last 84.25) - open/close 38069 (time 1761286072.41 total 1060.56 last 75.98) - open/close 38867 (time 1761286082.42 total 1070.56 last 79.74) - open/close 39687 (time 1761286092.42 total 1080.57 last 81.94) - open/close 40000 (time 1761286096.36 total 1084.51 last 79.44) - open/close 40800 (time 1761286106.37 total 1094.51 last 80.00) - open/close 41615 (time 1761286116.37 total 1104.51 last 81.49) - open/close 42465 (time 1761286126.38 total 1114.52 last 84.91) - open/close 43342 (time 1761286136.38 total 1124.52 last 87.70) - open/close 44146 (time 1761286146.38 total 1134.53 last 80.35) - open/close 44940 (time 1761286156.39 total 1144.53 last 79.39) - open/close 45742 (time 1761286166.39 total 1154.53 last 80.20) - open/close 46490 (time 1761286176.40 total 1164.55 last 74.67) - open/close 47319 (time 1761286186.41 total 1174.56 last 82.82) - open/close 48125 (time 1761286196.42 total 1184.57 last 80.51) - open/close 48914 (time 1761286206.43 total 1194.57 last 78.87) - open/close 49687 (time 1761286216.43 total 1204.58 last 77.28) total: 50000 open/close in 1209.09 seconds: 41.35 ops/second - unlinked 0 (time 1761286224 ; total 0 ; last 0) - unlinked 10000 (time 1761286350 ; total 126 ; last 126) - unlinked 20000 (time 1761286477 ; total 253 ; last 127) - unlinked 30000 (time 1761286593 ; total 369 ; last 116) - unlinked 40000 (time 1761286715 ; total 491 ; last 122) total: 50000 unlinks in 612 seconds: 81.699348 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg312-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg312-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg312-server stop mds service on oleg312-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg312-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg312-server' oleg312-server: oleg312-server.virtnet: executing unload_modules_local oleg312-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg312-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2 pdsh@oleg312-client: oleg312-client: ssh exited with exit code 2 pdsh@oleg312-client: oleg312-server: ssh exited with exit code 2