== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:27:39 (1772472459) start mds service on oleg343-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg343-server' oleg343-server: oleg343-server.virtnet: executing load_modules_local oleg343-server: Loading modules from /home/green/git/lustre-release/lustre oleg343-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg343-server: detected 4 online CPUs by sysfs oleg343-server: MODOPTS_LIBCFS= oleg343-server: Force libcfs to create 2 CPU partitions oleg343-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg343-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg343-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg343-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0000 oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 261 (time 1772472544.30 total 10.92 last 23.91) - open/close 539 (time 1772472555.19 total 21.81 last 25.52) - open/close 755 (time 1772472565.73 total 32.35 last 20.48) - open/close 977 (time 1772472576.77 total 43.39 last 20.12) - open/close 1181 (time 1772472587.01 total 53.63 last 19.92) - open/close 1329 (time 1772472598.01 total 64.63 last 13.45) - open/close 1487 (time 1772472608.88 total 75.50 last 14.53) - open/close 1666 (time 1772472618.89 total 85.51 last 17.88) - open/close 1815 (time 1772472629.68 total 96.30 last 13.81) - open/close 2005 (time 1772472640.47 total 107.08 last 17.62) - open/close 2107 (time 1772472650.48 total 117.10 last 10.18) - open/close 2265 (time 1772472660.48 total 127.10 last 15.79) - open/close 2474 (time 1772472671.20 total 137.82 last 19.50) - open/close 2643 (time 1772472681.34 total 147.96 last 16.68) - open/close 2874 (time 1772472692.42 total 159.04 last 20.85) - open/close 3056 (time 1772472703.42 total 170.04 last 16.54) - open/close 3303 (time 1772472714.28 total 180.90 last 22.74) - open/close 3566 (time 1772472725.31 total 191.93 last 23.85) - open/close 3796 (time 1772472735.34 total 201.96 last 22.93) - open/close 3940 (time 1772472745.34 total 211.96 last 14.39) - open/close 4181 (time 1772472756.01 total 222.63 last 22.58) - open/close 4418 (time 1772472766.45 total 233.07 last 22.71) - open/close 4688 (time 1772472776.46 total 243.08 last 26.96) - open/close 4919 (time 1772472786.48 total 253.10 last 23.07) - open/close 5160 (time 1772472797.22 total 263.84 last 22.44) - open/close 5350 (time 1772472807.23 total 273.84 last 18.99) - open/close 5559 (time 1772472818.18 total 284.80 last 19.08) - open/close 5742 (time 1772472828.20 total 294.82 last 18.27) - open/close 5962 (time 1772472838.20 total 304.82 last 21.98) - open/close 6212 (time 1772472848.21 total 314.83 last 24.98) - open/close 6464 (time 1772472858.24 total 324.86 last 25.12) - open/close 6664 (time 1772472868.98 total 335.60 last 18.63) - open/close 6924 (time 1772472880.02 total 346.64 last 23.54) - open/close 7175 (time 1772472890.04 total 356.66 last 25.05) - open/close 7381 (time 1772472900.21 total 366.83 last 20.25) - open/close 7634 (time 1772472911.02 total 377.64 last 23.41) - open/close 7886 (time 1772472921.92 total 388.54 last 23.12) - open/close 8146 (time 1772472932.07 total 398.69 last 25.61) - open/close 8375 (time 1772472942.18 total 408.80 last 22.66) - open/close 8614 (time 1772472953.09 total 419.71 last 21.91) - open/close 8746 (time 1772472963.11 total 429.73 last 13.17) - open/close 8923 (time 1772472973.40 total 440.01 last 17.22) - open/close 9067 (time 1772472983.66 total 450.28 last 14.03) - open/close 9341 (time 1772472993.66 total 460.28 last 27.39) - open/close 9536 (time 1772473003.66 total 470.28 last 19.50) - open/close 9773 (time 1772473013.99 total 480.61 last 22.95) - open/close 9991 (time 1772473024.04 total 490.66 last 21.68) - open/close 10193 (time 1772473034.63 total 501.25 last 19.08) - open/close 10319 (time 1772473045.12 total 511.74 last 12.01) - open/close 10482 (time 1772473056.11 total 522.72 last 14.84) - open/close 10659 (time 1772473066.13 total 532.75 last 17.66) - open/close 10837 (time 1772473076.13 total 542.75 last 17.79) - open/close 10956 (time 1772473086.14 total 552.76 last 11.89) - open/close 11172 (time 1772473096.49 total 563.11 last 20.86) - open/close 11408 (time 1772473108.18 total 574.80 last 20.20) - open/close 11555 (time 1772473118.44 total 585.06 last 14.32) - open/close 11748 (time 1772473128.46 total 595.08 last 19.27) - open/close 11919 (time 1772473138.93 total 605.55 last 16.33) - open/close 12029 (time 1772473149.61 total 616.23 last 10.29) - open/close 12148 (time 1772473159.62 total 626.24 last 11.89) - open/close 12268 (time 1772473169.65 total 636.26 last 11.97) - open/close 12482 (time 1772473179.83 total 646.45 last 21.01) - open/close 12721 (time 1772473190.18 total 656.80 last 23.09) - open/close 12938 (time 1772473200.18 total 666.80 last 21.69) - open/close 13012 (time 1772473210.83 total 677.45 last 6.95) - open/close 13116 (time 1772473220.85 total 687.47 last 10.38) - open/close 13204 (time 1772473230.88 total 697.50 last 8.78) - open/close 13314 (time 1772473241.39 total 708.01 last 10.46) - open/close 13534 (time 1772473252.16 total 718.78 last 20.43) - open/close 13764 (time 1772473262.25 total 728.87 last 22.79) - open/close 13975 (time 1772473272.27 total 738.89 last 21.07) - open/close 14161 (time 1772473283.16 total 749.78 last 17.07) - open/close 14331 (time 1772473293.17 total 759.79 last 16.98) - open/close 14523 (time 1772473303.19 total 769.81 last 19.16) - open/close 14720 (time 1772473313.22 total 779.84 last 19.65) - open/close 14937 (time 1772473323.65 total 790.27 last 20.81) - open/close 15083 (time 1772473333.68 total 800.29 last 14.56) - open/close 15292 (time 1772473344.72 total 811.34 last 18.93) - open/close 15444 (time 1772473355.12 total 821.74 last 14.62) - open/close 15657 (time 1772473365.13 total 831.75 last 21.26) - open/close 15837 (time 1772473376.19 total 842.81 last 16.28) - open/close 15962 (time 1772473386.83 total 853.45 last 11.74) - open/close 16137 (time 1772473397.08 total 863.69 last 17.09) - open/close 16368 (time 1772473407.62 total 874.24 last 21.91) - open/close 16647 (time 1772473417.63 total 884.25 last 27.86) - open/close 16883 (time 1772473428.41 total 895.03 last 21.89) - open/close 17176 (time 1772473439.34 total 905.96 last 26.81) - open/close 17416 (time 1772473449.38 total 916.00 last 23.92) - open/close 17681 (time 1772473459.38 total 926.00 last 26.49) - open/close 17931 (time 1772473470.18 total 936.80 last 23.14) - open/close 18181 (time 1772473481.13 total 947.75 last 22.85) - open/close 18420 (time 1772473491.37 total 957.99 last 23.34) - open/close 18642 (time 1772473501.44 total 968.06 last 22.05) - open/close 18825 (time 1772473511.44 total 978.06 last 18.30) - open/close 19043 (time 1772473522.29 total 988.91 last 20.08) - open/close 19416 (time 1772473532.30 total 998.91 last 37.29) - open/close 20000 (time 1772473538.32 total 1004.94 last 96.98) - open/close 20920 (time 1772473548.32 total 1014.94 last 91.96) - open/close 21909 (time 1772473558.32 total 1024.94 last 98.89) - open/close 22774 (time 1772473568.33 total 1034.95 last 86.45) - open/close 23632 (time 1772473578.33 total 1044.95 last 85.75) - open/close 24660 (time 1772473588.34 total 1054.96 last 102.79) - open/close 25680 (time 1772473598.34 total 1064.96 last 101.93) - open/close 26612 (time 1772473608.36 total 1074.98 last 93.05) - open/close 27595 (time 1772473618.36 total 1084.98 last 98.27) - open/close 28550 (time 1772473628.36 total 1094.98 last 95.49) - open/close 29582 (time 1772473638.37 total 1104.99 last 103.16) - open/close 30000 (time 1772473642.84 total 1109.46 last 93.41) - open/close 31035 (time 1772473652.85 total 1119.47 last 103.39) - open/close 31939 (time 1772473662.86 total 1129.48 last 90.34) - open/close 32932 (time 1772473672.86 total 1139.48 last 99.24) - open/close 33890 (time 1772473682.89 total 1149.50 last 95.59) - open/close 35048 (time 1772473692.89 total 1159.51 last 115.75) - open/close 36187 (time 1772473702.90 total 1169.52 last 113.80) - open/close 37329 (time 1772473712.90 total 1179.52 last 114.13) - open/close 38384 (time 1772473722.92 total 1189.54 last 105.37) - open/close 39315 (time 1772473732.92 total 1199.54 last 93.04) - open/close 40000 (time 1772473739.80 total 1206.42 last 99.61) - open/close 40836 (time 1772473749.81 total 1216.43 last 83.53) - open/close 41874 (time 1772473759.81 total 1226.43 last 103.78) - open/close 42940 (time 1772473769.81 total 1236.43 last 106.59) - open/close 44017 (time 1772473779.83 total 1246.45 last 107.55) - open/close 44993 (time 1772473789.83 total 1256.45 last 97.59) - open/close 46082 (time 1772473799.84 total 1266.46 last 108.77) - open/close 47201 (time 1772473809.84 total 1276.46 last 111.88) - open/close 48117 (time 1772473819.85 total 1286.47 last 91.53) - open/close 48836 (time 1772473829.86 total 1296.48 last 71.84) - open/close 49714 (time 1772473839.86 total 1306.48 last 87.74) total: 50000 open/close in 1309.50 seconds: 38.18 ops/second - unlinked 0 (time 1772473846 ; total 0 ; last 0) - unlinked 10000 (time 1772473939 ; total 93 ; last 93) - unlinked 20000 (time 1772474020 ; total 174 ; last 81) - unlinked 30000 (time 1772474074 ; total 228 ; last 54) - unlinked 40000 (time 1772474127 ; total 281 ; last 53) total: 50000 unlinks in 330 seconds: 151.515152 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg343-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg343-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg343-server stop mds service on oleg343-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg343-server unloading modules via unload_modules_local on: 'oleg343-server' oleg343-server: oleg343-server.virtnet: executing unload_modules_local oleg343-server: modules unloaded. pdsh@oleg343-client: oleg343-client: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-client: ssh exited with exit code 2