== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:13:15 (1780992795) start mds service on oleg234-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing load_modules_local oleg234-server: Loading modules from /home/green/git/lustre-release/lustre oleg234-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg234-server: detected 4 online CPUs by sysfs oleg234-server: MODOPTS_LIBCFS= oleg234-server: Force libcfs to create 2 CPU partitions oleg234-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg234-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg234-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg234-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 245 (time 1780992873.79 total 10.50 last 23.33) - open/close 326 (time 1780992883.81 total 20.52 last 8.09) - open/close 444 (time 1780992894.29 total 31.00 last 11.26) - open/close 618 (time 1780992904.72 total 41.42 last 16.69) - open/close 806 (time 1780992917.66 total 54.36 last 14.53) - open/close 1022 (time 1780992927.67 total 64.37 last 21.58) - open/close 1272 (time 1780992938.69 total 75.40 last 22.67) - open/close 1521 (time 1780992949.60 total 86.30 last 22.84) - open/close 1733 (time 1780992959.65 total 96.35 last 21.09) - open/close 1894 (time 1780992970.49 total 107.19 last 14.85) - open/close 2082 (time 1780992980.56 total 117.26 last 18.67) - open/close 2156 (time 1780992990.56 total 127.27 last 7.39) - open/close 2306 (time 1780993000.59 total 137.30 last 14.95) - open/close 2413 (time 1780993010.60 total 147.30 last 10.70) - open/close 2654 (time 1780993020.61 total 157.31 last 24.08) - open/close 2839 (time 1780993030.63 total 167.33 last 18.47) - open/close 3072 (time 1780993040.63 total 177.34 last 23.29) - open/close 3308 (time 1780993051.58 total 188.29 last 21.55) - open/close 3465 (time 1780993061.82 total 198.53 last 15.33) - open/close 3648 (time 1780993074.06 total 210.77 last 14.95) - open/close 3904 (time 1780993084.23 total 220.93 last 25.18) - open/close 4163 (time 1780993094.25 total 230.96 last 25.84) - open/close 4373 (time 1780993105.09 total 241.80 last 19.38) - open/close 4539 (time 1780993115.40 total 252.11 last 16.10) - open/close 4612 (time 1780993125.44 total 262.15 last 7.27) - open/close 4784 (time 1780993136.14 total 272.85 last 16.08) - open/close 5002 (time 1780993146.19 total 282.89 last 21.70) - open/close 5175 (time 1780993156.41 total 293.11 last 16.93) - open/close 5293 (time 1780993166.45 total 303.16 last 11.75) - open/close 5467 (time 1780993176.46 total 313.16 last 17.39) - open/close 5650 (time 1780993186.82 total 323.52 last 17.67) - open/close 5747 (time 1780993197.66 total 334.37 last 8.94) - open/close 5868 (time 1780993207.70 total 344.41 last 12.05) - open/close 6093 (time 1780993218.69 total 355.40 last 20.47) - open/close 6344 (time 1780993231.56 total 368.27 last 19.50) - open/close 6501 (time 1780993242.48 total 379.19 last 14.37) - open/close 6732 (time 1780993253.07 total 389.77 last 21.83) - open/close 7011 (time 1780993263.42 total 400.13 last 26.94) - open/close 7196 (time 1780993273.43 total 410.14 last 18.48) - open/close 7386 (time 1780993283.45 total 420.16 last 18.97) - open/close 7447 (time 1780993293.46 total 430.17 last 6.09) - open/close 7609 (time 1780993304.27 total 440.97 last 14.99) - open/close 7761 (time 1780993314.28 total 450.98 last 15.19) - open/close 7970 (time 1780993325.25 total 461.96 last 19.04) - open/close 8188 (time 1780993335.43 total 472.14 last 21.41) - open/close 8384 (time 1780993346.06 total 482.77 last 18.45) - open/close 8525 (time 1780993356.62 total 493.33 last 13.35) - open/close 8708 (time 1780993366.62 total 503.33 last 18.30) - open/close 8791 (time 1780993376.71 total 513.42 last 8.22) - open/close 8967 (time 1780993386.73 total 523.44 last 17.57) - open/close 9189 (time 1780993397.07 total 533.78 last 21.46) - open/close 9393 (time 1780993407.09 total 543.80 last 20.36) - open/close 9540 (time 1780993417.17 total 553.87 last 14.59) - open/close 9696 (time 1780993427.34 total 564.05 last 15.33) - open/close 9874 (time 1780993438.36 total 575.07 last 16.15) - open/close 10000 (time 1780993445.50 total 582.20 last 17.66) - open/close 10141 (time 1780993455.95 total 592.66 last 13.48) - open/close 10245 (time 1780993466.77 total 603.47 last 9.62) - open/close 10368 (time 1780993476.80 total 613.51 last 12.26) - open/close 10450 (time 1780993487.11 total 623.82 last 7.95) - open/close 10597 (time 1780993497.11 total 633.82 last 14.70) - open/close 10754 (time 1780993508.22 total 644.93 last 14.13) - open/close 10903 (time 1780993519.25 total 655.95 last 13.52) - open/close 10970 (time 1780993532.17 total 668.88 last 5.18) - open/close 11088 (time 1780993544.13 total 680.84 last 9.87) - open/close 11250 (time 1780993554.14 total 690.85 last 16.18) - open/close 11356 (time 1780993564.19 total 700.90 last 10.55) - open/close 11472 (time 1780993575.25 total 711.95 last 10.49) - open/close 11645 (time 1780993585.30 total 722.00 last 17.22) - open/close 11850 (time 1780993598.02 total 734.73 last 16.11) - open/close 11989 (time 1780993608.13 total 744.84 last 13.75) - open/close 12183 (time 1780993618.32 total 755.02 last 19.05) - open/close 12339 (time 1780993628.87 total 765.58 last 14.78) - open/close 12407 (time 1780993639.83 total 776.53 last 6.21) - open/close 12496 (time 1780993650.25 total 786.96 last 8.54) - open/close 12605 (time 1780993660.26 total 796.97 last 10.89) - open/close 12714 (time 1780993671.30 total 808.01 last 9.88) - open/close 12839 (time 1780993681.31 total 818.02 last 12.49) - open/close 12885 (time 1780993691.74 total 828.45 last 4.41) - open/close 13053 (time 1780993702.74 total 839.44 last 15.28) - open/close 13213 (time 1780993713.76 total 850.46 last 14.52) - open/close 13313 (time 1780993724.56 total 861.26 last 9.26) - open/close 13423 (time 1780993735.04 total 871.75 last 10.49) - open/close 13524 (time 1780993745.43 total 882.13 last 9.72) - open/close 13659 (time 1780993755.91 total 892.62 last 12.87) - open/close 13716 (time 1780993766.84 total 903.55 last 5.22) - open/close 13931 (time 1780993777.88 total 914.58 last 19.49) - open/close 14044 (time 1780993787.90 total 924.61 last 11.28) - open/close 14128 (time 1780993797.91 total 934.61 last 8.39) - open/close 14233 (time 1780993808.02 total 944.73 last 10.38) - open/close 14368 (time 1780993818.57 total 955.28 last 12.80) - open/close 14460 (time 1780993828.61 total 965.31 last 9.17) - open/close 14627 (time 1780993838.61 total 975.32 last 16.69) - open/close 14708 (time 1780993850.86 total 987.57 last 6.61) - open/close 14827 (time 1780993861.02 total 997.73 last 11.72) - open/close 14990 (time 1780993871.03 total 1007.73 last 16.29) - open/close 15061 (time 1780993881.05 total 1017.76 last 7.08) - open/close 15150 (time 1780993891.40 total 1028.11 last 8.60) - open/close 15221 (time 1780993901.41 total 1038.11 last 7.10) - open/close 15350 (time 1780993911.63 total 1048.33 last 12.62) - open/close 15464 (time 1780993921.74 total 1058.45 last 11.27) - open/close 15627 (time 1780993931.76 total 1068.47 last 16.27) - open/close 15767 (time 1780993941.77 total 1078.48 last 13.99) - open/close 15920 (time 1780993951.88 total 1088.59 last 15.13) - open/close 16029 (time 1780993961.88 total 1098.59 last 10.90) - open/close 16084 (time 1780993972.75 total 1109.46 last 5.06) - open/close 16666 (time 1780993982.76 total 1119.46 last 58.16) - open/close 17165 (time 1780993992.76 total 1129.46 last 49.90) - open/close 17800 (time 1780994002.78 total 1139.49 last 63.35) - open/close 18583 (time 1780994012.79 total 1149.50 last 78.21) - open/close 19362 (time 1780994022.80 total 1159.50 last 77.87) - open/close 20000 (time 1780994032.20 total 1168.91 last 67.83) - open/close 20714 (time 1780994042.21 total 1178.92 last 71.34) - open/close 21447 (time 1780994052.22 total 1188.92 last 73.24) - open/close 22031 (time 1780994062.23 total 1198.93 last 58.34) - open/close 22735 (time 1780994072.24 total 1208.94 last 70.34) - open/close 23402 (time 1780994082.25 total 1218.95 last 66.62) - open/close 24035 (time 1780994092.26 total 1228.97 last 63.22) - open/close 24610 (time 1780994102.28 total 1238.98 last 57.42) - open/close 25121 (time 1780994112.28 total 1248.99 last 51.07) - open/close 25862 (time 1780994122.30 total 1259.01 last 73.96) - open/close 26591 (time 1780994132.30 total 1269.01 last 72.89) - open/close 27260 (time 1780994142.31 total 1279.02 last 66.83) - open/close 27913 (time 1780994152.34 total 1289.05 last 65.13) - open/close 28584 (time 1780994162.34 total 1299.05 last 67.10) - open/close 29329 (time 1780994172.35 total 1309.05 last 74.43) - open/close 30000 (time 1780994182.09 total 1318.80 last 68.88) - open/close 30703 (time 1780994192.09 total 1328.80 last 70.27) - open/close 31398 (time 1780994202.11 total 1338.82 last 69.36) - open/close 32105 (time 1780994212.12 total 1348.82 last 70.67) - open/close 32899 (time 1780994222.13 total 1358.83 last 79.33) - open/close 33565 (time 1780994232.13 total 1368.83 last 66.59) - open/close 34189 (time 1780994242.15 total 1378.85 last 62.28) - open/close 34679 (time 1780994252.17 total 1388.88 last 48.89) - open/close 35181 (time 1780994262.18 total 1398.89 last 50.15) - open/close 35952 (time 1780994272.18 total 1408.89 last 77.09) - open/close 36712 (time 1780994282.19 total 1418.90 last 75.94) - open/close 37376 (time 1780994292.19 total 1428.90 last 66.40) - open/close 38025 (time 1780994302.19 total 1438.90 last 64.90) - open/close 38706 (time 1780994312.20 total 1448.90 last 68.07) - open/close 39284 (time 1780994322.20 total 1458.91 last 57.76) - open/close 40000 (time 1780994332.02 total 1468.73 last 72.93) - open/close 40687 (time 1780994342.03 total 1478.73 last 68.66) - open/close 41360 (time 1780994352.03 total 1488.74 last 67.24) - open/close 41978 (time 1780994362.04 total 1498.74 last 61.79) - open/close 42647 (time 1780994372.04 total 1508.75 last 66.85) - open/close 43392 (time 1780994382.05 total 1518.75 last 74.49) - open/close 43986 (time 1780994392.06 total 1528.76 last 59.33) - open/close 44643 (time 1780994402.07 total 1538.78 last 65.61) - open/close 45223 (time 1780994412.13 total 1548.84 last 57.64) - open/close 45940 (time 1780994422.14 total 1558.85 last 71.65) - open/close 46600 (time 1780994432.14 total 1568.85 last 65.98) - open/close 47352 (time 1780994442.15 total 1578.86 last 75.14) - open/close 48061 (time 1780994452.16 total 1588.87 last 70.84) - open/close 48822 (time 1780994462.17 total 1598.87 last 76.04) - open/close 49414 (time 1780994472.18 total 1608.89 last 59.11) - open/close 49965 (time 1780994482.18 total 1618.89 last 55.10) total: 50000 open/close in 1619.35 seconds: 30.88 ops/second - unlinked 0 (time 1780994487 ; total 0 ; last 0) - unlinked 10000 (time 1780994631 ; total 144 ; last 144) - unlinked 20000 (time 1780994804 ; total 317 ; last 173) - unlinked 30000 (time 1780994969 ; total 482 ; last 165) - unlinked 40000 (time 1780995139 ; total 652 ; last 170) total: 50000 unlinks in 804 seconds: 62.189056 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg234-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg234-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg234-server unloading modules via unload_modules_local on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing unload_modules_local oleg234-server: modules unloaded. pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2