== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:00:53 (1789113653) start mds service on oleg305-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg305-server' oleg305-server: oleg305-server.virtnet: executing load_modules_local oleg305-server: Loading modules from /home/green/git/lustre-release/lustre oleg305-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy oleg305-server: detected 4 online CPUs by sysfs oleg305-server: MODOPTS_LIBCFS= oleg305-server: Force libcfs to create 2 CPU partitions oleg305-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg305-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg305-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg305-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0000 oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 214 (time 1789113741.46 total 10.67 last 20.06) - open/close 392 (time 1789113751.51 total 20.72 last 17.71) - open/close 620 (time 1789113762.38 total 31.59 last 20.98) - open/close 831 (time 1789113773.42 total 42.63 last 19.11) - open/close 952 (time 1789113783.59 total 52.79 last 11.90) - open/close 1100 (time 1789113794.46 total 63.66 last 13.61) - open/close 1216 (time 1789113804.81 total 74.02 last 11.20) - open/close 1435 (time 1789113815.58 total 84.78 last 20.35) - open/close 1505 (time 1789113826.50 total 95.71 last 6.41) - open/close 1601 (time 1789113836.54 total 105.74 last 9.57) - open/close 1790 (time 1789113846.56 total 115.76 last 18.86) - open/close 1990 (time 1789113856.56 total 125.76 last 20.00) - open/close 2221 (time 1789113866.58 total 135.79 last 23.05) - open/close 2343 (time 1789113876.77 total 145.97 last 11.97) - open/close 2533 (time 1789113886.87 total 156.07 last 18.82) - open/close 2748 (time 1789113897.29 total 166.50 last 20.62) - open/close 2959 (time 1789113907.68 total 176.88 last 20.31) - open/close 3205 (time 1789113917.72 total 186.92 last 24.50) - open/close 3364 (time 1789113928.45 total 197.65 last 14.82) - open/close 3639 (time 1789113938.46 total 207.66 last 27.46) - open/close 3757 (time 1789113948.48 total 217.69 last 11.77) - open/close 3952 (time 1789113959.06 total 228.26 last 18.44) - open/close 4164 (time 1789113969.10 total 238.30 last 21.12) - open/close 4265 (time 1789113979.10 total 248.30 last 10.10) - open/close 4454 (time 1789113990.20 total 259.40 last 17.03) - open/close 4610 (time 1789114000.84 total 270.04 last 14.66) - open/close 4835 (time 1789114013.91 total 283.12 last 17.20) - open/close 5055 (time 1789114023.93 total 293.13 last 21.97) - open/close 5234 (time 1789114033.94 total 303.15 last 17.87) - open/close 5429 (time 1789114044.94 total 314.14 last 17.74) - open/close 5539 (time 1789114054.94 total 324.15 last 11.00) - open/close 5688 (time 1789114064.94 total 334.15 last 14.90) - open/close 5815 (time 1789114075.21 total 344.41 last 12.37) - open/close 5949 (time 1789114085.28 total 354.48 last 13.31) - open/close 6128 (time 1789114095.29 total 364.50 last 17.87) - open/close 6326 (time 1789114105.46 total 374.67 last 19.47) - open/close 6479 (time 1789114116.18 total 385.38 last 14.28) - open/close 6611 (time 1789114126.20 total 395.40 last 13.17) - open/close 6776 (time 1789114136.48 total 405.68 last 16.05) - open/close 7028 (time 1789114146.50 total 415.70 last 25.16) - open/close 7230 (time 1789114156.64 total 425.84 last 19.91) - open/close 7483 (time 1789114166.65 total 435.85 last 25.29) - open/close 7739 (time 1789114179.22 total 448.42 last 20.36) - open/close 7999 (time 1789114189.22 total 458.43 last 25.98) - open/close 8206 (time 1789114199.52 total 468.72 last 20.11) - open/close 8292 (time 1789114210.08 total 479.28 last 8.14) - open/close 8457 (time 1789114220.25 total 489.46 last 16.22) - open/close 8566 (time 1789114230.26 total 499.46 last 10.90) - open/close 8718 (time 1789114240.59 total 509.79 last 14.71) - open/close 8889 (time 1789114251.19 total 520.40 last 16.13) - open/close 9062 (time 1789114261.22 total 530.43 last 17.25) - open/close 9276 (time 1789114271.23 total 540.43 last 21.39) - open/close 9451 (time 1789114281.23 total 550.44 last 17.49) - open/close 9636 (time 1789114291.25 total 560.45 last 18.47) - open/close 9855 (time 1789114302.19 total 571.39 last 20.03) - open/close 10000 (time 1789114310.42 total 579.63 last 17.61) - open/close 10221 (time 1789114321.05 total 590.25 last 20.80) - open/close 10432 (time 1789114331.05 total 600.25 last 21.10) - open/close 10610 (time 1789114341.07 total 610.27 last 17.76) - open/close 10887 (time 1789114351.62 total 620.83 last 26.25) - open/close 11134 (time 1789114361.64 total 630.84 last 24.66) - open/close 11363 (time 1789114372.55 total 641.76 last 20.98) - open/close 11586 (time 1789114382.87 total 652.07 last 21.62) - open/close 11817 (time 1789114393.70 total 662.90 last 21.33) - open/close 12070 (time 1789114403.70 total 672.91 last 25.29) - open/close 12300 (time 1789114413.92 total 683.12 last 22.51) - open/close 12491 (time 1789114424.78 total 693.99 last 17.58) - open/close 12725 (time 1789114435.36 total 704.56 last 22.12) - open/close 12956 (time 1789114445.37 total 714.58 last 23.07) - open/close 13146 (time 1789114455.52 total 724.73 last 18.72) - open/close 13405 (time 1789114465.54 total 734.75 last 25.85) - open/close 13618 (time 1789114475.92 total 745.12 last 20.53) - open/close 13823 (time 1789114486.17 total 755.37 last 20.01) - open/close 13973 (time 1789114496.53 total 765.73 last 14.48) - open/close 14214 (time 1789114506.53 total 775.74 last 24.09) - open/close 14393 (time 1789114516.85 total 786.06 last 17.34) - open/close 14546 (time 1789114526.87 total 796.07 last 15.28) - open/close 14750 (time 1789114537.42 total 806.63 last 19.32) - open/close 14988 (time 1789114548.00 total 817.20 last 22.50) - open/close 15155 (time 1789114558.67 total 827.87 last 15.65) - open/close 15383 (time 1789114569.50 total 838.70 last 21.05) - open/close 15593 (time 1789114579.99 total 849.19 last 20.02) - open/close 15832 (time 1789114590.00 total 859.21 last 23.87) - open/close 16063 (time 1789114600.59 total 869.80 last 21.81) - open/close 16301 (time 1789114610.59 total 879.80 last 23.80) - open/close 16548 (time 1789114621.53 total 890.74 last 22.58) - open/close 16778 (time 1789114632.34 total 901.55 last 21.28) - open/close 16999 (time 1789114642.51 total 911.71 last 21.74) - open/close 17261 (time 1789114652.51 total 921.71 last 26.20) - open/close 17488 (time 1789114663.07 total 932.27 last 21.50) - open/close 17714 (time 1789114673.42 total 942.63 last 21.82) - open/close 17944 (time 1789114683.44 total 952.65 last 22.95) - open/close 18166 (time 1789114693.84 total 963.05 last 21.35) - open/close 18301 (time 1789114704.03 total 973.23 last 13.25) - open/close 18724 (time 1789114714.03 total 983.24 last 42.28) - open/close 19351 (time 1789114724.04 total 993.24 last 62.67) - open/close 19980 (time 1789114734.05 total 1003.25 last 62.83) - open/close 20568 (time 1789114744.05 total 1013.26 last 58.78) - open/close 21156 (time 1789114754.05 total 1023.26 last 58.79) - open/close 21825 (time 1789114764.07 total 1033.28 last 66.79) - open/close 22492 (time 1789114774.08 total 1043.28 last 66.65) - open/close 23059 (time 1789114784.09 total 1053.29 last 56.63) - open/close 23564 (time 1789114794.09 total 1063.29 last 50.49) - open/close 24204 (time 1789114804.09 total 1073.30 last 63.98) - open/close 24841 (time 1789114814.10 total 1083.30 last 63.66) - open/close 25434 (time 1789114824.12 total 1093.32 last 59.17) - open/close 26130 (time 1789114834.12 total 1103.32 last 69.60) - open/close 26867 (time 1789114844.13 total 1113.33 last 73.65) - open/close 27637 (time 1789114854.14 total 1123.34 last 76.93) - open/close 28279 (time 1789114864.15 total 1133.35 last 64.12) - open/close 28947 (time 1789114874.16 total 1143.36 last 66.74) - open/close 29624 (time 1789114884.17 total 1153.37 last 67.62) - open/close 30000 (time 1789114890.01 total 1159.21 last 64.43) - open/close 30635 (time 1789114900.01 total 1169.21 last 63.48) - open/close 31233 (time 1789114910.01 total 1179.21 last 59.79) - open/close 31784 (time 1789114920.02 total 1189.22 last 55.06) - open/close 32404 (time 1789114930.03 total 1199.24 last 61.91) - open/close 32982 (time 1789114940.03 total 1209.24 last 57.79) - open/close 33598 (time 1789114950.04 total 1219.24 last 61.56) - open/close 34241 (time 1789114960.04 total 1229.25 last 64.27) - open/close 34916 (time 1789114970.04 total 1239.25 last 67.49) - open/close 35572 (time 1789114980.05 total 1249.25 last 65.59) - open/close 36192 (time 1789114990.06 total 1259.26 last 61.94) - open/close 36812 (time 1789115000.07 total 1269.28 last 61.89) - open/close 37434 (time 1789115010.08 total 1279.28 last 62.16) - open/close 38032 (time 1789115020.08 total 1289.28 last 59.79) - open/close 38645 (time 1789115030.09 total 1299.29 last 61.26) - open/close 39243 (time 1789115040.09 total 1309.30 last 59.77) - open/close 39883 (time 1789115050.10 total 1319.31 last 63.92) - open/close 40533 (time 1789115060.10 total 1329.31 last 65.00) - open/close 41107 (time 1789115070.11 total 1339.31 last 57.36) - open/close 41631 (time 1789115080.13 total 1349.33 last 52.32) - open/close 42154 (time 1789115090.13 total 1359.34 last 52.27) - open/close 42671 (time 1789115100.14 total 1369.34 last 51.66) - open/close 43286 (time 1789115110.15 total 1379.35 last 61.46) - open/close 43859 (time 1789115120.16 total 1389.36 last 57.24) - open/close 44457 (time 1789115130.17 total 1399.38 last 59.71) - open/close 45094 (time 1789115140.19 total 1409.40 last 63.56) - open/close 45732 (time 1789115150.20 total 1419.40 last 63.76) - open/close 46369 (time 1789115160.20 total 1429.41 last 63.69) - open/close 46971 (time 1789115170.21 total 1439.42 last 60.13) - open/close 47575 (time 1789115180.22 total 1449.42 last 60.35) - open/close 48126 (time 1789115190.25 total 1459.46 last 54.91) - open/close 48672 (time 1789115200.26 total 1469.47 last 54.55) - open/close 49367 (time 1789115210.27 total 1479.47 last 69.48) total: 50000 open/close in 1489.40 seconds: 33.57 ops/second - unlinked 0 (time 1789115224 ; total 0 ; last 0) - unlinked 10000 (time 1789115397 ; total 173 ; last 173) - unlinked 20000 (time 1789115554 ; total 330 ; last 157) - unlinked 30000 (time 1789115720 ; total 496 ; last 166) - unlinked 40000 (time 1789115884 ; total 660 ; last 164) total: 50000 unlinks in 832 seconds: 60.096153 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg305-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg305-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg305-server stop mds service on oleg305-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg305-server unloading modules via unload_modules_local on: 'oleg305-server' oleg305-server: oleg305-server.virtnet: executing unload_modules_local oleg305-server: modules unloaded. pdsh@oleg305-client: oleg305-client: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-client: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2