== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:51:15 (1787363475) start mds service on oleg155-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg155-server' oleg155-server: oleg155-server.virtnet: executing load_modules_local oleg155-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg155-server: Loading modules from /home/green/git/lustre-release/lustre oleg155-server: detected 4 online CPUs by sysfs oleg155-server: MODOPTS_LIBCFS= oleg155-server: Force libcfs to create 2 CPU partitions oleg155-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg155-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg155-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg155-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-OST0000 oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg155-client.virtnet: -o user_xattr,flock 192.168.201.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 145 (time 1787363565.05 total 10.83 last 13.39) - open/close 244 (time 1787363575.76 total 21.54 last 9.24) - open/close 443 (time 1787363586.31 total 32.09 last 18.87) - open/close 646 (time 1787363596.69 total 42.47 last 19.56) - open/close 826 (time 1787363607.55 total 53.33 last 16.57) - open/close 961 (time 1787363618.24 total 64.02 last 12.63) - open/close 1171 (time 1787363628.25 total 74.03 last 20.99) - open/close 1338 (time 1787363638.27 total 84.05 last 16.67) - open/close 1478 (time 1787363648.28 total 94.06 last 13.98) - open/close 1578 (time 1787363658.30 total 104.08 last 9.97) - open/close 1734 (time 1787363669.25 total 115.03 last 14.25) - open/close 1835 (time 1787363679.27 total 125.05 last 10.08) - open/close 2062 (time 1787363689.65 total 135.43 last 21.88) - open/close 2301 (time 1787363699.68 total 145.46 last 23.82) - open/close 2365 (time 1787363710.06 total 155.84 last 6.17) - open/close 2501 (time 1787363720.89 total 166.67 last 12.56) - open/close 2602 (time 1787363731.07 total 176.85 last 9.92) - open/close 2732 (time 1787363741.39 total 187.17 last 12.59) - open/close 2816 (time 1787363752.97 total 198.75 last 7.25) - open/close 3056 (time 1787363762.99 total 208.77 last 23.96) - open/close 3188 (time 1787363774.00 total 219.78 last 11.98) - open/close 3365 (time 1787363784.03 total 229.81 last 17.64) - open/close 3628 (time 1787363794.06 total 239.84 last 26.23) - open/close 3857 (time 1787363805.05 total 250.83 last 20.84) - open/close 4099 (time 1787363815.81 total 261.59 last 22.48) - open/close 4358 (time 1787363826.81 total 272.59 last 23.55) - open/close 4581 (time 1787363837.02 total 282.80 last 21.84) - open/close 4852 (time 1787363847.04 total 292.82 last 27.06) - open/close 5088 (time 1787363857.13 total 302.91 last 23.39) - open/close 5272 (time 1787363868.02 total 313.80 last 16.89) - open/close 5540 (time 1787363878.03 total 323.81 last 26.77) - open/close 5751 (time 1787363889.27 total 335.05 last 18.77) - open/close 5877 (time 1787363899.27 total 345.05 last 12.60) - open/close 6074 (time 1787363909.30 total 355.08 last 19.64) - open/close 6285 (time 1787363919.86 total 365.64 last 19.99) - open/close 6536 (time 1787363929.87 total 375.65 last 25.07) - open/close 6626 (time 1787363939.88 total 385.66 last 9.00) - open/close 6883 (time 1787363949.96 total 395.74 last 25.48) - open/close 6964 (time 1787363960.00 total 405.78 last 8.07) - open/close 7143 (time 1787363970.62 total 416.40 last 16.85) - open/close 7347 (time 1787363982.76 total 428.54 last 16.81) - open/close 7588 (time 1787363992.77 total 438.55 last 24.07) - open/close 7786 (time 1787364003.01 total 448.79 last 19.34) - open/close 7988 (time 1787364013.12 total 458.90 last 19.99) - open/close 8176 (time 1787364025.34 total 471.12 last 15.38) - open/close 8376 (time 1787364036.09 total 481.87 last 18.61) - open/close 8423 (time 1787364046.65 total 492.43 last 4.45) - open/close 8538 (time 1787364056.65 total 502.43 last 11.50) - open/close 8746 (time 1787364067.73 total 513.51 last 18.77) - open/close 8980 (time 1787364078.54 total 524.32 last 21.65) - open/close 9188 (time 1787364088.55 total 534.33 last 20.77) - open/close 9348 (time 1787364098.88 total 544.66 last 15.49) - open/close 9455 (time 1787364109.50 total 555.28 last 10.08) - open/close 9638 (time 1787364120.60 total 566.38 last 16.49) - open/close 9774 (time 1787364131.34 total 577.12 last 12.66) - open/close 9965 (time 1787364141.35 total 587.13 last 19.09) - open/close 10000 (time 1787364144.24 total 590.02 last 12.08) - open/close 10187 (time 1787364154.29 total 600.07 last 18.62) - open/close 10365 (time 1787364164.31 total 610.09 last 17.76) - open/close 10622 (time 1787364174.33 total 620.11 last 25.65) - open/close 10837 (time 1787364184.45 total 630.23 last 21.25) - open/close 11004 (time 1787364194.82 total 640.60 last 16.10) - open/close 11181 (time 1787364206.77 total 652.55 last 14.81) - open/close 11311 (time 1787364217.22 total 663.00 last 12.43) - open/close 11357 (time 1787364227.68 total 673.46 last 4.40) - open/close 11540 (time 1787364238.43 total 684.21 last 17.03) - open/close 11775 (time 1787364248.88 total 694.66 last 22.48) - open/close 11992 (time 1787364259.19 total 704.97 last 21.05) - open/close 12108 (time 1787364269.50 total 715.28 last 11.26) - open/close 12204 (time 1787364280.20 total 725.98 last 8.97) - open/close 12368 (time 1787364290.22 total 736.00 last 16.38) - open/close 12541 (time 1787364300.24 total 746.02 last 17.26) - open/close 12828 (time 1787364311.18 total 756.96 last 26.24) - open/close 12989 (time 1787364321.33 total 767.11 last 15.87) - open/close 13171 (time 1787364331.33 total 777.11 last 18.20) - open/close 13346 (time 1787364342.41 total 788.19 last 15.79) - open/close 13574 (time 1787364353.08 total 798.86 last 21.36) - open/close 13813 (time 1787364363.47 total 809.25 last 23.01) - open/close 13883 (time 1787364374.02 total 819.80 last 6.64) - open/close 14097 (time 1787364384.04 total 829.82 last 21.36) - open/close 14273 (time 1787364394.64 total 840.42 last 16.60) - open/close 14552 (time 1787364404.64 total 850.42 last 27.89) - open/close 14763 (time 1787364414.92 total 860.70 last 20.53) - open/close 15004 (time 1787364425.97 total 871.75 last 21.81) - open/close 15092 (time 1787364436.12 total 881.90 last 8.67) - open/close 15291 (time 1787364446.17 total 891.95 last 19.81) - open/close 15530 (time 1787364456.19 total 901.97 last 23.83) - open/close 15749 (time 1787364466.27 total 912.05 last 21.73) - open/close 15982 (time 1787364476.79 total 922.57 last 22.15) - open/close 16228 (time 1787364486.80 total 932.58 last 24.58) - open/close 16422 (time 1787364497.53 total 943.31 last 18.08) - open/close 16606 (time 1787364507.97 total 953.75 last 17.61) - open/close 16776 (time 1787364518.65 total 964.43 last 15.92) - open/close 17041 (time 1787364529.20 total 974.98 last 25.11) - open/close 17234 (time 1787364539.83 total 985.61 last 18.16) - open/close 17426 (time 1787364550.80 total 996.58 last 17.51) - open/close 17710 (time 1787364560.93 total 1006.71 last 28.04) - open/close 17991 (time 1787364570.93 total 1016.71 last 28.10) - open/close 18873 (time 1787364580.94 total 1026.72 last 88.08) - open/close 19776 (time 1787364590.96 total 1036.74 last 90.09) - open/close 20000 (time 1787364593.28 total 1039.06 last 96.44) - open/close 20915 (time 1787364603.29 total 1049.07 last 91.42) - open/close 21742 (time 1787364613.29 total 1059.07 last 82.70) - open/close 22637 (time 1787364623.29 total 1069.07 last 89.49) - open/close 23360 (time 1787364633.30 total 1079.08 last 72.25) - open/close 24255 (time 1787364643.32 total 1089.10 last 89.33) - open/close 25086 (time 1787364653.33 total 1099.11 last 83.01) - open/close 25937 (time 1787364663.34 total 1109.12 last 85.06) - open/close 26855 (time 1787364673.34 total 1119.12 last 91.73) - open/close 27737 (time 1787364683.35 total 1129.13 last 88.11) - open/close 28629 (time 1787364693.36 total 1139.14 last 89.16) - open/close 29516 (time 1787364703.37 total 1149.15 last 88.58) - open/close 30000 (time 1787364710.02 total 1155.80 last 72.84) - open/close 30846 (time 1787364720.02 total 1165.80 last 84.59) - open/close 31718 (time 1787364730.03 total 1175.81 last 87.12) - open/close 32589 (time 1787364740.03 total 1185.81 last 87.08) - open/close 33504 (time 1787364750.03 total 1195.81 last 91.49) - open/close 34287 (time 1787364760.04 total 1205.82 last 78.21) - open/close 34918 (time 1787364770.05 total 1215.83 last 63.06) - open/close 35794 (time 1787364780.05 total 1225.83 last 87.56) - open/close 36671 (time 1787364790.06 total 1235.84 last 87.62) - open/close 37605 (time 1787364800.06 total 1245.84 last 93.36) - open/close 38485 (time 1787364810.07 total 1255.85 last 87.98) - open/close 39357 (time 1787364820.07 total 1265.85 last 87.19) - open/close 40000 (time 1787364827.68 total 1273.46 last 84.49) - open/close 40922 (time 1787364837.71 total 1283.49 last 91.88) - open/close 41845 (time 1787364847.71 total 1293.49 last 92.28) - open/close 42739 (time 1787364857.71 total 1303.49 last 89.40) - open/close 43574 (time 1787364867.72 total 1313.50 last 83.49) - open/close 44496 (time 1787364877.72 total 1323.50 last 92.18) - open/close 45382 (time 1787364887.72 total 1333.50 last 88.56) - open/close 46260 (time 1787364897.74 total 1343.52 last 87.67) - open/close 46896 (time 1787364907.76 total 1353.54 last 63.48) - open/close 47765 (time 1787364917.76 total 1363.54 last 86.82) - open/close 48515 (time 1787364927.77 total 1373.55 last 74.95) - open/close 49319 (time 1787364937.78 total 1383.56 last 80.35) total: 50000 open/close in 1391.66 seconds: 35.93 ops/second - unlinked 0 (time 1787364949 ; total 0 ; last 0) - unlinked 10000 (time 1787365066 ; total 117 ; last 117) - unlinked 20000 (time 1787365172 ; total 223 ; last 106) - unlinked 30000 (time 1787365243 ; total 294 ; last 71) - unlinked 40000 (time 1787365320 ; total 371 ; last 77) total: 50000 unlinks in 442 seconds: 113.122169 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg155-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg155-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg155-server stop mds service on oleg155-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg155-server unloading modules via unload_modules_local on: 'oleg155-server' oleg155-server: oleg155-server.virtnet: executing unload_modules_local oleg155-server: modules unloaded. pdsh@oleg155-client: oleg155-client: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-server: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-client: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-server: ssh exited with exit code 2