== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:11:37 (1788444697) start mds service on oleg606-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg606-server' oleg606-server: oleg606-server.virtnet: executing load_modules_local oleg606-server: Loading modules from /home/green/git/lustre-release/lustre oleg606-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg606-server: detected 4 online CPUs by sysfs oleg606-server: MODOPTS_LIBCFS= oleg606-server: Force libcfs to create 2 CPU partitions oleg606-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg606-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg606-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg606-server: oleg606-server.virtnet: executing set_default_debug -1 all pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg606-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg606-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg606-server: oleg606-server.virtnet: executing set_default_debug -1 all pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Started lustre-OST0000 oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg606-client.virtnet: -o user_xattr,flock 192.168.206.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 250 (time 1788444779.85 total 10.46 last 23.90) - open/close 477 (time 1788444789.96 total 20.57 last 22.45) - open/close 659 (time 1788444800.08 total 30.69 last 17.99) - open/close 765 (time 1788444810.96 total 41.58 last 9.74) - open/close 884 (time 1788444821.71 total 52.32 last 11.07) - open/close 1101 (time 1788444832.00 total 62.61 last 21.10) - open/close 1327 (time 1788444842.64 total 73.25 last 21.24) - open/close 1577 (time 1788444852.79 total 83.40 last 24.62) - open/close 1820 (time 1788444863.24 total 93.85 last 23.26) - open/close 2093 (time 1788444873.24 total 103.85 last 27.29) - open/close 2349 (time 1788444883.24 total 113.86 last 25.59) - open/close 2427 (time 1788444893.26 total 123.87 last 7.79) - open/close 2628 (time 1788444903.63 total 134.24 last 19.38) - open/close 2799 (time 1788444913.63 total 144.24 last 17.10) - open/close 3065 (time 1788444923.84 total 154.45 last 26.05) - open/close 3228 (time 1788444935.62 total 166.23 last 13.84) - open/close 3405 (time 1788444946.58 total 177.19 last 16.15) - open/close 3655 (time 1788444957.52 total 188.13 last 22.85) - open/close 3877 (time 1788444967.53 total 198.14 last 22.18) - open/close 4061 (time 1788444977.79 total 208.40 last 17.93) - open/close 4296 (time 1788444987.80 total 218.41 last 23.47) - open/close 4539 (time 1788444997.97 total 228.58 last 23.90) - open/close 4770 (time 1788445008.19 total 238.80 last 22.61) - open/close 5015 (time 1788445019.30 total 249.92 last 22.04) - open/close 5140 (time 1788445029.45 total 260.06 last 12.32) - open/close 5349 (time 1788445040.41 total 271.02 last 19.08) - open/close 5605 (time 1788445050.43 total 281.04 last 25.53) - open/close 5882 (time 1788445061.31 total 291.92 last 25.46) - open/close 6128 (time 1788445071.70 total 302.31 last 23.67) - open/close 6350 (time 1788445081.71 total 312.32 last 22.19) - open/close 6513 (time 1788445091.92 total 322.53 last 15.96) - open/close 6742 (time 1788445102.87 total 333.48 last 20.91) - open/close 6947 (time 1788445115.16 total 345.77 last 16.68) - open/close 7195 (time 1788445126.15 total 356.77 last 22.56) - open/close 7495 (time 1788445136.15 total 366.77 last 30.00) - open/close 7705 (time 1788445146.16 total 376.78 last 20.98) - open/close 7913 (time 1788445156.18 total 386.79 last 20.77) - open/close 8124 (time 1788445166.46 total 397.08 last 20.51) - open/close 8308 (time 1788445176.48 total 407.09 last 18.37) - open/close 8468 (time 1788445189.31 total 419.92 last 12.47) - open/close 8689 (time 1788445199.56 total 430.18 last 21.55) - open/close 8888 (time 1788445210.53 total 441.14 last 18.15) - open/close 9054 (time 1788445220.55 total 451.16 last 16.56) - open/close 9191 (time 1788445231.50 total 462.11 last 12.51) - open/close 9277 (time 1788445242.41 total 473.02 last 7.88) - open/close 9394 (time 1788445252.47 total 483.09 last 11.63) - open/close 9596 (time 1788445262.51 total 493.12 last 20.14) - open/close 9767 (time 1788445272.77 total 503.38 last 16.67) - open/close 10000 (time 1788445280.82 total 511.44 last 28.92) - open/close 10228 (time 1788445291.01 total 521.62 last 22.39) - open/close 10471 (time 1788445301.21 total 531.82 last 23.81) - open/close 10711 (time 1788445311.22 total 541.83 last 23.98) - open/close 10965 (time 1788445322.24 total 552.86 last 23.04) - open/close 11142 (time 1788445333.18 total 563.79 last 16.19) - open/close 11342 (time 1788445344.14 total 574.75 last 18.25) - open/close 11554 (time 1788445354.88 total 585.50 last 19.73) - open/close 11769 (time 1788445364.89 total 595.50 last 21.49) - open/close 11921 (time 1788445375.44 total 606.05 last 14.41) - open/close 12144 (time 1788445385.62 total 616.24 last 21.89) - open/close 12384 (time 1788445396.33 total 626.94 last 22.42) - open/close 12581 (time 1788445406.34 total 636.95 last 19.69) - open/close 12715 (time 1788445416.35 total 646.97 last 13.38) - open/close 12809 (time 1788445427.17 total 657.78 last 8.69) - open/close 12916 (time 1788445437.71 total 668.33 last 10.15) - open/close 13062 (time 1788445448.66 total 679.28 last 13.33) - open/close 13284 (time 1788445458.67 total 689.29 last 22.18) - open/close 13473 (time 1788445469.70 total 700.31 last 17.15) - open/close 13592 (time 1788445479.95 total 710.56 last 11.61) - open/close 13779 (time 1788445489.95 total 720.56 last 18.69) - open/close 14007 (time 1788445500.68 total 731.29 last 21.25) - open/close 14253 (time 1788445511.44 total 742.05 last 22.87) - open/close 14445 (time 1788445523.93 total 754.54 last 15.38) - open/close 14607 (time 1788445533.93 total 764.54 last 16.20) - open/close 14722 (time 1788445543.94 total 774.55 last 11.49) - open/close 14939 (time 1788445553.96 total 784.57 last 21.66) - open/close 15152 (time 1788445563.97 total 794.58 last 21.27) - open/close 15290 (time 1788445574.35 total 804.97 last 13.29) - open/close 15415 (time 1788445584.37 total 814.98 last 12.48) - open/close 15665 (time 1788445595.27 total 825.88 last 22.93) - open/close 15834 (time 1788445605.28 total 835.89 last 16.90) - open/close 15955 (time 1788445615.28 total 845.89 last 12.09) - open/close 16203 (time 1788445626.11 total 856.73 last 22.89) - open/close 16282 (time 1788445636.43 total 867.04 last 7.66) - open/close 16405 (time 1788445646.45 total 877.07 last 12.27) - open/close 16624 (time 1788445656.47 total 887.08 last 21.87) - open/close 16737 (time 1788445666.47 total 897.08 last 11.29) - open/close 16917 (time 1788445676.76 total 907.37 last 17.49) - open/close 17147 (time 1788445687.37 total 917.99 last 21.67) - open/close 17373 (time 1788445697.37 total 927.99 last 22.60) - open/close 17530 (time 1788445709.78 total 940.39 last 12.66) - open/close 17703 (time 1788445720.71 total 951.32 last 15.82) - open/close 18261 (time 1788445730.71 total 961.32 last 55.79) - open/close 18922 (time 1788445740.72 total 971.33 last 66.06) - open/close 19530 (time 1788445750.73 total 981.34 last 60.74) - open/close 20000 (time 1788445758.01 total 988.62 last 64.59) - open/close 20691 (time 1788445768.02 total 998.63 last 69.03) - open/close 21328 (time 1788445778.03 total 1008.64 last 63.59) - open/close 21970 (time 1788445788.04 total 1018.65 last 64.17) - open/close 22638 (time 1788445798.04 total 1028.65 last 66.77) - open/close 23167 (time 1788445808.06 total 1038.67 last 52.81) - open/close 23826 (time 1788445818.06 total 1048.68 last 65.86) - open/close 24527 (time 1788445828.08 total 1058.69 last 69.98) - open/close 25246 (time 1788445838.09 total 1068.70 last 71.84) - open/close 25940 (time 1788445848.10 total 1078.72 last 69.31) - open/close 26613 (time 1788445858.11 total 1088.72 last 67.25) - open/close 27252 (time 1788445868.12 total 1098.73 last 63.86) - open/close 27891 (time 1788445878.14 total 1108.75 last 63.74) - open/close 28561 (time 1788445888.15 total 1118.77 last 66.92) - open/close 29144 (time 1788445898.16 total 1128.77 last 58.27) - open/close 29779 (time 1788445908.18 total 1138.79 last 63.38) - open/close 30000 (time 1788445911.48 total 1142.09 last 67.02) - open/close 30691 (time 1788445921.49 total 1152.10 last 69.01) - open/close 31329 (time 1788445931.51 total 1162.12 last 63.69) - open/close 31982 (time 1788445941.52 total 1172.13 last 65.20) - open/close 32637 (time 1788445951.60 total 1182.21 last 65.01) - open/close 33186 (time 1788445961.61 total 1192.22 last 54.85) - open/close 33807 (time 1788445971.62 total 1202.23 last 62.00) - open/close 34365 (time 1788445981.63 total 1212.24 last 55.77) - open/close 34961 (time 1788445991.64 total 1222.25 last 59.51) - open/close 35608 (time 1788446001.65 total 1232.26 last 64.67) - open/close 36260 (time 1788446011.65 total 1242.27 last 65.15) - open/close 36868 (time 1788446021.66 total 1252.27 last 60.78) - open/close 37538 (time 1788446031.67 total 1262.28 last 66.93) - open/close 38224 (time 1788446041.67 total 1272.28 last 68.58) - open/close 38953 (time 1788446051.67 total 1282.29 last 72.87) - open/close 39646 (time 1788446061.68 total 1292.29 last 69.26) - open/close 40000 (time 1788446066.92 total 1297.53 last 67.59) - open/close 40616 (time 1788446076.92 total 1307.53 last 61.60) - open/close 41232 (time 1788446086.93 total 1317.54 last 61.52) - open/close 41918 (time 1788446096.94 total 1327.55 last 68.55) - open/close 42600 (time 1788446106.95 total 1337.56 last 68.11) - open/close 43307 (time 1788446116.96 total 1347.57 last 70.66) - open/close 43999 (time 1788446126.97 total 1357.58 last 69.12) - open/close 44704 (time 1788446136.97 total 1367.58 last 70.49) - open/close 45413 (time 1788446146.97 total 1377.58 last 70.88) - open/close 46052 (time 1788446156.98 total 1387.59 last 63.85) - open/close 46692 (time 1788446166.98 total 1397.59 last 63.98) - open/close 47403 (time 1788446177.00 total 1407.61 last 71.00) - open/close 48099 (time 1788446187.01 total 1417.62 last 69.52) - open/close 48762 (time 1788446197.01 total 1427.63 last 66.25) - open/close 49389 (time 1788446207.02 total 1437.63 last 62.69) - open/close 49999 (time 1788446217.04 total 1447.65 last 60.88) total: 50000 open/close in 1447.65 seconds: 34.54 ops/second - unlinked 0 (time 1788446223 ; total 0 ; last 0) - unlinked 10000 (time 1788446380 ; total 157 ; last 157) - unlinked 20000 (time 1788446538 ; total 315 ; last 158) - unlinked 30000 (time 1788446698 ; total 475 ; last 160) - unlinked 40000 (time 1788446855 ; total 632 ; last 157) total: 50000 unlinks in 787 seconds: 63.532402 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg606-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg606-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg606-server stop mds service on oleg606-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg606-server unloading modules via unload_modules_local on: 'oleg606-server' oleg606-server: oleg606-server.virtnet: executing unload_modules_local oleg606-server: modules unloaded. pdsh@oleg606-client: oleg606-client: ssh exited with exit code 2 pdsh@oleg606-client: oleg606-server: ssh exited with exit code 2 pdsh@oleg606-client: oleg606-client: ssh exited with exit code 2 pdsh@oleg606-client: oleg606-server: ssh exited with exit code 2