== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:59:59 (1788271199) start mds service on oleg151-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing load_modules_local oleg151-server: Loading modules from /home/green/git/lustre-release/lustre oleg151-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg151-server: detected 4 online CPUs by sysfs oleg151-server: MODOPTS_LIBCFS= oleg151-server: Force libcfs to create 2 CPU partitions oleg151-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg151-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg151-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg151-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 645 (time 1788271237.77 total 10.00 last 64.50) - open/close 1320 (time 1788271248.80 total 21.03 last 61.18) - open/close 1978 (time 1788271259.81 total 32.04 last 59.77) - open/close 2323 (time 1788271270.44 total 42.67 last 32.47) - open/close 2561 (time 1788271280.80 total 53.03 last 22.97) - open/close 2851 (time 1788271291.04 total 63.27 last 28.32) - open/close 3030 (time 1788271301.67 total 73.90 last 16.84) - open/close 3326 (time 1788271312.68 total 84.91 last 26.90) - open/close 3594 (time 1788271323.49 total 95.72 last 24.78) - open/close 3830 (time 1788271334.44 total 106.67 last 21.57) - open/close 4047 (time 1788271345.19 total 117.42 last 20.18) - open/close 4349 (time 1788271355.19 total 127.42 last 30.19) - open/close 4646 (time 1788271365.35 total 137.58 last 29.25) - open/close 4962 (time 1788271375.46 total 147.69 last 31.23) - open/close 5560 (time 1788271385.64 total 157.87 last 58.79) - open/close 5954 (time 1788271397.41 total 169.64 last 33.45) - open/close 6187 (time 1788271407.78 total 180.01 last 22.48) - open/close 6369 (time 1788271418.02 total 190.25 last 17.77) - open/close 6764 (time 1788271428.07 total 200.30 last 39.31) - open/close 6961 (time 1788271438.76 total 210.99 last 18.44) - open/close 7199 (time 1788271449.17 total 221.40 last 22.85) - open/close 7489 (time 1788271459.31 total 231.54 last 28.60) - open/close 7708 (time 1788271470.08 total 242.31 last 20.35) - open/close 7959 (time 1788271480.94 total 253.17 last 23.10) - open/close 8265 (time 1788271490.95 total 263.18 last 30.58) - open/close 8396 (time 1788271501.24 total 273.47 last 12.73) - open/close 8714 (time 1788271511.46 total 283.69 last 31.10) - open/close 8826 (time 1788271522.19 total 294.42 last 10.44) - open/close 9001 (time 1788271533.25 total 305.48 last 15.82) - open/close 9123 (time 1788271543.54 total 315.77 last 11.85) - open/close 9197 (time 1788271553.79 total 326.02 last 7.22) - open/close 9284 (time 1788271563.90 total 336.13 last 8.60) - open/close 9407 (time 1788271574.04 total 346.27 last 12.14) - open/close 9519 (time 1788271584.06 total 356.29 last 11.18) - open/close 9653 (time 1788271595.00 total 367.23 last 12.24) - open/close 9772 (time 1788271605.02 total 377.25 last 11.88) - open/close 9941 (time 1788271617.09 total 389.32 last 14.01) - open/close 10000 (time 1788271620.15 total 392.38 last 19.22) - open/close 10199 (time 1788271630.25 total 402.48 last 19.71) - open/close 10384 (time 1788271640.27 total 412.50 last 18.47) - open/close 10608 (time 1788271650.55 total 422.78 last 21.79) - open/close 10839 (time 1788271661.37 total 433.60 last 21.35) - open/close 11111 (time 1788271671.37 total 443.60 last 27.20) - open/close 11321 (time 1788271681.40 total 453.63 last 20.93) - open/close 11580 (time 1788271692.09 total 464.32 last 24.22) - open/close 11807 (time 1788271702.11 total 474.34 last 22.65) - open/close 12078 (time 1788271712.96 total 485.19 last 24.99) - open/close 12319 (time 1788271722.96 total 495.19 last 24.08) - open/close 12520 (time 1788271733.11 total 505.34 last 19.82) - open/close 12733 (time 1788271743.11 total 515.34 last 21.30) - open/close 12920 (time 1788271754.16 total 526.39 last 16.92) - open/close 13150 (time 1788271764.53 total 536.76 last 22.17) - open/close 13349 (time 1788271775.17 total 547.41 last 18.70) - open/close 13620 (time 1788271786.04 total 558.27 last 24.95) - open/close 13894 (time 1788271796.80 total 569.03 last 25.45) - open/close 14035 (time 1788271806.93 total 579.16 last 13.92) - open/close 14239 (time 1788271816.93 total 589.16 last 20.39) - open/close 14976 (time 1788271826.93 total 599.16 last 73.69) - open/close 15798 (time 1788271836.94 total 609.17 last 82.17) - open/close 16576 (time 1788271846.94 total 619.17 last 77.77) - open/close 17409 (time 1788271856.94 total 629.18 last 83.28) - open/close 18158 (time 1788271866.95 total 639.18 last 74.86) - open/close 19007 (time 1788271876.96 total 649.19 last 84.79) - open/close 19716 (time 1788271886.97 total 659.20 last 70.89) - open/close 20000 (time 1788271890.45 total 662.68 last 81.46) - open/close 20796 (time 1788271900.47 total 672.70 last 79.50) - open/close 21544 (time 1788271910.49 total 682.72 last 74.65) - open/close 22201 (time 1788271920.49 total 692.72 last 65.70) - open/close 22925 (time 1788271930.49 total 702.72 last 72.37) - open/close 23591 (time 1788271940.50 total 712.73 last 66.57) - open/close 24281 (time 1788271950.50 total 722.73 last 69.00) - open/close 24998 (time 1788271960.50 total 732.73 last 71.64) - open/close 25703 (time 1788271970.51 total 742.74 last 70.43) - open/close 26493 (time 1788271980.52 total 752.75 last 78.99) - open/close 27246 (time 1788271990.54 total 762.77 last 75.15) - open/close 27986 (time 1788272000.55 total 772.78 last 73.93) - open/close 28817 (time 1788272010.55 total 782.78 last 83.06) - open/close 29589 (time 1788272020.56 total 792.79 last 77.16) - open/close 30000 (time 1788272025.43 total 797.66 last 84.36) - open/close 30746 (time 1788272035.43 total 807.66 last 74.58) - open/close 31496 (time 1788272045.44 total 817.67 last 74.95) - open/close 32246 (time 1788272055.45 total 827.68 last 74.93) - open/close 32999 (time 1788272065.45 total 837.68 last 75.27) - open/close 33813 (time 1788272075.45 total 847.68 last 81.40) - open/close 34624 (time 1788272085.47 total 857.70 last 80.94) - open/close 35412 (time 1788272095.47 total 867.70 last 78.80) - open/close 36156 (time 1788272105.48 total 877.71 last 74.32) - open/close 36881 (time 1788272115.48 total 887.71 last 72.48) - open/close 37772 (time 1788272125.51 total 897.74 last 88.88) - open/close 38523 (time 1788272135.51 total 907.74 last 75.10) - open/close 39177 (time 1788272145.51 total 917.74 last 65.39) - open/close 39924 (time 1788272155.52 total 927.75 last 74.62) - open/close 40664 (time 1788272165.54 total 937.77 last 73.84) - open/close 41423 (time 1788272175.54 total 947.77 last 75.90) - open/close 42243 (time 1788272185.55 total 957.78 last 81.92) - open/close 43030 (time 1788272195.57 total 967.80 last 78.60) - open/close 43804 (time 1788272205.57 total 977.80 last 77.35) - open/close 44511 (time 1788272215.58 total 987.81 last 70.65) - open/close 45240 (time 1788272225.58 total 997.81 last 72.87) - open/close 46063 (time 1788272235.58 total 1007.81 last 82.29) - open/close 46898 (time 1788272245.59 total 1017.82 last 83.47) - open/close 47844 (time 1788272255.59 total 1027.82 last 94.55) - open/close 48678 (time 1788272265.61 total 1037.84 last 83.25) - open/close 49452 (time 1788272275.62 total 1047.85 last 77.34) total: 50000 open/close in 1054.58 seconds: 47.41 ops/second - unlinked 0 (time 1788272285 ; total 0 ; last 0) - unlinked 10000 (time 1788272474 ; total 189 ; last 189) - unlinked 20000 (time 1788272639 ; total 354 ; last 165) - unlinked 30000 (time 1788272802 ; total 517 ; last 163) - unlinked 40000 (time 1788272957 ; total 672 ; last 155) total: 50000 unlinks in 834 seconds: 59.952038 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg151-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server unloading modules via unload_modules_local on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing unload_modules_local oleg151-server: modules unloaded. pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2