== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:39:52 (1787395192) start mds service on oleg134-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing load_modules_local oleg134-server: Loading modules from /home/green/git/lustre-release/lustre oleg134-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg134-server: detected 4 online CPUs by sysfs oleg134-server: MODOPTS_LIBCFS= oleg134-server: Force libcfs to create 2 CPU partitions oleg134-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg134-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg134-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg134-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 155 (time 1787395273.29 total 10.43 last 14.86) - open/close 441 (time 1787395283.31 total 20.45 last 28.54) - open/close 724 (time 1787395293.33 total 30.47 last 28.25) - open/close 944 (time 1787395303.35 total 40.49 last 21.97) - open/close 1188 (time 1787395313.35 total 50.49 last 24.39) - open/close 1453 (time 1787395323.37 total 60.51 last 26.45) - open/close 1625 (time 1787395333.51 total 70.65 last 16.95) - open/close 1887 (time 1787395344.32 total 81.46 last 24.25) - open/close 2160 (time 1787395354.35 total 91.49 last 27.20) - open/close 2286 (time 1787395364.35 total 101.49 last 12.60) - open/close 2457 (time 1787395377.20 total 114.34 last 13.31) - open/close 2688 (time 1787395389.43 total 126.57 last 18.89) - open/close 2943 (time 1787395399.44 total 136.58 last 25.48) - open/close 3142 (time 1787395409.68 total 146.82 last 19.44) - open/close 3378 (time 1787395420.16 total 157.31 last 22.50) - open/close 3670 (time 1787395430.73 total 167.87 last 27.64) - open/close 3900 (time 1787395441.08 total 178.22 last 22.23) - open/close 4198 (time 1787395451.09 total 188.23 last 29.75) - open/close 4487 (time 1787395461.44 total 198.58 last 27.92) - open/close 4747 (time 1787395472.45 total 209.59 last 23.62) - open/close 4987 (time 1787395483.07 total 220.22 last 22.59) - open/close 5203 (time 1787395493.74 total 230.88 last 20.25) - open/close 5454 (time 1787395504.64 total 241.78 last 23.02) - open/close 5672 (time 1787395515.26 total 252.40 last 20.54) - open/close 5895 (time 1787395525.28 total 262.42 last 22.26) - open/close 6172 (time 1787395535.68 total 272.82 last 26.63) - open/close 6387 (time 1787395545.91 total 283.05 last 21.01) - open/close 6611 (time 1787395556.55 total 293.69 last 21.06) - open/close 6886 (time 1787395566.56 total 303.70 last 27.48) - open/close 7041 (time 1787395576.77 total 313.91 last 15.17) - open/close 7212 (time 1787395586.92 total 324.06 last 16.85) - open/close 7467 (time 1787395596.94 total 334.08 last 25.45) - open/close 7736 (time 1787395607.44 total 344.58 last 25.61) - open/close 7981 (time 1787395618.10 total 355.24 last 22.99) - open/close 8207 (time 1787395628.11 total 365.25 last 22.58) - open/close 8439 (time 1787395638.13 total 375.27 last 23.17) - open/close 8645 (time 1787395648.24 total 385.38 last 20.37) - open/close 8864 (time 1787395658.24 total 395.38 last 21.90) - open/close 9110 (time 1787395668.81 total 405.95 last 23.28) - open/close 9359 (time 1787395679.43 total 416.57 last 23.45) - open/close 9451 (time 1787395690.11 total 427.25 last 8.61) - open/close 9622 (time 1787395700.13 total 437.27 last 17.06) - open/close 9860 (time 1787395710.14 total 447.28 last 23.78) - open/close 9991 (time 1787395723.00 total 460.14 last 10.19) - open/close 10210 (time 1787395733.93 total 471.07 last 20.02) - open/close 10412 (time 1787395744.24 total 481.38 last 19.60) - open/close 10677 (time 1787395754.25 total 491.39 last 26.46) - open/close 10917 (time 1787395764.26 total 501.40 last 23.99) - open/close 11126 (time 1787395774.26 total 511.40 last 20.90) - open/close 11372 (time 1787395784.27 total 521.41 last 24.58) - open/close 11601 (time 1787395795.09 total 532.23 last 21.15) - open/close 11793 (time 1787395805.55 total 542.69 last 18.36) - open/close 12012 (time 1787395816.31 total 553.45 last 20.36) - open/close 12249 (time 1787395827.13 total 564.27 last 21.91) - open/close 12423 (time 1787395837.52 total 574.66 last 16.74) - open/close 12669 (time 1787395848.44 total 585.59 last 22.52) - open/close 12889 (time 1787395858.80 total 595.94 last 21.24) - open/close 13068 (time 1787395869.16 total 606.30 last 17.27) - open/close 13314 (time 1787395879.17 total 616.31 last 24.58) - open/close 13544 (time 1787395889.24 total 626.38 last 22.84) - open/close 13786 (time 1787395899.24 total 636.38 last 24.20) - open/close 14061 (time 1787395910.13 total 647.27 last 25.27) - open/close 14315 (time 1787395920.13 total 657.27 last 25.40) - open/close 14460 (time 1787395930.16 total 667.30 last 14.45) - open/close 14704 (time 1787395942.41 total 679.55 last 19.93) - open/close 14873 (time 1787395952.85 total 689.99 last 16.17) - open/close 14953 (time 1787395962.86 total 700.00 last 8.00) - open/close 15170 (time 1787395973.32 total 710.46 last 20.73) - open/close 15422 (time 1787395984.09 total 721.23 last 23.41) - open/close 15651 (time 1787395994.19 total 731.33 last 22.67) - open/close 15906 (time 1787396004.35 total 741.49 last 25.09) - open/close 16061 (time 1787396014.66 total 751.80 last 15.04) - open/close 16251 (time 1787396024.90 total 762.04 last 18.55) - open/close 16434 (time 1787396035.52 total 772.66 last 17.24) - open/close 16695 (time 1787396046.19 total 783.33 last 24.45) - open/close 16973 (time 1787396057.08 total 794.22 last 25.55) - open/close 17254 (time 1787396067.09 total 804.23 last 28.06) - open/close 17460 (time 1787396077.62 total 814.76 last 19.57) - open/close 17660 (time 1787396087.62 total 824.76 last 20.00) - open/close 17907 (time 1787396097.62 total 834.76 last 24.70) - open/close 18194 (time 1787396107.64 total 844.78 last 28.65) - open/close 18442 (time 1787396118.14 total 855.28 last 23.62) - open/close 18658 (time 1787396128.14 total 865.28 last 21.58) - open/close 18940 (time 1787396139.14 total 876.28 last 25.64) - open/close 19414 (time 1787396149.15 total 886.29 last 47.36) - open/close 20000 (time 1787396156.35 total 893.49 last 81.41) - open/close 20900 (time 1787396166.35 total 903.49 last 89.92) - open/close 21705 (time 1787396176.38 total 913.53 last 80.25) - open/close 22614 (time 1787396186.39 total 923.53 last 90.85) - open/close 23437 (time 1787396196.39 total 933.53 last 82.30) - open/close 24279 (time 1787396206.40 total 943.54 last 84.16) - open/close 25120 (time 1787396216.40 total 953.54 last 84.05) - open/close 26009 (time 1787396226.40 total 963.54 last 88.88) - open/close 26867 (time 1787396236.41 total 973.55 last 85.78) - open/close 27720 (time 1787396246.41 total 983.55 last 85.29) - open/close 28487 (time 1787396256.42 total 993.56 last 76.64) - open/close 29323 (time 1787396266.42 total 1003.56 last 83.56) - open/close 30000 (time 1787396274.25 total 1011.39 last 86.46) - open/close 30832 (time 1787396284.25 total 1021.39 last 83.20) - open/close 31568 (time 1787396294.25 total 1031.40 last 73.59) - open/close 32356 (time 1787396304.27 total 1041.41 last 78.72) - open/close 33200 (time 1787396314.27 total 1051.41 last 84.33) - open/close 33954 (time 1787396324.29 total 1061.43 last 75.29) - open/close 34828 (time 1787396334.30 total 1071.44 last 87.32) - open/close 35666 (time 1787396344.30 total 1081.44 last 83.78) - open/close 36418 (time 1787396354.31 total 1091.45 last 75.09) - open/close 37067 (time 1787396364.33 total 1101.47 last 64.83) - open/close 37804 (time 1787396374.33 total 1111.47 last 73.65) - open/close 38667 (time 1787396384.34 total 1121.48 last 86.26) - open/close 39414 (time 1787396394.34 total 1131.48 last 74.68) - open/close 40000 (time 1787396401.27 total 1138.41 last 84.56) - open/close 40741 (time 1787396411.28 total 1148.42 last 74.04) - open/close 41475 (time 1787396421.29 total 1158.43 last 73.33) - open/close 42238 (time 1787396431.29 total 1168.43 last 76.25) - open/close 42945 (time 1787396441.30 total 1178.44 last 70.64) - open/close 43563 (time 1787396451.31 total 1188.45 last 61.75) - open/close 44186 (time 1787396461.33 total 1198.47 last 62.17) - open/close 44756 (time 1787396471.33 total 1208.47 last 56.99) - open/close 45541 (time 1787396481.33 total 1218.47 last 78.49) - open/close 46242 (time 1787396491.35 total 1228.49 last 69.97) - open/close 46817 (time 1787396501.35 total 1238.49 last 57.50) - open/close 47432 (time 1787396511.35 total 1248.49 last 61.49) - open/close 48112 (time 1787396521.36 total 1258.50 last 67.95) - open/close 48786 (time 1787396531.37 total 1268.51 last 67.32) - open/close 49502 (time 1787396541.38 total 1278.52 last 71.56) total: 50000 open/close in 1285.80 seconds: 38.89 ops/second - unlinked 0 (time 1787396553 ; total 0 ; last 0) - unlinked 10000 (time 1787396686 ; total 133 ; last 133) - unlinked 20000 (time 1787396830 ; total 277 ; last 144) - unlinked 30000 (time 1787396965 ; total 412 ; last 135) - unlinked 40000 (time 1787397102 ; total 549 ; last 137) total: 50000 unlinks in 686 seconds: 72.886299 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server unloading modules via unload_modules_local on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing unload_modules_local oleg134-server: modules unloaded. pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2