== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:57:23 (1787821043) start mds service on oleg301-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing load_modules_local oleg301-server: Loading modules from /home/green/git/lustre-release/lustre oleg301-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg301-server: detected 4 online CPUs by sysfs oleg301-server: MODOPTS_LIBCFS= oleg301-server: Force libcfs to create 2 CPU partitions oleg301-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg301-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg301-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg301-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 261 (time 1787821127.70 total 10.98 last 23.76) - open/close 515 (time 1787821138.34 total 21.62 last 23.88) - open/close 752 (time 1787821148.34 total 31.62 last 23.70) - open/close 950 (time 1787821159.14 total 42.42 last 18.34) - open/close 1231 (time 1787821169.14 total 52.42 last 28.09) - open/close 1470 (time 1787821179.16 total 62.44 last 23.86) - open/close 1720 (time 1787821189.59 total 72.87 last 23.97) - open/close 1997 (time 1787821200.61 total 83.89 last 25.13) - open/close 2237 (time 1787821210.84 total 94.12 last 23.47) - open/close 2507 (time 1787821220.84 total 104.13 last 26.99) - open/close 2702 (time 1787821231.44 total 114.72 last 18.40) - open/close 2943 (time 1787821242.27 total 125.55 last 22.25) - open/close 3179 (time 1787821252.88 total 136.16 last 22.24) - open/close 3425 (time 1787821262.89 total 146.17 last 24.58) - open/close 3614 (time 1787821273.10 total 156.39 last 18.51) - open/close 3844 (time 1787821283.12 total 166.41 last 22.95) - open/close 4067 (time 1787821293.72 total 177.01 last 21.03) - open/close 4309 (time 1787821303.96 total 187.25 last 23.64) - open/close 4556 (time 1787821313.98 total 197.27 last 24.65) - open/close 4783 (time 1787821324.20 total 207.48 last 22.22) - open/close 5016 (time 1787821334.70 total 217.98 last 22.20) - open/close 5218 (time 1787821344.74 total 228.02 last 20.11) - open/close 5403 (time 1787821355.47 total 238.76 last 17.24) - open/close 5490 (time 1787821365.50 total 248.78 last 8.68) - open/close 5754 (time 1787821375.51 total 258.79 last 26.37) - open/close 5993 (time 1787821385.52 total 268.81 last 23.86) - open/close 6215 (time 1787821396.20 total 279.49 last 20.79) - open/close 6482 (time 1787821406.51 total 289.79 last 25.91) - open/close 6706 (time 1787821416.79 total 300.07 last 21.78) - open/close 6939 (time 1787821427.43 total 310.71 last 21.90) - open/close 7208 (time 1787821437.44 total 320.72 last 26.88) - open/close 7447 (time 1787821447.57 total 330.85 last 23.59) - open/close 7674 (time 1787821457.99 total 341.27 last 21.78) - open/close 7910 (time 1787821468.30 total 351.58 last 22.89) - open/close 8188 (time 1787821479.38 total 362.66 last 25.10) - open/close 8422 (time 1787821489.39 total 372.67 last 23.37) - open/close 8677 (time 1787821500.25 total 383.53 last 23.48) - open/close 8968 (time 1787821510.80 total 394.09 last 27.57) - open/close 9236 (time 1787821521.50 total 404.78 last 25.06) - open/close 9468 (time 1787821532.52 total 415.80 last 21.06) - open/close 9689 (time 1787821542.54 total 425.82 last 22.05) - open/close 9923 (time 1787821554.53 total 437.82 last 19.51) - open/close 10000 (time 1787821557.03 total 440.31 last 30.87) - open/close 10243 (time 1787821567.58 total 450.86 last 23.03) - open/close 10481 (time 1787821577.69 total 460.97 last 23.55) - open/close 10712 (time 1787821587.69 total 470.97 last 23.10) - open/close 10898 (time 1787821598.20 total 481.49 last 17.69) - open/close 11129 (time 1787821608.23 total 491.51 last 23.04) - open/close 11383 (time 1787821618.60 total 501.88 last 24.49) - open/close 11522 (time 1787821628.61 total 511.90 last 13.88) - open/close 11801 (time 1787821638.63 total 521.92 last 27.85) - open/close 12028 (time 1787821648.65 total 531.94 last 22.66) - open/close 12203 (time 1787821658.98 total 542.26 last 16.95) - open/close 12504 (time 1787821670.03 total 553.31 last 27.23) - open/close 12725 (time 1787821680.03 total 563.31 last 22.10) - open/close 12946 (time 1787821690.65 total 573.94 last 20.80) - open/close 13052 (time 1787821700.69 total 583.97 last 10.56) - open/close 13184 (time 1787821711.60 total 594.88 last 12.10) - open/close 13236 (time 1787821721.60 total 604.88 last 5.20) - open/close 13413 (time 1787821732.07 total 615.35 last 16.91) - open/close 13652 (time 1787821742.60 total 625.88 last 22.70) - open/close 13873 (time 1787821752.66 total 635.94 last 21.96) - open/close 14101 (time 1787821763.23 total 646.51 last 21.57) - open/close 14230 (time 1787821773.43 total 656.71 last 12.65) - open/close 14391 (time 1787821783.44 total 666.72 last 16.09) - open/close 14522 (time 1787821793.63 total 676.92 last 12.85) - open/close 14650 (time 1787821803.67 total 686.95 last 12.76) - open/close 14844 (time 1787821814.49 total 697.77 last 17.92) - open/close 15115 (time 1787821824.50 total 707.78 last 27.07) - open/close 15260 (time 1787821835.01 total 718.29 last 13.80) - open/close 15482 (time 1787821845.78 total 729.06 last 20.61) - open/close 15669 (time 1787821856.09 total 739.37 last 18.14) - open/close 15857 (time 1787821867.03 total 750.31 last 17.18) - open/close 16103 (time 1787821877.80 total 761.08 last 22.84) - open/close 16338 (time 1787821888.36 total 771.65 last 22.25) - open/close 16489 (time 1787821898.37 total 781.66 last 15.09) - open/close 16651 (time 1787821908.64 total 791.92 last 15.78) - open/close 16852 (time 1787821919.66 total 802.95 last 18.23) - open/close 17056 (time 1787821929.70 total 812.98 last 20.33) - open/close 17328 (time 1787821939.71 total 822.99 last 27.18) - open/close 17547 (time 1787821950.11 total 833.39 last 21.06) - open/close 17784 (time 1787821960.18 total 843.46 last 23.53) - open/close 17833 (time 1787821970.70 total 853.99 last 4.66) - open/close 18052 (time 1787821980.86 total 864.14 last 21.56) - open/close 18279 (time 1787821990.86 total 874.14 last 22.70) - open/close 18494 (time 1787822001.18 total 884.46 last 20.84) - open/close 18771 (time 1787822011.20 total 894.48 last 27.65) - open/close 19026 (time 1787822021.68 total 904.96 last 24.33) - open/close 19306 (time 1787822032.11 total 915.39 last 26.83) - open/close 19967 (time 1787822042.12 total 925.40 last 66.05) - open/close 20682 (time 1787822052.12 total 935.41 last 71.47) - open/close 21345 (time 1787822062.14 total 945.42 last 66.19) - open/close 22032 (time 1787822072.15 total 955.43 last 68.65) - open/close 22709 (time 1787822082.15 total 965.44 last 67.65) - open/close 23478 (time 1787822092.17 total 975.45 last 76.81) - open/close 24213 (time 1787822102.18 total 985.47 last 73.37) - open/close 24891 (time 1787822112.19 total 995.47 last 67.79) - open/close 25549 (time 1787822122.19 total 1005.47 last 65.77) - open/close 26234 (time 1787822132.19 total 1015.47 last 68.50) - open/close 26907 (time 1787822142.19 total 1025.48 last 67.28) - open/close 27578 (time 1787822152.20 total 1035.49 last 67.03) - open/close 28240 (time 1787822162.21 total 1045.49 last 66.18) - open/close 28902 (time 1787822172.22 total 1055.50 last 66.14) - open/close 29616 (time 1787822182.22 total 1065.51 last 71.34) - open/close 30000 (time 1787822188.49 total 1071.77 last 61.28) - open/close 30673 (time 1787822198.49 total 1081.77 last 67.29) - open/close 31370 (time 1787822208.51 total 1091.80 last 69.55) - open/close 32071 (time 1787822218.53 total 1101.81 last 69.98) - open/close 32733 (time 1787822228.53 total 1111.81 last 66.20) - open/close 33462 (time 1787822238.53 total 1121.82 last 72.89) - open/close 34044 (time 1787822248.56 total 1131.84 last 58.05) - open/close 34578 (time 1787822258.57 total 1141.85 last 53.36) - open/close 35154 (time 1787822268.59 total 1151.87 last 57.45) - open/close 35745 (time 1787822278.59 total 1161.88 last 59.09) - open/close 36436 (time 1787822288.61 total 1171.89 last 69.00) - open/close 37103 (time 1787822298.61 total 1181.89 last 66.69) - open/close 37864 (time 1787822308.61 total 1191.90 last 76.09) - open/close 38524 (time 1787822318.62 total 1201.90 last 65.97) - open/close 39171 (time 1787822328.63 total 1211.92 last 64.58) - open/close 39766 (time 1787822338.69 total 1221.97 last 59.20) - open/close 40000 (time 1787822342.22 total 1225.50 last 66.30) - open/close 40669 (time 1787822352.22 total 1235.50 last 66.87) - open/close 41315 (time 1787822362.23 total 1245.51 last 64.53) - open/close 42057 (time 1787822372.23 total 1255.51 last 74.19) - open/close 42820 (time 1787822382.24 total 1265.52 last 76.26) - open/close 43534 (time 1787822392.25 total 1275.53 last 71.30) - open/close 44271 (time 1787822402.25 total 1285.54 last 73.68) - open/close 45050 (time 1787822412.29 total 1295.58 last 77.59) - open/close 45808 (time 1787822422.30 total 1305.58 last 75.77) - open/close 46531 (time 1787822432.30 total 1315.58 last 72.30) - open/close 47236 (time 1787822442.30 total 1325.58 last 70.49) - open/close 47988 (time 1787822452.31 total 1335.59 last 75.12) - open/close 48752 (time 1787822462.32 total 1345.60 last 76.36) - open/close 49503 (time 1787822472.33 total 1355.61 last 75.00) total: 50000 open/close in 1362.03 seconds: 36.71 ops/second - unlinked 0 (time 1787822482 ; total 0 ; last 0) - unlinked 10000 (time 1787822618 ; total 136 ; last 136) - unlinked 20000 (time 1787822752 ; total 270 ; last 134) - unlinked 30000 (time 1787822889 ; total 407 ; last 137) - unlinked 40000 (time 1787823027 ; total 545 ; last 138) total: 50000 unlinks in 689 seconds: 72.568939 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg301-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server stop mds service on oleg301-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg301-server unloading modules via unload_modules_local on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing unload_modules_local oleg301-server: modules unloaded. pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2