== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:09:04 (1787576944) start mds service on oleg624-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg624-server' oleg624-server: oleg624-server.virtnet: executing load_modules_local oleg624-server: Loading modules from /home/green/git/lustre-release/lustre oleg624-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg624-server: detected 4 online CPUs by sysfs oleg624-server: MODOPTS_LIBCFS= oleg624-server: Force libcfs to create 2 CPU partitions oleg624-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg624-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg624-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg624-server: oleg624-server.virtnet: executing set_default_debug -1 all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg624-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg624-server: oleg624-server.virtnet: executing set_default_debug -1 all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Started lustre-OST0000 oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 213 (time 1787577033.01 total 10.89 last 19.55) - open/close 441 (time 1787577043.12 total 21.00 last 22.55) - open/close 725 (time 1787577054.06 total 31.95 last 25.96) - open/close 931 (time 1787577064.06 total 41.95 last 20.59) - open/close 1124 (time 1787577075.13 total 53.02 last 17.43) - open/close 1363 (time 1787577085.43 total 63.32 last 23.21) - open/close 1594 (time 1787577095.45 total 73.33 last 23.06) - open/close 1847 (time 1787577105.78 total 83.66 last 24.49) - open/close 2055 (time 1787577116.10 total 93.98 last 20.16) - open/close 2215 (time 1787577126.10 total 103.99 last 15.99) - open/close 2347 (time 1787577136.83 total 114.72 last 12.30) - open/close 2484 (time 1787577147.64 total 125.53 last 12.67) - open/close 2682 (time 1787577157.75 total 135.63 last 19.59) - open/close 2752 (time 1787577169.54 total 147.42 last 5.94) - open/close 2995 (time 1787577180.21 total 158.09 last 22.77) - open/close 3185 (time 1787577190.84 total 168.73 last 17.87) - open/close 3377 (time 1787577201.47 total 179.36 last 18.07) - open/close 3580 (time 1787577211.48 total 189.37 last 20.28) - open/close 3784 (time 1787577221.48 total 199.37 last 20.40) - open/close 3931 (time 1787577231.49 total 209.37 last 14.70) - open/close 4185 (time 1787577242.18 total 220.07 last 23.74) - open/close 4376 (time 1787577252.18 total 230.07 last 19.10) - open/close 4635 (time 1787577262.85 total 240.74 last 24.28) - open/close 4873 (time 1787577273.01 total 250.90 last 23.42) - open/close 5162 (time 1787577283.90 total 261.79 last 26.54) - open/close 5402 (time 1787577295.01 total 272.89 last 21.61) - open/close 5487 (time 1787577305.67 total 283.56 last 7.97) - open/close 5688 (time 1787577315.96 total 293.84 last 19.54) - open/close 5884 (time 1787577325.98 total 303.86 last 19.57) - open/close 6047 (time 1787577336.51 total 314.40 last 15.47) - open/close 6326 (time 1787577347.26 total 325.14 last 25.96) - open/close 6601 (time 1787577358.07 total 335.95 last 25.45) - open/close 6854 (time 1787577368.63 total 346.52 last 23.94) - open/close 7130 (time 1787577378.64 total 356.52 last 27.58) - open/close 7272 (time 1787577389.32 total 367.20 last 13.30) - open/close 7406 (time 1787577399.67 total 377.55 last 12.94) - open/close 7546 (time 1787577409.79 total 387.68 last 13.83) - open/close 7663 (time 1787577420.80 total 398.68 last 10.64) - open/close 7917 (time 1787577430.80 total 408.68 last 25.39) - open/close 8073 (time 1787577441.40 total 419.28 last 14.72) - open/close 8279 (time 1787577451.78 total 429.66 last 19.85) - open/close 8501 (time 1787577461.78 total 439.66 last 22.19) - open/close 8723 (time 1787577471.81 total 449.70 last 22.13) - open/close 8907 (time 1787577482.76 total 460.64 last 16.81) - open/close 9091 (time 1787577493.75 total 471.63 last 16.74) - open/close 9229 (time 1787577504.60 total 482.48 last 12.72) - open/close 9426 (time 1787577514.61 total 492.49 last 19.67) - open/close 9648 (time 1787577524.63 total 502.52 last 22.15) - open/close 9866 (time 1787577536.12 total 514.00 last 18.98) - open/close 9958 (time 1787577546.80 total 524.68 last 8.62) - open/close 10000 (time 1787577552.26 total 530.15 last 7.68) - open/close 10045 (time 1787577562.77 total 540.65 last 4.28) - open/close 10312 (time 1787577573.59 total 551.47 last 24.67) - open/close 10504 (time 1787577583.59 total 561.47 last 19.19) - open/close 10792 (time 1787577594.03 total 571.92 last 27.58) - open/close 10916 (time 1787577604.73 total 582.61 last 11.60) - open/close 11076 (time 1787577614.73 total 592.62 last 15.99) - open/close 11331 (time 1787577624.74 total 602.62 last 25.48) - open/close 11566 (time 1787577634.75 total 612.64 last 23.46) - open/close 11795 (time 1787577644.76 total 622.64 last 22.89) - open/close 12041 (time 1787577655.67 total 633.55 last 22.55) - open/close 12231 (time 1787577666.25 total 644.13 last 17.96) - open/close 12340 (time 1787577676.35 total 654.24 last 10.79) - open/close 12508 (time 1787577686.40 total 664.28 last 16.72) - open/close 12783 (time 1787577696.40 total 674.29 last 27.49) - open/close 12989 (time 1787577706.45 total 684.33 last 20.51) - open/close 13136 (time 1787577717.31 total 695.19 last 13.54) - open/close 13274 (time 1787577727.32 total 705.20 last 13.79) - open/close 13494 (time 1787577738.11 total 715.99 last 20.38) - open/close 13749 (time 1787577748.36 total 726.25 last 24.87) - open/close 13938 (time 1787577758.37 total 736.26 last 18.88) - open/close 14197 (time 1787577769.41 total 747.29 last 23.47) - open/close 14451 (time 1787577780.22 total 758.11 last 23.49) - open/close 14556 (time 1787577790.29 total 768.18 last 10.42) - open/close 14662 (time 1787577800.53 total 778.41 last 10.36) - open/close 14812 (time 1787577811.38 total 789.27 last 13.82) - open/close 15045 (time 1787577821.95 total 799.83 last 22.05) - open/close 15285 (time 1787577832.95 total 810.84 last 21.81) - open/close 15393 (time 1787577843.13 total 821.02 last 10.61) - open/close 15640 (time 1787577853.51 total 831.39 last 23.82) - open/close 15945 (time 1787577864.36 total 842.25 last 28.10) - open/close 16226 (time 1787577874.75 total 852.63 last 27.06) - open/close 16435 (time 1787577885.84 total 863.73 last 18.84) - open/close 16550 (time 1787577896.37 total 874.25 last 10.93) - open/close 16693 (time 1787577908.47 total 886.35 last 11.82) - open/close 16789 (time 1787577921.48 total 899.36 last 7.38) - open/close 16927 (time 1787577934.20 total 912.08 last 10.85) - open/close 17172 (time 1787577945.16 total 923.04 last 22.36) - open/close 17362 (time 1787577955.34 total 933.22 last 18.66) - open/close 17565 (time 1787577965.74 total 943.63 last 19.51) - open/close 17711 (time 1787577975.76 total 953.64 last 14.58) - open/close 17875 (time 1787577985.77 total 963.65 last 16.38) - open/close 18036 (time 1787577996.35 total 974.24 last 15.21) - open/close 18192 (time 1787578007.42 total 985.31 last 14.09) - open/close 18425 (time 1787578017.43 total 995.32 last 23.27) - open/close 19232 (time 1787578027.43 total 1005.32 last 80.70) - open/close 19948 (time 1787578037.44 total 1015.32 last 71.57) - open/close 20705 (time 1787578047.44 total 1025.33 last 75.67) - open/close 21346 (time 1787578057.45 total 1035.34 last 64.03) - open/close 22093 (time 1787578067.45 total 1045.34 last 74.69) - open/close 22793 (time 1787578077.46 total 1055.35 last 69.93) - open/close 23455 (time 1787578087.47 total 1065.35 last 66.18) - open/close 24224 (time 1787578097.47 total 1075.36 last 76.87) - open/close 25054 (time 1787578107.47 total 1085.36 last 82.98) - open/close 25814 (time 1787578117.48 total 1095.37 last 75.95) - open/close 26533 (time 1787578127.49 total 1105.37 last 71.86) - open/close 27149 (time 1787578137.50 total 1115.38 last 61.55) - open/close 27866 (time 1787578147.50 total 1125.38 last 71.69) - open/close 28506 (time 1787578157.50 total 1135.38 last 63.99) - open/close 29249 (time 1787578167.50 total 1145.39 last 74.26) - open/close 29866 (time 1787578177.52 total 1155.40 last 61.60) - open/close 30614 (time 1787578187.52 total 1165.41 last 74.77) - open/close 31175 (time 1787578197.55 total 1175.43 last 55.96) - open/close 31650 (time 1787578207.57 total 1185.45 last 47.41) - open/close 32233 (time 1787578217.58 total 1195.46 last 58.24) - open/close 32764 (time 1787578227.58 total 1205.47 last 53.08) - open/close 33370 (time 1787578237.58 total 1215.47 last 60.60) - open/close 34081 (time 1787578247.60 total 1225.49 last 70.97) - open/close 34864 (time 1787578257.61 total 1235.49 last 78.25) - open/close 35519 (time 1787578267.62 total 1245.50 last 65.43) - open/close 35997 (time 1787578277.62 total 1255.51 last 47.78) - open/close 36444 (time 1787578287.64 total 1265.52 last 44.64) - open/close 37058 (time 1787578297.64 total 1275.52 last 61.37) - open/close 37697 (time 1787578307.64 total 1285.53 last 63.89) - open/close 38380 (time 1787578317.64 total 1295.53 last 68.29) - open/close 39030 (time 1787578327.65 total 1305.53 last 64.98) - open/close 39719 (time 1787578337.65 total 1315.53 last 68.87) - open/close 40000 (time 1787578342.34 total 1320.23 last 59.87) - open/close 40691 (time 1787578352.35 total 1330.23 last 69.09) - open/close 41411 (time 1787578362.35 total 1340.23 last 71.96) - open/close 42160 (time 1787578372.35 total 1350.24 last 74.89) - open/close 42914 (time 1787578382.35 total 1360.24 last 75.39) - open/close 43660 (time 1787578392.38 total 1370.26 last 74.39) - open/close 44335 (time 1787578402.40 total 1380.28 last 67.39) - open/close 45069 (time 1787578412.41 total 1390.29 last 73.30) - open/close 45931 (time 1787578422.42 total 1400.30 last 86.14) - open/close 46777 (time 1787578432.43 total 1410.31 last 84.52) - open/close 47681 (time 1787578442.43 total 1420.32 last 90.35) - open/close 48484 (time 1787578452.45 total 1430.33 last 80.20) - open/close 49282 (time 1787578462.47 total 1440.35 last 79.66) total: 50000 open/close in 1448.78 seconds: 34.51 ops/second - unlinked 0 (time 1787578474 ; total 0 ; last 0) - unlinked 10000 (time 1787578630 ; total 156 ; last 156) - unlinked 20000 (time 1787578789 ; total 315 ; last 159) - unlinked 30000 (time 1787578932 ; total 458 ; last 143) - unlinked 40000 (time 1787579069 ; total 595 ; last 137) total: 50000 unlinks in 763 seconds: 65.530800 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg624-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg624-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg624-server stop mds service on oleg624-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg624-server unloading modules via unload_modules_local on: 'oleg624-server' oleg624-server: oleg624-server.virtnet: executing unload_modules_local oleg624-server: modules unloaded. pdsh@oleg624-client: oleg624-client: ssh exited with exit code 2 pdsh@oleg624-client: oleg624-server: ssh exited with exit code 2 pdsh@oleg624-client: oleg624-client: ssh exited with exit code 2 pdsh@oleg624-client: oleg624-server: ssh exited with exit code 2