== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:14:38 (1788747278) start mds service on oleg427-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 6 20:25 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: kmemleak disabled oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: detected 4 online CPUs by sysfs oleg427-server: MODOPTS_LIBCFS= oleg427-server: Force libcfs to create 2 CPU partitions oleg427-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg427-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg427-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg427-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg427-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg427-client.virtnet: -o user_xattr,flock 192.168.204.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 221 (time 1788747364.28 total 10.97 last 20.14) - open/close 461 (time 1788747374.39 total 21.09 last 23.72) - open/close 644 (time 1788747384.57 total 31.27 last 17.98) - open/close 844 (time 1788747394.57 total 41.27 last 19.99) - open/close 943 (time 1788747406.10 total 52.80 last 8.59) - open/close 1193 (time 1788747416.12 total 62.81 last 24.95) - open/close 1376 (time 1788747426.12 total 72.82 last 18.29) - open/close 1492 (time 1788747436.21 total 82.90 last 11.51) - open/close 1695 (time 1788747446.59 total 93.29 last 19.54) - open/close 1861 (time 1788747456.61 total 103.31 last 16.57) - open/close 1990 (time 1788747466.62 total 113.31 last 12.89) - open/close 2219 (time 1788747476.64 total 123.34 last 22.84) - open/close 2471 (time 1788747487.56 total 134.26 last 23.08) - open/close 2641 (time 1788747498.03 total 144.72 last 16.25) - open/close 2864 (time 1788747508.28 total 154.98 last 21.75) - open/close 3016 (time 1788747518.52 total 165.22 last 14.84) - open/close 3123 (time 1788747529.17 total 175.86 last 10.05) - open/close 3284 (time 1788747540.02 total 186.72 last 14.83) - open/close 3450 (time 1788747550.60 total 197.29 last 15.70) - open/close 3638 (time 1788747560.61 total 207.31 last 18.78) - open/close 3918 (time 1788747571.53 total 218.23 last 25.63) - open/close 3954 (time 1788747581.77 total 228.47 last 3.52) - open/close 4112 (time 1788747592.50 total 239.20 last 14.72) - open/close 4286 (time 1788747602.97 total 249.66 last 16.62) - open/close 4342 (time 1788747613.46 total 260.15 last 5.34) - open/close 4534 (time 1788747623.56 total 270.26 last 19.00) - open/close 4732 (time 1788747633.56 total 280.26 last 19.80) - open/close 4982 (time 1788747644.23 total 290.93 last 23.44) - open/close 5080 (time 1788747657.79 total 304.49 last 7.22) - open/close 5197 (time 1788747668.86 total 315.56 last 10.57) - open/close 5374 (time 1788747678.87 total 325.57 last 17.68) - open/close 5504 (time 1788747689.27 total 335.96 last 12.51) - open/close 5595 (time 1788747699.27 total 345.96 last 9.10) - open/close 5725 (time 1788747709.28 total 355.98 last 12.98) - open/close 5833 (time 1788747719.36 total 366.06 last 10.71) - open/close 5967 (time 1788747730.06 total 376.76 last 12.53) - open/close 6129 (time 1788747740.15 total 386.84 last 16.07) - open/close 6241 (time 1788747750.41 total 397.10 last 10.91) - open/close 6510 (time 1788747760.41 total 407.11 last 26.89) - open/close 6686 (time 1788747770.42 total 417.11 last 17.59) - open/close 6817 (time 1788747780.43 total 427.12 last 13.09) - open/close 6985 (time 1788747790.97 total 437.67 last 15.94) - open/close 7116 (time 1788747800.97 total 447.67 last 13.10) - open/close 7347 (time 1788747810.99 total 457.69 last 23.05) - open/close 7584 (time 1788747821.00 total 467.70 last 23.67) - open/close 7749 (time 1788747831.17 total 477.86 last 16.24) - open/close 7971 (time 1788747843.00 total 489.70 last 18.76) - open/close 8093 (time 1788747853.01 total 499.70 last 12.19) - open/close 8177 (time 1788747863.57 total 510.26 last 7.96) - open/close 8324 (time 1788747874.07 total 520.76 last 14.00) - open/close 8531 (time 1788747886.80 total 533.49 last 16.26) - open/close 8753 (time 1788747897.21 total 543.90 last 21.32) - open/close 8820 (time 1788747912.02 total 558.72 last 4.52) - open/close 8959 (time 1788747923.19 total 569.89 last 12.44) - open/close 9093 (time 1788747933.56 total 580.26 last 12.92) - open/close 9147 (time 1788747944.46 total 591.16 last 4.95) - open/close 9167 (time 1788747955.21 total 601.90 last 1.86) - open/close 9382 (time 1788747965.22 total 611.92 last 21.46) - open/close 9497 (time 1788747975.61 total 622.31 last 11.07) - open/close 9581 (time 1788747986.43 total 633.13 last 7.76) - open/close 9703 (time 1788747997.19 total 643.88 last 11.34) - open/close 9856 (time 1788748007.20 total 653.90 last 15.28) - open/close 10000 (time 1788748012.24 total 658.93 last 28.61) - open/close 10234 (time 1788748023.01 total 669.70 last 21.73) - open/close 10396 (time 1788748033.65 total 680.35 last 15.22) - open/close 10624 (time 1788748043.97 total 690.67 last 22.09) - open/close 10790 (time 1788748054.01 total 700.71 last 16.54) - open/close 10898 (time 1788748064.03 total 710.72 last 10.78) - open/close 11062 (time 1788748075.89 total 722.59 last 13.82) - open/close 11167 (time 1788748088.66 total 735.35 last 8.23) - open/close 11321 (time 1788748099.14 total 745.84 last 14.69) - open/close 11462 (time 1788748109.15 total 755.84 last 14.10) - open/close 11667 (time 1788748119.16 total 765.86 last 20.47) - open/close 11911 (time 1788748129.34 total 776.03 last 23.98) - open/close 12021 (time 1788748139.34 total 786.04 last 10.99) - open/close 12084 (time 1788748149.35 total 796.05 last 6.29) - open/close 12277 (time 1788748160.15 total 806.85 last 17.87) - open/close 12444 (time 1788748170.99 total 817.69 last 15.41) - open/close 12576 (time 1788748181.00 total 827.69 last 13.20) - open/close 12701 (time 1788748191.03 total 837.72 last 12.46) - open/close 12792 (time 1788748201.80 total 848.49 last 8.45) - open/close 12962 (time 1788748211.81 total 858.50 last 16.98) - open/close 13091 (time 1788748222.54 total 869.23 last 12.02) - open/close 13230 (time 1788748233.23 total 879.92 last 13.00) - open/close 13313 (time 1788748243.72 total 890.42 last 7.91) - open/close 13489 (time 1788748253.74 total 900.44 last 17.56) - open/close 13617 (time 1788748265.72 total 912.42 last 10.69) - open/close 13827 (time 1788748278.72 total 925.41 last 16.16) - open/close 14008 (time 1788748288.95 total 935.65 last 17.68) - open/close 14240 (time 1788748299.58 total 946.28 last 21.83) - open/close 14401 (time 1788748310.46 total 957.15 last 14.80) - open/close 14586 (time 1788748320.70 total 967.39 last 18.06) - open/close 14804 (time 1788748330.72 total 977.41 last 21.76) - open/close 14991 (time 1788748341.78 total 988.48 last 16.89) - open/close 15212 (time 1788748351.80 total 998.50 last 22.07) - open/close 15471 (time 1788748362.20 total 1008.89 last 24.91) - open/close 15711 (time 1788748372.85 total 1019.55 last 22.52) - open/close 15857 (time 1788748383.05 total 1029.75 last 14.31) - open/close 15982 (time 1788748393.91 total 1040.61 last 11.51) - open/close 16373 (time 1788748403.92 total 1050.62 last 39.08) - open/close 17028 (time 1788748413.93 total 1060.62 last 65.45) - open/close 17685 (time 1788748423.94 total 1070.63 last 65.64) - open/close 18277 (time 1788748433.95 total 1080.64 last 59.12) - open/close 18863 (time 1788748443.96 total 1090.65 last 58.54) - open/close 19513 (time 1788748453.97 total 1100.66 last 64.95) - open/close 20000 (time 1788748461.85 total 1108.54 last 61.79) - open/close 20620 (time 1788748471.85 total 1118.54 last 62.00) - open/close 21295 (time 1788748481.85 total 1128.54 last 67.50) - open/close 21909 (time 1788748491.85 total 1138.55 last 61.40) - open/close 22529 (time 1788748501.86 total 1148.56 last 61.93) - open/close 23125 (time 1788748511.89 total 1158.58 last 59.44) - open/close 23745 (time 1788748521.91 total 1168.60 last 61.87) - open/close 24352 (time 1788748531.91 total 1178.61 last 60.69) - open/close 24900 (time 1788748541.92 total 1188.62 last 54.74) - open/close 25466 (time 1788748551.93 total 1198.63 last 56.53) - open/close 26055 (time 1788748561.94 total 1208.63 last 58.88) - open/close 26694 (time 1788748571.95 total 1218.64 last 63.84) - open/close 27283 (time 1788748581.95 total 1228.64 last 58.89) - open/close 27808 (time 1788748591.97 total 1238.66 last 52.40) - open/close 28382 (time 1788748601.98 total 1248.67 last 57.34) - open/close 28922 (time 1788748611.98 total 1258.67 last 54.00) - open/close 29516 (time 1788748621.99 total 1268.68 last 59.34) - open/close 30000 (time 1788748629.77 total 1276.47 last 62.20) - open/close 30626 (time 1788748639.78 total 1286.48 last 62.51) - open/close 31241 (time 1788748649.80 total 1296.49 last 61.43) - open/close 31882 (time 1788748659.80 total 1306.50 last 64.06) - open/close 32546 (time 1788748669.82 total 1316.52 last 66.27) - open/close 33181 (time 1788748679.84 total 1326.54 last 63.35) - open/close 33812 (time 1788748689.86 total 1336.56 last 63.01) - open/close 34421 (time 1788748699.86 total 1346.56 last 60.89) - open/close 35006 (time 1788748709.88 total 1356.58 last 58.39) - open/close 35593 (time 1788748719.88 total 1366.58 last 58.69) - open/close 36185 (time 1788748729.89 total 1376.58 last 59.17) - open/close 36789 (time 1788748739.90 total 1386.60 last 60.30) - open/close 37392 (time 1788748749.91 total 1396.61 last 60.26) - open/close 38037 (time 1788748759.91 total 1406.61 last 64.49) - open/close 38640 (time 1788748769.91 total 1416.61 last 60.29) - open/close 39278 (time 1788748779.94 total 1426.63 last 63.66) - open/close 39924 (time 1788748789.96 total 1436.65 last 64.48) - open/close 40565 (time 1788748799.96 total 1446.65 last 64.09) - open/close 41227 (time 1788748809.97 total 1456.67 last 66.13) - open/close 41902 (time 1788748819.98 total 1466.68 last 67.43) - open/close 42540 (time 1788748829.99 total 1476.69 last 63.74) - open/close 43153 (time 1788748840.00 total 1486.69 last 61.26) - open/close 43831 (time 1788748850.01 total 1496.71 last 67.70) - open/close 44518 (time 1788748860.01 total 1506.71 last 68.69) - open/close 45230 (time 1788748870.02 total 1516.72 last 71.12) - open/close 45891 (time 1788748880.04 total 1526.74 last 65.97) - open/close 46530 (time 1788748890.05 total 1536.74 last 63.87) - open/close 47228 (time 1788748900.05 total 1546.74 last 69.78) - open/close 47855 (time 1788748910.06 total 1556.76 last 62.63) - open/close 48488 (time 1788748920.06 total 1566.76 last 63.27) - open/close 49141 (time 1788748930.07 total 1576.77 last 65.23) - open/close 49768 (time 1788748940.08 total 1586.78 last 62.66) total: 50000 open/close in 1590.55 seconds: 31.44 ops/second - unlinked 0 (time 1788748948 ; total 0 ; last 0) - unlinked 10000 (time 1788749109 ; total 161 ; last 161) - unlinked 20000 (time 1788749262 ; total 314 ; last 153) - unlinked 30000 (time 1788749402 ; total 454 ; last 140) - unlinked 40000 (time 1788749530 ; total 582 ; last 128) total: 50000 unlinks in 771 seconds: 64.850845 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg427-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server unloading modules via unload_modules_local on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing unload_modules_local oleg427-server: kmemleak disabled modules unloaded. pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2