== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:53:26 (1790103206) start mds service on oleg407-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1940568 Sep 22 12:10 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: kmemleak disabled oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: detected 4 online CPUs by sysfs oleg407-server: MODOPTS_LIBCFS= oleg407-server: Force libcfs to create 2 CPU partitions oleg407-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg407-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg407-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg407-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg407-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0000 oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 239 (time 1790103297.84 total 10.01 last 23.87) - open/close 532 (time 1790103307.86 total 20.03 last 29.24) - open/close 710 (time 1790103318.58 total 30.75 last 16.61) - open/close 893 (time 1790103328.59 total 40.76 last 18.27) - open/close 1008 (time 1790103338.61 total 50.78 last 11.48) - open/close 1227 (time 1790103349.57 total 61.74 last 19.98) - open/close 1314 (time 1790103360.02 total 72.19 last 8.33) - open/close 1429 (time 1790103370.49 total 82.66 last 10.98) - open/close 1526 (time 1790103380.49 total 92.66 last 9.70) - open/close 1695 (time 1790103390.49 total 102.66 last 16.89) - open/close 1834 (time 1790103400.67 total 112.84 last 13.66) - open/close 1999 (time 1790103410.74 total 122.91 last 16.38) - open/close 2132 (time 1790103424.32 total 136.49 last 9.80) - open/close 2222 (time 1790103435.38 total 147.55 last 8.13) - open/close 2361 (time 1790103445.87 total 158.04 last 13.26) - open/close 2455 (time 1790103456.36 total 168.53 last 8.95) - open/close 2554 (time 1790103467.38 total 179.55 last 8.99) - open/close 2687 (time 1790103477.82 total 189.99 last 12.73) - open/close 2800 (time 1790103487.82 total 199.99 last 11.30) - open/close 2929 (time 1790103498.87 total 211.04 last 11.67) - open/close 3020 (time 1790103508.90 total 221.07 last 9.07) - open/close 3147 (time 1790103519.60 total 231.77 last 11.87) - open/close 3263 (time 1790103529.66 total 241.83 last 11.54) - open/close 3331 (time 1790103539.70 total 251.87 last 6.77) - open/close 3483 (time 1790103549.70 total 261.87 last 15.20) - open/close 3713 (time 1790103559.70 total 271.87 last 22.99) - open/close 3942 (time 1790103570.41 total 282.58 last 21.39) - open/close 4092 (time 1790103580.65 total 292.82 last 14.65) - open/close 4315 (time 1790103590.67 total 302.84 last 22.26) - open/close 4508 (time 1790103600.69 total 312.86 last 19.26) - open/close 4651 (time 1790103611.39 total 323.56 last 13.36) - open/close 4849 (time 1790103621.91 total 334.08 last 18.82) - open/close 5076 (time 1790103634.11 total 346.28 last 18.61) - open/close 5229 (time 1790103644.11 total 356.28 last 15.29) - open/close 5417 (time 1790103654.12 total 366.29 last 18.78) - open/close 5565 (time 1790103664.51 total 376.68 last 14.25) - open/close 5668 (time 1790103675.40 total 387.57 last 9.46) - open/close 5849 (time 1790103687.54 total 399.71 last 14.90) - open/close 6081 (time 1790103698.09 total 410.26 last 22.00) - open/close 6285 (time 1790103708.53 total 420.70 last 19.54) - open/close 6525 (time 1790103719.10 total 431.27 last 22.72) - open/close 6719 (time 1790103729.20 total 441.37 last 19.20) - open/close 6914 (time 1790103739.21 total 451.38 last 19.49) - open/close 7096 (time 1790103749.21 total 461.38 last 18.19) - open/close 7344 (time 1790103759.22 total 471.39 last 24.79) - open/close 7552 (time 1790103769.90 total 482.07 last 19.47) - open/close 7787 (time 1790103779.91 total 492.08 last 23.47) - open/close 7999 (time 1790103789.94 total 502.11 last 21.14) - open/close 8256 (time 1790103799.95 total 512.12 last 25.68) - open/close 8477 (time 1790103810.49 total 522.66 last 20.98) - open/close 8725 (time 1790103820.50 total 532.67 last 24.77) - open/close 8941 (time 1790103830.77 total 542.94 last 21.03) - open/close 9126 (time 1790103840.78 total 552.95 last 18.48) - open/close 9366 (time 1790103850.79 total 562.96 last 23.98) - open/close 9615 (time 1790103861.29 total 573.46 last 23.71) - open/close 9853 (time 1790103871.78 total 583.95 last 22.68) - open/close 10000 (time 1790103876.32 total 588.49 last 32.39) - open/close 10194 (time 1790103886.65 total 598.82 last 18.79) - open/close 10402 (time 1790103896.65 total 608.82 last 20.78) - open/close 10639 (time 1790103906.66 total 618.83 last 23.70) - open/close 10842 (time 1790103917.16 total 629.33 last 19.33) - open/close 11102 (time 1790103927.16 total 639.33 last 25.99) - open/close 11308 (time 1790103937.83 total 650.00 last 19.31) - open/close 11538 (time 1790103947.85 total 660.02 last 22.96) - open/close 11772 (time 1790103958.84 total 671.01 last 21.29) - open/close 12023 (time 1790103968.84 total 681.01 last 25.10) - open/close 12189 (time 1790103978.85 total 691.02 last 16.59) - open/close 12293 (time 1790103989.05 total 701.22 last 10.19) - open/close 12421 (time 1790103999.05 total 711.22 last 12.80) - open/close 12529 (time 1790104009.96 total 722.13 last 9.91) - open/close 12717 (time 1790104020.10 total 732.27 last 18.53) - open/close 12855 (time 1790104030.53 total 742.70 last 13.24) - open/close 12938 (time 1790104040.82 total 752.99 last 8.06) - open/close 13090 (time 1790104050.85 total 763.02 last 15.16) - open/close 13225 (time 1790104061.16 total 773.33 last 13.09) - open/close 13318 (time 1790104071.52 total 783.69 last 8.98) - open/close 13415 (time 1790104082.36 total 794.53 last 8.95) - open/close 13509 (time 1790104092.46 total 804.63 last 9.31) - open/close 13726 (time 1790104102.47 total 814.64 last 21.67) - open/close 13891 (time 1790104112.54 total 824.71 last 16.39) - open/close 14000 (time 1790104122.68 total 834.85 last 10.75) - open/close 14147 (time 1790104132.87 total 845.04 last 14.42) - open/close 14392 (time 1790104143.41 total 855.58 last 23.24) - open/close 14567 (time 1790104154.49 total 866.66 last 15.80) - open/close 14817 (time 1790104164.49 total 876.66 last 24.99) - open/close 15050 (time 1790104175.28 total 887.45 last 21.59) - open/close 15171 (time 1790104186.37 total 898.54 last 10.91) - open/close 15416 (time 1790104196.38 total 908.55 last 24.50) - open/close 15644 (time 1790104206.89 total 919.06 last 21.68) - open/close 15872 (time 1790104217.84 total 930.01 last 20.83) - open/close 16049 (time 1790104227.86 total 940.03 last 17.66) - open/close 16271 (time 1790104238.65 total 950.82 last 20.58) - open/close 16460 (time 1790104248.96 total 961.13 last 18.34) - open/close 16599 (time 1790104259.05 total 971.22 last 13.77) - open/close 16850 (time 1790104269.05 total 981.22 last 25.10) - open/close 17094 (time 1790104280.05 total 992.22 last 22.19) - open/close 17291 (time 1790104290.41 total 1002.58 last 19.01) - open/close 17503 (time 1790104301.17 total 1013.34 last 19.71) - open/close 17740 (time 1790104312.17 total 1024.34 last 21.54) - open/close 18145 (time 1790104322.19 total 1034.36 last 40.44) - open/close 18776 (time 1790104332.20 total 1044.37 last 63.03) - open/close 19421 (time 1790104342.23 total 1054.40 last 64.31) - open/close 20000 (time 1790104350.23 total 1062.40 last 72.33) - open/close 20583 (time 1790104360.24 total 1072.41 last 58.25) - open/close 21292 (time 1790104370.25 total 1082.42 last 70.87) - open/close 21972 (time 1790104380.26 total 1092.43 last 67.94) - open/close 22678 (time 1790104390.27 total 1102.44 last 70.53) - open/close 23332 (time 1790104400.28 total 1112.45 last 65.34) - open/close 23937 (time 1790104410.29 total 1122.46 last 60.45) - open/close 24545 (time 1790104420.30 total 1132.47 last 60.73) - open/close 25197 (time 1790104430.30 total 1142.47 last 65.19) - open/close 25879 (time 1790104440.30 total 1152.47 last 68.18) - open/close 26430 (time 1790104450.31 total 1162.48 last 55.03) - open/close 27030 (time 1790104460.32 total 1172.49 last 59.97) - open/close 27624 (time 1790104470.34 total 1182.51 last 59.31) - open/close 28112 (time 1790104480.34 total 1192.51 last 48.80) - open/close 28604 (time 1790104490.34 total 1202.51 last 49.19) - open/close 29130 (time 1790104500.35 total 1212.52 last 52.56) - open/close 29762 (time 1790104510.36 total 1222.53 last 63.09) - open/close 30000 (time 1790104513.49 total 1225.66 last 76.23) - open/close 30604 (time 1790104523.49 total 1235.66 last 60.36) - open/close 31203 (time 1790104533.49 total 1245.66 last 59.89) - open/close 31824 (time 1790104543.49 total 1255.66 last 62.09) - open/close 32363 (time 1790104553.50 total 1265.67 last 53.87) - open/close 32975 (time 1790104563.50 total 1275.67 last 61.20) - open/close 33607 (time 1790104573.51 total 1285.68 last 63.16) - open/close 34161 (time 1790104583.53 total 1295.70 last 55.29) - open/close 34824 (time 1790104593.53 total 1305.70 last 66.26) - open/close 35471 (time 1790104603.55 total 1315.72 last 64.62) - open/close 36129 (time 1790104613.55 total 1325.72 last 65.77) - open/close 36730 (time 1790104623.55 total 1335.72 last 60.09) - open/close 37270 (time 1790104633.56 total 1345.73 last 53.99) - open/close 37868 (time 1790104643.56 total 1355.73 last 59.78) - open/close 38479 (time 1790104653.57 total 1365.74 last 61.06) - open/close 39099 (time 1790104663.57 total 1375.74 last 61.96) - open/close 39678 (time 1790104673.58 total 1385.75 last 57.83) - open/close 40000 (time 1790104679.82 total 1391.99 last 51.63) - open/close 40659 (time 1790104689.83 total 1402.00 last 65.85) - open/close 41245 (time 1790104699.87 total 1412.04 last 58.35) - open/close 41756 (time 1790104709.88 total 1422.05 last 51.06) - open/close 42327 (time 1790104719.88 total 1432.05 last 57.07) - open/close 42936 (time 1790104729.90 total 1442.07 last 60.83) - open/close 43507 (time 1790104739.90 total 1452.07 last 57.06) - open/close 44138 (time 1790104749.91 total 1462.08 last 63.05) - open/close 44661 (time 1790104759.92 total 1472.09 last 52.26) - open/close 45155 (time 1790104769.92 total 1482.09 last 49.40) - open/close 45611 (time 1790104779.95 total 1492.12 last 45.46) - open/close 46119 (time 1790104789.95 total 1502.12 last 50.79) - open/close 46729 (time 1790104799.95 total 1512.12 last 61.00) - open/close 47378 (time 1790104809.96 total 1522.13 last 64.86) - open/close 48045 (time 1790104819.96 total 1532.13 last 66.69) - open/close 48655 (time 1790104829.98 total 1542.15 last 60.83) - open/close 49242 (time 1790104839.99 total 1552.16 last 58.69) - open/close 49827 (time 1790104849.99 total 1562.16 last 58.48) total: 50000 open/close in 1564.93 seconds: 31.95 ops/second - unlinked 0 (time 1790104857 ; total 0 ; last 0) - unlinked 10000 (time 1790105033 ; total 176 ; last 176) - unlinked 20000 (time 1790105212 ; total 355 ; last 179) - unlinked 30000 (time 1790105397 ; total 540 ; last 185) - unlinked 40000 (time 1790105572 ; total 715 ; last 175) total: 50000 unlinks in 897 seconds: 55.741360 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg407-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server unloading modules via unload_modules_local on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing unload_modules_local oleg407-server: kmemleak disabled modules unloaded. pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2