== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:01:59 (1788332519) start mds service on oleg160-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 2 00:53 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing load_modules_local oleg160-server: kmemleak disabled oleg160-server: Loading modules from /home/green/git/lustre-release/lustre oleg160-server: detected 4 online CPUs by sysfs oleg160-server: MODOPTS_LIBCFS= oleg160-server: Force libcfs to create 2 CPU partitions oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg160-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg160-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg160-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-OST0000 oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg160-client.virtnet: -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 214 (time 1788332617.56 total 10.98 last 19.49) - open/close 292 (time 1788332627.86 total 21.28 last 7.57) - open/close 487 (time 1788332638.62 total 32.04 last 18.12) - open/close 633 (time 1788332649.27 total 42.70 last 13.70) - open/close 738 (time 1788332660.10 total 53.52 last 9.70) - open/close 897 (time 1788332670.12 total 63.54 last 15.86) - open/close 1133 (time 1788332680.14 total 73.57 last 23.55) - open/close 1360 (time 1788332690.16 total 83.58 last 22.66) - open/close 1421 (time 1788332700.22 total 93.64 last 6.07) - open/close 1638 (time 1788332712.89 total 106.31 last 17.13) - open/close 1769 (time 1788332723.91 total 117.33 last 11.89) - open/close 1930 (time 1788332733.92 total 127.34 last 16.08) - open/close 2153 (time 1788332743.94 total 137.36 last 22.27) - open/close 2293 (time 1788332754.34 total 147.76 last 13.46) - open/close 2451 (time 1788332765.23 total 158.65 last 14.51) - open/close 2668 (time 1788332775.24 total 168.66 last 21.67) - open/close 2877 (time 1788332785.62 total 179.04 last 20.13) - open/close 3115 (time 1788332795.63 total 189.05 last 23.78) - open/close 3296 (time 1788332805.66 total 199.08 last 18.05) - open/close 3506 (time 1788332816.09 total 209.51 last 20.13) - open/close 3755 (time 1788332826.11 total 219.53 last 24.85) - open/close 3966 (time 1788332836.26 total 229.68 last 20.79) - open/close 4131 (time 1788332846.79 total 240.21 last 15.68) - open/close 4335 (time 1788332857.46 total 250.88 last 19.11) - open/close 4481 (time 1788332868.02 total 261.44 last 13.83) - open/close 4777 (time 1788332878.70 total 272.12 last 27.72) - open/close 4998 (time 1788332888.72 total 282.14 last 22.07) - open/close 5202 (time 1788332899.62 total 293.04 last 18.70) - open/close 5383 (time 1788332909.74 total 303.16 last 17.89) - open/close 5559 (time 1788332919.75 total 313.18 last 17.58) - open/close 5644 (time 1788332931.05 total 324.47 last 7.53) - open/close 5834 (time 1788332941.83 total 335.25 last 17.62) - open/close 5967 (time 1788332952.68 total 346.10 last 12.25) - open/close 6142 (time 1788332963.50 total 356.92 last 16.18) - open/close 6308 (time 1788332974.36 total 367.78 last 15.28) - open/close 6476 (time 1788332984.48 total 377.90 last 16.59) - open/close 6590 (time 1788332995.29 total 388.71 last 10.55) - open/close 6840 (time 1788333006.31 total 399.73 last 22.68) - open/close 7047 (time 1788333016.31 total 409.74 last 20.70) - open/close 7283 (time 1788333026.32 total 419.74 last 23.60) - open/close 7484 (time 1788333037.09 total 430.51 last 18.66) - open/close 7768 (time 1788333048.11 total 441.53 last 25.76) - open/close 8027 (time 1788333058.12 total 451.54 last 25.89) - open/close 8266 (time 1788333069.16 total 462.58 last 21.65) - open/close 8433 (time 1788333079.17 total 472.59 last 16.68) - open/close 8581 (time 1788333089.17 total 482.60 last 14.80) - open/close 8631 (time 1788333099.22 total 492.64 last 4.98) - open/close 8791 (time 1788333109.22 total 502.64 last 16.00) - open/close 8884 (time 1788333119.23 total 512.65 last 9.29) - open/close 8990 (time 1788333129.27 total 522.69 last 10.56) - open/close 9098 (time 1788333139.90 total 533.32 last 10.16) - open/close 9323 (time 1788333150.14 total 543.56 last 21.99) - open/close 9488 (time 1788333160.24 total 553.67 last 16.32) - open/close 9623 (time 1788333170.68 total 564.10 last 12.94) - open/close 9637 (time 1788333181.44 total 574.86 last 1.30) - open/close 9801 (time 1788333191.80 total 585.22 last 15.84) - open/close 10000 (time 1788333200.92 total 594.34 last 21.81) - open/close 10122 (time 1788333210.94 total 604.36 last 12.18) - open/close 10326 (time 1788333221.03 total 614.45 last 20.22) - open/close 10425 (time 1788333231.73 total 625.15 last 9.25) - open/close 10578 (time 1788333241.74 total 635.16 last 15.28) - open/close 10826 (time 1788333251.75 total 645.17 last 24.78) - open/close 10975 (time 1788333261.77 total 655.19 last 14.87) - open/close 11191 (time 1788333272.77 total 666.19 last 19.63) - open/close 11399 (time 1788333282.78 total 676.21 last 20.77) - open/close 11543 (time 1788333293.28 total 686.70 last 13.72) - open/close 11634 (time 1788333303.29 total 696.71 last 9.09) - open/close 11773 (time 1788333314.23 total 707.65 last 12.70) - open/close 11896 (time 1788333324.25 total 717.67 last 12.27) - open/close 11960 (time 1788333334.64 total 728.06 last 6.16) - open/close 12033 (time 1788333344.65 total 738.07 last 7.29) - open/close 12181 (time 1788333355.49 total 748.91 last 13.65) - open/close 12304 (time 1788333365.93 total 759.35 last 11.78) - open/close 12552 (time 1788333375.93 total 769.35 last 24.79) - open/close 12706 (time 1788333388.72 total 782.14 last 12.05) - open/close 12849 (time 1788333399.38 total 792.80 last 13.41) - open/close 12988 (time 1788333409.42 total 802.84 last 13.85) - open/close 13124 (time 1788333419.43 total 812.85 last 13.58) - open/close 13362 (time 1788333429.44 total 822.86 last 23.79) - open/close 13576 (time 1788333439.99 total 833.41 last 20.27) - open/close 13817 (time 1788333450.87 total 844.29 last 22.15) - open/close 14025 (time 1788333463.14 total 856.56 last 16.95) - open/close 14165 (time 1788333473.64 total 867.06 last 13.34) - open/close 14396 (time 1788333484.56 total 877.98 last 21.14) - open/close 14635 (time 1788333495.66 total 889.08 last 21.53) - open/close 14829 (time 1788333505.97 total 899.39 last 18.83) - open/close 14948 (time 1788333516.44 total 909.87 last 11.36) - open/close 15082 (time 1788333527.33 total 920.75 last 12.31) - open/close 15178 (time 1788333537.44 total 930.86 last 9.49) - open/close 15297 (time 1788333548.16 total 941.58 last 11.10) - open/close 15423 (time 1788333560.63 total 954.05 last 10.11) - open/close 15601 (time 1788333571.07 total 964.50 last 17.04) - open/close 15808 (time 1788333581.09 total 974.51 last 20.67) - open/close 15951 (time 1788333592.82 total 986.25 last 12.18) - open/close 16040 (time 1788333603.37 total 996.79 last 8.44) - open/close 16166 (time 1788333613.79 total 1007.21 last 12.09) - open/close 16353 (time 1788333624.36 total 1017.78 last 17.69) - open/close 16571 (time 1788333637.35 total 1030.77 last 16.78) - open/close 16753 (time 1788333647.35 total 1040.77 last 18.19) - open/close 17085 (time 1788333657.36 total 1050.78 last 33.18) - open/close 17852 (time 1788333667.37 total 1060.79 last 76.63) - open/close 18503 (time 1788333677.37 total 1070.79 last 65.07) - open/close 19102 (time 1788333687.41 total 1080.84 last 59.65) - open/close 19738 (time 1788333697.42 total 1090.84 last 63.58) - open/close 20000 (time 1788333701.13 total 1094.55 last 70.62) - open/close 20645 (time 1788333711.14 total 1104.56 last 64.45) - open/close 21391 (time 1788333721.14 total 1114.56 last 74.56) - open/close 22058 (time 1788333731.18 total 1124.60 last 66.44) - open/close 22674 (time 1788333741.23 total 1134.65 last 61.28) - open/close 23226 (time 1788333751.24 total 1144.66 last 55.15) - open/close 23709 (time 1788333761.25 total 1154.67 last 48.26) - open/close 24330 (time 1788333771.26 total 1164.68 last 62.06) - open/close 25014 (time 1788333781.26 total 1174.68 last 68.40) - open/close 25754 (time 1788333791.27 total 1184.69 last 73.94) - open/close 26447 (time 1788333801.28 total 1194.70 last 69.22) - open/close 27139 (time 1788333811.28 total 1204.70 last 69.15) - open/close 27900 (time 1788333821.32 total 1214.74 last 75.86) - open/close 28600 (time 1788333831.32 total 1224.74 last 69.98) - open/close 29311 (time 1788333841.33 total 1234.75 last 71.02) - open/close 30000 (time 1788333850.37 total 1243.79 last 76.20) - open/close 30789 (time 1788333860.39 total 1253.81 last 78.78) - open/close 31451 (time 1788333870.39 total 1263.81 last 66.16) - open/close 32096 (time 1788333880.41 total 1273.83 last 64.38) - open/close 32798 (time 1788333890.41 total 1283.83 last 70.19) - open/close 33433 (time 1788333900.42 total 1293.84 last 63.44) - open/close 34091 (time 1788333910.43 total 1303.85 last 65.73) - open/close 34802 (time 1788333920.45 total 1313.87 last 70.97) - open/close 35454 (time 1788333930.46 total 1323.88 last 65.14) - open/close 36015 (time 1788333940.47 total 1333.89 last 56.06) - open/close 36698 (time 1788333950.47 total 1343.89 last 68.29) - open/close 37403 (time 1788333960.47 total 1353.89 last 70.47) - open/close 38042 (time 1788333970.48 total 1363.90 last 63.88) - open/close 38677 (time 1788333980.49 total 1373.91 last 63.42) - open/close 39395 (time 1788333990.49 total 1383.91 last 71.78) - open/close 40000 (time 1788333999.56 total 1392.98 last 66.75) - open/close 40652 (time 1788334009.56 total 1402.99 last 65.14) - open/close 41302 (time 1788334019.56 total 1412.99 last 65.00) - open/close 41977 (time 1788334029.57 total 1422.99 last 67.50) - open/close 42651 (time 1788334039.58 total 1433.00 last 67.31) - open/close 43347 (time 1788334049.58 total 1443.01 last 69.56) - open/close 44043 (time 1788334059.58 total 1453.01 last 69.60) - open/close 44766 (time 1788334069.59 total 1463.01 last 72.30) - open/close 45436 (time 1788334079.59 total 1473.01 last 67.00) - open/close 46075 (time 1788334089.60 total 1483.02 last 63.80) - open/close 46703 (time 1788334099.63 total 1493.05 last 62.64) - open/close 47427 (time 1788334109.64 total 1503.06 last 72.34) - open/close 48139 (time 1788334119.64 total 1513.06 last 71.19) - open/close 48820 (time 1788334129.64 total 1523.06 last 68.06) - open/close 49601 (time 1788334139.65 total 1533.07 last 78.05) total: 50000 open/close in 1538.44 seconds: 32.50 ops/second - unlinked 0 (time 1788334149 ; total 0 ; last 0) - unlinked 10000 (time 1788334296 ; total 147 ; last 147) - unlinked 20000 (time 1788334437 ; total 288 ; last 141) - unlinked 30000 (time 1788334594 ; total 445 ; last 157) - unlinked 40000 (time 1788334747 ; total 598 ; last 153) total: 50000 unlinks in 752 seconds: 66.489365 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg160-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server unloading modules via unload_modules_local on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing unload_modules_local oleg160-server: kmemleak disabled modules unloaded. pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2