== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:49:56 (1788576596) start mds service on oleg356-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1944872 Sep 4 04:21 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing load_modules_local oleg356-server: kmemleak disabled oleg356-server: Loading modules from /home/green/git/lustre-release/lustre oleg356-server: detected 4 online CPUs by sysfs oleg356-server: MODOPTS_LIBCFS= oleg356-server: Force libcfs to create 2 CPU partitions oleg356-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg356-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg356-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg356-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg356-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 208 (time 1788576676.21 total 10.00 last 20.79) - open/close 463 (time 1788576686.86 total 20.65 last 23.94) - open/close 721 (time 1788576697.63 total 31.43 last 23.94) - open/close 936 (time 1788576707.64 total 41.44 last 21.49) - open/close 1206 (time 1788576718.54 total 52.34 last 24.77) - open/close 1428 (time 1788576728.55 total 62.35 last 22.19) - open/close 1634 (time 1788576738.56 total 72.35 last 20.59) - open/close 1859 (time 1788576748.57 total 82.37 last 22.47) - open/close 2032 (time 1788576758.58 total 92.38 last 17.28) - open/close 2231 (time 1788576768.92 total 102.72 last 19.25) - open/close 2464 (time 1788576779.03 total 112.83 last 23.05) - open/close 2681 (time 1788576789.78 total 123.58 last 20.19) - open/close 2861 (time 1788576799.92 total 133.72 last 17.75) - open/close 3112 (time 1788576810.59 total 144.39 last 23.53) - open/close 3323 (time 1788576820.94 total 154.73 last 20.39) - open/close 3541 (time 1788576831.50 total 165.30 last 20.64) - open/close 3779 (time 1788576841.50 total 175.30 last 23.79) - open/close 3992 (time 1788576852.05 total 185.85 last 20.20) - open/close 4213 (time 1788576862.74 total 196.54 last 20.67) - open/close 4428 (time 1788576873.43 total 207.23 last 20.12) - open/close 4598 (time 1788576883.72 total 217.52 last 16.52) - open/close 4779 (time 1788576894.75 total 228.54 last 16.42) - open/close 5040 (time 1788576905.82 total 239.62 last 23.57) - open/close 5259 (time 1788576916.24 total 250.04 last 21.01) - open/close 5511 (time 1788576926.93 total 260.73 last 23.58) - open/close 5750 (time 1788576937.49 total 271.28 last 22.64) - open/close 5984 (time 1788576948.13 total 281.93 last 21.99) - open/close 6232 (time 1788576961.18 total 294.98 last 19.00) - open/close 6464 (time 1788576971.20 total 304.99 last 23.17) - open/close 6726 (time 1788576981.99 total 315.79 last 24.27) - open/close 6907 (time 1788576992.62 total 326.42 last 17.03) - open/close 7159 (time 1788577002.65 total 336.44 last 25.14) - open/close 7403 (time 1788577012.65 total 346.45 last 24.39) - open/close 7588 (time 1788577023.26 total 357.06 last 17.44) - open/close 7813 (time 1788577033.27 total 367.07 last 22.47) - open/close 8048 (time 1788577043.28 total 377.07 last 23.49) - open/close 8239 (time 1788577053.53 total 387.33 last 18.63) - open/close 8426 (time 1788577064.54 total 398.34 last 16.98) - open/close 8570 (time 1788577076.98 total 410.77 last 11.58) - open/close 8772 (time 1788577088.01 total 421.80 last 18.31) - open/close 8932 (time 1788577098.01 total 431.81 last 15.99) - open/close 9086 (time 1788577108.02 total 441.81 last 15.39) - open/close 9336 (time 1788577118.67 total 452.47 last 23.47) - open/close 9500 (time 1788577128.97 total 462.76 last 15.93) - open/close 9732 (time 1788577139.10 total 472.89 last 22.90) - open/close 9917 (time 1788577150.17 total 483.96 last 16.71) - open/close 10000 (time 1788577152.99 total 486.79 last 29.39) - open/close 10239 (time 1788577163.09 total 496.89 last 23.65) - open/close 10415 (time 1788577173.78 total 507.58 last 16.47) - open/close 10639 (time 1788577184.14 total 517.94 last 21.63) - open/close 10890 (time 1788577194.18 total 527.97 last 25.01) - open/close 11140 (time 1788577204.46 total 538.26 last 24.31) - open/close 11355 (time 1788577214.94 total 548.74 last 20.52) - open/close 11563 (time 1788577225.76 total 559.55 last 19.23) - open/close 11800 (time 1788577236.25 total 570.05 last 22.58) - open/close 12001 (time 1788577246.25 total 580.05 last 20.10) - open/close 12151 (time 1788577256.91 total 590.71 last 14.07) - open/close 12355 (time 1788577267.87 total 601.67 last 18.62) - open/close 12533 (time 1788577278.48 total 612.28 last 16.77) - open/close 12673 (time 1788577288.49 total 622.29 last 13.99) - open/close 12918 (time 1788577299.23 total 633.03 last 22.81) - open/close 13146 (time 1788577311.39 total 645.19 last 18.75) - open/close 13291 (time 1788577321.49 total 655.29 last 14.36) - open/close 13523 (time 1788577332.05 total 665.84 last 21.98) - open/close 13765 (time 1788577342.05 total 675.84 last 24.20) - open/close 13844 (time 1788577352.48 total 686.28 last 7.57) - open/close 13935 (time 1788577364.57 total 698.37 last 7.53) - open/close 14040 (time 1788577374.93 total 708.72 last 10.14) - open/close 14261 (time 1788577385.42 total 719.22 last 21.07) - open/close 14434 (time 1788577396.82 total 730.61 last 15.18) - open/close 14647 (time 1788577406.83 total 740.62 last 21.28) - open/close 14851 (time 1788577417.17 total 750.97 last 19.72) - open/close 15085 (time 1788577427.75 total 761.55 last 22.12) - open/close 15315 (time 1788577438.02 total 771.81 last 22.41) - open/close 15525 (time 1788577448.02 total 781.82 last 20.98) - open/close 15636 (time 1788577459.11 total 792.91 last 10.01) - open/close 15813 (time 1788577469.15 total 802.94 last 17.64) - open/close 15997 (time 1788577479.38 total 813.17 last 17.99) - open/close 16178 (time 1788577489.95 total 823.75 last 17.12) - open/close 16326 (time 1788577499.95 total 833.75 last 14.80) - open/close 16513 (time 1788577509.96 total 843.75 last 18.69) - open/close 16720 (time 1788577520.73 total 854.53 last 19.22) - open/close 16859 (time 1788577531.03 total 864.83 last 13.49) - open/close 16965 (time 1788577541.13 total 874.93 last 10.50) - open/close 17203 (time 1788577552.65 total 886.44 last 20.66) - open/close 17347 (time 1788577562.65 total 896.45 last 14.39) - open/close 17608 (time 1788577573.65 total 907.45 last 23.72) - open/close 18260 (time 1788577583.66 total 917.46 last 65.15) - open/close 18849 (time 1788577593.67 total 927.47 last 58.83) - open/close 19481 (time 1788577603.68 total 937.48 last 63.13) - open/close 20000 (time 1788577611.68 total 945.48 last 64.92) - open/close 20704 (time 1788577621.68 total 955.48 last 70.39) - open/close 21365 (time 1788577631.69 total 965.48 last 66.06) - open/close 22018 (time 1788577641.70 total 975.50 last 65.21) - open/close 22674 (time 1788577651.71 total 985.51 last 65.55) - open/close 23376 (time 1788577661.72 total 995.52 last 70.09) - open/close 24038 (time 1788577671.73 total 1005.52 last 66.17) - open/close 24721 (time 1788577681.73 total 1015.53 last 68.28) - open/close 25378 (time 1788577691.73 total 1025.53 last 65.68) - open/close 26080 (time 1788577701.74 total 1035.54 last 70.13) - open/close 26780 (time 1788577711.74 total 1045.54 last 70.00) - open/close 27394 (time 1788577721.75 total 1055.55 last 61.34) - open/close 28043 (time 1788577731.76 total 1065.56 last 64.87) - open/close 28779 (time 1788577741.77 total 1075.57 last 73.51) - open/close 29447 (time 1788577751.79 total 1085.58 last 66.69) - open/close 30000 (time 1788577760.25 total 1094.05 last 65.32) - open/close 30607 (time 1788577770.26 total 1104.06 last 60.66) - open/close 31255 (time 1788577780.27 total 1114.07 last 64.72) - open/close 31887 (time 1788577790.27 total 1124.07 last 63.19) - open/close 32522 (time 1788577800.27 total 1134.07 last 63.50) - open/close 33128 (time 1788577810.28 total 1144.08 last 60.56) - open/close 33753 (time 1788577820.29 total 1154.08 last 62.46) - open/close 34440 (time 1788577830.30 total 1164.10 last 68.60) - open/close 35088 (time 1788577840.31 total 1174.11 last 64.72) - open/close 35720 (time 1788577850.33 total 1184.13 last 63.08) - open/close 36289 (time 1788577860.33 total 1194.13 last 56.88) - open/close 36905 (time 1788577870.34 total 1204.14 last 61.54) - open/close 37583 (time 1788577880.35 total 1214.14 last 67.79) - open/close 38250 (time 1788577890.37 total 1224.17 last 66.52) - open/close 38885 (time 1788577900.38 total 1234.17 last 63.48) - open/close 39563 (time 1788577910.38 total 1244.18 last 67.74) - open/close 40000 (time 1788577916.97 total 1250.77 last 66.33) - open/close 40599 (time 1788577926.99 total 1260.79 last 59.78) - open/close 41281 (time 1788577937.00 total 1270.80 last 68.16) - open/close 41859 (time 1788577947.01 total 1280.81 last 57.74) - open/close 42445 (time 1788577957.02 total 1290.82 last 58.54) - open/close 43017 (time 1788577967.04 total 1300.83 last 57.10) - open/close 43600 (time 1788577977.05 total 1310.85 last 58.21) - open/close 44297 (time 1788577987.07 total 1320.87 last 69.58) - open/close 44972 (time 1788577997.11 total 1330.91 last 67.22) - open/close 45679 (time 1788578007.12 total 1340.92 last 70.66) - open/close 46374 (time 1788578017.13 total 1350.93 last 69.40) - open/close 46998 (time 1788578027.15 total 1360.95 last 62.27) - open/close 47608 (time 1788578037.17 total 1370.97 last 60.90) - open/close 48293 (time 1788578047.18 total 1380.97 last 68.45) - open/close 48950 (time 1788578057.18 total 1390.98 last 65.68) - open/close 49548 (time 1788578067.19 total 1400.99 last 59.71) total: 50000 open/close in 1408.79 seconds: 35.49 ops/second - unlinked 0 (time 1788578079 ; total 0 ; last 0) - unlinked 10000 (time 1788578241 ; total 162 ; last 162) - unlinked 20000 (time 1788578397 ; total 318 ; last 156) - unlinked 30000 (time 1788578548 ; total 469 ; last 151) - unlinked 40000 (time 1788578698 ; total 619 ; last 150) total: 50000 unlinks in 769 seconds: 65.019508 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg356-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server unloading modules via unload_modules_local on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing unload_modules_local oleg356-server: kmemleak disabled modules unloaded. pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2