== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:14:51 (1788837291) start mds service on oleg327-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1754936 Sep 7 17:53 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing load_modules_local oleg327-server: kmemleak disabled oleg327-server: Loading modules from /home/green/git/lustre-release/lustre oleg327-server: detected 4 online CPUs by sysfs oleg327-server: MODOPTS_LIBCFS= oleg327-server: Force libcfs to create 2 CPU partitions oleg327-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg327-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg327-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg327-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg327-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-OST0000 oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg327-client.virtnet: -o user_xattr,flock 192.168.203.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 222 (time 1788837371.70 total 10.00 last 22.19) - open/close 434 (time 1788837382.07 total 20.37 last 20.45) - open/close 703 (time 1788837392.70 total 31.00 last 25.31) - open/close 873 (time 1788837402.73 total 41.02 last 16.96) - open/close 1093 (time 1788837412.74 total 51.04 last 21.96) - open/close 1154 (time 1788837422.75 total 61.05 last 6.10) - open/close 1371 (time 1788837433.07 total 71.37 last 21.02) - open/close 1476 (time 1788837443.46 total 81.76 last 10.11) - open/close 1637 (time 1788837453.90 total 92.19 last 15.43) - open/close 1782 (time 1788837464.77 total 103.07 last 13.34) - open/close 1989 (time 1788837477.75 total 116.05 last 15.94) - open/close 2192 (time 1788837488.13 total 126.43 last 19.56) - open/close 2406 (time 1788837498.14 total 136.44 last 21.38) - open/close 2594 (time 1788837508.31 total 146.61 last 18.49) - open/close 2836 (time 1788837519.07 total 157.37 last 22.49) - open/close 2856 (time 1788837529.91 total 168.21 last 1.84) - open/close 3011 (time 1788837540.60 total 178.90 last 14.50) - open/close 3169 (time 1788837550.91 total 189.21 last 15.32) - open/close 3407 (time 1788837561.54 total 199.84 last 22.38) - open/close 3585 (time 1788837571.55 total 209.85 last 17.79) - open/close 3798 (time 1788837581.69 total 219.99 last 21.00) - open/close 3984 (time 1788837591.72 total 230.02 last 18.55) - open/close 4171 (time 1788837603.07 total 241.37 last 16.48) - open/close 4346 (time 1788837614.01 total 252.31 last 15.99) - open/close 4500 (time 1788837624.71 total 263.01 last 14.39) - open/close 4541 (time 1788837636.23 total 274.53 last 3.56) - open/close 4714 (time 1788837646.97 total 285.27 last 16.10) - open/close 4888 (time 1788837656.98 total 295.28 last 17.39) - open/close 5095 (time 1788837668.03 total 306.33 last 18.74) - open/close 5209 (time 1788837678.03 total 316.33 last 11.39) - open/close 5339 (time 1788837688.04 total 326.34 last 12.99) - open/close 5425 (time 1788837698.82 total 337.12 last 7.97) - open/close 5536 (time 1788837709.70 total 347.99 last 10.21) - open/close 5697 (time 1788837719.73 total 358.03 last 16.04) - open/close 5845 (time 1788837730.24 total 368.54 last 14.08) - open/close 5965 (time 1788837740.25 total 378.55 last 11.99) - open/close 6038 (time 1788837750.27 total 388.57 last 7.28) - open/close 6269 (time 1788837762.29 total 400.59 last 19.22) - open/close 6483 (time 1788837772.30 total 410.60 last 21.38) - open/close 6704 (time 1788837782.32 total 420.62 last 22.06) - open/close 6887 (time 1788837792.51 total 430.81 last 17.97) - open/close 7070 (time 1788837804.21 total 442.51 last 15.64) - open/close 7216 (time 1788837814.99 total 453.29 last 13.55) - open/close 7444 (time 1788837824.99 total 463.29 last 22.79) - open/close 7525 (time 1788837835.00 total 473.30 last 8.10) - open/close 7644 (time 1788837845.24 total 483.54 last 11.62) - open/close 7879 (time 1788837855.60 total 493.90 last 22.68) - open/close 8074 (time 1788837865.60 total 503.90 last 19.50) - open/close 8289 (time 1788837877.81 total 516.11 last 17.61) - open/close 8535 (time 1788837887.82 total 526.11 last 24.59) - open/close 8749 (time 1788837897.82 total 536.12 last 21.39) - open/close 8992 (time 1788837908.73 total 547.03 last 22.27) - open/close 9195 (time 1788837918.74 total 557.03 last 20.29) - open/close 9365 (time 1788837928.76 total 567.06 last 16.96) - open/close 9457 (time 1788837939.29 total 577.59 last 8.74) - open/close 9578 (time 1788837949.50 total 587.80 last 11.85) - open/close 9702 (time 1788837960.01 total 598.31 last 11.81) - open/close 9803 (time 1788837970.01 total 608.31 last 10.09) - open/close 9865 (time 1788837980.05 total 618.35 last 6.18) - open/close 9929 (time 1788837991.36 total 629.66 last 5.66) - open/close 10000 (time 1788837993.89 total 632.19 last 28.03) - open/close 10168 (time 1788838004.63 total 642.93 last 15.64) - open/close 10318 (time 1788838016.44 total 654.74 last 12.70) - open/close 10537 (time 1788838027.13 total 665.43 last 20.48) - open/close 10716 (time 1788838037.14 total 675.44 last 17.88) - open/close 10898 (time 1788838047.29 total 685.59 last 17.93) - open/close 11133 (time 1788838057.30 total 695.60 last 23.48) - open/close 11369 (time 1788838067.31 total 705.61 last 23.58) - open/close 11453 (time 1788838077.43 total 715.73 last 8.30) - open/close 11605 (time 1788838087.94 total 726.24 last 14.46) - open/close 11791 (time 1788838097.95 total 736.25 last 18.58) - open/close 11977 (time 1788838108.08 total 746.38 last 18.36) - open/close 12022 (time 1788838119.68 total 757.98 last 3.88) - open/close 12133 (time 1788838129.70 total 768.00 last 11.08) - open/close 12304 (time 1788838139.84 total 778.14 last 16.88) - open/close 12600 (time 1788838150.85 total 789.14 last 26.89) - open/close 12787 (time 1788838160.85 total 799.15 last 18.69) - open/close 12923 (time 1788838170.86 total 809.15 last 13.60) - open/close 13125 (time 1788838180.87 total 819.17 last 20.17) - open/close 13398 (time 1788838191.89 total 830.19 last 24.79) - open/close 13619 (time 1788838201.90 total 840.20 last 22.08) - open/close 13817 (time 1788838212.62 total 850.92 last 18.46) - open/close 13865 (time 1788838222.72 total 861.01 last 4.76) - open/close 14150 (time 1788838232.72 total 871.02 last 28.48) - open/close 14181 (time 1788838242.74 total 881.04 last 3.10) - open/close 14327 (time 1788838253.17 total 891.47 last 13.99) - open/close 14407 (time 1788838265.46 total 903.76 last 6.51) - open/close 14534 (time 1788838275.64 total 913.94 last 12.48) - open/close 14731 (time 1788838286.57 total 924.87 last 18.02) - open/close 14964 (time 1788838296.76 total 935.06 last 22.87) - open/close 15031 (time 1788838307.80 total 946.10 last 6.07) - open/close 15148 (time 1788838318.21 total 956.51 last 11.23) - open/close 15377 (time 1788838328.39 total 966.69 last 22.51) - open/close 15676 (time 1788838338.40 total 976.70 last 29.87) - open/close 15868 (time 1788838348.68 total 986.98 last 18.67) - open/close 16094 (time 1788838359.42 total 997.71 last 21.06) - open/close 16322 (time 1788838370.43 total 1008.73 last 20.69) - open/close 16438 (time 1788838380.44 total 1018.74 last 11.59) - open/close 16615 (time 1788838390.72 total 1029.02 last 17.22) - open/close 17175 (time 1788838400.74 total 1039.04 last 55.86) - open/close 17777 (time 1788838410.76 total 1049.06 last 60.11) - open/close 18436 (time 1788838420.77 total 1059.07 last 65.83) - open/close 19112 (time 1788838430.80 total 1069.09 last 67.43) - open/close 19773 (time 1788838440.81 total 1079.11 last 66.02) - open/close 20000 (time 1788838443.88 total 1082.18 last 73.92) - open/close 20662 (time 1788838453.88 total 1092.18 last 66.16) - open/close 21284 (time 1788838463.89 total 1102.19 last 62.18) - open/close 21885 (time 1788838473.90 total 1112.20 last 60.02) - open/close 22519 (time 1788838483.91 total 1122.20 last 63.37) - open/close 23148 (time 1788838493.92 total 1132.21 last 62.84) - open/close 23762 (time 1788838503.93 total 1142.23 last 61.31) - open/close 24342 (time 1788838513.98 total 1152.28 last 57.73) - open/close 24899 (time 1788838523.98 total 1162.28 last 55.68) - open/close 25511 (time 1788838533.99 total 1172.29 last 61.15) - open/close 26035 (time 1788838544.00 total 1182.29 last 52.36) - open/close 26680 (time 1788838554.00 total 1192.30 last 64.50) - open/close 27320 (time 1788838564.00 total 1202.30 last 64.00) - open/close 27968 (time 1788838574.01 total 1212.31 last 64.71) - open/close 28607 (time 1788838584.04 total 1222.34 last 63.73) - open/close 29264 (time 1788838594.04 total 1232.34 last 65.67) - open/close 29932 (time 1788838604.06 total 1242.36 last 66.67) - open/close 30493 (time 1788838614.06 total 1252.36 last 56.09) - open/close 31193 (time 1788838624.08 total 1262.38 last 69.86) - open/close 31755 (time 1788838634.09 total 1272.39 last 56.13) - open/close 32414 (time 1788838644.10 total 1282.40 last 65.84) - open/close 33005 (time 1788838654.12 total 1292.42 last 59.02) - open/close 33687 (time 1788838664.13 total 1302.42 last 68.14) - open/close 34327 (time 1788838674.15 total 1312.44 last 63.88) - open/close 34986 (time 1788838684.15 total 1322.45 last 65.88) - open/close 35608 (time 1788838694.15 total 1332.45 last 62.20) - open/close 36290 (time 1788838704.15 total 1342.45 last 68.20) - open/close 36947 (time 1788838714.15 total 1352.45 last 65.67) - open/close 37370 (time 1788838724.16 total 1362.46 last 42.25) - open/close 37880 (time 1788838734.18 total 1372.47 last 50.94) - open/close 38541 (time 1788838744.19 total 1382.49 last 66.01) - open/close 39135 (time 1788838754.20 total 1392.50 last 59.33) - open/close 39781 (time 1788838764.21 total 1402.51 last 64.54) - open/close 40000 (time 1788838767.35 total 1405.65 last 69.78) - open/close 40643 (time 1788838777.35 total 1415.65 last 64.29) - open/close 41263 (time 1788838787.35 total 1425.65 last 61.99) - open/close 41937 (time 1788838797.36 total 1435.66 last 67.37) - open/close 42556 (time 1788838807.36 total 1445.66 last 61.86) - open/close 43147 (time 1788838817.38 total 1455.68 last 59.01) - open/close 43720 (time 1788838827.40 total 1465.69 last 57.21) - open/close 44369 (time 1788838837.40 total 1475.70 last 64.86) - open/close 45076 (time 1788838847.41 total 1485.71 last 70.63) - open/close 45716 (time 1788838857.42 total 1495.71 last 63.98) - open/close 46393 (time 1788838867.42 total 1505.72 last 67.66) - open/close 47058 (time 1788838877.44 total 1515.74 last 66.37) - open/close 47747 (time 1788838887.45 total 1525.74 last 68.88) - open/close 48405 (time 1788838897.45 total 1535.75 last 65.74) - open/close 49034 (time 1788838907.47 total 1545.76 last 62.83) - open/close 49648 (time 1788838917.48 total 1555.78 last 61.31) total: 50000 open/close in 1561.97 seconds: 32.01 ops/second - unlinked 0 (time 1788838928 ; total 0 ; last 0) - unlinked 10000 (time 1788839097 ; total 169 ; last 169) - unlinked 20000 (time 1788839257 ; total 329 ; last 160) - unlinked 30000 (time 1788839417 ; total 489 ; last 160) - unlinked 40000 (time 1788839581 ; total 653 ; last 164) total: 50000 unlinks in 812 seconds: 61.576355 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg327-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg327-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg327-server stop mds service on oleg327-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg327-server unloading modules via unload_modules_local on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing unload_modules_local oleg327-server: kmemleak disabled modules unloaded. pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2