== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:09:16 (1788624556) start mds service on oleg333-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 4 17:42 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing load_modules_local oleg333-server: kmemleak disabled oleg333-server: Loading modules from /home/green/git/lustre-release/lustre oleg333-server: detected 4 online CPUs by sysfs oleg333-server: MODOPTS_LIBCFS= oleg333-server: Force libcfs to create 2 CPU partitions oleg333-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg333-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg333-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg333-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg333-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-OST0000 oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg333-client.virtnet: -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 228 (time 1788624644.02 total 10.95 last 20.82) - open/close 427 (time 1788624654.07 total 21.00 last 19.80) - open/close 691 (time 1788624664.58 total 31.51 last 25.12) - open/close 938 (time 1788624675.57 total 42.50 last 22.48) - open/close 1157 (time 1788624686.02 total 52.95 last 20.96) - open/close 1382 (time 1788624696.66 total 63.59 last 21.15) - open/close 1562 (time 1788624706.67 total 73.60 last 17.99) - open/close 1775 (time 1788624716.67 total 83.60 last 21.29) - open/close 2037 (time 1788624726.68 total 93.61 last 26.18) - open/close 2199 (time 1788624738.50 total 105.43 last 13.71) - open/close 2321 (time 1788624748.51 total 115.44 last 12.18) - open/close 2482 (time 1788624758.52 total 125.45 last 16.08) - open/close 2701 (time 1788624768.82 total 135.74 last 21.28) - open/close 2923 (time 1788624779.53 total 146.46 last 20.72) - open/close 3111 (time 1788624789.54 total 156.47 last 18.77) - open/close 3334 (time 1788624800.25 total 167.18 last 20.83) - open/close 3547 (time 1788624810.93 total 177.86 last 19.94) - open/close 3802 (time 1788624821.96 total 188.89 last 23.12) - open/close 3989 (time 1788624832.25 total 199.18 last 18.17) - open/close 4220 (time 1788624842.26 total 209.19 last 23.08) - open/close 4374 (time 1788624852.86 total 219.78 last 14.54) - open/close 4501 (time 1788624862.89 total 229.82 last 12.65) - open/close 4618 (time 1788624875.00 total 241.93 last 9.66) - open/close 4769 (time 1788624885.57 total 252.50 last 14.29) - open/close 4832 (time 1788624896.33 total 263.26 last 5.86) - open/close 5003 (time 1788624908.35 total 275.28 last 14.22) - open/close 5146 (time 1788624919.12 total 286.04 last 13.28) - open/close 5245 (time 1788624930.24 total 297.17 last 8.90) - open/close 5365 (time 1788624940.81 total 307.74 last 11.36) - open/close 5604 (time 1788624950.82 total 317.75 last 23.87) - open/close 5840 (time 1788624960.83 total 327.76 last 23.59) - open/close 6080 (time 1788624970.83 total 337.76 last 24.00) - open/close 6315 (time 1788624981.36 total 348.29 last 22.31) - open/close 6574 (time 1788624991.36 total 358.29 last 25.90) - open/close 6803 (time 1788625001.77 total 368.70 last 22.00) - open/close 7016 (time 1788625011.83 total 378.76 last 21.18) - open/close 7255 (time 1788625022.31 total 389.24 last 22.79) - open/close 7490 (time 1788625032.35 total 399.28 last 23.41) - open/close 7730 (time 1788625042.88 total 409.80 last 22.81) - open/close 7899 (time 1788625053.18 total 420.11 last 16.40) - open/close 8142 (time 1788625064.20 total 431.13 last 22.05) - open/close 8322 (time 1788625074.23 total 441.15 last 17.95) - open/close 8522 (time 1788625084.23 total 451.16 last 19.98) - open/close 8743 (time 1788625095.19 total 462.12 last 20.17) - open/close 8981 (time 1788625105.79 total 472.72 last 22.45) - open/close 9221 (time 1788625116.42 total 483.35 last 22.58) - open/close 9451 (time 1788625126.58 total 493.51 last 22.64) - open/close 9678 (time 1788625137.48 total 504.40 last 20.83) - open/close 9874 (time 1788625148.52 total 515.45 last 17.74) - open/close 10000 (time 1788625153.82 total 520.75 last 23.77) - open/close 10140 (time 1788625163.85 total 530.78 last 13.96) - open/close 10239 (time 1788625175.11 total 542.04 last 8.80) - open/close 10320 (time 1788625185.34 total 552.26 last 7.92) - open/close 10528 (time 1788625197.74 total 564.67 last 16.77) - open/close 10597 (time 1788625208.26 total 575.19 last 6.56) - open/close 10726 (time 1788625218.99 total 585.92 last 12.02) - open/close 10959 (time 1788625229.00 total 595.92 last 23.28) - open/close 11102 (time 1788625239.48 total 606.41 last 13.64) - open/close 11254 (time 1788625250.25 total 617.18 last 14.12) - open/close 11327 (time 1788625260.40 total 627.33 last 7.19) - open/close 11364 (time 1788625271.74 total 638.66 last 3.27) - open/close 11567 (time 1788625282.30 total 649.23 last 19.22) - open/close 11743 (time 1788625292.31 total 659.24 last 17.58) - open/close 11907 (time 1788625302.33 total 669.26 last 16.36) - open/close 12087 (time 1788625312.34 total 679.26 last 17.99) - open/close 12265 (time 1788625322.56 total 689.49 last 17.41) - open/close 12479 (time 1788625332.82 total 699.75 last 20.85) - open/close 12673 (time 1788625343.00 total 709.93 last 19.07) - open/close 12899 (time 1788625353.55 total 720.48 last 21.41) - open/close 13161 (time 1788625365.49 total 732.41 last 21.96) - open/close 13403 (time 1788625375.86 total 742.79 last 23.32) - open/close 13627 (time 1788625385.88 total 752.81 last 22.35) - open/close 13839 (time 1788625396.01 total 762.94 last 20.93) - open/close 14021 (time 1788625406.02 total 772.95 last 18.18) - open/close 14213 (time 1788625418.55 total 785.48 last 15.33) - open/close 14402 (time 1788625428.94 total 795.86 last 18.19) - open/close 14617 (time 1788625439.23 total 806.15 last 20.89) - open/close 14825 (time 1788625449.71 total 816.64 last 19.84) - open/close 15101 (time 1788625459.97 total 826.90 last 26.91) - open/close 15348 (time 1788625470.93 total 837.86 last 22.54) - open/close 15602 (time 1788625481.97 total 848.90 last 22.99) - open/close 15816 (time 1788625492.41 total 859.34 last 20.51) - open/close 16071 (time 1788625502.41 total 869.34 last 25.49) - open/close 16283 (time 1788625513.21 total 880.14 last 19.63) - open/close 16358 (time 1788625524.23 total 891.15 last 6.81) - open/close 16482 (time 1788625534.26 total 901.19 last 12.35) - open/close 16600 (time 1788625544.29 total 911.22 last 11.77) - open/close 16821 (time 1788625554.95 total 921.88 last 20.73) - open/close 17032 (time 1788625565.32 total 932.25 last 20.35) - open/close 17135 (time 1788625576.00 total 942.92 last 9.65) - open/close 17313 (time 1788625586.99 total 953.92 last 16.19) - open/close 17428 (time 1788625597.57 total 964.49 last 10.87) - open/close 17621 (time 1788625607.93 total 974.85 last 18.63) - open/close 18092 (time 1788625617.94 total 984.87 last 47.05) - open/close 18737 (time 1788625627.94 total 994.87 last 64.50) - open/close 19346 (time 1788625637.94 total 1004.87 last 60.87) - open/close 20000 (time 1788625647.70 total 1014.63 last 67.02) - open/close 20675 (time 1788625657.73 total 1024.66 last 67.32) - open/close 21325 (time 1788625667.73 total 1034.66 last 64.96) - open/close 21963 (time 1788625677.74 total 1044.67 last 63.78) - open/close 22573 (time 1788625687.75 total 1054.68 last 60.90) - open/close 23270 (time 1788625697.76 total 1064.69 last 69.67) - open/close 23894 (time 1788625707.76 total 1074.69 last 62.37) - open/close 24561 (time 1788625717.78 total 1084.71 last 66.59) - open/close 25183 (time 1788625727.78 total 1094.71 last 62.18) - open/close 25775 (time 1788625737.80 total 1104.72 last 59.13) - open/close 26314 (time 1788625747.81 total 1114.74 last 53.83) - open/close 26837 (time 1788625757.82 total 1124.74 last 52.26) - open/close 27477 (time 1788625767.82 total 1134.75 last 63.97) - open/close 28152 (time 1788625777.83 total 1144.76 last 67.43) - open/close 28766 (time 1788625787.83 total 1154.76 last 61.38) - open/close 29486 (time 1788625797.84 total 1164.77 last 71.94) - open/close 30000 (time 1788625805.85 total 1172.78 last 64.18) - open/close 30622 (time 1788625815.86 total 1182.78 last 62.17) - open/close 31261 (time 1788625825.87 total 1192.80 last 63.81) - open/close 31950 (time 1788625835.91 total 1202.83 last 68.66) - open/close 32571 (time 1788625845.92 total 1212.84 last 62.04) - open/close 33212 (time 1788625855.93 total 1222.85 last 64.04) - open/close 33854 (time 1788625865.93 total 1232.86 last 64.15) - open/close 34458 (time 1788625875.95 total 1242.87 last 60.34) - open/close 35074 (time 1788625885.95 total 1252.88 last 61.56) - open/close 35854 (time 1788625895.96 total 1262.89 last 77.91) - open/close 36415 (time 1788625905.98 total 1272.91 last 56.00) - open/close 36995 (time 1788625915.99 total 1282.92 last 57.96) - open/close 37620 (time 1788625925.99 total 1292.92 last 62.46) - open/close 38269 (time 1788625936.01 total 1302.93 last 64.82) - open/close 38860 (time 1788625946.01 total 1312.94 last 59.06) - open/close 39478 (time 1788625956.01 total 1322.94 last 61.80) - open/close 40000 (time 1788625964.42 total 1331.35 last 62.06) - open/close 40654 (time 1788625974.43 total 1341.36 last 65.34) - open/close 41264 (time 1788625984.44 total 1351.36 last 60.98) - open/close 41886 (time 1788625994.45 total 1361.38 last 62.09) - open/close 42532 (time 1788626004.46 total 1371.39 last 64.55) - open/close 43196 (time 1788626014.47 total 1381.40 last 66.31) - open/close 43781 (time 1788626024.49 total 1391.42 last 58.42) - open/close 44422 (time 1788626034.49 total 1401.42 last 64.05) - open/close 45121 (time 1788626044.50 total 1411.43 last 69.87) - open/close 45686 (time 1788626054.51 total 1421.43 last 56.46) - open/close 46337 (time 1788626064.51 total 1431.44 last 65.05) - open/close 46900 (time 1788626074.52 total 1441.45 last 56.26) - open/close 47436 (time 1788626084.53 total 1451.45 last 53.57) - open/close 48122 (time 1788626094.53 total 1461.46 last 68.56) - open/close 48764 (time 1788626104.54 total 1471.47 last 64.11) - open/close 49339 (time 1788626114.56 total 1481.49 last 57.40) - open/close 49879 (time 1788626124.57 total 1491.49 last 53.98) total: 50000 open/close in 1493.59 seconds: 33.48 ops/second - unlinked 0 (time 1788626130 ; total 0 ; last 0) - unlinked 10000 (time 1788626284 ; total 154 ; last 154) - unlinked 20000 (time 1788626447 ; total 317 ; last 163) - unlinked 30000 (time 1788626607 ; total 477 ; last 160) - unlinked 40000 (time 1788626765 ; total 635 ; last 158) total: 50000 unlinks in 807 seconds: 61.957870 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg333-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg333-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server unloading modules via unload_modules_local on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing unload_modules_local oleg333-server: kmemleak disabled modules unloaded. pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2