== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:20:31 (1789183231) start mds service on oleg658-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 11 11:59 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg658-server' oleg658-server: oleg658-server.virtnet: executing load_modules_local oleg658-server: kmemleak disabled oleg658-server: Loading modules from /home/green/git/lustre-release/lustre oleg658-server: detected 4 online CPUs by sysfs oleg658-server: MODOPTS_LIBCFS= oleg658-server: Force libcfs to create 2 CPU partitions oleg658-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg658-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg658-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg658-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg658-server: oleg658-server.virtnet: executing set_default_debug -1 all pdsh@oleg658-client: oleg658-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg658-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg658-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg658-server: oleg658-server.virtnet: executing set_default_debug -1 all pdsh@oleg658-client: oleg658-server: ssh exited with exit code 1 Started lustre-OST0000 oleg658-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg658-client.virtnet: -o user_xattr,flock 192.168.206.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 239 (time 1789183308.77 total 10.55 last 22.66) - open/close 431 (time 1789183319.59 total 21.37 last 17.74) - open/close 700 (time 1789183330.21 total 31.99 last 25.33) - open/close 941 (time 1789183340.64 total 42.42 last 23.10) - open/close 1162 (time 1789183350.65 total 52.43 last 22.08) - open/close 1378 (time 1789183360.65 total 62.43 last 21.60) - open/close 1608 (time 1789183370.74 total 72.51 last 22.81) - open/close 1817 (time 1789183380.74 total 82.52 last 20.90) - open/close 2027 (time 1789183390.77 total 92.55 last 20.92) - open/close 2305 (time 1789183400.81 total 102.59 last 27.71) - open/close 2562 (time 1789183410.82 total 112.59 last 25.68) - open/close 2786 (time 1789183421.36 total 123.14 last 21.24) - open/close 2998 (time 1789183432.12 total 133.90 last 19.71) - open/close 3244 (time 1789183442.12 total 143.90 last 24.59) - open/close 3452 (time 1789183452.26 total 154.04 last 20.51) - open/close 3666 (time 1789183462.27 total 164.05 last 21.39) - open/close 3946 (time 1789183473.25 total 175.03 last 25.50) - open/close 4175 (time 1789183483.30 total 185.08 last 22.79) - open/close 4408 (time 1789183493.73 total 195.51 last 22.33) - open/close 4634 (time 1789183503.77 total 205.55 last 22.51) - open/close 4860 (time 1789183513.84 total 215.62 last 22.46) - open/close 5011 (time 1789183524.76 total 226.54 last 13.82) - open/close 5270 (time 1789183535.59 total 237.37 last 23.92) - open/close 5492 (time 1789183546.23 total 248.01 last 20.87) - open/close 5774 (time 1789183557.15 total 258.93 last 25.82) - open/close 6007 (time 1789183567.33 total 269.11 last 22.87) - open/close 6239 (time 1789183577.34 total 279.12 last 23.18) - open/close 6494 (time 1789183588.06 total 289.84 last 23.79) - open/close 6729 (time 1789183598.24 total 300.02 last 23.08) - open/close 6935 (time 1789183609.26 total 311.04 last 18.69) - open/close 7190 (time 1789183619.29 total 321.07 last 25.43) - open/close 7404 (time 1789183629.99 total 331.77 last 20.00) - open/close 7669 (time 1789183640.43 total 342.21 last 25.38) - open/close 7812 (time 1789183650.67 total 352.45 last 13.97) - open/close 7945 (time 1789183660.77 total 362.54 last 13.17) - open/close 8132 (time 1789183671.46 total 373.24 last 17.49) - open/close 8391 (time 1789183682.34 total 384.12 last 23.79) - open/close 8642 (time 1789183692.35 total 394.13 last 25.07) - open/close 8855 (time 1789183702.36 total 404.14 last 21.28) - open/close 9094 (time 1789183712.74 total 414.52 last 23.03) - open/close 9312 (time 1789183722.75 total 424.53 last 21.79) - open/close 9546 (time 1789183733.53 total 435.31 last 21.70) - open/close 9781 (time 1789183743.58 total 445.36 last 23.38) - open/close 10000 (time 1789183751.51 total 453.29 last 27.62) - open/close 10257 (time 1789183762.09 total 463.87 last 24.29) - open/close 10403 (time 1789183772.27 total 474.05 last 14.35) - open/close 10648 (time 1789183782.88 total 484.66 last 23.09) - open/close 10905 (time 1789183793.78 total 495.56 last 23.59) - open/close 11082 (time 1789183804.32 total 506.10 last 16.79) - open/close 11357 (time 1789183815.27 total 517.05 last 25.10) - open/close 11645 (time 1789183825.97 total 527.75 last 26.93) - open/close 11883 (time 1789183836.38 total 538.16 last 22.85) - open/close 12092 (time 1789183846.69 total 548.47 last 20.29) - open/close 12300 (time 1789183857.51 total 559.29 last 19.22) - open/close 12570 (time 1789183868.39 total 570.17 last 24.81) - open/close 12828 (time 1789183879.28 total 581.06 last 23.69) - open/close 13044 (time 1789183889.90 total 591.68 last 20.35) - open/close 13337 (time 1789183900.33 total 602.11 last 28.08) - open/close 13543 (time 1789183910.62 total 612.40 last 20.02) - open/close 13774 (time 1789183920.69 total 622.47 last 22.94) - open/close 14030 (time 1789183931.25 total 633.03 last 24.24) - open/close 14256 (time 1789183942.19 total 643.97 last 20.65) - open/close 14450 (time 1789183952.38 total 654.16 last 19.05) - open/close 14712 (time 1789183963.24 total 665.02 last 24.11) - open/close 14895 (time 1789183973.42 total 675.20 last 17.99) - open/close 15137 (time 1789183983.43 total 685.21 last 24.16) - open/close 15370 (time 1789183993.83 total 695.61 last 22.41) - open/close 15596 (time 1789184003.85 total 705.63 last 22.55) - open/close 15827 (time 1789184013.86 total 715.64 last 23.07) - open/close 16077 (time 1789184024.17 total 725.95 last 24.25) - open/close 16189 (time 1789184034.61 total 736.39 last 10.73) - open/close 16340 (time 1789184045.22 total 747.00 last 14.23) - open/close 16613 (time 1789184055.49 total 757.27 last 26.59) - open/close 16751 (time 1789184065.49 total 767.27 last 13.80) - open/close 16951 (time 1789184075.50 total 777.28 last 19.99) - open/close 17203 (time 1789184086.56 total 788.34 last 22.78) - open/close 17411 (time 1789184097.58 total 799.36 last 18.87) - open/close 17632 (time 1789184108.45 total 810.23 last 20.34) - open/close 17891 (time 1789184118.69 total 820.47 last 25.28) - open/close 18144 (time 1789184129.32 total 831.10 last 23.80) - open/close 18382 (time 1789184139.69 total 841.47 last 22.96) - open/close 18595 (time 1789184149.69 total 851.47 last 21.30) - open/close 18814 (time 1789184160.11 total 861.89 last 21.02) - open/close 18892 (time 1789184170.11 total 871.89 last 7.80) - open/close 19035 (time 1789184180.12 total 881.90 last 14.29) - open/close 19612 (time 1789184190.12 total 891.90 last 57.70) - open/close 20000 (time 1789184195.38 total 897.16 last 73.75) - open/close 20683 (time 1789184205.40 total 907.18 last 68.18) - open/close 21415 (time 1789184215.41 total 917.19 last 73.15) - open/close 22099 (time 1789184225.41 total 927.19 last 68.39) - open/close 22780 (time 1789184235.41 total 937.19 last 68.09) - open/close 23393 (time 1789184245.42 total 947.20 last 61.25) - open/close 24048 (time 1789184255.44 total 957.22 last 65.37) - open/close 24780 (time 1789184265.47 total 967.24 last 73.00) - open/close 25429 (time 1789184275.48 total 977.26 last 64.82) - open/close 26092 (time 1789184285.49 total 987.27 last 66.20) - open/close 26787 (time 1789184295.50 total 997.28 last 69.43) - open/close 27521 (time 1789184305.51 total 1007.29 last 73.35) - open/close 28250 (time 1789184315.52 total 1017.30 last 72.84) - open/close 28948 (time 1789184325.52 total 1027.30 last 69.79) - open/close 29679 (time 1789184335.53 total 1037.31 last 73.04) - open/close 30000 (time 1789184340.82 total 1042.60 last 60.62) - open/close 30676 (time 1789184350.83 total 1052.61 last 67.58) - open/close 31423 (time 1789184360.84 total 1062.61 last 74.64) - open/close 32213 (time 1789184370.85 total 1072.63 last 78.85) - open/close 32846 (time 1789184380.87 total 1082.65 last 63.21) - open/close 33486 (time 1789184390.88 total 1092.66 last 63.95) - open/close 34226 (time 1789184400.88 total 1102.66 last 73.98) - open/close 34975 (time 1789184410.88 total 1112.66 last 74.87) - open/close 35753 (time 1789184420.89 total 1122.67 last 77.77) - open/close 36417 (time 1789184430.90 total 1132.68 last 66.30) - open/close 37084 (time 1789184440.92 total 1142.70 last 66.58) - open/close 37697 (time 1789184450.93 total 1152.71 last 61.25) - open/close 38396 (time 1789184460.93 total 1162.71 last 69.87) - open/close 39079 (time 1789184470.94 total 1172.72 last 68.28) - open/close 39828 (time 1789184480.95 total 1182.72 last 74.84) - open/close 40574 (time 1789184490.95 total 1192.73 last 74.54) - open/close 41200 (time 1789184500.96 total 1202.74 last 62.53) - open/close 41885 (time 1789184510.97 total 1212.75 last 68.46) - open/close 42578 (time 1789184520.97 total 1222.75 last 69.27) - open/close 43278 (time 1789184530.98 total 1232.76 last 69.96) - open/close 43970 (time 1789184540.99 total 1242.77 last 69.15) - open/close 44690 (time 1789184550.99 total 1252.77 last 71.99) - open/close 45349 (time 1789184561.00 total 1262.78 last 65.85) - open/close 46035 (time 1789184571.00 total 1272.78 last 68.56) - open/close 46754 (time 1789184581.00 total 1282.78 last 71.89) - open/close 47385 (time 1789184591.00 total 1292.78 last 63.09) - open/close 48086 (time 1789184601.01 total 1302.79 last 70.06) - open/close 48759 (time 1789184611.01 total 1312.79 last 67.27) - open/close 49464 (time 1789184621.02 total 1322.80 last 70.47) total: 50000 open/close in 1330.64 seconds: 37.58 ops/second - unlinked 0 (time 1789184632 ; total 0 ; last 0) - unlinked 10000 (time 1789184781 ; total 149 ; last 149) - unlinked 20000 (time 1789184925 ; total 293 ; last 144) - unlinked 30000 (time 1789185078 ; total 446 ; last 153) - unlinked 40000 (time 1789185221 ; total 589 ; last 143) total: 50000 unlinks in 737 seconds: 67.842606 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg658-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg658-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg658-server stop mds service on oleg658-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg658-server unloading modules via unload_modules_local on: 'oleg658-server' oleg658-server: oleg658-server.virtnet: executing unload_modules_local oleg658-server: kmemleak disabled modules unloaded. pdsh@oleg658-client: oleg658-client: ssh exited with exit code 2 pdsh@oleg658-client: oleg658-server: ssh exited with exit code 2 pdsh@oleg658-client: oleg658-client: ssh exited with exit code 2 pdsh@oleg658-client: oleg658-server: ssh exited with exit code 2