== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:04:39 (1788577479) start mds service on oleg401-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1944872 Sep 4 04:21 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg401-server' oleg401-server: oleg401-server.virtnet: executing load_modules_local oleg401-server: kmemleak disabled oleg401-server: Loading modules from /home/green/git/lustre-release/lustre oleg401-server: detected 4 online CPUs by sysfs oleg401-server: MODOPTS_LIBCFS= oleg401-server: Force libcfs to create 2 CPU partitions oleg401-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg401-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg401-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg401-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg401-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0000 oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 218 (time 1788577560.50 total 10.41 last 20.94) - open/close 303 (time 1788577570.74 total 20.66 last 8.30) - open/close 426 (time 1788577581.55 total 31.47 last 11.38) - open/close 659 (time 1788577591.75 total 41.67 last 22.84) - open/close 920 (time 1788577602.55 total 52.46 last 24.18) - open/close 1105 (time 1788577613.37 total 63.28 last 17.10) - open/close 1416 (time 1788577624.04 total 73.95 last 29.14) - open/close 1661 (time 1788577637.18 total 87.10 last 18.64) - open/close 1918 (time 1788577647.19 total 97.10 last 25.68) - open/close 2136 (time 1788577657.19 total 107.11 last 21.80) - open/close 2290 (time 1788577667.30 total 117.22 last 15.23) - open/close 2551 (time 1788577678.27 total 128.18 last 23.80) - open/close 2737 (time 1788577689.07 total 138.98 last 17.23) - open/close 2935 (time 1788577699.14 total 149.05 last 19.66) - open/close 3126 (time 1788577709.81 total 159.72 last 17.90) - open/close 3365 (time 1788577720.56 total 170.48 last 22.22) - open/close 3615 (time 1788577731.00 total 180.91 last 23.96) - open/close 3852 (time 1788577741.18 total 191.09 last 23.28) - open/close 4116 (time 1788577751.81 total 201.73 last 24.82) - open/close 4399 (time 1788577762.67 total 212.58 last 26.07) - open/close 4529 (time 1788577773.12 total 223.04 last 12.44) - open/close 4709 (time 1788577783.35 total 233.26 last 17.60) - open/close 4981 (time 1788577794.03 total 243.94 last 25.47) - open/close 5203 (time 1788577804.03 total 253.94 last 22.20) - open/close 5428 (time 1788577814.57 total 264.49 last 21.33) - open/close 5645 (time 1788577824.58 total 274.49 last 21.69) - open/close 5875 (time 1788577834.81 total 284.73 last 22.47) - open/close 6098 (time 1788577845.95 total 295.86 last 20.03) - open/close 6328 (time 1788577856.89 total 306.80 last 21.02) - open/close 6525 (time 1788577866.90 total 316.82 last 19.67) - open/close 6739 (time 1788577877.37 total 327.28 last 20.45) - open/close 6900 (time 1788577888.05 total 337.96 last 15.07) - open/close 7159 (time 1788577898.70 total 348.61 last 24.32) - open/close 7313 (time 1788577908.71 total 358.62 last 15.39) - open/close 7540 (time 1788577919.12 total 369.03 last 21.80) - open/close 7784 (time 1788577929.90 total 379.81 last 22.63) - open/close 8016 (time 1788577939.90 total 389.82 last 23.20) - open/close 8141 (time 1788577950.25 total 400.16 last 12.08) - open/close 8367 (time 1788577960.37 total 410.29 last 22.32) - open/close 8540 (time 1788577970.48 total 420.40 last 17.11) - open/close 8661 (time 1788577980.49 total 430.40 last 12.09) - open/close 8885 (time 1788577990.84 total 440.76 last 21.64) - open/close 9053 (time 1788578001.14 total 451.05 last 16.31) - open/close 9282 (time 1788578011.62 total 461.54 last 21.84) - open/close 9517 (time 1788578022.06 total 471.97 last 22.52) - open/close 9755 (time 1788578032.82 total 482.73 last 22.12) - open/close 9996 (time 1788578043.07 total 492.99 last 23.50) - open/close 10201 (time 1788578054.13 total 504.04 last 18.54) - open/close 10289 (time 1788578064.14 total 514.06 last 8.79) - open/close 10408 (time 1788578074.81 total 524.73 last 11.15) - open/close 10625 (time 1788578084.98 total 534.89 last 21.35) - open/close 10905 (time 1788578096.06 total 545.98 last 25.26) - open/close 11159 (time 1788578106.68 total 556.59 last 23.93) - open/close 11452 (time 1788578116.68 total 566.60 last 29.28) - open/close 11677 (time 1788578126.69 total 576.61 last 22.48) - open/close 11962 (time 1788578137.65 total 587.57 last 26.01) - open/close 12125 (time 1788578147.66 total 597.58 last 16.28) - open/close 12280 (time 1788578157.66 total 607.58 last 15.50) - open/close 12439 (time 1788578167.85 total 617.76 last 15.61) - open/close 12612 (time 1788578177.85 total 627.77 last 17.29) - open/close 12856 (time 1788578188.80 total 638.72 last 22.28) - open/close 13041 (time 1788578199.15 total 649.07 last 17.88) - open/close 13263 (time 1788578209.15 total 659.07 last 22.19) - open/close 13497 (time 1788578219.58 total 669.50 last 22.44) - open/close 13635 (time 1788578232.43 total 682.34 last 10.74) - open/close 13886 (time 1788578242.96 total 692.88 last 23.83) - open/close 14109 (time 1788578253.24 total 703.16 last 21.68) - open/close 14354 (time 1788578263.25 total 713.16 last 24.49) - open/close 14597 (time 1788578273.28 total 723.20 last 24.21) - open/close 14837 (time 1788578283.29 total 733.20 last 23.99) - open/close 15067 (time 1788578293.29 total 743.21 last 22.99) - open/close 15230 (time 1788578303.41 total 753.33 last 16.11) - open/close 15502 (time 1788578313.43 total 763.34 last 27.16) - open/close 15772 (time 1788578324.13 total 774.05 last 25.22) - open/close 16032 (time 1788578334.63 total 784.55 last 24.76) - open/close 16289 (time 1788578345.46 total 795.38 last 23.72) - open/close 16544 (time 1788578355.88 total 805.80 last 24.48) - open/close 16768 (time 1788578366.83 total 816.74 last 20.47) - open/close 17021 (time 1788578376.83 total 826.74 last 25.30) - open/close 17231 (time 1788578386.92 total 836.83 last 20.81) - open/close 17380 (time 1788578397.10 total 847.02 last 14.63) - open/close 17599 (time 1788578408.04 total 857.96 last 20.02) - open/close 17899 (time 1788578418.68 total 868.60 last 28.19) - open/close 18154 (time 1788578428.96 total 878.88 last 24.81) - open/close 18410 (time 1788578439.22 total 889.13 last 24.96) - open/close 18718 (time 1788578449.77 total 899.68 last 29.19) - open/close 18985 (time 1788578459.77 total 909.69 last 26.70) - open/close 19231 (time 1788578469.78 total 919.69 last 24.58) - open/close 19942 (time 1788578479.79 total 929.70 last 71.05) - open/close 20657 (time 1788578489.81 total 939.73 last 71.31) - open/close 21394 (time 1788578499.82 total 949.73 last 73.66) - open/close 22193 (time 1788578509.82 total 959.73 last 79.88) - open/close 23004 (time 1788578519.83 total 969.74 last 81.03) - open/close 23761 (time 1788578529.83 total 979.74 last 75.69) - open/close 24540 (time 1788578539.85 total 989.76 last 77.76) - open/close 25312 (time 1788578549.85 total 999.76 last 77.18) - open/close 26045 (time 1788578559.85 total 1009.77 last 73.26) - open/close 26774 (time 1788578569.86 total 1019.78 last 72.85) - open/close 27509 (time 1788578579.87 total 1029.78 last 73.45) - open/close 28251 (time 1788578589.88 total 1039.79 last 74.14) - open/close 29015 (time 1788578599.89 total 1049.80 last 76.33) - open/close 29755 (time 1788578609.89 total 1059.81 last 73.94) - open/close 30000 (time 1788578612.83 total 1062.74 last 83.46) - open/close 30734 (time 1788578622.83 total 1072.75 last 73.39) - open/close 31482 (time 1788578632.84 total 1082.75 last 74.76) - open/close 32218 (time 1788578642.84 total 1092.75 last 73.57) - open/close 32898 (time 1788578652.85 total 1102.76 last 67.94) - open/close 33570 (time 1788578662.85 total 1112.77 last 67.15) - open/close 34314 (time 1788578672.86 total 1122.77 last 74.36) - open/close 35131 (time 1788578682.86 total 1132.78 last 81.69) - open/close 35963 (time 1788578692.87 total 1142.78 last 83.16) - open/close 36761 (time 1788578702.87 total 1152.78 last 79.79) - open/close 37587 (time 1788578712.88 total 1162.79 last 82.54) - open/close 38307 (time 1788578722.89 total 1172.80 last 71.93) - open/close 39098 (time 1788578732.89 total 1182.81 last 79.03) - open/close 39904 (time 1788578742.90 total 1192.82 last 80.53) - open/close 40719 (time 1788578752.91 total 1202.83 last 81.43) - open/close 41562 (time 1788578762.92 total 1212.83 last 84.27) - open/close 42411 (time 1788578772.94 total 1222.86 last 84.68) - open/close 43205 (time 1788578782.97 total 1232.88 last 79.18) - open/close 43842 (time 1788578792.98 total 1242.90 last 63.63) - open/close 44450 (time 1788578802.99 total 1252.91 last 60.72) - open/close 45049 (time 1788578813.01 total 1262.92 last 59.82) - open/close 45740 (time 1788578823.01 total 1272.93 last 69.07) - open/close 46434 (time 1788578833.02 total 1282.94 last 69.33) - open/close 47158 (time 1788578843.03 total 1292.94 last 72.36) - open/close 47865 (time 1788578853.04 total 1302.95 last 70.62) - open/close 48617 (time 1788578863.05 total 1312.96 last 75.14) - open/close 49316 (time 1788578873.06 total 1322.97 last 69.83) total: 50000 open/close in 1332.74 seconds: 37.52 ops/second - unlinked 0 (time 1788578886 ; total 0 ; last 0) - unlinked 10000 (time 1788579031 ; total 145 ; last 145) - unlinked 20000 (time 1788579171 ; total 285 ; last 140) - unlinked 30000 (time 1788579319 ; total 433 ; last 148) - unlinked 40000 (time 1788579463 ; total 577 ; last 144) total: 50000 unlinks in 718 seconds: 69.637886 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg401-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg401-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg401-server stop mds service on oleg401-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg401-server unloading modules via unload_modules_local on: 'oleg401-server' oleg401-server: oleg401-server.virtnet: executing unload_modules_local oleg401-server: kmemleak disabled modules unloaded. pdsh@oleg401-client: oleg401-client: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-server: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-server: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-client: ssh exited with exit code 2