== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:16:20 (1788585380) start mds service on oleg110-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1765344 Sep 4 08:26 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg110-server' oleg110-server: oleg110-server.virtnet: executing load_modules_local oleg110-server: kmemleak disabled oleg110-server: Loading modules from /home/green/git/lustre-release/lustre oleg110-server: detected 4 online CPUs by sysfs oleg110-server: MODOPTS_LIBCFS= oleg110-server: Force libcfs to create 2 CPU partitions oleg110-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg110-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg110-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg110-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg110-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0000 oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 86 (time 1788585482.04 total 10.73 last 8.02) - open/close 206 (time 1788585492.40 total 21.09 last 11.58) - open/close 434 (time 1788585503.43 total 32.12 last 20.67) - open/close 554 (time 1788585513.44 total 42.13 last 11.99) - open/close 630 (time 1788585526.45 total 55.14 last 5.84) - open/close 744 (time 1788585536.48 total 65.17 last 11.36) - open/close 809 (time 1788585546.96 total 75.65 last 6.20) - open/close 1030 (time 1788585557.50 total 86.19 last 20.97) - open/close 1178 (time 1788585567.92 total 96.61 last 14.21) - open/close 1307 (time 1788585578.92 total 107.61 last 11.72) - open/close 1450 (time 1788585588.94 total 117.63 last 14.27) - open/close 1645 (time 1788585599.98 total 128.67 last 17.66) - open/close 1863 (time 1788585609.99 total 138.68 last 21.79) - open/close 2041 (time 1788585620.22 total 148.91 last 17.39) - open/close 2227 (time 1788585630.24 total 158.93 last 18.57) - open/close 2425 (time 1788585640.24 total 168.93 last 19.79) - open/close 2546 (time 1788585650.86 total 179.55 last 11.39) - open/close 2609 (time 1788585660.88 total 189.57 last 6.29) - open/close 2752 (time 1788585670.90 total 199.58 last 14.28) - open/close 2934 (time 1788585680.93 total 209.62 last 18.14) - open/close 3186 (time 1788585691.91 total 220.60 last 22.95) - open/close 3350 (time 1788585701.92 total 230.61 last 16.38) - open/close 3601 (time 1788585711.94 total 240.63 last 25.05) - open/close 3778 (time 1788585722.18 total 250.87 last 17.29) - open/close 3978 (time 1788585732.19 total 260.88 last 19.98) - open/close 4203 (time 1788585743.59 total 272.28 last 19.72) - open/close 4462 (time 1788585753.61 total 282.30 last 25.87) - open/close 4675 (time 1788585763.62 total 292.31 last 21.27) - open/close 4800 (time 1788585773.82 total 302.51 last 12.25) - open/close 5042 (time 1788585784.07 total 312.76 last 23.61) - open/close 5258 (time 1788585794.18 total 322.87 last 21.37) - open/close 5509 (time 1788585805.13 total 333.82 last 22.92) - open/close 5600 (time 1788585815.49 total 344.18 last 8.78) - open/close 5815 (time 1788585825.77 total 354.46 last 20.91) - open/close 5972 (time 1788585836.55 total 365.24 last 14.56) - open/close 6123 (time 1788585848.39 total 377.08 last 12.76) - open/close 6283 (time 1788585858.39 total 387.08 last 15.99) - open/close 6427 (time 1788585868.65 total 397.33 last 14.04) - open/close 6654 (time 1788585878.65 total 407.34 last 22.69) - open/close 6878 (time 1788585890.60 total 419.29 last 18.75) - open/close 6996 (time 1788585900.61 total 429.30 last 11.79) - open/close 7160 (time 1788585910.87 total 439.56 last 15.98) - open/close 7292 (time 1788585921.15 total 449.83 last 12.84) - open/close 7506 (time 1788585931.18 total 459.87 last 21.33) - open/close 7578 (time 1788585947.00 total 475.69 last 4.55) - open/close 7791 (time 1788585957.84 total 486.53 last 19.65) - open/close 7884 (time 1788585968.40 total 497.09 last 8.81) - open/close 8016 (time 1788585978.70 total 507.39 last 12.81) - open/close 8193 (time 1788585988.96 total 517.65 last 17.26) - open/close 8278 (time 1788585999.29 total 527.98 last 8.23) - open/close 8443 (time 1788586009.44 total 538.13 last 16.25) - open/close 8615 (time 1788586019.48 total 548.17 last 17.12) - open/close 8691 (time 1788586029.89 total 558.58 last 7.30) - open/close 8792 (time 1788586040.62 total 569.31 last 9.41) - open/close 8974 (time 1788586050.64 total 579.33 last 18.16) - open/close 9177 (time 1788586060.70 total 589.39 last 20.18) - open/close 9290 (time 1788586071.36 total 600.05 last 10.60) - open/close 9437 (time 1788586082.24 total 610.93 last 13.51) - open/close 9525 (time 1788586092.89 total 621.58 last 8.26) - open/close 9672 (time 1788586103.11 total 631.80 last 14.38) - open/close 9804 (time 1788586113.41 total 642.09 last 12.82) - open/close 9992 (time 1788586124.33 total 653.02 last 17.21) - open/close 10182 (time 1788586135.26 total 663.95 last 17.38) - open/close 10364 (time 1788586145.58 total 674.27 last 17.65) - open/close 10578 (time 1788586155.60 total 684.29 last 21.35) - open/close 10822 (time 1788586165.61 total 694.30 last 24.37) - open/close 10965 (time 1788586175.63 total 704.31 last 14.28) - open/close 11129 (time 1788586185.63 total 714.32 last 16.39) - open/close 11371 (time 1788586195.65 total 724.34 last 24.16) - open/close 11541 (time 1788586206.53 total 735.22 last 15.63) - open/close 11585 (time 1788586217.23 total 745.92 last 4.11) - open/close 11826 (time 1788586227.24 total 755.93 last 24.07) - open/close 11983 (time 1788586240.10 total 768.79 last 12.20) - open/close 12119 (time 1788586250.66 total 779.35 last 12.88) - open/close 12248 (time 1788586260.87 total 789.56 last 12.64) - open/close 12443 (time 1788586270.89 total 799.58 last 19.47) - open/close 12500 (time 1788586281.86 total 810.55 last 5.19) - open/close 12631 (time 1788586292.77 total 821.46 last 12.01) - open/close 12765 (time 1788586302.80 total 831.49 last 13.36) - open/close 12914 (time 1788586312.82 total 841.51 last 14.87) - open/close 13014 (time 1788586322.83 total 851.52 last 9.99) - open/close 13132 (time 1788586333.04 total 861.73 last 11.56) - open/close 13332 (time 1788586343.91 total 872.60 last 18.40) - open/close 13495 (time 1788586353.91 total 882.60 last 16.30) - open/close 13687 (time 1788586364.82 total 893.50 last 17.60) - open/close 13894 (time 1788586375.30 total 903.99 last 19.74) - open/close 13963 (time 1788586385.98 total 914.67 last 6.46) - open/close 14060 (time 1788586396.01 total 924.70 last 9.67) - open/close 14169 (time 1788586406.02 total 934.71 last 10.89) - open/close 14251 (time 1788586417.47 total 946.16 last 7.16) - open/close 14491 (time 1788586428.26 total 956.95 last 22.24) - open/close 14708 (time 1788586438.38 total 967.07 last 21.45) - open/close 14885 (time 1788586448.39 total 977.08 last 17.68) - open/close 14997 (time 1788586459.46 total 988.15 last 10.12) - open/close 15060 (time 1788586470.38 total 999.06 last 5.77) - open/close 15277 (time 1788586480.39 total 1009.08 last 21.66) - open/close 15456 (time 1788586490.87 total 1019.56 last 17.09) - open/close 15541 (time 1788586502.56 total 1031.25 last 7.27) - open/close 15690 (time 1788586513.02 total 1041.71 last 14.24) - open/close 15746 (time 1788586523.04 total 1051.73 last 5.59) - open/close 15945 (time 1788586533.76 total 1062.45 last 18.55) - open/close 16182 (time 1788586544.02 total 1072.71 last 23.12) - open/close 16797 (time 1788586554.03 total 1082.72 last 61.42) - open/close 17482 (time 1788586564.04 total 1092.73 last 68.46) - open/close 18170 (time 1788586574.04 total 1102.73 last 68.75) - open/close 18844 (time 1788586584.05 total 1112.74 last 67.38) - open/close 19521 (time 1788586594.05 total 1122.74 last 67.67) - open/close 20000 (time 1788586602.77 total 1131.46 last 54.93) - open/close 20685 (time 1788586612.77 total 1141.46 last 68.48) - open/close 21297 (time 1788586622.78 total 1151.47 last 61.14) - open/close 21905 (time 1788586632.79 total 1161.47 last 60.79) - open/close 22582 (time 1788586642.80 total 1171.48 last 67.63) - open/close 23146 (time 1788586652.83 total 1181.52 last 56.20) - open/close 23777 (time 1788586662.86 total 1191.55 last 62.94) - open/close 24746 (time 1788586672.87 total 1201.56 last 96.78) - open/close 25413 (time 1788586682.87 total 1211.56 last 66.69) - open/close 26061 (time 1788586692.87 total 1221.56 last 64.80) - open/close 26639 (time 1788586702.88 total 1231.56 last 57.77) - open/close 27295 (time 1788586712.88 total 1241.57 last 65.59) - open/close 27909 (time 1788586722.89 total 1251.58 last 61.34) - open/close 28442 (time 1788586732.91 total 1261.59 last 53.21) - open/close 29005 (time 1788586742.91 total 1271.60 last 56.26) - open/close 29691 (time 1788586752.93 total 1281.62 last 68.49) - open/close 30000 (time 1788586757.41 total 1286.10 last 68.97) - open/close 30612 (time 1788586767.42 total 1296.11 last 61.15) - open/close 31241 (time 1788586777.43 total 1306.12 last 62.79) - open/close 31899 (time 1788586787.44 total 1316.13 last 65.74) - open/close 32575 (time 1788586797.45 total 1326.14 last 67.54) - open/close 33390 (time 1788586807.46 total 1336.15 last 81.41) - open/close 34027 (time 1788586817.47 total 1346.16 last 63.66) - open/close 34721 (time 1788586827.48 total 1356.17 last 69.34) - open/close 35406 (time 1788586837.49 total 1366.18 last 68.42) - open/close 36036 (time 1788586847.50 total 1376.19 last 62.93) - open/close 36601 (time 1788586857.51 total 1386.20 last 56.44) - open/close 37262 (time 1788586867.54 total 1396.23 last 65.92) - open/close 37798 (time 1788586877.57 total 1406.26 last 53.46) - open/close 38521 (time 1788586887.57 total 1416.26 last 72.27) - open/close 39222 (time 1788586897.58 total 1426.27 last 70.02) - open/close 39828 (time 1788586907.59 total 1436.28 last 60.53) - open/close 40000 (time 1788586911.05 total 1439.74 last 49.75) - open/close 40580 (time 1788586921.06 total 1449.75 last 57.93) - open/close 41197 (time 1788586931.08 total 1459.77 last 61.61) - open/close 41931 (time 1788586941.08 total 1469.77 last 73.37) - open/close 42461 (time 1788586951.09 total 1479.78 last 52.98) - open/close 43101 (time 1788586961.09 total 1489.78 last 64.00) - open/close 43822 (time 1788586971.09 total 1499.78 last 72.08) - open/close 44523 (time 1788586981.10 total 1509.79 last 70.06) - open/close 45215 (time 1788586991.10 total 1519.79 last 69.19) - open/close 45963 (time 1788587001.11 total 1529.80 last 74.71) - open/close 46561 (time 1788587011.11 total 1539.80 last 59.80) - open/close 47181 (time 1788587021.13 total 1549.82 last 61.88) - open/close 47732 (time 1788587031.13 total 1559.82 last 55.10) - open/close 48268 (time 1788587041.14 total 1569.82 last 53.57) - open/close 48825 (time 1788587051.14 total 1579.83 last 55.65) - open/close 49421 (time 1788587061.15 total 1589.84 last 59.55) total: 50000 open/close in 1598.72 seconds: 31.28 ops/second - unlinked 0 (time 1788587074 ; total 0 ; last 0) - unlinked 10000 (time 1788587237 ; total 163 ; last 163) - unlinked 20000 (time 1788587395 ; total 321 ; last 158) - unlinked 30000 (time 1788587582 ; total 508 ; last 187) - unlinked 40000 (time 1788587755 ; total 681 ; last 173) total: 50000 unlinks in 859 seconds: 58.207218 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg110-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg110-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg110-server stop mds service on oleg110-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg110-server unloading modules via unload_modules_local on: 'oleg110-server' oleg110-server: oleg110-server.virtnet: executing unload_modules_local oleg110-server: kmemleak disabled modules unloaded. pdsh@oleg110-client: oleg110-client: ssh exited with exit code 2 pdsh@oleg110-client: oleg110-server: ssh exited with exit code 2 pdsh@oleg110-client: oleg110-client: ssh exited with exit code 2 pdsh@oleg110-client: oleg110-server: ssh exited with exit code 2