== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:07:54 (1789301274) start mds service on oleg443-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 12 19:14 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing load_modules_local oleg443-server: kmemleak disabled oleg443-server: Loading modules from /home/green/git/lustre-release/lustre oleg443-server: detected 4 online CPUs by sysfs oleg443-server: MODOPTS_LIBCFS= oleg443-server: Force libcfs to create 2 CPU partitions oleg443-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg443-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg443-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg443-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg443-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0000 oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg443-client.virtnet: -o user_xattr,flock 192.168.204.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 109 (time 1789301368.09 total 10.68 last 10.21) - open/close 233 (time 1789301378.67 total 21.26 last 11.71) - open/close 381 (time 1789301388.69 total 31.28 last 14.77) - open/close 497 (time 1789301399.24 total 41.83 last 11.00) - open/close 655 (time 1789301411.28 total 53.87 last 13.12) - open/close 877 (time 1789301422.10 total 64.69 last 20.51) - open/close 981 (time 1789301432.35 total 74.94 last 10.15) - open/close 1174 (time 1789301442.36 total 84.95 last 19.28) - open/close 1362 (time 1789301452.38 total 94.97 last 18.76) - open/close 1606 (time 1789301462.39 total 104.98 last 24.37) - open/close 1771 (time 1789301472.92 total 115.51 last 15.67) - open/close 2019 (time 1789301482.94 total 125.52 last 24.76) - open/close 2263 (time 1789301492.95 total 135.54 last 24.37) - open/close 2324 (time 1789301502.97 total 145.56 last 6.09) - open/close 2412 (time 1789301513.86 total 156.44 last 8.08) - open/close 2511 (time 1789301523.93 total 166.52 last 9.83) - open/close 2606 (time 1789301535.24 total 177.83 last 8.40) - open/close 2705 (time 1789301545.55 total 188.14 last 9.60) - open/close 2801 (time 1789301555.57 total 198.16 last 9.59) - open/close 2918 (time 1789301565.59 total 208.18 last 11.67) - open/close 2968 (time 1789301576.35 total 218.94 last 4.65) - open/close 3069 (time 1789301587.41 total 230.00 last 9.13) - open/close 3155 (time 1789301600.04 total 242.63 last 6.81) - open/close 3326 (time 1789301610.04 total 252.63 last 17.09) - open/close 3547 (time 1789301620.06 total 262.65 last 22.08) - open/close 3591 (time 1789301630.54 total 273.13 last 4.20) - open/close 3793 (time 1789301642.79 total 285.38 last 16.50) - open/close 4006 (time 1789301653.00 total 295.59 last 20.85) - open/close 4183 (time 1789301663.00 total 305.59 last 17.69) - open/close 4313 (time 1789301673.50 total 316.09 last 12.39) - open/close 4426 (time 1789301683.50 total 326.09 last 11.29) - open/close 4542 (time 1789301693.92 total 336.51 last 11.14) - open/close 4711 (time 1789301704.51 total 347.10 last 15.96) - open/close 4853 (time 1789301714.78 total 357.37 last 13.83) - open/close 5082 (time 1789301724.79 total 367.38 last 22.88) - open/close 5246 (time 1789301734.79 total 377.38 last 16.39) - open/close 5299 (time 1789301745.63 total 388.22 last 4.89) - open/close 5401 (time 1789301756.60 total 399.19 last 9.30) - open/close 5548 (time 1789301767.27 total 409.86 last 13.77) - open/close 5665 (time 1789301777.29 total 419.88 last 11.68) - open/close 5816 (time 1789301790.35 total 432.94 last 11.56) - open/close 5978 (time 1789301800.53 total 443.12 last 15.92) - open/close 6053 (time 1789301810.73 total 453.32 last 7.35) - open/close 6155 (time 1789301820.95 total 463.54 last 9.98) - open/close 6351 (time 1789301831.39 total 473.98 last 18.77) - open/close 6572 (time 1789301841.88 total 484.47 last 21.06) - open/close 6782 (time 1789301852.19 total 494.78 last 20.38) - open/close 6896 (time 1789301862.20 total 504.79 last 11.38) - open/close 6980 (time 1789301872.21 total 514.79 last 8.40) - open/close 7158 (time 1789301883.86 total 526.44 last 15.28) - open/close 7349 (time 1789301894.69 total 537.28 last 17.63) - open/close 7476 (time 1789301904.78 total 547.37 last 12.58) - open/close 7606 (time 1789301914.79 total 557.38 last 12.98) - open/close 7727 (time 1789301925.59 total 568.18 last 11.21) - open/close 7859 (time 1789301935.60 total 578.19 last 13.18) - open/close 8123 (time 1789301945.82 total 588.41 last 25.82) - open/close 8318 (time 1789301956.64 total 599.23 last 18.03) - open/close 8446 (time 1789301966.67 total 609.26 last 12.76) - open/close 8557 (time 1789301976.78 total 619.37 last 10.98) - open/close 8673 (time 1789301987.16 total 629.75 last 11.17) - open/close 8831 (time 1789301997.22 total 639.81 last 15.70) - open/close 8977 (time 1789302007.23 total 649.82 last 14.59) - open/close 9076 (time 1789302017.79 total 660.38 last 9.38) - open/close 9248 (time 1789302027.81 total 670.40 last 17.16) - open/close 9462 (time 1789302037.82 total 680.41 last 21.39) - open/close 9613 (time 1789302049.43 total 692.02 last 13.00) - open/close 9768 (time 1789302059.44 total 702.03 last 15.48) - open/close 9978 (time 1789302069.85 total 712.43 last 20.18) - open/close 10000 (time 1789302079.78 total 722.37 last 2.21) - open/close 10104 (time 1789302089.78 total 732.37 last 10.40) - open/close 10269 (time 1789302099.79 total 742.38 last 16.48) - open/close 10422 (time 1789302110.70 total 753.29 last 14.02) - open/close 10632 (time 1789302120.70 total 763.29 last 21.00) - open/close 10772 (time 1789302131.66 total 774.25 last 12.78) - open/close 10944 (time 1789302142.88 total 785.47 last 15.33) - open/close 11152 (time 1789302152.90 total 795.48 last 20.77) - open/close 11381 (time 1789302163.54 total 806.13 last 21.50) - open/close 11585 (time 1789302174.68 total 817.27 last 18.32) - open/close 11812 (time 1789302185.18 total 827.77 last 21.61) - open/close 11960 (time 1789302195.74 total 838.33 last 14.02) - open/close 12167 (time 1789302205.75 total 848.34 last 20.67) - open/close 12263 (time 1789302215.97 total 858.56 last 9.40) - open/close 12440 (time 1789302225.99 total 868.58 last 17.67) - open/close 12569 (time 1789302236.65 total 879.24 last 12.09) - open/close 12666 (time 1789302247.04 total 889.62 last 9.34) - open/close 12772 (time 1789302258.04 total 900.63 last 9.63) - open/close 12960 (time 1789302268.50 total 911.09 last 17.97) - open/close 13179 (time 1789302278.52 total 921.11 last 21.87) - open/close 13285 (time 1789302289.43 total 932.02 last 9.71) - open/close 13439 (time 1789302299.96 total 942.55 last 14.63) - open/close 13494 (time 1789302310.49 total 953.08 last 5.22) - open/close 13597 (time 1789302321.37 total 963.96 last 9.46) - open/close 13651 (time 1789302332.11 total 974.70 last 5.03) - open/close 13879 (time 1789302342.82 total 985.41 last 21.29) - open/close 14030 (time 1789302352.83 total 995.41 last 15.09) - open/close 14248 (time 1789302362.83 total 1005.42 last 21.79) - open/close 14423 (time 1789302372.84 total 1015.43 last 17.49) - open/close 14500 (time 1789302382.84 total 1025.43 last 7.70) - open/close 14638 (time 1789302392.92 total 1035.51 last 13.69) - open/close 14818 (time 1789302403.74 total 1046.33 last 16.64) - open/close 15090 (time 1789302413.91 total 1056.50 last 26.74) - open/close 15318 (time 1789302423.91 total 1066.50 last 22.79) - open/close 15553 (time 1789302436.50 total 1079.09 last 18.67) - open/close 15974 (time 1789302446.52 total 1089.10 last 42.04) - open/close 16578 (time 1789302456.52 total 1099.11 last 60.35) - open/close 17162 (time 1789302466.55 total 1109.14 last 58.27) - open/close 17699 (time 1789302476.56 total 1119.15 last 53.65) - open/close 18283 (time 1789302486.57 total 1129.16 last 58.32) - open/close 18932 (time 1789302496.59 total 1139.18 last 64.74) - open/close 19482 (time 1789302506.61 total 1149.20 last 54.91) - open/close 20000 (time 1789302514.70 total 1157.29 last 64.02) - open/close 20614 (time 1789302524.70 total 1167.29 last 61.39) - open/close 21242 (time 1789302534.71 total 1177.30 last 62.73) - open/close 21864 (time 1789302544.72 total 1187.31 last 62.14) - open/close 22406 (time 1789302554.73 total 1197.32 last 54.17) - open/close 23032 (time 1789302564.73 total 1207.32 last 62.57) - open/close 23682 (time 1789302574.74 total 1217.33 last 64.99) - open/close 24244 (time 1789302584.75 total 1227.34 last 56.13) - open/close 24791 (time 1789302594.77 total 1237.36 last 54.60) - open/close 25368 (time 1789302604.77 total 1247.36 last 57.67) - open/close 25981 (time 1789302614.77 total 1257.36 last 61.30) - open/close 26581 (time 1789302624.79 total 1267.38 last 59.90) - open/close 27150 (time 1789302634.79 total 1277.38 last 56.87) - open/close 27661 (time 1789302644.80 total 1287.39 last 51.05) - open/close 28257 (time 1789302654.82 total 1297.41 last 59.48) - open/close 28852 (time 1789302664.83 total 1307.42 last 59.45) - open/close 29423 (time 1789302674.84 total 1317.43 last 57.07) - open/close 29955 (time 1789302684.84 total 1327.43 last 53.17) - open/close 30496 (time 1789302694.85 total 1337.44 last 54.04) - open/close 31101 (time 1789302704.86 total 1347.45 last 60.44) - open/close 31706 (time 1789302714.88 total 1357.47 last 60.41) - open/close 32381 (time 1789302724.89 total 1367.48 last 67.43) - open/close 32990 (time 1789302734.91 total 1377.50 last 60.79) - open/close 33590 (time 1789302744.92 total 1387.51 last 59.94) - open/close 34183 (time 1789302754.92 total 1397.51 last 59.27) - open/close 34799 (time 1789302764.94 total 1407.53 last 61.46) - open/close 35366 (time 1789302774.96 total 1417.55 last 56.62) - open/close 35892 (time 1789302784.96 total 1427.55 last 52.59) - open/close 36492 (time 1789302794.96 total 1437.55 last 59.99) - open/close 37132 (time 1789302804.96 total 1447.55 last 63.99) - open/close 37694 (time 1789302814.97 total 1457.56 last 56.13) - open/close 38272 (time 1789302824.98 total 1467.57 last 57.78) - open/close 38795 (time 1789302834.98 total 1477.57 last 52.28) - open/close 39322 (time 1789302844.98 total 1487.57 last 52.70) - open/close 39849 (time 1789302855.04 total 1497.63 last 52.37) - open/close 40000 (time 1789302858.04 total 1500.62 last 50.47) - open/close 40551 (time 1789302868.06 total 1510.65 last 54.98) - open/close 41172 (time 1789302878.06 total 1520.65 last 62.05) - open/close 41766 (time 1789302888.10 total 1530.69 last 59.22) - open/close 42331 (time 1789302898.10 total 1540.69 last 56.49) - open/close 42900 (time 1789302908.10 total 1550.69 last 56.87) - open/close 43477 (time 1789302918.11 total 1560.70 last 57.66) - open/close 44060 (time 1789302928.12 total 1570.71 last 58.26) - open/close 44618 (time 1789302938.12 total 1580.71 last 55.76) - open/close 45241 (time 1789302948.13 total 1590.72 last 62.26) - open/close 45883 (time 1789302958.13 total 1600.72 last 64.19) - open/close 46433 (time 1789302968.14 total 1610.73 last 54.94) - open/close 46983 (time 1789302978.16 total 1620.75 last 54.90) - open/close 47618 (time 1789302988.20 total 1630.79 last 63.26) - open/close 48232 (time 1789302998.20 total 1640.79 last 61.39) - open/close 48830 (time 1789303008.20 total 1650.79 last 59.78) - open/close 49415 (time 1789303018.21 total 1660.80 last 58.48) total: 50000 open/close in 1670.62 seconds: 29.93 ops/second - unlinked 0 (time 1789303032 ; total 0 ; last 0) - unlinked 10000 (time 1789303206 ; total 174 ; last 174) - unlinked 20000 (time 1789303376 ; total 344 ; last 170) - unlinked 30000 (time 1789303549 ; total 517 ; last 173) - unlinked 40000 (time 1789303720 ; total 688 ; last 171) total: 50000 unlinks in 860 seconds: 58.139534 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg443-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg443-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg443-server stop mds service on oleg443-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg443-server unloading modules via unload_modules_local on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing unload_modules_local oleg443-server: kmemleak disabled modules unloaded. pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-client: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-client: ssh exited with exit code 2