== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:13:18 (1789063998) start mds service on oleg203-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 10 02:33 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing load_modules_local oleg203-server: kmemleak disabled oleg203-server: Loading modules from /home/green/git/lustre-release/lustre oleg203-server: detected 4 online CPUs by sysfs oleg203-server: MODOPTS_LIBCFS= oleg203-server: Force libcfs to create 2 CPU partitions oleg203-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg203-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg203-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg203-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg203-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 74 (time 1789064093.97 total 10.00 last 7.40) - open/close 260 (time 1789064104.77 total 20.81 last 17.21) - open/close 391 (time 1789064114.79 total 30.82 last 13.08) - open/close 549 (time 1789064125.15 total 41.19 last 15.24) - open/close 728 (time 1789064135.16 total 51.20 last 17.88) - open/close 896 (time 1789064145.19 total 61.23 last 16.75) - open/close 970 (time 1789064155.46 total 71.50 last 7.21) - open/close 1039 (time 1789064168.13 total 84.17 last 5.45) - open/close 1177 (time 1789064178.70 total 94.74 last 13.06) - open/close 1255 (time 1789064189.83 total 105.87 last 7.01) - open/close 1383 (time 1789064199.99 total 116.02 last 12.60) - open/close 1520 (time 1789064210.47 total 126.51 last 13.07) - open/close 1608 (time 1789064221.93 total 137.96 last 7.68) - open/close 1825 (time 1789064232.77 total 148.81 last 20.00) - open/close 2002 (time 1789064242.79 total 158.83 last 17.67) - open/close 2185 (time 1789064253.39 total 169.43 last 17.26) - open/close 2362 (time 1789064265.37 total 181.41 last 14.77) - open/close 2484 (time 1789064275.42 total 191.46 last 12.14) - open/close 2689 (time 1789064285.77 total 201.81 last 19.80) - open/close 2982 (time 1789064298.85 total 214.89 last 22.40) - open/close 3115 (time 1789064308.89 total 224.93 last 13.24) - open/close 3317 (time 1789064319.30 total 235.34 last 19.41) - open/close 3399 (time 1789064329.73 total 245.77 last 7.86) - open/close 3537 (time 1789064339.79 total 255.83 last 13.72) - open/close 3704 (time 1789064350.80 total 266.84 last 15.17) - open/close 3729 (time 1789064360.90 total 276.94 last 2.47) - open/close 3883 (time 1789064371.40 total 287.44 last 14.67) - open/close 3965 (time 1789064381.71 total 297.75 last 7.95) - open/close 4036 (time 1789064394.89 total 310.92 last 5.39) - open/close 4108 (time 1789064405.13 total 321.16 last 7.03) - open/close 4242 (time 1789064415.50 total 331.54 last 12.92) - open/close 4377 (time 1789064425.51 total 341.55 last 13.49) - open/close 4672 (time 1789064435.54 total 351.58 last 29.40) - open/close 4810 (time 1789064447.69 total 363.73 last 11.36) - open/close 4968 (time 1789064458.32 total 374.36 last 14.86) - open/close 5075 (time 1789064469.27 total 385.30 last 9.78) - open/close 5196 (time 1789064479.52 total 395.56 last 11.80) - open/close 5428 (time 1789064490.51 total 406.55 last 21.12) - open/close 5675 (time 1789064500.52 total 416.55 last 24.68) - open/close 5804 (time 1789064510.81 total 426.84 last 12.54) - open/close 5938 (time 1789064520.83 total 436.86 last 13.37) - open/close 6144 (time 1789064531.28 total 447.32 last 19.71) - open/close 6261 (time 1789064541.29 total 457.33 last 11.68) - open/close 6452 (time 1789064551.44 total 467.48 last 18.82) - open/close 6579 (time 1789064561.46 total 477.50 last 12.68) - open/close 6742 (time 1789064571.73 total 487.76 last 15.88) - open/close 6889 (time 1789064582.37 total 498.41 last 13.81) - open/close 7068 (time 1789064592.37 total 508.41 last 17.89) - open/close 7184 (time 1789064603.19 total 519.22 last 10.73) - open/close 7411 (time 1789064614.10 total 530.13 last 20.81) - open/close 7519 (time 1789064624.86 total 540.90 last 10.03) - open/close 7687 (time 1789064635.25 total 551.29 last 16.16) - open/close 7853 (time 1789064646.22 total 562.26 last 15.13) - open/close 7996 (time 1789064656.27 total 572.31 last 14.23) - open/close 8160 (time 1789064666.28 total 582.31 last 16.39) - open/close 8266 (time 1789064676.37 total 592.41 last 10.50) - open/close 8363 (time 1789064687.26 total 603.30 last 8.91) - open/close 8579 (time 1789064698.12 total 614.16 last 19.89) - open/close 8642 (time 1789064708.12 total 624.16 last 6.30) - open/close 8805 (time 1789064718.73 total 634.77 last 15.36) - open/close 9080 (time 1789064729.16 total 645.19 last 26.39) - open/close 9229 (time 1789064739.73 total 655.77 last 14.09) - open/close 9367 (time 1789064749.74 total 665.78 last 13.78) - open/close 9416 (time 1789064760.54 total 676.57 last 4.54) - open/close 9446 (time 1789064770.70 total 686.74 last 2.95) - open/close 9563 (time 1789064780.73 total 696.77 last 11.66) - open/close 9731 (time 1789064791.65 total 707.69 last 15.38) - open/close 9889 (time 1789064801.66 total 717.70 last 15.79) - open/close 10000 (time 1789064809.33 total 725.36 last 14.48) - open/close 10225 (time 1789064820.26 total 736.30 last 20.58) - open/close 10352 (time 1789064830.35 total 746.38 last 12.59) - open/close 10502 (time 1789064841.43 total 757.47 last 13.53) - open/close 10741 (time 1789064851.53 total 767.56 last 23.67) - open/close 10938 (time 1789064861.66 total 777.69 last 19.45) - open/close 11084 (time 1789064871.95 total 787.99 last 14.18) - open/close 11300 (time 1789064881.98 total 798.02 last 21.53) - open/close 11511 (time 1789064892.01 total 808.05 last 21.04) - open/close 11716 (time 1789064902.36 total 818.40 last 19.80) - open/close 11812 (time 1789064912.78 total 828.82 last 9.21) - open/close 11919 (time 1789064922.79 total 838.83 last 10.69) - open/close 12120 (time 1789064933.39 total 849.43 last 18.97) - open/close 12278 (time 1789064943.57 total 859.61 last 15.52) - open/close 12494 (time 1789064953.66 total 869.70 last 21.40) - open/close 12597 (time 1789064963.68 total 879.72 last 10.28) - open/close 12781 (time 1789064976.04 total 892.08 last 14.89) - open/close 12942 (time 1789064986.62 total 902.66 last 15.22) - open/close 13178 (time 1789064996.75 total 912.79 last 23.30) - open/close 13330 (time 1789065006.76 total 922.80 last 15.19) - open/close 13507 (time 1789065017.10 total 933.14 last 17.11) - open/close 13733 (time 1789065027.42 total 943.45 last 21.91) - open/close 13975 (time 1789065038.36 total 954.39 last 22.12) - open/close 14234 (time 1789065049.23 total 965.27 last 23.83) - open/close 14343 (time 1789065061.51 total 977.55 last 8.88) - open/close 14571 (time 1789065072.45 total 988.49 last 20.84) - open/close 14716 (time 1789065083.15 total 999.18 last 13.56) - open/close 14917 (time 1789065094.04 total 1010.08 last 18.45) - open/close 15100 (time 1789065104.85 total 1020.89 last 16.93) - open/close 15231 (time 1789065115.21 total 1031.25 last 12.64) - open/close 15338 (time 1789065126.04 total 1042.08 last 9.88) - open/close 15516 (time 1789065137.04 total 1053.08 last 16.19) - open/close 15684 (time 1789065147.06 total 1063.10 last 16.76) - open/close 15875 (time 1789065157.08 total 1073.12 last 19.06) - open/close 16019 (time 1789065167.18 total 1083.21 last 14.26) - open/close 16159 (time 1789065178.01 total 1094.05 last 12.93) - open/close 16748 (time 1789065188.01 total 1104.05 last 58.90) - open/close 17407 (time 1789065198.02 total 1114.06 last 65.83) - open/close 18100 (time 1789065208.03 total 1124.07 last 69.24) - open/close 18828 (time 1789065218.04 total 1134.07 last 72.74) - open/close 19485 (time 1789065228.04 total 1144.08 last 65.69) - open/close 20000 (time 1789065235.44 total 1151.48 last 69.54) - open/close 20694 (time 1789065245.45 total 1161.49 last 69.35) - open/close 21354 (time 1789065255.46 total 1171.49 last 65.96) - open/close 22069 (time 1789065265.47 total 1181.51 last 71.37) - open/close 22646 (time 1789065275.48 total 1191.51 last 57.69) - open/close 23242 (time 1789065285.48 total 1201.52 last 59.55) - open/close 23886 (time 1789065295.50 total 1211.54 last 64.29) - open/close 24451 (time 1789065305.51 total 1221.55 last 56.44) - open/close 24964 (time 1789065315.54 total 1231.58 last 51.15) - open/close 25355 (time 1789065325.55 total 1241.58 last 39.08) - open/close 25753 (time 1789065335.55 total 1251.59 last 39.78) - open/close 26330 (time 1789065345.58 total 1261.62 last 57.52) - open/close 26778 (time 1789065355.58 total 1271.62 last 44.80) - open/close 27280 (time 1789065365.59 total 1281.62 last 50.17) - open/close 27768 (time 1789065375.59 total 1291.62 last 48.80) - open/close 28186 (time 1789065385.60 total 1301.64 last 41.73) - open/close 28627 (time 1789065395.65 total 1311.68 last 43.91) - open/close 29070 (time 1789065405.65 total 1321.68 last 44.30) - open/close 29638 (time 1789065415.65 total 1331.69 last 56.78) - open/close 30000 (time 1789065420.79 total 1336.82 last 70.51) - open/close 30555 (time 1789065430.79 total 1346.83 last 55.48) - open/close 31089 (time 1789065440.81 total 1356.85 last 53.29) - open/close 31594 (time 1789065450.84 total 1366.88 last 50.36) - open/close 32074 (time 1789065460.85 total 1376.89 last 47.95) - open/close 32666 (time 1789065470.85 total 1386.89 last 59.18) - open/close 33170 (time 1789065480.86 total 1396.89 last 50.39) - open/close 33743 (time 1789065490.88 total 1406.91 last 57.18) - open/close 34298 (time 1789065500.89 total 1416.92 last 55.44) - open/close 34797 (time 1789065510.90 total 1426.93 last 49.86) - open/close 35353 (time 1789065520.90 total 1436.94 last 55.56) - open/close 35858 (time 1789065530.91 total 1446.95 last 50.45) - open/close 36352 (time 1789065540.93 total 1456.96 last 49.33) - open/close 36889 (time 1789065550.93 total 1466.97 last 53.69) - open/close 37466 (time 1789065560.94 total 1476.97 last 57.65) - open/close 38041 (time 1789065570.94 total 1486.98 last 57.47) - open/close 38692 (time 1789065580.95 total 1496.98 last 65.08) - open/close 39268 (time 1789065590.97 total 1507.00 last 57.50) - open/close 39858 (time 1789065600.98 total 1517.02 last 58.92) - open/close 40000 (time 1789065604.53 total 1520.57 last 39.96) - open/close 40510 (time 1789065614.56 total 1530.60 last 50.86) - open/close 40920 (time 1789065624.57 total 1540.61 last 40.96) - open/close 41411 (time 1789065634.60 total 1550.64 last 48.95) - open/close 41913 (time 1789065644.61 total 1560.65 last 50.16) - open/close 42451 (time 1789065654.62 total 1570.66 last 53.72) - open/close 42923 (time 1789065664.63 total 1580.67 last 47.16) - open/close 43496 (time 1789065674.65 total 1590.68 last 57.22) - open/close 44158 (time 1789065684.66 total 1600.70 last 66.10) - open/close 44810 (time 1789065694.66 total 1610.70 last 65.19) - open/close 45297 (time 1789065704.70 total 1620.74 last 48.50) - open/close 45742 (time 1789065714.74 total 1630.77 last 44.36) - open/close 46228 (time 1789065724.74 total 1640.78 last 48.58) - open/close 46766 (time 1789065734.76 total 1650.80 last 53.70) - open/close 47250 (time 1789065744.78 total 1660.82 last 48.29) - open/close 47745 (time 1789065754.79 total 1670.83 last 49.46) - open/close 48216 (time 1789065764.80 total 1680.84 last 47.04) - open/close 48679 (time 1789065774.81 total 1690.85 last 46.28) - open/close 49196 (time 1789065784.81 total 1700.85 last 51.68) - open/close 49872 (time 1789065794.82 total 1710.85 last 67.56) total: 50000 open/close in 1713.51 seconds: 29.18 ops/second - unlinked 0 (time 1789065804 ; total 0 ; last 0) - unlinked 10000 (time 1789065994 ; total 190 ; last 190) - unlinked 20000 (time 1789066161 ; total 357 ; last 167) - unlinked 30000 (time 1789066316 ; total 512 ; last 155) - unlinked 40000 (time 1789066463 ; total 659 ; last 147) total: 50000 unlinks in 787 seconds: 63.532402 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server unloading modules via unload_modules_local on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing unload_modules_local oleg203-server: kmemleak disabled modules unloaded. pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2