== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:59:18 (1790661558) start mds service on oleg260-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1940568 Sep 28 12:40 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: kmemleak disabled oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions oleg260-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg260-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg260-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg260-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg260-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg260-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 197 (time 1790661690.62 total 10.01 last 19.69) - open/close 293 (time 1790661700.65 total 20.03 last 9.58) - open/close 392 (time 1790661711.72 total 31.10 last 8.94) - open/close 489 (time 1790661722.74 total 42.13 last 8.80) - open/close 673 (time 1790661733.28 total 52.66 last 17.46) - open/close 806 (time 1790661743.88 total 63.27 last 12.55) - open/close 899 (time 1790661754.23 total 73.61 last 8.99) - open/close 1104 (time 1790661764.24 total 83.63 last 20.47) - open/close 1304 (time 1790661774.70 total 94.08 last 19.13) - open/close 1537 (time 1790661785.41 total 104.79 last 21.75) - open/close 1789 (time 1790661795.41 total 114.80 last 25.19) - open/close 2026 (time 1790661806.14 total 125.52 last 22.09) - open/close 2166 (time 1790661816.15 total 135.54 last 13.98) - open/close 2374 (time 1790661826.16 total 145.55 last 20.78) - open/close 2588 (time 1790661836.17 total 155.55 last 21.40) - open/close 2764 (time 1790661846.89 total 166.28 last 16.40) - open/close 2966 (time 1790661856.91 total 176.30 last 20.16) - open/close 3154 (time 1790661867.72 total 187.11 last 17.39) - open/close 3392 (time 1790661877.88 total 197.26 last 23.43) - open/close 3639 (time 1790661888.46 total 207.84 last 23.36) - open/close 3795 (time 1790661899.37 total 218.76 last 14.29) - open/close 4049 (time 1790661909.39 total 228.77 last 25.36) - open/close 4109 (time 1790661920.11 total 239.49 last 5.60) - open/close 4260 (time 1790661930.48 total 249.87 last 14.56) - open/close 4501 (time 1790661940.50 total 259.89 last 24.06) - open/close 4627 (time 1790661951.48 total 270.86 last 11.48) - open/close 4795 (time 1790661961.79 total 281.18 last 16.29) - open/close 5051 (time 1790661972.36 total 291.74 last 24.22) - open/close 5278 (time 1790661982.36 total 301.75 last 22.69) - open/close 5516 (time 1790661995.37 total 314.76 last 18.29) - open/close 5717 (time 1790662005.38 total 324.76 last 20.09) - open/close 5948 (time 1790662016.06 total 335.44 last 21.63) - open/close 6218 (time 1790662026.07 total 345.45 last 26.98) - open/close 6435 (time 1790662036.43 total 355.81 last 20.94) - open/close 6564 (time 1790662047.19 total 366.58 last 11.98) - open/close 6787 (time 1790662062.72 total 382.11 last 14.36) - open/close 7048 (time 1790662072.74 total 392.12 last 26.07) - open/close 7286 (time 1790662082.74 total 402.13 last 23.78) - open/close 7464 (time 1790662093.56 total 412.95 last 16.45) - open/close 7670 (time 1790662103.88 total 423.26 last 19.97) - open/close 7751 (time 1790662114.56 total 433.95 last 7.58) - open/close 7907 (time 1790662125.01 total 444.39 last 14.93) - open/close 8179 (time 1790662135.03 total 454.41 last 27.14) - open/close 8403 (time 1790662145.79 total 465.18 last 20.81) - open/close 8720 (time 1790662156.60 total 475.99 last 29.33) - open/close 8970 (time 1790662167.10 total 486.49 last 23.81) - open/close 9071 (time 1790662177.21 total 496.59 last 10.00) - open/close 9373 (time 1790662187.38 total 506.77 last 29.68) - open/close 9623 (time 1790662197.95 total 517.33 last 23.66) - open/close 9850 (time 1790662208.43 total 527.82 last 21.65) - open/close 10000 (time 1790662214.86 total 534.24 last 23.36) - open/close 10194 (time 1790662225.01 total 544.40 last 19.10) - open/close 10392 (time 1790662235.62 total 555.01 last 18.66) - open/close 10516 (time 1790662245.63 total 565.01 last 12.40) - open/close 10725 (time 1790662256.57 total 575.95 last 19.10) - open/close 10896 (time 1790662266.57 total 585.96 last 17.10) - open/close 11146 (time 1790662276.58 total 595.96 last 24.98) - open/close 11306 (time 1790662286.72 total 606.10 last 15.78) - open/close 11563 (time 1790662297.77 total 617.16 last 23.25) - open/close 11850 (time 1790662307.78 total 627.16 last 28.68) - open/close 12092 (time 1790662317.89 total 637.27 last 23.94) - open/close 12317 (time 1790662327.90 total 647.28 last 22.47) - open/close 12478 (time 1790662339.54 total 658.93 last 13.83) - open/close 12658 (time 1790662350.29 total 669.68 last 16.75) - open/close 12893 (time 1790662361.02 total 680.41 last 21.90) - open/close 13122 (time 1790662372.47 total 691.86 last 20.00) - open/close 13343 (time 1790662383.47 total 702.85 last 20.10) - open/close 13613 (time 1790662393.48 total 712.86 last 26.97) - open/close 13756 (time 1790662403.63 total 723.01 last 14.09) - open/close 14007 (time 1790662413.65 total 733.03 last 25.06) - open/close 14271 (time 1790662424.69 total 744.08 last 23.90) - open/close 14523 (time 1790662435.02 total 754.40 last 24.41) - open/close 14810 (time 1790662445.75 total 765.14 last 26.74) - open/close 15131 (time 1790662456.31 total 775.70 last 30.39) - open/close 15265 (time 1790662467.31 total 786.69 last 12.19) - open/close 15480 (time 1790662477.33 total 796.71 last 21.46) - open/close 15752 (time 1790662487.35 total 806.73 last 27.15) - open/close 15952 (time 1790662497.41 total 816.79 last 19.88) - open/close 16170 (time 1790662507.43 total 826.82 last 21.75) - open/close 16411 (time 1790662517.45 total 836.83 last 24.06) - open/close 16671 (time 1790662527.46 total 846.84 last 25.97) - open/close 16905 (time 1790662537.93 total 857.31 last 22.35) - open/close 17116 (time 1790662547.93 total 867.31 last 21.10) - open/close 17332 (time 1790662557.97 total 877.35 last 21.52) - open/close 17578 (time 1790662567.98 total 887.36 last 24.57) - open/close 17804 (time 1790662579.01 total 898.39 last 20.50) - open/close 18081 (time 1790662589.93 total 909.32 last 25.35) - open/close 18335 (time 1790662601.69 total 921.07 last 21.61) - open/close 18410 (time 1790662612.81 total 932.19 last 6.74) - open/close 18549 (time 1790662622.83 total 942.21 last 13.87) - open/close 18738 (time 1790662632.84 total 952.23 last 18.87) - open/close 18973 (time 1790662643.76 total 963.15 last 21.52) - open/close 19133 (time 1790662653.77 total 973.16 last 15.99) - open/close 19719 (time 1790662663.78 total 983.17 last 58.56) - open/close 20000 (time 1790662667.51 total 986.89 last 75.37) - open/close 20877 (time 1790662677.51 total 996.89 last 87.70) - open/close 21798 (time 1790662687.52 total 1006.90 last 92.04) - open/close 22793 (time 1790662697.52 total 1016.91 last 99.45) - open/close 23736 (time 1790662707.55 total 1026.93 last 94.04) - open/close 24764 (time 1790662717.57 total 1036.95 last 102.61) - open/close 25778 (time 1790662727.57 total 1046.96 last 101.35) - open/close 26852 (time 1790662737.58 total 1056.96 last 107.34) - open/close 27919 (time 1790662747.58 total 1066.96 last 106.70) - open/close 29039 (time 1790662757.58 total 1076.97 last 111.95) - open/close 29971 (time 1790662767.59 total 1086.97 last 93.14) - open/close 30987 (time 1790662777.59 total 1096.97 last 101.60) - open/close 31977 (time 1790662787.59 total 1106.98 last 98.95) - open/close 32938 (time 1790662797.60 total 1116.99 last 96.03) - open/close 33798 (time 1790662807.61 total 1126.99 last 85.96) - open/close 34680 (time 1790662817.61 total 1136.99 last 88.17) - open/close 35634 (time 1790662827.61 total 1147.00 last 95.38) - open/close 36324 (time 1790662837.63 total 1157.02 last 68.85) - open/close 37086 (time 1790662847.64 total 1167.02 last 76.17) - open/close 38018 (time 1790662857.64 total 1177.03 last 93.15) - open/close 38874 (time 1790662867.65 total 1187.03 last 85.53) - open/close 39820 (time 1790662877.66 total 1197.04 last 94.53) - open/close 40802 (time 1790662887.67 total 1207.05 last 98.12) - open/close 41740 (time 1790662897.67 total 1217.05 last 93.75) - open/close 42640 (time 1790662907.67 total 1227.06 last 89.98) - open/close 43564 (time 1790662917.68 total 1237.06 last 92.33) - open/close 44496 (time 1790662927.70 total 1247.08 last 93.06) - open/close 45326 (time 1790662937.70 total 1257.08 last 82.98) - open/close 46075 (time 1790662947.70 total 1267.09 last 74.88) - open/close 46931 (time 1790662957.72 total 1277.10 last 85.48) - open/close 47788 (time 1790662967.72 total 1287.10 last 85.67) - open/close 48672 (time 1790662977.73 total 1297.11 last 88.33) - open/close 49527 (time 1790662987.73 total 1307.11 last 85.48) total: 50000 open/close in 1312.03 seconds: 38.11 ops/second - unlinked 0 (time 1790662996 ; total 0 ; last 0) - unlinked 10000 (time 1790663123 ; total 127 ; last 127) - unlinked 20000 (time 1790663248 ; total 252 ; last 125) - unlinked 30000 (time 1790663327 ; total 331 ; last 79) - unlinked 40000 (time 1790663404 ; total 408 ; last 77) total: 50000 unlinks in 485 seconds: 103.092781 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg260-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server unloading modules via unload_modules_local on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing unload_modules_local oleg260-server: kmemleak disabled modules unloaded. pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2