== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:17:02 (1788668222) start mds service on oleg636-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1922328 Sep 5 15:39 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg636-server' oleg636-server: oleg636-server.virtnet: executing load_modules_local oleg636-server: kmemleak disabled oleg636-server: Loading modules from /home/green/git/lustre-release/lustre oleg636-server: detected 4 online CPUs by sysfs oleg636-server: MODOPTS_LIBCFS= oleg636-server: Force libcfs to create 2 CPU partitions oleg636-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg636-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg636-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg636-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg636-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg636-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg636-server: oleg636-server.virtnet: executing set_default_debug -1 all pdsh@oleg636-client: oleg636-server: ssh exited with exit code 1 Started lustre-OST0000 oleg636-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg636-client.virtnet: -o user_xattr,flock 192.168.206.136@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 193 (time 1788668305.53 total 10.68 last 18.08) - open/close 493 (time 1788668315.54 total 20.68 last 29.98) - open/close 706 (time 1788668325.54 total 30.69 last 21.30) - open/close 918 (time 1788668335.63 total 40.77 last 21.01) - open/close 1162 (time 1788668345.64 total 50.79 last 24.37) - open/close 1276 (time 1788668355.68 total 60.82 last 11.36) - open/close 1454 (time 1788668366.53 total 71.67 last 16.41) - open/close 1711 (time 1788668376.55 total 81.70 last 25.63) - open/close 1882 (time 1788668386.96 total 92.10 last 16.43) - open/close 2123 (time 1788668398.66 total 103.80 last 20.59) - open/close 2327 (time 1788668408.66 total 113.80 last 20.40) - open/close 2479 (time 1788668418.67 total 123.81 last 15.19) - open/close 2687 (time 1788668428.67 total 133.82 last 20.79) - open/close 2906 (time 1788668439.13 total 144.27 last 20.95) - open/close 3145 (time 1788668449.13 total 154.27 last 23.89) - open/close 3348 (time 1788668459.47 total 164.61 last 19.64) - open/close 3567 (time 1788668469.69 total 174.84 last 21.41) - open/close 3816 (time 1788668480.32 total 185.46 last 23.43) - open/close 4018 (time 1788668490.38 total 195.52 last 20.09) - open/close 4273 (time 1788668501.28 total 206.42 last 23.39) - open/close 4481 (time 1788668511.30 total 216.45 last 20.74) - open/close 4729 (time 1788668522.25 total 227.40 last 22.65) - open/close 5006 (time 1788668532.55 total 237.69 last 26.90) - open/close 5242 (time 1788668542.85 total 248.00 last 22.90) - open/close 5443 (time 1788668553.23 total 258.38 last 19.36) - open/close 5723 (time 1788668563.65 total 268.79 last 26.89) - open/close 5929 (time 1788668575.88 total 281.02 last 16.85) - open/close 6166 (time 1788668586.82 total 291.97 last 21.65) - open/close 6440 (time 1788668596.82 total 301.97 last 27.40) - open/close 6649 (time 1788668607.75 total 312.90 last 19.12) - open/close 6907 (time 1788668618.63 total 323.77 last 23.72) - open/close 7139 (time 1788668628.76 total 333.90 last 22.91) - open/close 7374 (time 1788668639.17 total 344.31 last 22.57) - open/close 7623 (time 1788668649.99 total 355.13 last 23.02) - open/close 7878 (time 1788668660.09 total 365.23 last 25.25) - open/close 7980 (time 1788668670.29 total 375.43 last 10.00) - open/close 8243 (time 1788668680.31 total 385.45 last 26.24) - open/close 8483 (time 1788668690.33 total 395.47 last 23.96) - open/close 8700 (time 1788668701.33 total 406.47 last 19.73) - open/close 9005 (time 1788668712.13 total 417.27 last 28.23) - open/close 9304 (time 1788668722.36 total 427.51 last 29.21) - open/close 9540 (time 1788668734.98 total 440.12 last 18.71) - open/close 9763 (time 1788668745.79 total 450.93 last 20.63) - open/close 9960 (time 1788668755.91 total 461.05 last 19.47) - open/close 10170 (time 1788668766.29 total 471.43 last 20.23) - open/close 10433 (time 1788668776.85 total 481.99 last 24.89) - open/close 10662 (time 1788668786.86 total 492.00 last 22.89) - open/close 10893 (time 1788668796.87 total 502.01 last 23.07) - open/close 11108 (time 1788668807.57 total 512.71 last 20.09) - open/close 11345 (time 1788668817.99 total 523.13 last 22.74) - open/close 11581 (time 1788668828.75 total 533.90 last 21.93) - open/close 11908 (time 1788668838.77 total 543.91 last 32.66) - open/close 12163 (time 1788668849.81 total 554.95 last 23.09) - open/close 12392 (time 1788668860.18 total 565.33 last 22.07) - open/close 12613 (time 1788668870.19 total 575.33 last 22.08) - open/close 12900 (time 1788668880.40 total 585.55 last 28.10) - open/close 13091 (time 1788668890.78 total 595.92 last 18.41) - open/close 13307 (time 1788668900.89 total 606.03 last 21.37) - open/close 13537 (time 1788668910.89 total 616.03 last 23.00) - open/close 13804 (time 1788668921.47 total 626.62 last 25.23) - open/close 14031 (time 1788668931.48 total 636.62 last 22.68) - open/close 14240 (time 1788668941.49 total 646.63 last 20.89) - open/close 14449 (time 1788668951.49 total 656.63 last 20.90) - open/close 14695 (time 1788668961.49 total 666.63 last 24.60) - open/close 14885 (time 1788668972.26 total 677.40 last 17.64) - open/close 15145 (time 1788668982.27 total 687.41 last 25.98) - open/close 15355 (time 1788668992.30 total 697.44 last 20.94) - open/close 15590 (time 1788669002.30 total 707.45 last 23.48) - open/close 15754 (time 1788669013.15 total 718.29 last 15.13) - open/close 15914 (time 1788669023.15 total 728.30 last 15.99) - open/close 16078 (time 1788669035.68 total 740.83 last 13.09) - open/close 16294 (time 1788669046.67 total 751.81 last 19.66) - open/close 16516 (time 1788669058.51 total 763.65 last 18.75) - open/close 16749 (time 1788669069.59 total 774.73 last 21.04) - open/close 17002 (time 1788669080.53 total 785.68 last 23.11) - open/close 17255 (time 1788669090.54 total 795.68 last 25.29) - open/close 17461 (time 1788669100.87 total 806.01 last 19.94) - open/close 17700 (time 1788669111.52 total 816.66 last 22.44) - open/close 17920 (time 1788669121.52 total 826.66 last 22.00) - open/close 18193 (time 1788669131.79 total 836.93 last 26.60) - open/close 18438 (time 1788669142.05 total 847.20 last 23.87) - open/close 18685 (time 1788669152.26 total 857.40 last 24.21) - open/close 18922 (time 1788669162.37 total 867.52 last 23.42) - open/close 19317 (time 1788669172.38 total 877.53 last 39.46) - open/close 20000 (time 1788669182.05 total 887.20 last 70.63) - open/close 20646 (time 1788669192.06 total 897.20 last 64.57) - open/close 21303 (time 1788669202.06 total 907.21 last 65.66) - open/close 22000 (time 1788669212.08 total 917.22 last 69.62) - open/close 22743 (time 1788669222.08 total 927.22 last 74.28) - open/close 23404 (time 1788669232.09 total 937.24 last 66.00) - open/close 24073 (time 1788669242.12 total 947.26 last 66.75) - open/close 24729 (time 1788669252.13 total 957.27 last 65.54) - open/close 25457 (time 1788669262.14 total 967.29 last 72.67) - open/close 26205 (time 1788669272.15 total 977.30 last 74.73) - open/close 26939 (time 1788669282.15 total 987.30 last 73.39) - open/close 27674 (time 1788669292.16 total 997.30 last 73.46) - open/close 28395 (time 1788669302.16 total 1007.30 last 72.10) - open/close 29135 (time 1788669312.17 total 1017.31 last 73.95) - open/close 29892 (time 1788669322.17 total 1027.31 last 75.69) - open/close 30526 (time 1788669332.17 total 1037.32 last 63.37) - open/close 31246 (time 1788669342.18 total 1047.32 last 71.95) - open/close 31935 (time 1788669352.18 total 1057.33 last 68.89) - open/close 32655 (time 1788669362.21 total 1067.35 last 71.81) - open/close 33403 (time 1788669372.22 total 1077.36 last 74.75) - open/close 34102 (time 1788669382.22 total 1087.37 last 69.85) - open/close 34795 (time 1788669392.23 total 1097.37 last 69.27) - open/close 35364 (time 1788669402.23 total 1107.38 last 56.85) - open/close 36092 (time 1788669412.25 total 1117.39 last 72.71) - open/close 36843 (time 1788669422.25 total 1127.40 last 75.05) - open/close 37584 (time 1788669432.26 total 1137.40 last 74.07) - open/close 38286 (time 1788669442.27 total 1147.41 last 70.13) - open/close 38960 (time 1788669452.28 total 1157.42 last 67.33) - open/close 39684 (time 1788669462.28 total 1167.42 last 72.40) - open/close 40000 (time 1788669466.84 total 1171.99 last 69.23) - open/close 40664 (time 1788669476.85 total 1181.99 last 66.36) - open/close 41379 (time 1788669486.85 total 1191.99 last 71.49) - open/close 42092 (time 1788669496.85 total 1202.00 last 71.27) - open/close 42821 (time 1788669506.86 total 1212.00 last 72.89) - open/close 43523 (time 1788669516.86 total 1222.00 last 70.19) - open/close 44201 (time 1788669526.87 total 1232.01 last 67.74) - open/close 44928 (time 1788669536.88 total 1242.02 last 72.62) - open/close 45604 (time 1788669546.89 total 1252.03 last 67.52) - open/close 46314 (time 1788669556.90 total 1262.04 last 70.93) - open/close 47049 (time 1788669566.90 total 1272.05 last 73.46) - open/close 47716 (time 1788669576.91 total 1282.05 last 66.68) - open/close 48377 (time 1788669586.91 total 1292.05 last 66.07) - open/close 48977 (time 1788669596.91 total 1302.06 last 59.98) - open/close 49563 (time 1788669606.92 total 1312.07 last 58.55) total: 50000 open/close in 1319.12 seconds: 37.90 ops/second - unlinked 0 (time 1788669617 ; total 0 ; last 0) - unlinked 10000 (time 1788669761 ; total 144 ; last 144) - unlinked 20000 (time 1788669900 ; total 283 ; last 139) - unlinked 30000 (time 1788670047 ; total 430 ; last 147) - unlinked 40000 (time 1788670189 ; total 572 ; last 142) total: 50000 unlinks in 717 seconds: 69.735008 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg636-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg636-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg636-server stop mds service on oleg636-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg636-server unloading modules via unload_modules_local on: 'oleg636-server' oleg636-server: oleg636-server.virtnet: executing unload_modules_local oleg636-server: kmemleak disabled modules unloaded. pdsh@oleg636-client: oleg636-server: ssh exited with exit code 2 pdsh@oleg636-client: oleg636-client: ssh exited with exit code 2 pdsh@oleg636-client: oleg636-server: ssh exited with exit code 2 pdsh@oleg636-client: oleg636-client: ssh exited with exit code 2