== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:07:57 (1788865677) start mds service on oleg331-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg331-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg331-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 139 (time 1788865762.89 total 10.57 last 13.16) - open/close 384 (time 1788865773.75 total 21.43 last 22.56) - open/close 595 (time 1788865784.13 total 31.81 last 20.33) - open/close 766 (time 1788865794.14 total 41.82 last 17.08) - open/close 943 (time 1788865804.16 total 51.84 last 17.67) - open/close 1085 (time 1788865814.85 total 62.53 last 13.28) - open/close 1219 (time 1788865824.91 total 72.59 last 13.32) - open/close 1432 (time 1788865835.13 total 82.81 last 20.84) - open/close 1633 (time 1788865845.56 total 93.24 last 19.28) - open/close 1853 (time 1788865855.69 total 103.37 last 21.72) - open/close 2051 (time 1788865865.71 total 113.39 last 19.75) - open/close 2222 (time 1788865875.72 total 123.40 last 17.09) - open/close 2471 (time 1788865885.73 total 133.41 last 24.88) - open/close 2708 (time 1788865895.73 total 143.41 last 23.70) - open/close 2846 (time 1788865905.73 total 153.41 last 13.79) - open/close 3055 (time 1788865915.74 total 163.42 last 20.88) - open/close 3289 (time 1788865925.75 total 173.43 last 23.40) - open/close 3471 (time 1788865935.78 total 183.46 last 18.14) - open/close 3684 (time 1788865946.26 total 193.94 last 20.32) - open/close 3899 (time 1788865956.34 total 204.02 last 21.32) - open/close 4112 (time 1788865966.77 total 214.45 last 20.42) - open/close 4271 (time 1788865976.79 total 224.47 last 15.87) - open/close 4463 (time 1788865987.12 total 234.80 last 18.58) - open/close 4692 (time 1788865997.39 total 245.07 last 22.32) - open/close 4941 (time 1788866008.25 total 255.93 last 22.92) - open/close 5202 (time 1788866018.26 total 265.94 last 26.08) - open/close 5415 (time 1788866028.53 total 276.21 last 20.73) - open/close 5580 (time 1788866038.53 total 286.21 last 16.50) - open/close 5709 (time 1788866049.31 total 296.99 last 11.97) - open/close 5827 (time 1788866061.06 total 308.74 last 10.04) - open/close 6025 (time 1788866072.13 total 319.81 last 17.89) - open/close 6287 (time 1788866082.45 total 330.13 last 25.39) - open/close 6490 (time 1788866092.49 total 340.17 last 20.23) - open/close 6684 (time 1788866102.60 total 350.28 last 19.18) - open/close 6870 (time 1788866112.62 total 360.30 last 18.58) - open/close 7095 (time 1788866123.26 total 370.93 last 21.15) - open/close 7238 (time 1788866133.72 total 381.40 last 13.66) - open/close 7395 (time 1788866143.73 total 391.41 last 15.69) - open/close 7610 (time 1788866154.82 total 402.50 last 19.40) - open/close 7849 (time 1788866165.63 total 413.31 last 22.10) - open/close 8093 (time 1788866175.64 total 423.32 last 24.38) - open/close 8300 (time 1788866185.64 total 433.32 last 20.70) - open/close 8553 (time 1788866196.54 total 444.22 last 23.22) - open/close 8793 (time 1788866206.54 total 454.22 last 23.99) - open/close 9051 (time 1788866216.54 total 464.22 last 25.79) - open/close 9251 (time 1788866226.55 total 474.23 last 19.99) - open/close 9486 (time 1788866237.07 total 484.74 last 22.34) - open/close 9708 (time 1788866247.13 total 494.81 last 22.05) - open/close 9973 (time 1788866258.50 total 506.18 last 23.32) - open/close 10188 (time 1788866268.52 total 516.20 last 21.46) - open/close 10409 (time 1788866278.52 total 526.20 last 22.10) - open/close 10618 (time 1788866288.76 total 536.44 last 20.40) - open/close 10842 (time 1788866299.01 total 546.69 last 21.86) - open/close 11067 (time 1788866309.44 total 557.12 last 21.57) - open/close 11299 (time 1788866320.40 total 568.08 last 21.18) - open/close 11457 (time 1788866330.77 total 578.45 last 15.24) - open/close 11633 (time 1788866341.75 total 589.43 last 16.02) - open/close 11827 (time 1788866351.76 total 599.44 last 19.38) - open/close 12041 (time 1788866362.23 total 609.91 last 20.43) - open/close 12289 (time 1788866373.06 total 620.74 last 22.91) - open/close 12545 (time 1788866383.07 total 630.75 last 25.57) - open/close 12798 (time 1788866393.35 total 641.03 last 24.61) - open/close 12993 (time 1788866403.96 total 651.64 last 18.38) - open/close 13239 (time 1788866414.01 total 661.69 last 24.48) - open/close 13434 (time 1788866425.11 total 672.79 last 17.56) - open/close 13683 (time 1788866435.83 total 683.51 last 23.23) - open/close 13918 (time 1788866446.40 total 694.08 last 22.23) - open/close 14151 (time 1788866456.40 total 704.08 last 23.30) - open/close 14372 (time 1788866466.42 total 714.10 last 22.06) - open/close 14594 (time 1788866476.87 total 724.55 last 21.24) - open/close 14827 (time 1788866487.35 total 735.03 last 22.23) - open/close 15039 (time 1788866497.78 total 745.46 last 20.32) - open/close 15253 (time 1788866508.30 total 755.98 last 20.35) - open/close 15343 (time 1788866518.32 total 766.00 last 8.98) - open/close 15551 (time 1788866528.33 total 776.01 last 20.78) - open/close 15737 (time 1788866539.14 total 786.82 last 17.21) - open/close 15994 (time 1788866549.14 total 796.82 last 25.69) - open/close 16225 (time 1788866560.09 total 807.77 last 21.11) - open/close 16421 (time 1788866570.65 total 818.32 last 18.56) - open/close 16631 (time 1788866580.93 total 828.61 last 20.43) - open/close 16786 (time 1788866591.34 total 839.02 last 14.88) - open/close 16960 (time 1788866601.35 total 849.03 last 17.39) - open/close 17140 (time 1788866611.96 total 859.64 last 16.97) - open/close 17372 (time 1788866623.03 total 870.71 last 20.94) - open/close 17584 (time 1788866633.73 total 881.41 last 19.82) - open/close 17762 (time 1788866643.75 total 891.43 last 17.77) - open/close 17839 (time 1788866653.95 total 901.63 last 7.55) - open/close 18036 (time 1788866663.96 total 911.64 last 19.68) - open/close 18241 (time 1788866674.37 total 922.05 last 19.68) - open/close 18495 (time 1788866685.24 total 932.92 last 23.38) - open/close 19041 (time 1788866695.24 total 942.92 last 54.56) - open/close 19839 (time 1788866705.26 total 952.94 last 79.64) - open/close 20000 (time 1788866707.66 total 955.34 last 67.17) - open/close 20676 (time 1788866717.68 total 965.36 last 67.49) - open/close 21356 (time 1788866727.70 total 975.38 last 67.86) - open/close 22011 (time 1788866737.71 total 985.39 last 65.40) - open/close 22700 (time 1788866747.72 total 995.40 last 68.83) - open/close 23359 (time 1788866757.73 total 1005.41 last 65.86) - open/close 24058 (time 1788866767.77 total 1015.45 last 69.59) - open/close 24753 (time 1788866777.78 total 1025.46 last 69.48) - open/close 25470 (time 1788866787.79 total 1035.47 last 71.62) - open/close 26174 (time 1788866797.79 total 1045.47 last 70.37) - open/close 26887 (time 1788866807.80 total 1055.48 last 71.24) - open/close 27597 (time 1788866817.80 total 1065.48 last 70.98) - open/close 28264 (time 1788866827.81 total 1075.49 last 66.67) - open/close 28939 (time 1788866837.82 total 1085.50 last 67.45) - open/close 29620 (time 1788866847.82 total 1095.50 last 68.08) - open/close 30000 (time 1788866853.28 total 1100.96 last 69.58) - open/close 30609 (time 1788866863.29 total 1110.97 last 60.82) - open/close 31230 (time 1788866873.30 total 1120.97 last 62.08) - open/close 31847 (time 1788866883.31 total 1130.99 last 61.58) - open/close 32526 (time 1788866893.32 total 1141.00 last 67.88) - open/close 33159 (time 1788866903.32 total 1151.00 last 63.27) - open/close 33834 (time 1788866913.35 total 1161.03 last 67.33) - open/close 34509 (time 1788866923.36 total 1171.04 last 67.42) - open/close 35153 (time 1788866933.36 total 1181.04 last 64.35) - open/close 35798 (time 1788866943.38 total 1191.06 last 64.42) - open/close 36472 (time 1788866953.38 total 1201.06 last 67.37) - open/close 37110 (time 1788866963.38 total 1211.06 last 63.79) - open/close 37802 (time 1788866973.39 total 1221.07 last 69.17) - open/close 38480 (time 1788866983.39 total 1231.07 last 67.78) - open/close 38965 (time 1788866993.39 total 1241.07 last 48.49) - open/close 39480 (time 1788867003.43 total 1251.11 last 51.31) - open/close 40000 (time 1788867012.85 total 1260.53 last 55.24) - open/close 40669 (time 1788867022.85 total 1270.53 last 66.89) - open/close 41256 (time 1788867032.86 total 1280.54 last 58.62) - open/close 41899 (time 1788867042.87 total 1290.54 last 64.27) - open/close 42558 (time 1788867052.87 total 1300.55 last 65.84) - open/close 43204 (time 1788867062.88 total 1310.56 last 64.55) - open/close 43887 (time 1788867072.91 total 1320.59 last 68.08) - open/close 44481 (time 1788867082.91 total 1330.59 last 59.40) - open/close 45039 (time 1788867092.92 total 1340.60 last 55.79) - open/close 45664 (time 1788867102.93 total 1350.61 last 62.39) - open/close 46332 (time 1788867112.94 total 1360.62 last 66.74) - open/close 46991 (time 1788867122.96 total 1370.63 last 65.82) - open/close 47670 (time 1788867132.96 total 1380.64 last 67.88) - open/close 48310 (time 1788867142.97 total 1390.65 last 63.92) - open/close 48958 (time 1788867152.97 total 1400.65 last 64.80) - open/close 49554 (time 1788867162.97 total 1410.65 last 59.59) total: 50000 open/close in 1417.65 seconds: 35.27 ops/second - unlinked 0 (time 1788867174 ; total 0 ; last 0) - unlinked 10000 (time 1788867333 ; total 159 ; last 159) - unlinked 20000 (time 1788867490 ; total 316 ; last 157) - unlinked 30000 (time 1788867647 ; total 473 ; last 157) - unlinked 40000 (time 1788867813 ; total 639 ; last 166) total: 50000 unlinks in 797 seconds: 62.735256 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server unloading modules via unload_modules_local on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local oleg331-server: kmemleak disabled modules unloaded. pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2