== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:16:24 (1788639384) start mds service on oleg628-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 4 18:18 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg628-server' oleg628-server: oleg628-server.virtnet: executing load_modules_local oleg628-server: kmemleak disabled oleg628-server: Loading modules from /home/green/git/lustre-release/lustre oleg628-server: detected 4 online CPUs by sysfs oleg628-server: MODOPTS_LIBCFS= oleg628-server: Force libcfs to create 2 CPU partitions oleg628-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg628-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg628-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg628-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg628-server: oleg628-server.virtnet: executing set_default_debug -1 all pdsh@oleg628-client: oleg628-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg628-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg628-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg628-server: oleg628-server.virtnet: executing set_default_debug -1 all pdsh@oleg628-client: oleg628-server: ssh exited with exit code 1 Started lustre-OST0000 oleg628-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg628-client.virtnet: -o user_xattr,flock 192.168.206.128@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 228 (time 1788639459.22 total 10.01 last 22.79) - open/close 424 (time 1788639469.24 total 20.03 last 19.55) - open/close 667 (time 1788639479.44 total 30.23 last 23.84) - open/close 895 (time 1788639489.54 total 40.33 last 22.58) - open/close 1108 (time 1788639499.96 total 50.75 last 20.43) - open/close 1321 (time 1788639510.47 total 61.25 last 20.28) - open/close 1567 (time 1788639521.17 total 71.96 last 22.99) - open/close 1836 (time 1788639531.18 total 81.96 last 26.88) - open/close 2108 (time 1788639541.83 total 92.62 last 25.52) - open/close 2387 (time 1788639552.64 total 103.43 last 25.82) - open/close 2615 (time 1788639562.64 total 113.43 last 22.79) - open/close 2827 (time 1788639573.32 total 124.11 last 19.86) - open/close 3090 (time 1788639583.32 total 134.11 last 26.30) - open/close 3298 (time 1788639593.32 total 144.11 last 20.80) - open/close 3508 (time 1788639603.33 total 154.12 last 20.98) - open/close 3745 (time 1788639613.66 total 164.44 last 22.95) - open/close 3915 (time 1788639623.67 total 174.46 last 16.97) - open/close 4143 (time 1788639633.67 total 184.46 last 22.79) - open/close 4284 (time 1788639643.85 total 194.64 last 13.85) - open/close 4516 (time 1788639653.85 total 204.64 last 23.19) - open/close 4716 (time 1788639664.39 total 215.18 last 18.98) - open/close 4942 (time 1788639674.44 total 225.23 last 22.48) - open/close 5187 (time 1788639684.54 total 235.33 last 24.26) - open/close 5408 (time 1788639694.55 total 245.33 last 22.09) - open/close 5596 (time 1788639704.55 total 255.34 last 18.80) - open/close 5815 (time 1788639715.39 total 266.18 last 20.20) - open/close 5970 (time 1788639725.39 total 276.18 last 15.49) - open/close 6144 (time 1788639736.15 total 286.94 last 16.17) - open/close 6332 (time 1788639746.16 total 296.95 last 18.79) - open/close 6492 (time 1788639756.62 total 307.41 last 15.29) - open/close 6621 (time 1788639766.72 total 317.51 last 12.76) - open/close 6824 (time 1788639776.73 total 327.51 last 20.30) - open/close 7052 (time 1788639787.27 total 338.06 last 21.63) - open/close 7296 (time 1788639797.45 total 348.24 last 23.96) - open/close 7530 (time 1788639808.35 total 359.13 last 21.48) - open/close 7802 (time 1788639818.36 total 369.15 last 27.16) - open/close 7997 (time 1788639829.96 total 380.74 last 16.82) - open/close 8214 (time 1788639840.47 total 391.26 last 20.64) - open/close 8390 (time 1788639850.51 total 401.30 last 17.53) - open/close 8616 (time 1788639860.51 total 411.30 last 22.60) - open/close 8821 (time 1788639870.52 total 421.31 last 20.48) - open/close 9005 (time 1788639880.78 total 431.56 last 17.95) - open/close 9231 (time 1788639891.27 total 442.05 last 21.54) - open/close 9454 (time 1788639902.17 total 452.95 last 20.46) - open/close 9676 (time 1788639912.40 total 463.18 last 21.70) - open/close 9906 (time 1788639922.95 total 473.73 last 21.80) - open/close 10000 (time 1788639926.53 total 477.32 last 26.22) - open/close 10178 (time 1788639937.23 total 488.02 last 16.64) - open/close 10376 (time 1788639947.24 total 498.02 last 19.79) - open/close 10493 (time 1788639957.51 total 508.30 last 11.38) - open/close 10722 (time 1788639967.62 total 518.40 last 22.67) - open/close 10961 (time 1788639977.62 total 528.41 last 23.89) - open/close 11180 (time 1788639988.35 total 539.14 last 20.40) - open/close 11409 (time 1788639999.31 total 550.09 last 20.91) - open/close 11663 (time 1788640009.93 total 560.72 last 23.91) - open/close 11912 (time 1788640020.54 total 571.33 last 23.46) - open/close 12137 (time 1788640030.73 total 581.52 last 22.09) - open/close 12349 (time 1788640040.86 total 591.65 last 20.91) - open/close 12600 (time 1788640050.89 total 601.68 last 25.04) - open/close 12838 (time 1788640060.90 total 611.69 last 23.77) - open/close 13058 (time 1788640071.81 total 622.60 last 20.16) - open/close 13299 (time 1788640081.81 total 632.60 last 24.10) - open/close 13467 (time 1788640092.04 total 642.83 last 16.43) - open/close 13668 (time 1788640102.08 total 652.87 last 20.03) - open/close 13845 (time 1788640112.09 total 662.87 last 17.68) - open/close 14000 (time 1788640122.09 total 672.87 last 15.50) - open/close 14193 (time 1788640132.11 total 682.90 last 19.25) - open/close 14427 (time 1788640142.13 total 692.92 last 23.36) - open/close 14674 (time 1788640152.84 total 703.63 last 23.05) - open/close 14922 (time 1788640162.86 total 713.65 last 24.76) - open/close 15108 (time 1788640172.86 total 723.65 last 18.59) - open/close 15300 (time 1788640183.38 total 734.17 last 18.26) - open/close 15551 (time 1788640193.38 total 744.17 last 25.09) - open/close 15786 (time 1788640203.85 total 754.64 last 22.45) - open/close 15994 (time 1788640214.22 total 765.01 last 20.05) - open/close 16144 (time 1788640225.16 total 775.95 last 13.71) - open/close 16324 (time 1788640235.22 total 786.01 last 17.90) - open/close 16561 (time 1788640245.24 total 796.03 last 23.65) - open/close 16729 (time 1788640255.56 total 806.35 last 16.28) - open/close 16936 (time 1788640266.83 total 817.62 last 18.36) - open/close 17100 (time 1788640277.37 total 828.16 last 15.56) - open/close 17325 (time 1788640287.64 total 838.43 last 21.91) - open/close 17554 (time 1788640300.11 total 850.90 last 18.37) - open/close 17788 (time 1788640310.22 total 861.01 last 23.14) - open/close 18006 (time 1788640320.24 total 871.03 last 21.76) - open/close 18436 (time 1788640330.24 total 881.03 last 42.98) - open/close 19080 (time 1788640340.25 total 891.04 last 64.35) - open/close 19708 (time 1788640350.25 total 901.04 last 62.78) - open/close 20000 (time 1788640355.21 total 906.00 last 58.92) - open/close 20607 (time 1788640365.21 total 916.00 last 60.69) - open/close 21250 (time 1788640375.22 total 926.00 last 64.28) - open/close 21876 (time 1788640385.22 total 936.01 last 62.54) - open/close 22635 (time 1788640395.23 total 946.01 last 75.88) - open/close 23337 (time 1788640405.23 total 956.02 last 70.17) - open/close 24021 (time 1788640415.25 total 966.03 last 68.30) - open/close 24704 (time 1788640425.25 total 976.04 last 68.27) - open/close 25246 (time 1788640435.25 total 986.04 last 54.20) - open/close 25947 (time 1788640445.26 total 996.05 last 70.01) - open/close 26599 (time 1788640455.27 total 1006.06 last 65.15) - open/close 27217 (time 1788640465.27 total 1016.06 last 61.78) - open/close 27850 (time 1788640475.29 total 1026.07 last 63.22) - open/close 28514 (time 1788640485.29 total 1036.08 last 66.39) - open/close 29165 (time 1788640495.29 total 1046.08 last 65.07) - open/close 29869 (time 1788640505.30 total 1056.09 last 70.35) - open/close 30516 (time 1788640515.30 total 1066.09 last 64.68) - open/close 31131 (time 1788640525.31 total 1076.09 last 61.48) - open/close 31715 (time 1788640535.31 total 1086.09 last 58.40) - open/close 32341 (time 1788640545.31 total 1096.10 last 62.56) - open/close 33000 (time 1788640555.33 total 1106.12 last 65.79) - open/close 33664 (time 1788640565.33 total 1116.12 last 66.37) - open/close 34321 (time 1788640575.34 total 1126.13 last 65.68) - open/close 34894 (time 1788640585.35 total 1136.14 last 57.25) - open/close 35431 (time 1788640595.35 total 1146.14 last 53.68) - open/close 35823 (time 1788640605.37 total 1156.16 last 39.12) - open/close 36411 (time 1788640615.39 total 1166.18 last 58.68) - open/close 37018 (time 1788640625.40 total 1176.18 last 60.68) - open/close 37625 (time 1788640635.41 total 1186.20 last 60.62) - open/close 38288 (time 1788640645.42 total 1196.21 last 66.21) - open/close 38858 (time 1788640655.43 total 1206.21 last 56.99) - open/close 39434 (time 1788640665.43 total 1216.22 last 57.57) - open/close 40000 (time 1788640673.94 total 1224.73 last 66.53) - open/close 40614 (time 1788640683.95 total 1234.74 last 61.34) - open/close 41286 (time 1788640693.97 total 1244.76 last 67.07) - open/close 41919 (time 1788640704.00 total 1254.78 last 63.12) - open/close 42399 (time 1788640714.00 total 1264.79 last 47.97) - open/close 42939 (time 1788640724.02 total 1274.80 last 53.92) - open/close 43453 (time 1788640734.02 total 1284.81 last 51.36) - open/close 44083 (time 1788640744.04 total 1294.83 last 62.91) - open/close 44797 (time 1788640754.04 total 1304.83 last 71.38) - open/close 45502 (time 1788640764.05 total 1314.83 last 70.47) - open/close 46100 (time 1788640774.05 total 1324.84 last 59.79) - open/close 46636 (time 1788640784.07 total 1334.85 last 53.50) - open/close 47226 (time 1788640794.09 total 1344.87 last 58.89) - open/close 47866 (time 1788640804.09 total 1354.88 last 63.96) - open/close 48468 (time 1788640814.09 total 1364.88 last 60.19) - open/close 49050 (time 1788640824.09 total 1374.88 last 58.19) - open/close 49708 (time 1788640834.10 total 1384.88 last 65.79) total: 50000 open/close in 1390.15 seconds: 35.97 ops/second - unlinked 0 (time 1788640844 ; total 0 ; last 0) - unlinked 10000 (time 1788641008 ; total 164 ; last 164) - unlinked 20000 (time 1788641169 ; total 325 ; last 161) - unlinked 30000 (time 1788641324 ; total 480 ; last 155) - unlinked 40000 (time 1788641479 ; total 635 ; last 155) total: 50000 unlinks in 789 seconds: 63.371357 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg628-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg628-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg628-server stop mds service on oleg628-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg628-server unloading modules via unload_modules_local on: 'oleg628-server' oleg628-server: oleg628-server.virtnet: executing unload_modules_local oleg628-server: kmemleak disabled modules unloaded. pdsh@oleg628-client: oleg628-server: ssh exited with exit code 2 pdsh@oleg628-client: oleg628-client: ssh exited with exit code 2 pdsh@oleg628-client: oleg628-server: ssh exited with exit code 2 pdsh@oleg628-client: oleg628-client: ssh exited with exit code 2