== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:55:02 (1788839702) start mds service on oleg624-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1754936 Sep 7 17:41 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg624-server' oleg624-server: oleg624-server.virtnet: executing load_modules_local oleg624-server: kmemleak disabled oleg624-server: Loading modules from /home/green/git/lustre-release/lustre oleg624-server: detected 4 online CPUs by sysfs oleg624-server: MODOPTS_LIBCFS= oleg624-server: Force libcfs to create 2 CPU partitions oleg624-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg624-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg624-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg624-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg624-server: oleg624-server.virtnet: executing set_default_debug -1 all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg624-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg624-server: oleg624-server.virtnet: executing set_default_debug -1 all pdsh@oleg624-client: oleg624-server: ssh exited with exit code 1 Started lustre-OST0000 oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg624-client.virtnet: -o user_xattr,flock 192.168.206.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 246 (time 1788839781.36 total 10.94 last 22.48) - open/close 504 (time 1788839792.29 total 21.88 last 23.59) - open/close 759 (time 1788839803.04 total 32.63 last 23.73) - open/close 1041 (time 1788839813.05 total 42.63 last 28.18) - open/close 1256 (time 1788839823.08 total 52.66 last 21.44) - open/close 1431 (time 1788839833.44 total 63.03 last 16.88) - open/close 1702 (time 1788839843.46 total 73.04 last 27.06) - open/close 1882 (time 1788839853.47 total 83.06 last 17.97) - open/close 2117 (time 1788839864.17 total 93.76 last 21.97) - open/close 2362 (time 1788839875.18 total 104.77 last 22.25) - open/close 2605 (time 1788839885.35 total 114.93 last 23.90) - open/close 2925 (time 1788839895.36 total 124.94 last 31.97) - open/close 3159 (time 1788839906.09 total 135.68 last 21.79) - open/close 3370 (time 1788839916.13 total 145.72 last 21.02) - open/close 3575 (time 1788839926.82 total 156.40 last 19.19) - open/close 3770 (time 1788839936.82 total 166.41 last 19.48) - open/close 4002 (time 1788839947.11 total 176.69 last 22.56) - open/close 4156 (time 1788839957.34 total 186.93 last 15.05) - open/close 4381 (time 1788839967.78 total 197.36 last 21.55) - open/close 4637 (time 1788839978.40 total 207.98 last 24.11) - open/close 4912 (time 1788839988.40 total 217.99 last 27.49) - open/close 5153 (time 1788839999.08 total 228.67 last 22.57) - open/close 5381 (time 1788840009.65 total 239.24 last 21.57) - open/close 5623 (time 1788840020.46 total 250.04 last 22.40) - open/close 5876 (time 1788840031.22 total 260.80 last 23.52) - open/close 6127 (time 1788840041.22 total 270.81 last 25.09) - open/close 6357 (time 1788840051.23 total 280.81 last 22.99) - open/close 6591 (time 1788840062.28 total 291.87 last 21.16) - open/close 6817 (time 1788840072.29 total 301.87 last 22.59) - open/close 7045 (time 1788840082.47 total 312.05 last 22.40) - open/close 7297 (time 1788840093.41 total 323.00 last 23.03) - open/close 7564 (time 1788840103.43 total 333.01 last 26.66) - open/close 7758 (time 1788840113.44 total 343.02 last 19.38) - open/close 7983 (time 1788840124.13 total 353.72 last 21.04) - open/close 8239 (time 1788840135.14 total 364.73 last 23.25) - open/close 8506 (time 1788840145.44 total 375.03 last 25.92) - open/close 8794 (time 1788840156.19 total 385.78 last 26.79) - open/close 9082 (time 1788840166.21 total 395.79 last 28.76) - open/close 9361 (time 1788840176.90 total 406.48 last 26.10) - open/close 9605 (time 1788840186.90 total 416.49 last 24.39) - open/close 9814 (time 1788840197.88 total 427.47 last 19.04) - open/close 10000 (time 1788840205.63 total 435.21 last 24.01) - open/close 10292 (time 1788840215.72 total 445.30 last 28.93) - open/close 10543 (time 1788840226.53 total 456.12 last 23.21) - open/close 10776 (time 1788840236.97 total 466.55 last 22.32) - open/close 11027 (time 1788840246.97 total 476.56 last 25.09) - open/close 11285 (time 1788840257.34 total 486.92 last 24.89) - open/close 11564 (time 1788840267.35 total 496.93 last 27.87) - open/close 11806 (time 1788840277.92 total 507.50 last 22.89) - open/close 12030 (time 1788840287.93 total 517.51 last 22.38) - open/close 12285 (time 1788840297.97 total 527.55 last 25.41) - open/close 12517 (time 1788840307.97 total 537.55 last 23.20) - open/close 12766 (time 1788840318.26 total 547.85 last 24.19) - open/close 13076 (time 1788840328.26 total 557.85 last 30.99) - open/close 13297 (time 1788840338.27 total 567.86 last 22.08) - open/close 13594 (time 1788840349.18 total 578.77 last 27.22) - open/close 13819 (time 1788840359.22 total 588.81 last 22.42) - open/close 14086 (time 1788840369.77 total 599.35 last 25.31) - open/close 14331 (time 1788840380.73 total 610.32 last 22.35) - open/close 14597 (time 1788840391.27 total 620.85 last 25.24) - open/close 14828 (time 1788840401.29 total 630.87 last 23.06) - open/close 15095 (time 1788840411.29 total 640.87 last 26.69) - open/close 15369 (time 1788840422.32 total 651.90 last 24.85) - open/close 15628 (time 1788840432.32 total 661.91 last 25.88) - open/close 15884 (time 1788840443.23 total 672.82 last 23.46) - open/close 16106 (time 1788840453.24 total 682.83 last 22.18) - open/close 16343 (time 1788840463.25 total 692.84 last 23.68) - open/close 16630 (time 1788840474.08 total 703.67 last 26.49) - open/close 16921 (time 1788840484.94 total 714.52 last 26.82) - open/close 17163 (time 1788840494.95 total 724.54 last 24.15) - open/close 17427 (time 1788840505.45 total 735.04 last 25.14) - open/close 17685 (time 1788840515.83 total 745.42 last 24.86) - open/close 17921 (time 1788840526.00 total 755.58 last 23.22) - open/close 18188 (time 1788840537.08 total 766.67 last 24.09) - open/close 18433 (time 1788840547.70 total 777.29 last 23.07) - open/close 18695 (time 1788840558.43 total 788.02 last 24.41) - open/close 18964 (time 1788840568.45 total 798.04 last 26.85) - open/close 19544 (time 1788840578.46 total 808.05 last 57.96) - open/close 20000 (time 1788840584.48 total 814.07 last 75.76) - open/close 20792 (time 1788840594.49 total 824.07 last 79.16) - open/close 21619 (time 1788840604.50 total 834.08 last 82.62) - open/close 22421 (time 1788840614.55 total 844.13 last 79.78) - open/close 23089 (time 1788840624.55 total 854.14 last 66.76) - open/close 23836 (time 1788840634.56 total 864.14 last 74.68) - open/close 24624 (time 1788840644.58 total 874.17 last 78.60) - open/close 25307 (time 1788840654.59 total 884.18 last 68.23) - open/close 26120 (time 1788840664.60 total 894.19 last 81.23) - open/close 26877 (time 1788840674.61 total 904.20 last 75.63) - open/close 27691 (time 1788840684.61 total 914.20 last 81.39) - open/close 28493 (time 1788840694.62 total 924.20 last 80.16) - open/close 29286 (time 1788840704.62 total 934.20 last 79.29) - open/close 30000 (time 1788840713.12 total 942.71 last 83.95) - open/close 30857 (time 1788840723.13 total 952.72 last 85.63) - open/close 31686 (time 1788840733.14 total 962.72 last 82.85) - open/close 32544 (time 1788840743.15 total 972.73 last 85.73) - open/close 33395 (time 1788840753.16 total 982.74 last 85.02) - open/close 34233 (time 1788840763.16 total 992.74 last 83.78) - open/close 35085 (time 1788840773.17 total 1002.75 last 85.14) - open/close 35892 (time 1788840783.17 total 1012.75 last 80.69) - open/close 36724 (time 1788840793.18 total 1022.76 last 83.13) - open/close 37519 (time 1788840803.18 total 1032.76 last 79.47) - open/close 38376 (time 1788840813.18 total 1042.77 last 85.69) - open/close 39189 (time 1788840823.18 total 1052.77 last 81.26) - open/close 40000 (time 1788840832.86 total 1062.45 last 83.78) - open/close 40804 (time 1788840842.87 total 1072.46 last 80.34) - open/close 41622 (time 1788840852.88 total 1082.46 last 81.74) - open/close 42454 (time 1788840862.89 total 1092.47 last 83.13) - open/close 43240 (time 1788840872.89 total 1102.47 last 78.59) - open/close 44052 (time 1788840882.89 total 1112.48 last 81.18) - open/close 44867 (time 1788840892.90 total 1122.49 last 81.42) - open/close 45699 (time 1788840902.90 total 1132.49 last 83.19) - open/close 46555 (time 1788840912.91 total 1142.49 last 85.58) - open/close 47390 (time 1788840922.91 total 1152.50 last 83.44) - open/close 48226 (time 1788840932.92 total 1162.50 last 83.58) - open/close 49149 (time 1788840942.92 total 1172.50 last 92.26) total: 50000 open/close in 1181.76 seconds: 42.31 ops/second - unlinked 0 (time 1788840954 ; total 0 ; last 0) - unlinked 10000 (time 1788841074 ; total 120 ; last 120) - unlinked 20000 (time 1788841198 ; total 244 ; last 124) - unlinked 30000 (time 1788841318 ; total 364 ; last 120) - unlinked 40000 (time 1788841448 ; total 494 ; last 130) total: 50000 unlinks in 616 seconds: 81.168831 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg624-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg624-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg624-server stop mds service on oleg624-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg624-server unloading modules via unload_modules_local on: 'oleg624-server' oleg624-server: oleg624-server.virtnet: executing unload_modules_local oleg624-server: kmemleak disabled modules unloaded. pdsh@oleg624-client: oleg624-client: ssh exited with exit code 2 pdsh@oleg624-client: oleg624-server: ssh exited with exit code 2 pdsh@oleg624-client: oleg624-client: ssh exited with exit code 2 pdsh@oleg624-client: oleg624-server: ssh exited with exit code 2