== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:04:47 (1789686287) start mds service on oleg110-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 17 16:20 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg110-server' oleg110-server: oleg110-server.virtnet: executing load_modules_local oleg110-server: kmemleak disabled oleg110-server: Loading modules from /home/green/git/lustre-release/lustre oleg110-server: detected 4 online CPUs by sysfs oleg110-server: MODOPTS_LIBCFS= oleg110-server: Force libcfs to create 2 CPU partitions oleg110-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg110-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg110-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg110-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg110-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0000 oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 181 (time 1789686377.79 total 12.32 last 14.69) - open/close 347 (time 1789686387.80 total 22.33 last 16.60) - open/close 522 (time 1789686398.39 total 32.92 last 16.52) - open/close 635 (time 1789686409.40 total 43.93 last 10.27) - open/close 718 (time 1789686419.94 total 54.47 last 7.88) - open/close 893 (time 1789686430.83 total 65.36 last 16.06) - open/close 1153 (time 1789686441.60 total 76.13 last 24.15) - open/close 1443 (time 1789686452.23 total 86.76 last 27.28) - open/close 1692 (time 1789686463.04 total 97.57 last 23.03) - open/close 1867 (time 1789686473.05 total 107.58 last 17.49) - open/close 2004 (time 1789686483.05 total 117.58 last 13.70) - open/close 2206 (time 1789686493.05 total 127.58 last 20.19) - open/close 2490 (time 1789686503.06 total 137.59 last 28.39) - open/close 2693 (time 1789686513.10 total 147.63 last 20.21) - open/close 2955 (time 1789686524.04 total 158.57 last 23.95) - open/close 3175 (time 1789686534.06 total 168.59 last 21.96) - open/close 3412 (time 1789686544.77 total 179.30 last 22.13) - open/close 3621 (time 1789686554.89 total 189.42 last 20.65) - open/close 3828 (time 1789686566.32 total 200.85 last 18.11) - open/close 4049 (time 1789686576.37 total 210.90 last 21.98) - open/close 4280 (time 1789686587.08 total 221.61 last 21.57) - open/close 4499 (time 1789686598.01 total 232.55 last 20.03) - open/close 4750 (time 1789686608.03 total 242.56 last 25.07) - open/close 4981 (time 1789686618.05 total 252.58 last 23.06) - open/close 5225 (time 1789686628.07 total 262.60 last 24.34) - open/close 5401 (time 1789686638.13 total 272.66 last 17.50) - open/close 5743 (time 1789686649.09 total 283.62 last 31.21) - open/close 5977 (time 1789686659.89 total 294.42 last 21.66) - open/close 6192 (time 1789686669.90 total 304.43 last 21.48) - open/close 6408 (time 1789686680.49 total 315.02 last 20.39) - open/close 6561 (time 1789686690.74 total 325.27 last 14.93) - open/close 6721 (time 1789686701.11 total 335.64 last 15.43) - open/close 6842 (time 1789686711.92 total 346.45 last 11.20) - open/close 7008 (time 1789686722.37 total 356.90 last 15.89) - open/close 7191 (time 1789686733.17 total 367.70 last 16.94) - open/close 7356 (time 1789686743.94 total 378.47 last 15.33) - open/close 7483 (time 1789686754.03 total 388.56 last 12.58) - open/close 7620 (time 1789686764.04 total 398.57 last 13.69) - open/close 7815 (time 1789686774.45 total 408.98 last 18.72) - open/close 7896 (time 1789686784.88 total 419.41 last 7.77) - open/close 8045 (time 1789686795.76 total 430.29 last 13.70) - open/close 8244 (time 1789686806.44 total 440.97 last 18.63) - open/close 8393 (time 1789686817.41 total 451.94 last 13.58) - open/close 8536 (time 1789686827.63 total 462.16 last 14.00) - open/close 8758 (time 1789686837.87 total 472.40 last 21.68) - open/close 8889 (time 1789686847.87 total 482.40 last 13.09) - open/close 8984 (time 1789686858.22 total 492.75 last 9.18) - open/close 9190 (time 1789686868.72 total 503.25 last 19.63) - open/close 9462 (time 1789686879.15 total 513.68 last 26.09) - open/close 9683 (time 1789686889.77 total 524.30 last 20.80) - open/close 9921 (time 1789686903.54 total 538.07 last 17.29) - open/close 10000 (time 1789686907.33 total 541.86 last 20.85) - open/close 10282 (time 1789686918.12 total 552.65 last 26.12) - open/close 10586 (time 1789686929.01 total 563.54 last 27.93) - open/close 10899 (time 1789686939.01 total 573.54 last 31.30) - open/close 11202 (time 1789686949.87 total 584.40 last 27.91) - open/close 11590 (time 1789686960.62 total 595.15 last 36.08) - open/close 11824 (time 1789686971.06 total 605.59 last 22.41) - open/close 12049 (time 1789686981.30 total 615.83 last 21.98) - open/close 12284 (time 1789686991.92 total 626.45 last 22.13) - open/close 12487 (time 1789687001.96 total 636.49 last 20.20) - open/close 12741 (time 1789687012.33 total 646.86 last 24.51) - open/close 12993 (time 1789687022.33 total 656.86 last 25.20) - open/close 13917 (time 1789687032.33 total 666.86 last 92.40) - open/close 15186 (time 1789687042.33 total 676.86 last 126.87) - open/close 16518 (time 1789687052.34 total 686.87 last 133.16) - open/close 17883 (time 1789687062.34 total 696.87 last 136.43) - open/close 19336 (time 1789687072.34 total 706.87 last 145.26) - open/close 20000 (time 1789687076.55 total 711.08 last 157.94) - open/close 21467 (time 1789687086.55 total 721.08 last 146.69) - open/close 22901 (time 1789687096.55 total 731.08 last 143.39) - open/close 24744 (time 1789687106.55 total 741.08 last 184.24) - open/close 26280 (time 1789687116.56 total 751.09 last 153.53) - open/close 27841 (time 1789687126.56 total 761.09 last 156.01) - open/close 29280 (time 1789687136.57 total 771.10 last 143.73) - open/close 30000 (time 1789687141.95 total 776.48 last 133.96) - open/close 31645 (time 1789687151.96 total 786.49 last 164.34) - open/close 33071 (time 1789687161.96 total 796.49 last 142.59) - open/close 34117 (time 1789687171.97 total 806.50 last 104.54) - open/close 34994 (time 1789687181.98 total 816.51 last 87.60) - open/close 35506 (time 1789687192.00 total 826.53 last 51.09) - open/close 35870 (time 1789687202.01 total 836.54 last 36.36) - open/close 36246 (time 1789687212.02 total 846.55 last 37.54) - open/close 36644 (time 1789687222.03 total 856.56 last 39.77) - open/close 37056 (time 1789687232.04 total 866.57 last 41.18) - open/close 37698 (time 1789687242.04 total 876.57 last 64.16) - open/close 38505 (time 1789687252.05 total 886.58 last 80.67) - open/close 39233 (time 1789687262.05 total 896.58 last 72.74) - open/close 39854 (time 1789687272.08 total 906.61 last 61.95) - open/close 40000 (time 1789687275.30 total 909.83 last 45.29) - open/close 40490 (time 1789687285.31 total 919.84 last 48.97) - open/close 41043 (time 1789687295.31 total 929.84 last 55.28) - open/close 41933 (time 1789687305.31 total 939.84 last 88.98) - open/close 42685 (time 1789687315.31 total 949.84 last 75.20) - open/close 43375 (time 1789687325.34 total 959.87 last 68.79) - open/close 44236 (time 1789687335.35 total 969.88 last 86.07) - open/close 45121 (time 1789687345.35 total 979.88 last 88.48) - open/close 45991 (time 1789687355.36 total 989.89 last 86.93) - open/close 46900 (time 1789687365.36 total 999.89 last 90.90) - open/close 47746 (time 1789687375.37 total 1009.90 last 84.52) - open/close 48628 (time 1789687385.37 total 1019.90 last 88.15) - open/close 49587 (time 1789687395.38 total 1029.91 last 95.83) total: 50000 open/close in 1035.40 seconds: 48.29 ops/second - unlinked 0 (time 1789687404 ; total 0 ; last 0) - unlinked 10000 (time 1789687541 ; total 137 ; last 137) - unlinked 20000 (time 1789687639 ; total 235 ; last 98) - unlinked 30000 (time 1789687721 ; total 317 ; last 82) - unlinked 40000 (time 1789687802 ; total 398 ; last 81) total: 50000 unlinks in 483 seconds: 103.519669 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg110-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg110-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg110-server stop mds service on oleg110-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg110-server unloading modules via unload_modules_local on: 'oleg110-server' oleg110-server: oleg110-server.virtnet: executing unload_modules_local oleg110-server: kmemleak disabled modules unloaded. pdsh@oleg110-client: oleg110-client: ssh exited with exit code 2 pdsh@oleg110-client: oleg110-server: ssh exited with exit code 2 pdsh@oleg110-client: oleg110-server: ssh exited with exit code 2 pdsh@oleg110-client: oleg110-client: ssh exited with exit code 2