== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:51:12 (1788641472) start mds service on oleg316-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 4 18:18 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg316-server' oleg316-server: oleg316-server.virtnet: executing load_modules_local oleg316-server: kmemleak disabled oleg316-server: Loading modules from /home/green/git/lustre-release/lustre oleg316-server: detected 4 online CPUs by sysfs oleg316-server: MODOPTS_LIBCFS= oleg316-server: Force libcfs to create 2 CPU partitions oleg316-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg316-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg316-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg316-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg316-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg316-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Started lustre-OST0000 oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 219 (time 1788641575.80 total 10.10 last 21.68) - open/close 501 (time 1788641585.81 total 20.11 last 28.18) - open/close 686 (time 1788641595.81 total 30.11 last 18.49) - open/close 841 (time 1788641606.03 total 40.33 last 15.17) - open/close 967 (time 1788641616.83 total 51.13 last 11.67) - open/close 1082 (time 1788641627.32 total 61.63 last 10.96) - open/close 1333 (time 1788641637.33 total 71.63 last 25.09) - open/close 1586 (time 1788641647.33 total 81.63 last 25.29) - open/close 1787 (time 1788641657.99 total 92.29 last 18.86) - open/close 2094 (time 1788641668.01 total 102.31 last 30.65) - open/close 2336 (time 1788641678.27 total 112.57 last 23.57) - open/close 2620 (time 1788641688.28 total 122.58 last 28.38) - open/close 2862 (time 1788641698.30 total 132.60 last 24.15) - open/close 3103 (time 1788641708.30 total 142.60 last 24.09) - open/close 3315 (time 1788641719.03 total 153.34 last 19.75) - open/close 3582 (time 1788641729.48 total 163.79 last 25.55) - open/close 3891 (time 1788641740.16 total 174.46 last 28.95) - open/close 4152 (time 1788641750.16 total 184.47 last 26.08) - open/close 4383 (time 1788641760.55 total 194.85 last 22.25) - open/close 4582 (time 1788641771.74 total 206.04 last 17.78) - open/close 4740 (time 1788641782.39 total 216.70 last 14.83) - open/close 4894 (time 1788641792.41 total 226.71 last 15.38) - open/close 5120 (time 1788641802.43 total 236.73 last 22.56) - open/close 5348 (time 1788641813.13 total 247.43 last 21.31) - open/close 5549 (time 1788641823.88 total 258.18 last 18.69) - open/close 5801 (time 1788641834.23 total 268.53 last 24.35) - open/close 6040 (time 1788641844.41 total 278.71 last 23.48) - open/close 6279 (time 1788641854.43 total 288.73 last 23.85) - open/close 6501 (time 1788641864.50 total 298.80 last 22.05) - open/close 6868 (time 1788641874.52 total 308.82 last 36.60) - open/close 7113 (time 1788641884.54 total 318.85 last 24.45) - open/close 7335 (time 1788641894.55 total 328.86 last 22.18) - open/close 7643 (time 1788641904.56 total 338.86 last 30.77) - open/close 7935 (time 1788641915.00 total 349.30 last 27.98) - open/close 8158 (time 1788641925.37 total 359.67 last 21.50) - open/close 8381 (time 1788641935.38 total 369.68 last 22.29) - open/close 8612 (time 1788641945.41 total 379.71 last 23.03) - open/close 8873 (time 1788641956.27 total 390.58 last 24.02) - open/close 9084 (time 1788641966.28 total 400.58 last 21.10) - open/close 9368 (time 1788641976.28 total 410.58 last 28.40) - open/close 9518 (time 1788641986.30 total 420.60 last 14.97) - open/close 9782 (time 1788641996.31 total 430.61 last 26.36) - open/close 10000 (time 1788642005.22 total 439.53 last 24.45) - open/close 10273 (time 1788642016.27 total 450.57 last 24.73) - open/close 10533 (time 1788642026.30 total 460.60 last 25.90) - open/close 10742 (time 1788642037.51 total 471.81 last 18.65) - open/close 10935 (time 1788642047.62 total 481.92 last 19.08) - open/close 11151 (time 1788642057.70 total 492.00 last 21.43) - open/close 11422 (time 1788642068.75 total 503.05 last 24.53) - open/close 11637 (time 1788642080.82 total 515.12 last 17.81) - open/close 11914 (time 1788642090.83 total 525.13 last 27.68) - open/close 12159 (time 1788642101.01 total 535.31 last 24.07) - open/close 12394 (time 1788642111.80 total 546.10 last 21.77) - open/close 12646 (time 1788642122.29 total 556.59 last 24.02) - open/close 12857 (time 1788642132.30 total 566.60 last 21.09) - open/close 13045 (time 1788642142.30 total 576.60 last 18.79) - open/close 13109 (time 1788642155.22 total 589.52 last 4.95) - open/close 13338 (time 1788642165.24 total 599.54 last 22.86) - open/close 13540 (time 1788642175.35 total 609.65 last 19.99) - open/close 13714 (time 1788642185.66 total 619.97 last 16.87) - open/close 13989 (time 1788642195.68 total 629.98 last 27.46) - open/close 14208 (time 1788642206.16 total 640.46 last 20.89) - open/close 14466 (time 1788642217.16 total 651.46 last 23.45) - open/close 14719 (time 1788642227.71 total 662.01 last 24.00) - open/close 15001 (time 1788642238.79 total 673.09 last 25.45) - open/close 15272 (time 1788642248.82 total 683.12 last 27.02) - open/close 15463 (time 1788642259.87 total 694.17 last 17.28) - open/close 15652 (time 1788642270.66 total 704.96 last 17.52) - open/close 15849 (time 1788642280.70 total 715.01 last 19.61) - open/close 16119 (time 1788642291.72 total 726.02 last 24.52) - open/close 16304 (time 1788642304.49 total 738.80 last 14.48) - open/close 16457 (time 1788642314.51 total 748.81 last 15.27) - open/close 16669 (time 1788642324.88 total 759.19 last 20.44) - open/close 16946 (time 1788642334.89 total 769.20 last 27.67) - open/close 17205 (time 1788642344.90 total 779.20 last 25.89) - open/close 17465 (time 1788642355.71 total 790.01 last 24.05) - open/close 17678 (time 1788642365.72 total 800.02 last 21.28) - open/close 17940 (time 1788642375.72 total 810.02 last 26.20) - open/close 18192 (time 1788642385.73 total 820.03 last 25.18) - open/close 18345 (time 1788642396.13 total 830.44 last 14.70) - open/close 18525 (time 1788642407.11 total 841.41 last 16.40) - open/close 18665 (time 1788642419.35 total 853.65 last 11.44) - open/close 18883 (time 1788642429.94 total 864.24 last 20.57) - open/close 19081 (time 1788642440.97 total 875.27 last 17.95) - open/close 19320 (time 1788642451.74 total 886.04 last 22.21) - open/close 19551 (time 1788642462.22 total 896.52 last 22.04) - open/close 19808 (time 1788642472.77 total 907.07 last 24.35) - open/close 20000 (time 1788642482.27 total 916.57 last 20.22) - open/close 20249 (time 1788642492.85 total 927.15 last 23.53) - open/close 20563 (time 1788642502.86 total 937.16 last 31.39) - open/close 21298 (time 1788642512.86 total 947.17 last 73.44) - open/close 21970 (time 1788642522.87 total 957.18 last 67.13) - open/close 22741 (time 1788642532.88 total 967.18 last 77.07) - open/close 23482 (time 1788642542.90 total 977.20 last 73.93) - open/close 24320 (time 1788642552.90 total 987.20 last 83.79) - open/close 25052 (time 1788642562.90 total 997.21 last 73.18) - open/close 25830 (time 1788642572.92 total 1007.22 last 77.66) - open/close 26586 (time 1788642582.92 total 1017.22 last 75.59) - open/close 27326 (time 1788642592.94 total 1027.24 last 73.86) - open/close 28071 (time 1788642602.95 total 1037.26 last 74.41) - open/close 28796 (time 1788642612.96 total 1047.26 last 72.47) - open/close 29506 (time 1788642622.98 total 1057.28 last 70.88) - open/close 30000 (time 1788642629.51 total 1063.81 last 75.59) - open/close 30699 (time 1788642639.54 total 1073.84 last 69.73) - open/close 31452 (time 1788642649.54 total 1083.84 last 75.29) - open/close 32259 (time 1788642659.54 total 1093.84 last 80.68) - open/close 33065 (time 1788642669.55 total 1103.85 last 80.49) - open/close 33875 (time 1788642679.57 total 1113.87 last 80.89) - open/close 34575 (time 1788642689.58 total 1123.88 last 69.94) - open/close 35217 (time 1788642699.58 total 1133.88 last 64.18) - open/close 35892 (time 1788642709.59 total 1143.89 last 67.40) - open/close 36664 (time 1788642719.61 total 1153.91 last 77.07) - open/close 37308 (time 1788642729.61 total 1163.92 last 64.37) - open/close 38037 (time 1788642739.62 total 1173.92 last 72.87) - open/close 38775 (time 1788642749.62 total 1183.92 last 73.76) - open/close 39543 (time 1788642759.63 total 1193.93 last 76.74) - open/close 40000 (time 1788642765.36 total 1199.66 last 79.82) - open/close 40748 (time 1788642775.36 total 1209.67 last 74.73) - open/close 41521 (time 1788642785.38 total 1219.68 last 77.18) - open/close 42255 (time 1788642795.39 total 1229.69 last 73.36) - open/close 43006 (time 1788642805.40 total 1239.70 last 75.01) - open/close 43743 (time 1788642815.41 total 1249.71 last 73.59) - open/close 44450 (time 1788642825.42 total 1259.72 last 70.66) - open/close 45235 (time 1788642835.42 total 1269.72 last 78.48) - open/close 46009 (time 1788642845.43 total 1279.74 last 77.28) - open/close 46755 (time 1788642855.44 total 1289.74 last 74.58) - open/close 47515 (time 1788642865.44 total 1299.75 last 75.95) - open/close 48219 (time 1788642875.45 total 1309.75 last 70.39) - open/close 48993 (time 1788642885.47 total 1319.77 last 77.24) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1328.85 seconds: 37.35 ops/second - unlinked 0 (time 1788642899 ; total 0 ; last 0) - unlinked 10000 (time 1788643037 ; total 138 ; last 138) - unlinked 20000 (time 1788643180 ; total 281 ; last 143) - unlinked 30000 (time 1788643316 ; total 417 ; last 136) - unlinked 40000 (time 1788643454 ; total 555 ; last 138) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 699 seconds: 71.002861 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg316-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg316-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg316-server stop mds service on oleg316-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg316-server stop mds service on oleg316-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg316-server unloading modules via unload_modules_local on: 'oleg316-server' oleg316-server: oleg316-server.virtnet: executing unload_modules_local oleg316-server: kmemleak disabled modules unloaded. pdsh@oleg316-client: oleg316-server: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-client: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-client: ssh exited with exit code 2 pdsh@oleg316-client: oleg316-server: ssh exited with exit code 2