== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:44:41 (1788655481) start mds service on oleg304-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 4 18:31 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg304-server' oleg304-server: oleg304-server.virtnet: executing load_modules_local oleg304-server: kmemleak disabled oleg304-server: Loading modules from /home/green/git/lustre-release/lustre oleg304-server: detected 4 online CPUs by sysfs oleg304-server: MODOPTS_LIBCFS= oleg304-server: Force libcfs to create 2 CPU partitions oleg304-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg304-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg304-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg304-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg304-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg304-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-OST0000 oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 271 (time 1788655576.42 total 10.02 last 27.05) - open/close 531 (time 1788655587.25 total 20.85 last 24.01) - open/close 800 (time 1788655597.43 total 31.02 last 26.43) - open/close 997 (time 1788655607.83 total 41.42 last 18.94) - open/close 1290 (time 1788655617.84 total 51.43 last 29.27) - open/close 1556 (time 1788655628.00 total 61.59 last 26.18) - open/close 1830 (time 1788655638.49 total 72.09 last 26.11) - open/close 2099 (time 1788655648.99 total 82.58 last 25.62) - open/close 2399 (time 1788655658.99 total 92.59 last 29.99) - open/close 2670 (time 1788655669.01 total 102.61 last 27.05) - open/close 2921 (time 1788655679.13 total 112.73 last 24.80) - open/close 3184 (time 1788655690.16 total 123.75 last 23.86) - open/close 3458 (time 1788655701.03 total 134.63 last 25.19) - open/close 3711 (time 1788655711.04 total 144.64 last 25.27) - open/close 3972 (time 1788655723.36 total 156.96 last 21.19) - open/close 4186 (time 1788655734.30 total 167.90 last 19.57) - open/close 4372 (time 1788655744.79 total 178.39 last 17.73) - open/close 4504 (time 1788655754.81 total 188.40 last 13.18) - open/close 4727 (time 1788655765.79 total 199.39 last 20.30) - open/close 4939 (time 1788655775.80 total 209.39 last 21.19) - open/close 5169 (time 1788655786.39 total 219.99 last 21.71) - open/close 5475 (time 1788655797.01 total 230.61 last 28.80) - open/close 5737 (time 1788655807.02 total 240.61 last 26.19) - open/close 6018 (time 1788655817.02 total 250.62 last 28.08) - open/close 6325 (time 1788655827.71 total 261.31 last 28.73) - open/close 6586 (time 1788655838.01 total 271.60 last 25.35) - open/close 6864 (time 1788655848.67 total 282.27 last 26.07) - open/close 7179 (time 1788655858.72 total 292.32 last 31.33) - open/close 7283 (time 1788655868.72 total 302.32 last 10.40) - open/close 7530 (time 1788655878.73 total 312.33 last 24.68) - open/close 7685 (time 1788655889.79 total 323.39 last 14.02) - open/close 7755 (time 1788655899.80 total 333.40 last 6.99) - open/close 7862 (time 1788655910.50 total 344.10 last 10.00) - open/close 7914 (time 1788655920.98 total 354.58 last 4.96) - open/close 8138 (time 1788655931.39 total 364.98 last 21.54) - open/close 8430 (time 1788655942.30 total 375.89 last 26.76) - open/close 8701 (time 1788655952.94 total 386.54 last 25.46) - open/close 9023 (time 1788655963.89 total 397.48 last 29.41) - open/close 9142 (time 1788655973.90 total 407.50 last 11.89) - open/close 9382 (time 1788655983.90 total 417.50 last 23.99) - open/close 9636 (time 1788655993.91 total 427.51 last 25.37) - open/close 9882 (time 1788656005.03 total 438.63 last 22.13) - open/close 10000 (time 1788656009.92 total 443.52 last 24.12) - open/close 10231 (time 1788656020.02 total 453.61 last 22.88) - open/close 10504 (time 1788656030.02 total 463.62 last 27.29) - open/close 10748 (time 1788656040.19 total 473.79 last 23.98) - open/close 10988 (time 1788656051.10 total 484.70 last 22.00) - open/close 11281 (time 1788656062.07 total 495.67 last 26.71) - open/close 11552 (time 1788656072.09 total 505.68 last 27.05) - open/close 11862 (time 1788656083.10 total 516.70 last 28.14) - open/close 12090 (time 1788656094.06 total 527.66 last 20.81) - open/close 12311 (time 1788656104.09 total 537.68 last 22.04) - open/close 12553 (time 1788656114.16 total 547.76 last 24.02) - open/close 12771 (time 1788656124.78 total 558.38 last 20.53) - open/close 13042 (time 1788656135.56 total 569.16 last 25.14) - open/close 13338 (time 1788656145.58 total 579.18 last 29.55) - open/close 13596 (time 1788656155.59 total 589.19 last 25.78) - open/close 13869 (time 1788656165.59 total 599.19 last 27.29) - open/close 14133 (time 1788656175.91 total 609.51 last 25.59) - open/close 14408 (time 1788656186.28 total 619.88 last 26.50) - open/close 14676 (time 1788656196.30 total 629.89 last 26.77) - open/close 14917 (time 1788656206.31 total 639.90 last 24.08) - open/close 15217 (time 1788656216.74 total 650.34 last 28.75) - open/close 15407 (time 1788656227.17 total 660.76 last 18.22) - open/close 15708 (time 1788656237.18 total 670.77 last 30.07) - open/close 15962 (time 1788656248.13 total 681.72 last 23.20) - open/close 16224 (time 1788656259.18 total 692.78 last 23.69) - open/close 16391 (time 1788656270.04 total 703.64 last 15.37) - open/close 16688 (time 1788656280.55 total 714.14 last 28.29) - open/close 16926 (time 1788656290.93 total 724.52 last 22.93) - open/close 17184 (time 1788656300.94 total 734.53 last 25.77) - open/close 17405 (time 1788656311.20 total 744.80 last 21.53) - open/close 17685 (time 1788656321.75 total 755.35 last 26.54) - open/close 17960 (time 1788656331.75 total 765.35 last 27.50) - open/close 18078 (time 1788656342.55 total 776.15 last 10.93) - open/close 18190 (time 1788656352.57 total 786.17 last 11.18) - open/close 18422 (time 1788656362.59 total 796.18 last 23.17) - open/close 18705 (time 1788656372.60 total 806.20 last 28.26) - open/close 18984 (time 1788656383.46 total 817.06 last 25.69) - open/close 19110 (time 1788656394.11 total 827.71 last 11.83) - open/close 19308 (time 1788656404.14 total 837.73 last 19.76) - open/close 19593 (time 1788656414.14 total 847.74 last 28.48) - open/close 19794 (time 1788656424.87 total 858.47 last 18.73) - open/close 20000 (time 1788656432.01 total 865.61 last 28.85) - open/close 20258 (time 1788656442.02 total 875.61 last 25.79) - open/close 20549 (time 1788656452.73 total 886.33 last 27.16) - open/close 20751 (time 1788656462.74 total 896.34 last 20.19) - open/close 21538 (time 1788656472.74 total 906.34 last 78.68) - open/close 22430 (time 1788656482.74 total 916.34 last 89.18) - open/close 23356 (time 1788656492.75 total 926.35 last 92.52) - open/close 24300 (time 1788656502.76 total 936.35 last 94.37) - open/close 25257 (time 1788656512.76 total 946.36 last 95.68) - open/close 26179 (time 1788656522.76 total 956.36 last 92.20) - open/close 27043 (time 1788656532.76 total 966.36 last 86.37) - open/close 27932 (time 1788656542.77 total 976.37 last 88.81) - open/close 28806 (time 1788656552.78 total 986.38 last 87.35) - open/close 29659 (time 1788656562.78 total 996.38 last 85.26) - open/close 30000 (time 1788656566.95 total 1000.54 last 81.90) - open/close 30965 (time 1788656576.95 total 1010.54 last 96.50) - open/close 31846 (time 1788656586.96 total 1020.55 last 88.01) - open/close 32713 (time 1788656596.96 total 1030.55 last 86.69) - open/close 33588 (time 1788656606.96 total 1040.56 last 87.43) - open/close 34457 (time 1788656616.97 total 1050.56 last 86.88) - open/close 35350 (time 1788656626.97 total 1060.57 last 89.29) - open/close 36242 (time 1788656636.98 total 1070.57 last 89.13) - open/close 37133 (time 1788656646.99 total 1080.59 last 88.99) - open/close 37959 (time 1788656656.99 total 1090.59 last 82.57) - open/close 38868 (time 1788656667.01 total 1100.61 last 90.76) - open/close 39763 (time 1788656677.01 total 1110.61 last 89.47) - open/close 40000 (time 1788656679.86 total 1113.46 last 83.18) - open/close 40806 (time 1788656689.88 total 1123.48 last 80.47) - open/close 41629 (time 1788656699.89 total 1133.49 last 82.20) - open/close 42454 (time 1788656709.89 total 1143.49 last 82.48) - open/close 43310 (time 1788656719.89 total 1153.49 last 85.60) - open/close 44205 (time 1788656729.91 total 1163.51 last 89.35) - open/close 44976 (time 1788656739.91 total 1173.51 last 77.10) - open/close 45771 (time 1788656749.92 total 1183.51 last 79.46) - open/close 46533 (time 1788656759.92 total 1193.52 last 76.14) - open/close 47147 (time 1788656769.95 total 1203.55 last 61.23) - open/close 47801 (time 1788656779.95 total 1213.55 last 65.39) - open/close 48580 (time 1788656789.96 total 1223.55 last 77.89) - open/close 49437 (time 1788656799.96 total 1233.56 last 85.66) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1235.72 seconds: 40.16 ops/second - unlinked 0 (time 1788656806 ; total 0 ; last 0) - unlinked 10000 (time 1788656922 ; total 116 ; last 116) - unlinked 20000 (time 1788657040 ; total 234 ; last 118) - unlinked 30000 (time 1788657108 ; total 302 ; last 68) - unlinked 40000 (time 1788657175 ; total 369 ; last 67) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 430 seconds: 115.420929 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg304-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg304-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg304-server stop mds service on oleg304-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg304-server stop mds service on oleg304-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg304-server unloading modules via unload_modules_local on: 'oleg304-server' oleg304-server: oleg304-server.virtnet: executing unload_modules_local oleg304-server: kmemleak disabled modules unloaded. pdsh@oleg304-client: oleg304-server: ssh exited with exit code 2 pdsh@oleg304-client: oleg304-client: ssh exited with exit code 2 pdsh@oleg304-client: oleg304-client: ssh exited with exit code 2 pdsh@oleg304-client: oleg304-server: ssh exited with exit code 2