== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:58:33 (1788749913) start mds service on oleg610-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 6 20:24 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg610-server' oleg610-server: oleg610-server.virtnet: executing load_modules_local oleg610-server: kmemleak disabled oleg610-server: Loading modules from /home/green/git/lustre-release/lustre oleg610-server: detected 4 online CPUs by sysfs oleg610-server: MODOPTS_LIBCFS= oleg610-server: Force libcfs to create 2 CPU partitions oleg610-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg610-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg610-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg610-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg610-server: oleg610-server.virtnet: executing set_default_debug -1 all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg610-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg610-server: oleg610-server.virtnet: executing set_default_debug -1 all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg610-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg610-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg610-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg610-server: oleg610-server.virtnet: executing set_default_debug -1 all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-OST0000 oleg610-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 196 (time 1788749969.11 total 10.21 last 19.20) - open/close 369 (time 1788749979.16 total 20.25 last 17.22) - open/close 656 (time 1788749990.11 total 31.20 last 26.22) - open/close 854 (time 1788750000.47 total 41.57 last 19.10) - open/close 997 (time 1788750011.48 total 52.57 last 13.00) - open/close 1209 (time 1788750021.73 total 62.82 last 20.69) - open/close 1454 (time 1788750032.43 total 73.52 last 22.90) - open/close 1678 (time 1788750043.13 total 84.23 last 20.92) - open/close 1856 (time 1788750055.28 total 96.37 last 14.65) - open/close 1937 (time 1788750066.35 total 107.44 last 7.32) - open/close 2162 (time 1788750076.35 total 117.44 last 22.49) - open/close 2340 (time 1788750086.69 total 127.78 last 17.21) - open/close 2584 (time 1788750097.15 total 138.25 last 23.32) - open/close 2725 (time 1788750107.16 total 148.25 last 14.10) - open/close 2911 (time 1788750117.68 total 158.77 last 17.67) - open/close 3133 (time 1788750128.67 total 169.77 last 20.19) - open/close 3379 (time 1788750138.91 total 180.00 last 24.03) - open/close 3544 (time 1788750149.02 total 190.11 last 16.32) - open/close 3695 (time 1788750159.08 total 200.17 last 15.02) - open/close 3873 (time 1788750169.18 total 210.27 last 17.61) - open/close 4067 (time 1788750179.82 total 220.92 last 18.23) - open/close 4289 (time 1788750190.82 total 231.91 last 20.19) - open/close 4588 (time 1788750200.82 total 241.91 last 29.90) - open/close 4863 (time 1788750210.84 total 251.94 last 27.43) - open/close 5074 (time 1788750221.80 total 262.90 last 19.26) - open/close 5288 (time 1788750232.81 total 273.90 last 19.44) - open/close 5516 (time 1788750243.81 total 284.90 last 20.72) - open/close 5751 (time 1788750253.83 total 294.92 last 23.46) - open/close 5984 (time 1788750264.09 total 305.19 last 22.70) - open/close 6261 (time 1788750274.11 total 315.20 last 27.66) - open/close 6552 (time 1788750284.46 total 325.55 last 28.12) - open/close 6857 (time 1788750294.47 total 335.56 last 30.47) - open/close 7126 (time 1788750305.24 total 346.34 last 24.96) - open/close 7406 (time 1788750315.69 total 356.79 last 26.80) - open/close 7671 (time 1788750326.44 total 367.53 last 24.66) - open/close 8016 (time 1788750337.44 total 378.54 last 31.35) - open/close 8305 (time 1788750348.44 total 389.54 last 26.27) - open/close 8549 (time 1788750359.56 total 400.65 last 21.96) - open/close 8823 (time 1788750369.57 total 410.66 last 27.37) - open/close 9054 (time 1788750380.13 total 421.22 last 21.87) - open/close 9355 (time 1788750390.81 total 431.91 last 28.17) - open/close 9615 (time 1788750400.82 total 441.91 last 25.99) - open/close 9894 (time 1788750410.82 total 451.91 last 27.89) - open/close 10000 (time 1788750414.26 total 455.35 last 30.86) - open/close 10294 (time 1788750424.27 total 465.37 last 29.35) - open/close 10576 (time 1788750434.55 total 475.64 last 27.45) - open/close 10811 (time 1788750445.21 total 486.31 last 22.03) - open/close 11096 (time 1788750456.04 total 497.13 last 26.32) - open/close 11376 (time 1788750466.24 total 507.33 last 27.46) - open/close 11649 (time 1788750476.46 total 517.56 last 26.70) - open/close 11901 (time 1788750486.70 total 527.79 last 24.62) - open/close 12243 (time 1788750497.69 total 538.79 last 31.11) - open/close 12515 (time 1788750508.03 total 549.12 last 26.32) - open/close 12799 (time 1788750520.16 total 561.25 last 23.41) - open/close 13039 (time 1788750530.36 total 571.46 last 23.52) - open/close 13258 (time 1788750540.38 total 581.48 last 21.85) - open/close 13449 (time 1788750550.95 total 592.04 last 18.08) - open/close 13692 (time 1788750561.21 total 602.30 last 23.69) - open/close 13944 (time 1788750571.94 total 613.03 last 23.48) - open/close 14229 (time 1788750581.94 total 623.04 last 28.49) - open/close 14497 (time 1788750592.63 total 633.73 last 25.06) - open/close 14849 (time 1788750603.36 total 644.45 last 32.82) - open/close 15115 (time 1788750614.11 total 655.21 last 24.74) - open/close 15357 (time 1788750624.13 total 665.22 last 24.17) - open/close 15518 (time 1788750636.31 total 677.40 last 13.22) - open/close 15704 (time 1788750646.57 total 687.66 last 18.12) - open/close 15979 (time 1788750656.58 total 697.67 last 27.49) - open/close 16194 (time 1788750666.60 total 707.69 last 21.46) - open/close 16437 (time 1788750677.34 total 718.43 last 22.62) - open/close 16668 (time 1788750689.95 total 731.04 last 18.32) - open/close 16996 (time 1788750699.97 total 741.06 last 32.74) - open/close 17393 (time 1788750709.97 total 751.06 last 39.68) - open/close 18327 (time 1788750719.97 total 761.07 last 93.38) - open/close 19313 (time 1788750729.98 total 771.07 last 98.55) - open/close 20000 (time 1788750737.77 total 778.86 last 88.17) - open/close 20880 (time 1788750747.78 total 788.87 last 87.96) - open/close 21766 (time 1788750757.78 total 798.88 last 88.53) - open/close 22602 (time 1788750767.80 total 808.89 last 83.45) - open/close 23447 (time 1788750777.80 total 818.89 last 84.50) - open/close 24432 (time 1788750787.80 total 828.90 last 98.49) - open/close 25364 (time 1788750797.81 total 838.90 last 93.13) - open/close 26303 (time 1788750807.81 total 848.91 last 93.87) - open/close 27258 (time 1788750817.82 total 858.91 last 95.47) - open/close 28103 (time 1788750827.82 total 868.92 last 84.44) - open/close 28983 (time 1788750837.83 total 878.93 last 87.91) - open/close 29891 (time 1788750847.84 total 888.93 last 90.74) - open/close 30898 (time 1788750857.84 total 898.94 last 100.67) - open/close 31878 (time 1788750867.85 total 908.94 last 97.96) - open/close 32857 (time 1788750877.85 total 918.94 last 97.87) - open/close 33825 (time 1788750887.85 total 928.95 last 96.77) - open/close 34781 (time 1788750897.86 total 938.95 last 95.58) - open/close 35762 (time 1788750907.86 total 948.95 last 98.08) - open/close 36699 (time 1788750917.86 total 958.96 last 93.66) - open/close 37729 (time 1788750927.87 total 968.96 last 102.94) - open/close 38669 (time 1788750937.87 total 978.96 last 93.98) - open/close 39733 (time 1788750947.87 total 988.97 last 106.38) - open/close 40000 (time 1788750950.32 total 991.41 last 109.06) - open/close 40945 (time 1788750960.34 total 1001.43 last 94.37) - open/close 41921 (time 1788750970.34 total 1011.44 last 97.54) - open/close 42885 (time 1788750980.35 total 1021.44 last 96.36) - open/close 43842 (time 1788750990.35 total 1031.44 last 95.69) - open/close 44899 (time 1788751000.35 total 1041.44 last 105.68) - open/close 45861 (time 1788751010.36 total 1051.45 last 96.09) - open/close 46830 (time 1788751020.37 total 1061.46 last 96.84) - open/close 47814 (time 1788751030.37 total 1071.47 last 98.34) - open/close 48785 (time 1788751040.38 total 1081.47 last 97.07) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1090.15 seconds: 45.53 ops/second - unlinked 0 (time 1788751052 ; total 0 ; last 0) - unlinked 10000 (time 1788751155 ; total 103 ; last 103) - unlinked 20000 (time 1788751231 ; total 179 ; last 76) - unlinked 30000 (time 1788751288 ; total 236 ; last 57) - unlinked 40000 (time 1788751344 ; total 292 ; last 56) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 342 seconds: 145.119888 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg610-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg610-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg610-server stop mds service on oleg610-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg610-server stop mds service on oleg610-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg610-server unloading modules via unload_modules_local on: 'oleg610-server' oleg610-server: oleg610-server.virtnet: executing unload_modules_local oleg610-server: kmemleak disabled modules unloaded. pdsh@oleg610-client: oleg610-client: ssh exited with exit code 2 pdsh@oleg610-client: oleg610-server: ssh exited with exit code 2 pdsh@oleg610-client: oleg610-client: ssh exited with exit code 2 pdsh@oleg610-client: oleg610-server: ssh exited with exit code 2