== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:12:48 (1788444768) start mds service on oleg447-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing load_modules_local oleg447-server: kmemleak disabled oleg447-server: Loading modules from /home/green/git/lustre-release/lustre oleg447-server: detected 4 online CPUs by sysfs oleg447-server: MODOPTS_LIBCFS= oleg447-server: Force libcfs to create 2 CPU partitions oleg447-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg447-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg447-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg447-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg447-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg447-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 145 (time 1788444884.07 total 10.06 last 14.41) - open/close 337 (time 1788444894.81 total 20.79 last 17.89) - open/close 487 (time 1788444905.81 total 31.80 last 13.63) - open/close 610 (time 1788444916.64 total 42.63 last 11.36) - open/close 805 (time 1788444927.20 total 53.19 last 18.46) - open/close 964 (time 1788444937.41 total 63.40 last 15.57) - open/close 1137 (time 1788444949.00 total 74.99 last 14.93) - open/close 1329 (time 1788444960.78 total 86.77 last 16.30) - open/close 1477 (time 1788444971.40 total 97.39 last 13.94) - open/close 1603 (time 1788444981.40 total 107.39 last 12.60) - open/close 1820 (time 1788444991.49 total 117.48 last 21.50) - open/close 1981 (time 1788445002.05 total 128.04 last 15.25) - open/close 2186 (time 1788445012.08 total 138.06 last 20.44) - open/close 2345 (time 1788445022.85 total 148.84 last 14.75) - open/close 2568 (time 1788445032.97 total 158.96 last 22.04) - open/close 2779 (time 1788445043.74 total 169.73 last 19.59) - open/close 2992 (time 1788445053.75 total 179.74 last 21.29) - open/close 3239 (time 1788445064.20 total 190.19 last 23.63) - open/close 3355 (time 1788445074.96 total 200.95 last 10.78) - open/close 3571 (time 1788445085.45 total 211.44 last 20.59) - open/close 3735 (time 1788445096.64 total 222.62 last 14.66) - open/close 3951 (time 1788445106.64 total 232.62 last 21.60) - open/close 4080 (time 1788445118.21 total 244.20 last 11.14) - open/close 4240 (time 1788445128.24 total 254.23 last 15.96) - open/close 4461 (time 1788445138.27 total 264.26 last 22.03) - open/close 4666 (time 1788445148.27 total 274.26 last 20.49) - open/close 4949 (time 1788445158.29 total 284.28 last 28.25) - open/close 5170 (time 1788445168.29 total 294.28 last 22.10) - open/close 5367 (time 1788445178.73 total 304.71 last 18.88) - open/close 5551 (time 1788445189.01 total 314.99 last 17.90) - open/close 5758 (time 1788445199.01 total 325.00 last 20.69) - open/close 6009 (time 1788445209.01 total 335.00 last 25.09) - open/close 6183 (time 1788445219.49 total 345.48 last 16.60) - open/close 6284 (time 1788445229.79 total 355.78 last 9.81) - open/close 6419 (time 1788445239.79 total 365.78 last 13.49) - open/close 6586 (time 1788445253.13 total 379.12 last 12.52) - open/close 6868 (time 1788445263.14 total 389.13 last 28.17) - open/close 7138 (time 1788445273.68 total 399.67 last 25.62) - open/close 7431 (time 1788445283.69 total 409.67 last 29.27) - open/close 7689 (time 1788445294.53 total 420.52 last 23.78) - open/close 7868 (time 1788445304.91 total 430.90 last 17.25) - open/close 8152 (time 1788445315.93 total 441.92 last 25.77) - open/close 8331 (time 1788445325.94 total 451.92 last 17.89) - open/close 8496 (time 1788445336.60 total 462.59 last 15.47) - open/close 8605 (time 1788445348.29 total 474.28 last 9.33) - open/close 8830 (time 1788445358.30 total 484.28 last 22.49) - open/close 9042 (time 1788445369.30 total 495.28 last 19.28) - open/close 9243 (time 1788445381.53 total 507.52 last 16.43) - open/close 9373 (time 1788445392.00 total 517.99 last 12.41) - open/close 9571 (time 1788445402.44 total 528.43 last 18.97) - open/close 9796 (time 1788445412.46 total 538.44 last 22.46) - open/close 10000 (time 1788445420.05 total 546.04 last 26.87) - open/close 10173 (time 1788445432.33 total 558.32 last 14.08) - open/close 10430 (time 1788445442.57 total 568.56 last 25.09) - open/close 10545 (time 1788445453.39 total 579.38 last 10.63) - open/close 10628 (time 1788445463.97 total 589.96 last 7.84) - open/close 10806 (time 1788445473.99 total 599.98 last 17.77) - open/close 11052 (time 1788445484.10 total 610.09 last 24.34) - open/close 11275 (time 1788445494.12 total 620.11 last 22.25) - open/close 11534 (time 1788445504.78 total 630.77 last 24.29) - open/close 11822 (time 1788445514.80 total 640.79 last 28.74) - open/close 12028 (time 1788445525.39 total 651.38 last 19.47) - open/close 12251 (time 1788445535.70 total 661.69 last 21.63) - open/close 12477 (time 1788445546.23 total 672.22 last 21.46) - open/close 12664 (time 1788445557.26 total 683.25 last 16.95) - open/close 12941 (time 1788445567.38 total 693.37 last 27.38) - open/close 13148 (time 1788445577.39 total 703.38 last 20.67) - open/close 13427 (time 1788445587.40 total 713.38 last 27.90) - open/close 13706 (time 1788445598.21 total 724.20 last 25.79) - open/close 13918 (time 1788445608.23 total 734.21 last 21.17) - open/close 14178 (time 1788445618.23 total 744.22 last 25.99) - open/close 14278 (time 1788445629.00 total 754.99 last 9.29) - open/close 14355 (time 1788445639.12 total 765.11 last 7.61) - open/close 14552 (time 1788445649.85 total 775.84 last 18.36) - open/close 14845 (time 1788445660.44 total 786.43 last 27.67) - open/close 14960 (time 1788445671.76 total 797.75 last 10.16) - open/close 15165 (time 1788445681.77 total 807.76 last 20.49) - open/close 15298 (time 1788445691.78 total 817.77 last 13.29) - open/close 15436 (time 1788445707.77 total 833.76 last 8.63) - open/close 15634 (time 1788445718.66 total 844.65 last 18.19) - open/close 15913 (time 1788445729.28 total 855.27 last 26.27) - open/close 16056 (time 1788445739.29 total 865.28 last 14.29) - open/close 16183 (time 1788445749.29 total 875.28 last 12.70) - open/close 16400 (time 1788445761.56 total 887.55 last 17.69) - open/close 16534 (time 1788445772.46 total 898.45 last 12.29) - open/close 16771 (time 1788445783.05 total 909.03 last 22.38) - open/close 17038 (time 1788445793.07 total 919.05 last 26.65) - open/close 17221 (time 1788445803.67 total 929.66 last 17.26) - open/close 17438 (time 1788445816.06 total 942.05 last 17.51) - open/close 17642 (time 1788445826.78 total 952.77 last 19.04) - open/close 17827 (time 1788445837.49 total 963.47 last 17.28) - open/close 17922 (time 1788445847.65 total 973.64 last 9.34) - open/close 18128 (time 1788445857.66 total 983.65 last 20.58) - open/close 18300 (time 1788445867.77 total 993.76 last 17.01) - open/close 18589 (time 1788445877.78 total 1003.77 last 28.89) - open/close 18868 (time 1788445888.27 total 1014.26 last 26.58) - open/close 19160 (time 1788445898.28 total 1024.27 last 29.18) - open/close 20000 (time 1788445908.24 total 1034.23 last 84.35) - open/close 20917 (time 1788445918.24 total 1044.23 last 91.67) - open/close 21730 (time 1788445928.25 total 1054.23 last 81.28) - open/close 22395 (time 1788445938.25 total 1064.23 last 66.49) - open/close 23024 (time 1788445948.25 total 1074.24 last 62.89) - open/close 23665 (time 1788445958.26 total 1084.24 last 64.05) - open/close 24400 (time 1788445968.27 total 1094.25 last 73.42) - open/close 25259 (time 1788445978.27 total 1104.26 last 85.87) - open/close 26165 (time 1788445988.28 total 1114.27 last 90.49) - open/close 27027 (time 1788445998.29 total 1124.28 last 86.16) - open/close 27817 (time 1788446008.30 total 1134.29 last 78.90) - open/close 28619 (time 1788446018.31 total 1144.30 last 80.14) - open/close 29450 (time 1788446028.32 total 1154.31 last 83.02) - open/close 30000 (time 1788446034.94 total 1160.93 last 83.00) - open/close 30809 (time 1788446044.95 total 1170.93 last 80.89) - open/close 31491 (time 1788446054.96 total 1180.95 last 68.10) - open/close 31932 (time 1788446064.98 total 1190.97 last 44.02) - open/close 32608 (time 1788446074.98 total 1200.97 last 67.58) - open/close 33375 (time 1788446084.99 total 1210.98 last 76.63) - open/close 34160 (time 1788446094.99 total 1220.98 last 78.50) - open/close 35039 (time 1788446104.99 total 1230.98 last 87.87) - open/close 35852 (time 1788446115.00 total 1240.99 last 81.25) - open/close 36585 (time 1788446125.00 total 1250.99 last 73.29) - open/close 37275 (time 1788446135.00 total 1260.99 last 68.98) - open/close 38012 (time 1788446145.01 total 1271.00 last 73.63) - open/close 38780 (time 1788446155.02 total 1281.01 last 76.77) - open/close 39670 (time 1788446165.02 total 1291.01 last 88.99) - open/close 40000 (time 1788446168.50 total 1294.49 last 94.67) - open/close 40758 (time 1788446178.51 total 1304.49 last 75.78) - open/close 41427 (time 1788446188.52 total 1314.51 last 66.83) - open/close 42201 (time 1788446198.54 total 1324.52 last 77.27) - open/close 42999 (time 1788446208.55 total 1334.53 last 79.71) - open/close 43764 (time 1788446218.56 total 1344.55 last 76.40) - open/close 44513 (time 1788446228.58 total 1354.57 last 74.75) - open/close 45269 (time 1788446238.58 total 1364.57 last 75.57) - open/close 46005 (time 1788446248.58 total 1374.57 last 73.59) - open/close 46825 (time 1788446258.61 total 1384.60 last 81.78) - open/close 47605 (time 1788446268.62 total 1394.61 last 77.95) - open/close 48419 (time 1788446278.63 total 1404.62 last 81.30) - open/close 49147 (time 1788446288.65 total 1414.64 last 72.62) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1421.19 seconds: 34.92 ops/second - unlinked 0 (time 1788446300 ; total 0 ; last 0) - unlinked 10000 (time 1788446422 ; total 122 ; last 122) - unlinked 20000 (time 1788446542 ; total 242 ; last 120) - unlinked 30000 (time 1788446665 ; total 365 ; last 123) - unlinked 40000 (time 1788446787 ; total 487 ; last 122) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 601 seconds: 82.580696 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg447-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg447-server unloading modules via unload_modules_local on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing unload_modules_local oleg447-server: kmemleak disabled modules unloaded. pdsh@oleg447-client: oleg447-client: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-client: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2