== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:23:29 (1789442609) start mds service on oleg304-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 14 12:06 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg304-server' oleg304-server: oleg304-server.virtnet: executing load_modules_local oleg304-server: kmemleak disabled oleg304-server: Loading modules from /home/green/git/lustre-release/lustre oleg304-server: detected 4 online CPUs by sysfs oleg304-server: MODOPTS_LIBCFS= oleg304-server: Force libcfs to create 2 CPU partitions oleg304-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg304-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg304-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg304-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg304-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg304-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-OST0000 oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 84 (time 1789442723.79 total 10.02 last 8.38) - open/close 277 (time 1789442734.67 total 20.90 last 17.74) - open/close 570 (time 1789442744.96 total 31.20 last 28.47) - open/close 801 (time 1789442755.20 total 41.43 last 22.56) - open/close 957 (time 1789442765.21 total 51.44 last 15.59) - open/close 1205 (time 1789442775.94 total 62.17 last 23.11) - open/close 1460 (time 1789442786.04 total 72.28 last 25.24) - open/close 1614 (time 1789442796.37 total 82.60 last 14.92) - open/close 1823 (time 1789442807.05 total 93.28 last 19.57) - open/close 1901 (time 1789442817.06 total 103.30 last 7.79) - open/close 2141 (time 1789442828.43 total 114.66 last 21.12) - open/close 2314 (time 1789442839.44 total 125.67 last 15.71) - open/close 2528 (time 1789442850.18 total 136.42 last 19.91) - open/close 2658 (time 1789442861.07 total 147.30 last 11.95) - open/close 2873 (time 1789442872.08 total 158.31 last 19.53) - open/close 3098 (time 1789442882.32 total 168.55 last 21.97) - open/close 3234 (time 1789442892.32 total 178.56 last 13.59) - open/close 3395 (time 1789442902.67 total 188.90 last 15.57) - open/close 3643 (time 1789442913.16 total 199.39 last 23.64) - open/close 3916 (time 1789442924.24 total 210.48 last 24.62) - open/close 4176 (time 1789442934.26 total 220.49 last 25.97) - open/close 4418 (time 1789442944.27 total 230.51 last 24.16) - open/close 4681 (time 1789442955.02 total 241.25 last 24.47) - open/close 4866 (time 1789442966.84 total 253.08 last 15.64) - open/close 5068 (time 1789442977.16 total 263.39 last 19.58) - open/close 5254 (time 1789442987.19 total 273.42 last 18.55) - open/close 5404 (time 1789442997.64 total 283.87 last 14.35) - open/close 5557 (time 1789443010.05 total 296.28 last 12.33) - open/close 5685 (time 1789443020.10 total 306.33 last 12.74) - open/close 5966 (time 1789443030.22 total 316.45 last 27.76) - open/close 6229 (time 1789443040.32 total 326.56 last 26.03) - open/close 6470 (time 1789443051.09 total 337.32 last 22.38) - open/close 6630 (time 1789443061.09 total 347.33 last 15.99) - open/close 6747 (time 1789443071.81 total 358.04 last 10.92) - open/close 6881 (time 1789443081.93 total 368.16 last 13.24) - open/close 7049 (time 1789443092.64 total 378.87 last 15.69) - open/close 7153 (time 1789443102.99 total 389.23 last 10.04) - open/close 7271 (time 1789443113.94 total 400.17 last 10.78) - open/close 7372 (time 1789443123.96 total 410.19 last 10.08) - open/close 7447 (time 1789443133.96 total 420.19 last 7.50) - open/close 7644 (time 1789443143.96 total 430.19 last 19.70) - open/close 7749 (time 1789443154.34 total 440.58 last 10.11) - open/close 8013 (time 1789443164.36 total 450.60 last 26.35) - open/close 8164 (time 1789443174.79 total 461.02 last 14.49) - open/close 8340 (time 1789443185.11 total 471.34 last 17.05) - open/close 8602 (time 1789443195.12 total 481.35 last 26.18) - open/close 8706 (time 1789443205.46 total 491.70 last 10.06) - open/close 8816 (time 1789443216.12 total 502.36 last 10.32) - open/close 9052 (time 1789443226.13 total 512.36 last 23.59) - open/close 9267 (time 1789443236.54 total 522.78 last 20.65) - open/close 9547 (time 1789443246.54 total 532.78 last 28.00) - open/close 9727 (time 1789443256.64 total 542.87 last 17.84) - open/close 9992 (time 1789443267.34 total 553.58 last 24.75) - open/close 10241 (time 1789443278.35 total 564.58 last 22.63) - open/close 10347 (time 1789443288.36 total 574.60 last 10.58) - open/close 10618 (time 1789443299.00 total 585.24 last 25.46) - open/close 10866 (time 1789443309.41 total 595.64 last 23.83) - open/close 11086 (time 1789443319.42 total 605.65 last 21.98) - open/close 11298 (time 1789443329.42 total 615.66 last 21.19) - open/close 11534 (time 1789443339.44 total 625.67 last 23.56) - open/close 11671 (time 1789443350.46 total 636.70 last 12.43) - open/close 11872 (time 1789443360.92 total 647.15 last 19.23) - open/close 11905 (time 1789443370.97 total 657.20 last 3.28) - open/close 12146 (time 1789443380.97 total 667.21 last 24.09) - open/close 12273 (time 1789443390.98 total 677.21 last 12.69) - open/close 12486 (time 1789443401.51 total 687.75 last 20.22) - open/close 12760 (time 1789443411.93 total 698.16 last 26.31) - open/close 12935 (time 1789443422.13 total 708.37 last 17.15) - open/close 13027 (time 1789443433.04 total 719.27 last 8.44) - open/close 13143 (time 1789443443.04 total 729.27 last 11.60) - open/close 13338 (time 1789443453.21 total 739.45 last 19.17) - open/close 13493 (time 1789443463.49 total 749.73 last 15.08) - open/close 13701 (time 1789443474.08 total 760.32 last 19.64) - open/close 13814 (time 1789443484.08 total 770.32 last 11.30) - open/close 13941 (time 1789443495.31 total 781.55 last 11.31) - open/close 14111 (time 1789443506.13 total 792.36 last 15.72) - open/close 14332 (time 1789443517.71 total 803.94 last 19.09) - open/close 14539 (time 1789443527.72 total 813.96 last 20.67) - open/close 14798 (time 1789443537.73 total 823.96 last 25.89) - open/close 15014 (time 1789443547.73 total 833.96 last 21.60) - open/close 15256 (time 1789443558.35 total 844.58 last 22.79) - open/close 15490 (time 1789443568.58 total 854.81 last 22.87) - open/close 15784 (time 1789443580.14 total 866.37 last 25.43) - open/close 15978 (time 1789443590.15 total 876.38 last 19.38) - open/close 16262 (time 1789443600.15 total 886.38 last 28.40) - open/close 16510 (time 1789443610.15 total 896.38 last 24.80) - open/close 16757 (time 1789443620.16 total 906.39 last 24.67) - open/close 16985 (time 1789443630.46 total 916.70 last 22.13) - open/close 17222 (time 1789443640.91 total 927.14 last 22.69) - open/close 17419 (time 1789443651.01 total 937.24 last 19.51) - open/close 17693 (time 1789443661.02 total 947.25 last 27.37) - open/close 17909 (time 1789443671.03 total 957.27 last 21.56) - open/close 18109 (time 1789443681.04 total 967.28 last 19.99) - open/close 18250 (time 1789443691.06 total 977.29 last 14.08) - open/close 18410 (time 1789443701.96 total 988.19 last 14.68) - open/close 18438 (time 1789443711.97 total 998.21 last 2.80) - open/close 18723 (time 1789443721.99 total 1008.22 last 28.46) - open/close 19472 (time 1789443732.01 total 1018.25 last 74.72) - open/close 20000 (time 1789443739.02 total 1025.25 last 75.36) - open/close 20839 (time 1789443749.02 total 1035.25 last 83.88) - open/close 21625 (time 1789443759.03 total 1045.26 last 78.52) - open/close 22312 (time 1789443769.04 total 1055.27 last 68.64) - open/close 22935 (time 1789443779.07 total 1065.30 last 62.13) - open/close 23672 (time 1789443789.08 total 1075.31 last 73.65) - open/close 24474 (time 1789443799.09 total 1085.32 last 80.11) - open/close 25238 (time 1789443809.09 total 1095.33 last 76.36) - open/close 26035 (time 1789443819.10 total 1105.34 last 79.61) - open/close 26825 (time 1789443829.11 total 1115.34 last 78.96) - open/close 27543 (time 1789443839.11 total 1125.35 last 71.77) - open/close 28351 (time 1789443849.12 total 1135.35 last 80.76) - open/close 29071 (time 1789443859.12 total 1145.36 last 71.96) - open/close 29861 (time 1789443869.13 total 1155.36 last 78.94) - open/close 30662 (time 1789443879.14 total 1165.37 last 80.05) - open/close 31472 (time 1789443889.14 total 1175.37 last 80.98) - open/close 32260 (time 1789443899.14 total 1185.38 last 78.76) - open/close 32965 (time 1789443909.16 total 1195.39 last 70.41) - open/close 33684 (time 1789443919.16 total 1205.39 last 71.90) - open/close 34504 (time 1789443929.16 total 1215.39 last 81.97) - open/close 35300 (time 1789443939.17 total 1225.40 last 79.57) - open/close 36135 (time 1789443949.18 total 1235.42 last 83.35) - open/close 36847 (time 1789443959.18 total 1245.42 last 71.20) - open/close 37600 (time 1789443969.19 total 1255.43 last 75.24) - open/close 38361 (time 1789443979.23 total 1265.46 last 75.84) - open/close 39136 (time 1789443989.23 total 1275.46 last 77.49) - open/close 39849 (time 1789443999.25 total 1285.49 last 71.12) - open/close 40000 (time 1789444001.81 total 1288.04 last 59.17) - open/close 40809 (time 1789444011.82 total 1298.05 last 80.80) - open/close 41486 (time 1789444021.83 total 1308.06 last 67.64) - open/close 42218 (time 1789444031.83 total 1318.06 last 73.18) - open/close 42951 (time 1789444041.84 total 1328.08 last 73.21) - open/close 43736 (time 1789444051.84 total 1338.08 last 78.48) - open/close 44522 (time 1789444061.86 total 1348.10 last 78.47) - open/close 45342 (time 1789444071.87 total 1358.11 last 81.90) - open/close 46093 (time 1789444081.88 total 1368.12 last 75.02) - open/close 46864 (time 1789444091.89 total 1378.12 last 77.06) - open/close 47580 (time 1789444101.90 total 1388.13 last 71.53) - open/close 48345 (time 1789444111.92 total 1398.15 last 76.38) - open/close 49073 (time 1789444121.93 total 1408.16 last 72.71) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1416.03 seconds: 35.05 ops/second - unlinked 0 (time 1789444134 ; total 0 ; last 0) - unlinked 10000 (time 1789444272 ; total 138 ; last 138) - unlinked 20000 (time 1789444400 ; total 266 ; last 128) - unlinked 30000 (time 1789444536 ; total 402 ; last 136) - unlinked 40000 (time 1789444667 ; total 533 ; last 131) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 669 seconds: 74.186844 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg304-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg304-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg304-server stop mds service on oleg304-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg304-server stop mds service on oleg304-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg304-server unloading modules via unload_modules_local on: 'oleg304-server' oleg304-server: oleg304-server.virtnet: executing unload_modules_local oleg304-server: kmemleak disabled modules unloaded. pdsh@oleg304-client: oleg304-client: ssh exited with exit code 2 pdsh@oleg304-client: oleg304-server: ssh exited with exit code 2 pdsh@oleg304-client: oleg304-server: ssh exited with exit code 2 pdsh@oleg304-client: oleg304-client: ssh exited with exit code 2