== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:10:42 (1788473442) start mds service on oleg635-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921072 Sep 3 15:45 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg635-server' oleg635-server: oleg635-server.virtnet: executing load_modules_local oleg635-server: kmemleak disabled oleg635-server: Loading modules from /home/green/git/lustre-release/lustre oleg635-server: detected 4 online CPUs by sysfs oleg635-server: MODOPTS_LIBCFS= oleg635-server: Force libcfs to create 2 CPU partitions oleg635-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg635-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg635-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg635-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg635-server: oleg635-server.virtnet: executing set_default_debug -1 all pdsh@oleg635-client: oleg635-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg635-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg635-server: oleg635-server.virtnet: executing set_default_debug -1 all pdsh@oleg635-client: oleg635-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg635-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg635-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg635-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg635-server: oleg635-server.virtnet: executing set_default_debug -1 all pdsh@oleg635-client: oleg635-server: ssh exited with exit code 1 Started lustre-OST0000 oleg635-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg635-client.virtnet: -o user_xattr,flock 192.168.206.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 179 (time 1788473555.29 total 10.58 last 16.91) - open/close 347 (time 1788473566.01 total 21.30 last 15.67) - open/close 618 (time 1788473576.58 total 31.87 last 25.64) - open/close 806 (time 1788473587.08 total 42.37 last 17.91) - open/close 1000 (time 1788473597.58 total 52.87 last 18.47) - open/close 1236 (time 1788473608.16 total 63.45 last 22.31) - open/close 1435 (time 1788473618.16 total 73.46 last 19.89) - open/close 1532 (time 1788473628.28 total 83.58 last 9.58) - open/close 1739 (time 1788473638.30 total 93.59 last 20.68) - open/close 1991 (time 1788473648.30 total 103.59 last 25.18) - open/close 2187 (time 1788473658.31 total 113.60 last 19.59) - open/close 2267 (time 1788473668.42 total 123.71 last 7.91) - open/close 2396 (time 1788473679.04 total 134.33 last 12.14) - open/close 2567 (time 1788473689.47 total 144.77 last 16.39) - open/close 2835 (time 1788473700.55 total 155.84 last 24.20) - open/close 3088 (time 1788473711.37 total 166.66 last 23.39) - open/close 3238 (time 1788473721.37 total 176.66 last 14.99) - open/close 3485 (time 1788473732.95 total 188.24 last 21.33) - open/close 3551 (time 1788473743.25 total 198.54 last 6.41) - open/close 3719 (time 1788473753.74 total 209.03 last 16.02) - open/close 3866 (time 1788473764.06 total 219.35 last 14.25) - open/close 4065 (time 1788473774.06 total 229.36 last 19.89) - open/close 4138 (time 1788473784.07 total 239.36 last 7.29) - open/close 4377 (time 1788473794.12 total 249.42 last 23.77) - open/close 4561 (time 1788473805.01 total 260.30 last 16.91) - open/close 4654 (time 1788473815.19 total 270.49 last 9.13) - open/close 4793 (time 1788473825.21 total 280.50 last 13.88) - open/close 4989 (time 1788473835.21 total 290.50 last 19.59) - open/close 5131 (time 1788473845.23 total 300.52 last 14.18) - open/close 5267 (time 1788473856.26 total 311.56 last 12.32) - open/close 5579 (time 1788473866.88 total 322.17 last 29.39) - open/close 5739 (time 1788473877.13 total 332.42 last 15.62) - open/close 5996 (time 1788473889.80 total 345.09 last 20.28) - open/close 6157 (time 1788473900.80 total 356.09 last 14.64) - open/close 6290 (time 1788473910.94 total 366.24 last 13.11) - open/close 6507 (time 1788473920.96 total 376.26 last 21.66) - open/close 6659 (time 1788473931.13 total 386.42 last 14.95) - open/close 6972 (time 1788473941.51 total 396.80 last 30.17) - open/close 7183 (time 1788473952.38 total 407.67 last 19.41) - open/close 7380 (time 1788473962.62 total 417.92 last 19.23) - open/close 7565 (time 1788473973.39 total 428.68 last 17.18) - open/close 7707 (time 1788473983.40 total 438.70 last 14.18) - open/close 7820 (time 1788473993.43 total 448.72 last 11.27) - open/close 8046 (time 1788474003.71 total 459.01 last 21.98) - open/close 8243 (time 1788474013.97 total 469.26 last 19.21) - open/close 8430 (time 1788474024.77 total 480.07 last 17.31) - open/close 8612 (time 1788474035.39 total 490.68 last 17.14) - open/close 8843 (time 1788474045.40 total 500.69 last 23.08) - open/close 9047 (time 1788474055.99 total 511.28 last 19.26) - open/close 9256 (time 1788474066.00 total 521.29 last 20.89) - open/close 9379 (time 1788474076.99 total 532.29 last 11.19) - open/close 9601 (time 1788474087.19 total 542.48 last 21.78) - open/close 9843 (time 1788474097.20 total 552.50 last 24.16) - open/close 9873 (time 1788474109.44 total 564.73 last 2.45) - open/close 10000 (time 1788474115.07 total 570.36 last 22.56) - open/close 10268 (time 1788474126.11 total 581.40 last 24.27) - open/close 10478 (time 1788474136.52 total 591.81 last 20.18) - open/close 10741 (time 1788474147.46 total 602.75 last 24.04) - open/close 10952 (time 1788474157.82 total 613.11 last 20.36) - open/close 11248 (time 1788474167.84 total 623.13 last 29.55) - open/close 11535 (time 1788474178.65 total 633.94 last 26.55) - open/close 11791 (time 1788474189.38 total 644.67 last 23.85) - open/close 11992 (time 1788474199.39 total 654.68 last 20.09) - open/close 12134 (time 1788474209.39 total 664.68 last 14.20) - open/close 12250 (time 1788474219.98 total 675.27 last 10.95) - open/close 12452 (time 1788474229.98 total 685.27 last 20.20) - open/close 12518 (time 1788474240.42 total 695.72 last 6.32) - open/close 12783 (time 1788474253.49 total 708.79 last 20.27) - open/close 12987 (time 1788474264.58 total 719.87 last 18.40) - open/close 13183 (time 1788474274.61 total 729.90 last 19.55) - open/close 13376 (time 1788474287.09 total 742.38 last 15.46) - open/close 13550 (time 1788474297.70 total 752.99 last 16.40) - open/close 13809 (time 1788474307.71 total 763.01 last 25.87) - open/close 13998 (time 1788474317.78 total 773.07 last 18.78) - open/close 14183 (time 1788474329.73 total 785.02 last 15.48) - open/close 14376 (time 1788474340.55 total 795.84 last 17.83) - open/close 14569 (time 1788474350.56 total 805.85 last 19.29) - open/close 14817 (time 1788474361.46 total 816.76 last 22.73) - open/close 15080 (time 1788474372.39 total 827.69 last 24.06) - open/close 15275 (time 1788474383.23 total 838.52 last 17.99) - open/close 15441 (time 1788474395.42 total 850.71 last 13.62) - open/close 15642 (time 1788474405.43 total 860.73 last 20.07) - open/close 15859 (time 1788474415.44 total 870.73 last 21.69) - open/close 16069 (time 1788474425.85 total 881.15 last 20.16) - open/close 16306 (time 1788474436.10 total 891.40 last 23.12) - open/close 16555 (time 1788474446.12 total 901.41 last 24.86) - open/close 16714 (time 1788474456.96 total 912.25 last 14.67) - open/close 16912 (time 1788474467.97 total 923.26 last 17.98) - open/close 17102 (time 1788474478.87 total 934.16 last 17.44) - open/close 17330 (time 1788474488.88 total 944.17 last 22.78) - open/close 17572 (time 1788474499.01 total 954.30 last 23.88) - open/close 17794 (time 1788474509.75 total 965.04 last 20.67) - open/close 18039 (time 1788474520.13 total 975.42 last 23.60) - open/close 18317 (time 1788474531.02 total 986.31 last 25.54) - open/close 18513 (time 1788474541.98 total 997.27 last 17.88) - open/close 18738 (time 1788474552.19 total 1007.48 last 22.04) - open/close 19058 (time 1788474562.19 total 1017.48 last 31.99) - open/close 19763 (time 1788474572.19 total 1027.48 last 70.50) - open/close 20000 (time 1788474575.03 total 1030.32 last 83.56) - open/close 20821 (time 1788474585.05 total 1040.34 last 81.91) - open/close 21520 (time 1788474595.05 total 1050.35 last 69.87) - open/close 22273 (time 1788474605.06 total 1060.35 last 75.27) - open/close 23036 (time 1788474615.06 total 1070.35 last 76.27) - open/close 23837 (time 1788474625.07 total 1080.36 last 80.04) - open/close 24531 (time 1788474635.07 total 1090.36 last 69.40) - open/close 25170 (time 1788474645.07 total 1100.37 last 63.87) - open/close 25913 (time 1788474655.09 total 1110.38 last 74.18) - open/close 26692 (time 1788474665.09 total 1120.38 last 77.89) - open/close 27435 (time 1788474675.10 total 1130.40 last 74.21) - open/close 28097 (time 1788474685.11 total 1140.40 last 66.16) - open/close 28771 (time 1788474695.11 total 1150.41 last 67.38) - open/close 29557 (time 1788474705.12 total 1160.41 last 78.55) - open/close 30000 (time 1788474711.32 total 1166.62 last 71.39) - open/close 30756 (time 1788474721.33 total 1176.62 last 75.54) - open/close 31560 (time 1788474731.35 total 1186.65 last 80.23) - open/close 32319 (time 1788474741.36 total 1196.65 last 75.84) - open/close 33039 (time 1788474751.37 total 1206.66 last 71.95) - open/close 33798 (time 1788474761.38 total 1216.67 last 75.85) - open/close 34582 (time 1788474771.39 total 1226.68 last 78.31) - open/close 35340 (time 1788474781.39 total 1236.68 last 75.78) - open/close 36096 (time 1788474791.39 total 1246.68 last 75.58) - open/close 36861 (time 1788474801.39 total 1256.69 last 76.49) - open/close 37647 (time 1788474811.40 total 1266.69 last 78.56) - open/close 38418 (time 1788474821.41 total 1276.71 last 76.99) - open/close 39205 (time 1788474831.42 total 1286.71 last 78.63) - open/close 40000 (time 1788474841.02 total 1296.31 last 82.85) - open/close 40824 (time 1788474851.02 total 1306.31 last 82.36) - open/close 41602 (time 1788474861.03 total 1316.33 last 77.71) - open/close 42381 (time 1788474871.03 total 1326.33 last 77.90) - open/close 43102 (time 1788474881.07 total 1336.36 last 71.86) - open/close 43841 (time 1788474891.07 total 1346.36 last 73.87) - open/close 44635 (time 1788474901.08 total 1356.37 last 79.32) - open/close 45403 (time 1788474911.08 total 1366.37 last 76.80) - open/close 46142 (time 1788474921.09 total 1376.38 last 73.83) - open/close 46802 (time 1788474931.09 total 1386.38 last 65.99) - open/close 47476 (time 1788474941.13 total 1396.42 last 67.15) - open/close 48210 (time 1788474951.14 total 1406.43 last 73.30) - open/close 48909 (time 1788474961.17 total 1416.47 last 69.68) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1425.64 seconds: 34.81 ops/second - unlinked 0 (time 1788474974 ; total 0 ; last 0) - unlinked 10000 (time 1788475108 ; total 134 ; last 134) - unlinked 20000 (time 1788475240 ; total 266 ; last 132) - unlinked 30000 (time 1788475378 ; total 404 ; last 138) - unlinked 40000 (time 1788475524 ; total 550 ; last 146) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 677 seconds: 73.310188 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg635-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg635-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg635-server stop mds service on oleg635-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg635-server stop mds service on oleg635-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg635-server unloading modules via unload_modules_local on: 'oleg635-server' oleg635-server: oleg635-server.virtnet: executing unload_modules_local oleg635-server: kmemleak disabled modules unloaded. pdsh@oleg635-client: oleg635-client: ssh exited with exit code 2 pdsh@oleg635-client: oleg635-server: ssh exited with exit code 2 pdsh@oleg635-client: oleg635-client: ssh exited with exit code 2 pdsh@oleg635-client: oleg635-server: ssh exited with exit code 2