== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:19:24 (1789600764) start mds service on oleg652-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 16 11:14 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg652-server' oleg652-server: oleg652-server.virtnet: executing load_modules_local oleg652-server: kmemleak disabled oleg652-server: Loading modules from /home/green/git/lustre-release/lustre oleg652-server: detected 4 online CPUs by sysfs oleg652-server: MODOPTS_LIBCFS= oleg652-server: Force libcfs to create 2 CPU partitions oleg652-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg652-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg652-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg652-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg652-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg652-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-OST0000 oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg652-client.virtnet: -o user_xattr,flock 192.168.206.152@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 302 (time 1789600868.59 total 12.09 last 24.98) - open/close 571 (time 1789600879.33 total 22.83 last 25.04) - open/close 865 (time 1789600889.76 total 33.27 last 28.17) - open/close 1120 (time 1789600899.85 total 43.35 last 25.29) - open/close 1423 (time 1789600909.87 total 53.38 last 30.22) - open/close 1720 (time 1789600919.88 total 63.38 last 29.68) - open/close 1902 (time 1789600929.89 total 73.39 last 18.19) - open/close 2141 (time 1789600939.91 total 83.41 last 23.86) - open/close 2429 (time 1789600950.94 total 94.44 last 26.10) - open/close 2702 (time 1789600961.84 total 105.34 last 25.06) - open/close 2995 (time 1789600972.84 total 116.34 last 26.64) - open/close 3287 (time 1789600982.84 total 126.34 last 29.19) - open/close 3561 (time 1789600992.85 total 136.35 last 27.38) - open/close 3864 (time 1789601003.61 total 147.11 last 28.15) - open/close 4131 (time 1789601014.57 total 158.07 last 24.36) - open/close 4406 (time 1789601024.58 total 168.08 last 27.48) - open/close 4758 (time 1789601035.30 total 178.80 last 32.83) - open/close 5050 (time 1789601045.67 total 189.17 last 28.16) - open/close 5345 (time 1789601055.68 total 199.19 last 29.46) - open/close 5571 (time 1789601066.23 total 209.73 last 21.43) - open/close 5854 (time 1789601076.84 total 220.34 last 26.67) - open/close 6107 (time 1789601086.88 total 230.38 last 25.20) - open/close 6433 (time 1789601096.88 total 240.39 last 32.59) - open/close 6663 (time 1789601106.89 total 250.40 last 22.98) - open/close 6889 (time 1789601116.97 total 260.47 last 22.43) - open/close 7083 (time 1789601127.65 total 271.15 last 18.16) - open/close 7295 (time 1789601139.37 total 282.87 last 18.09) - open/close 7594 (time 1789601149.96 total 293.47 last 28.22) - open/close 7867 (time 1789601159.97 total 303.47 last 27.29) - open/close 8144 (time 1789601170.67 total 314.17 last 25.88) - open/close 8419 (time 1789601181.30 total 324.80 last 25.88) - open/close 8711 (time 1789601192.10 total 335.61 last 27.03) - open/close 9029 (time 1789601203.06 total 346.56 last 29.04) - open/close 9326 (time 1789601213.87 total 357.37 last 27.47) - open/close 9620 (time 1789601223.88 total 367.39 last 29.35) - open/close 9833 (time 1789601234.73 total 378.23 last 19.63) - open/close 10000 (time 1789601239.14 total 382.64 last 37.91) - open/close 10234 (time 1789601249.44 total 392.95 last 22.71) - open/close 10456 (time 1789601259.45 total 402.95 last 22.19) - open/close 10682 (time 1789601269.45 total 412.96 last 22.59) - open/close 10940 (time 1789601280.04 total 423.54 last 24.38) - open/close 11246 (time 1789601290.53 total 434.03 last 29.16) - open/close 11512 (time 1789601301.49 total 444.99 last 24.27) - open/close 11793 (time 1789601311.78 total 455.29 last 27.29) - open/close 11984 (time 1789601322.79 total 466.30 last 17.35) - open/close 12163 (time 1789601333.66 total 477.17 last 16.47) - open/close 12470 (time 1789601344.17 total 487.68 last 29.21) - open/close 12743 (time 1789601354.73 total 498.23 last 25.86) - open/close 13028 (time 1789601364.75 total 508.25 last 28.46) - open/close 13261 (time 1789601375.20 total 518.70 last 22.29) - open/close 13492 (time 1789601385.77 total 529.27 last 21.86) - open/close 13765 (time 1789601395.78 total 539.28 last 27.27) - open/close 14018 (time 1789601405.78 total 549.29 last 25.29) - open/close 14274 (time 1789601416.81 total 560.31 last 23.23) - open/close 14608 (time 1789601426.93 total 570.43 last 33.00) - open/close 14889 (time 1789601437.34 total 580.84 last 26.98) - open/close 15074 (time 1789601447.36 total 590.86 last 18.47) - open/close 15328 (time 1789601457.37 total 600.88 last 25.37) - open/close 15607 (time 1789601468.06 total 611.57 last 26.09) - open/close 15874 (time 1789601478.07 total 621.57 last 26.70) - open/close 16151 (time 1789601488.07 total 631.58 last 27.68) - open/close 16376 (time 1789601498.34 total 641.84 last 21.92) - open/close 16617 (time 1789601509.06 total 652.56 last 22.48) - open/close 16855 (time 1789601519.06 total 662.56 last 23.79) - open/close 17120 (time 1789601529.50 total 673.00 last 25.38) - open/close 17463 (time 1789601539.51 total 683.01 last 34.27) - open/close 17721 (time 1789601549.51 total 693.01 last 25.79) - open/close 18036 (time 1789601560.29 total 703.80 last 29.21) - open/close 18342 (time 1789601571.23 total 714.73 last 27.98) - open/close 18548 (time 1789601581.49 total 724.99 last 20.07) - open/close 18792 (time 1789601591.51 total 735.01 last 24.36) - open/close 19049 (time 1789601601.52 total 745.02 last 25.68) - open/close 19320 (time 1789601611.87 total 755.37 last 26.17) - open/close 19584 (time 1789601622.38 total 765.88 last 25.12) - open/close 19843 (time 1789601632.94 total 776.45 last 24.52) - open/close 20000 (time 1789601638.37 total 781.87 last 28.94) - open/close 20256 (time 1789601648.44 total 791.94 last 25.42) - open/close 20542 (time 1789601659.39 total 802.90 last 26.11) - open/close 20718 (time 1789601669.39 total 812.90 last 17.60) - open/close 20928 (time 1789601679.40 total 822.90 last 20.99) - open/close 21157 (time 1789601689.40 total 832.91 last 22.89) - open/close 21434 (time 1789601699.44 total 842.94 last 27.61) - open/close 21665 (time 1789601710.18 total 853.68 last 21.51) - open/close 22291 (time 1789601720.18 total 863.69 last 62.56) - open/close 23110 (time 1789601730.19 total 873.69 last 81.85) - open/close 24000 (time 1789601740.19 total 883.69 last 89.00) - open/close 24789 (time 1789601750.20 total 893.70 last 78.86) - open/close 25580 (time 1789601760.23 total 903.74 last 78.80) - open/close 26405 (time 1789601770.24 total 913.75 last 82.43) - open/close 27173 (time 1789601780.24 total 923.75 last 76.78) - open/close 28079 (time 1789601790.25 total 933.75 last 90.58) - open/close 29023 (time 1789601800.25 total 943.76 last 94.34) - open/close 29934 (time 1789601810.25 total 953.76 last 91.08) - open/close 30950 (time 1789601820.26 total 963.77 last 101.50) - open/close 31907 (time 1789601830.27 total 973.77 last 95.66) - open/close 32827 (time 1789601840.27 total 983.77 last 91.99) - open/close 33753 (time 1789601850.27 total 993.78 last 92.58) - open/close 34740 (time 1789601860.28 total 1003.78 last 98.61) - open/close 35719 (time 1789601870.28 total 1013.79 last 97.89) - open/close 36718 (time 1789601880.29 total 1023.79 last 99.82) - open/close 37566 (time 1789601890.29 total 1033.79 last 84.79) - open/close 38522 (time 1789601900.30 total 1043.80 last 95.53) - open/close 39492 (time 1789601910.31 total 1053.82 last 96.84) - open/close 40000 (time 1789601915.83 total 1059.33 last 92.16) - open/close 40961 (time 1789601925.83 total 1069.33 last 96.07) - open/close 41990 (time 1789601935.84 total 1079.34 last 102.80) - open/close 42912 (time 1789601945.85 total 1089.35 last 92.13) - open/close 43773 (time 1789601955.86 total 1099.36 last 86.02) - open/close 44549 (time 1789601965.87 total 1109.37 last 77.53) - open/close 45513 (time 1789601975.88 total 1119.38 last 96.29) - open/close 46326 (time 1789601985.88 total 1129.38 last 81.30) - open/close 47247 (time 1789601995.89 total 1139.39 last 92.03) - open/close 47985 (time 1789602005.90 total 1149.40 last 73.69) - open/close 48943 (time 1789602015.91 total 1159.41 last 95.72) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1167.16 seconds: 42.52 ops/second - unlinked 0 (time 1789602027 ; total 0 ; last 0) - unlinked 10000 (time 1789602130 ; total 103 ; last 103) - unlinked 20000 (time 1789602235 ; total 208 ; last 105) - unlinked 30000 (time 1789602294 ; total 267 ; last 59) - unlinked 40000 (time 1789602351 ; total 324 ; last 57) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 381 seconds: 130.265091 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg652-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg652-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg652-server stop mds service on oleg652-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg652-server stop mds service on oleg652-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg652-server unloading modules via unload_modules_local on: 'oleg652-server' oleg652-server: oleg652-server.virtnet: executing unload_modules_local oleg652-server: kmemleak disabled modules unloaded. pdsh@oleg652-client: oleg652-client: ssh exited with exit code 2 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 2 pdsh@oleg652-client: oleg652-client: ssh exited with exit code 2 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 2