== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:08:48 (1788840528) start mds service on oleg647-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1754936 Sep 7 17:40 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg647-server' oleg647-server: oleg647-server.virtnet: executing load_modules_local oleg647-server: kmemleak disabled oleg647-server: Loading modules from /home/green/git/lustre-release/lustre oleg647-server: detected 4 online CPUs by sysfs oleg647-server: MODOPTS_LIBCFS= oleg647-server: Force libcfs to create 2 CPU partitions oleg647-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg647-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg647-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg647-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg647-server: oleg647-server.virtnet: executing set_default_debug -1 all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg647-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg647-server: oleg647-server.virtnet: executing set_default_debug -1 all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg647-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg647-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg647-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg647-server: oleg647-server.virtnet: executing set_default_debug -1 all pdsh@oleg647-client: oleg647-server: ssh exited with exit code 1 Started lustre-OST0000 oleg647-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg647-client.virtnet: -o user_xattr,flock 192.168.206.147@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 85 (time 1788840640.30 total 11.18 last 7.61) - open/close 252 (time 1788840650.32 total 21.19 last 16.67) - open/close 470 (time 1788840660.32 total 31.19 last 21.80) - open/close 737 (time 1788840671.00 total 41.87 last 25.00) - open/close 1002 (time 1788840681.18 total 52.05 last 26.04) - open/close 1244 (time 1788840691.19 total 62.06 last 24.17) - open/close 1485 (time 1788840701.41 total 72.28 last 23.59) - open/close 1782 (time 1788840711.41 total 82.29 last 29.68) - open/close 2008 (time 1788840722.00 total 92.87 last 21.35) - open/close 2273 (time 1788840732.82 total 103.69 last 24.49) - open/close 2544 (time 1788840743.71 total 114.58 last 24.88) - open/close 2800 (time 1788840754.07 total 124.94 last 24.71) - open/close 3065 (time 1788840765.09 total 135.96 last 24.06) - open/close 3313 (time 1788840775.09 total 145.97 last 24.78) - open/close 3558 (time 1788840785.11 total 155.98 last 24.47) - open/close 3777 (time 1788840795.54 total 166.41 last 21.00) - open/close 4034 (time 1788840805.72 total 176.59 last 25.24) - open/close 4297 (time 1788840815.73 total 186.60 last 26.27) - open/close 4516 (time 1788840825.74 total 196.61 last 21.89) - open/close 4807 (time 1788840835.80 total 206.67 last 28.91) - open/close 5045 (time 1788840846.10 total 216.97 last 23.11) - open/close 5322 (time 1788840856.74 total 227.62 last 26.02) - open/close 5593 (time 1788840867.28 total 238.15 last 25.72) - open/close 5871 (time 1788840877.98 total 248.85 last 26.00) - open/close 6142 (time 1788840888.47 total 259.34 last 25.82) - open/close 6412 (time 1788840898.80 total 269.68 last 26.13) - open/close 6724 (time 1788840909.59 total 280.46 last 28.92) - open/close 6986 (time 1788840920.54 total 291.41 last 23.94) - open/close 7259 (time 1788840930.55 total 301.42 last 27.28) - open/close 7462 (time 1788840941.46 total 312.34 last 18.59) - open/close 7653 (time 1788840952.33 total 323.20 last 17.57) - open/close 7799 (time 1788840962.64 total 333.51 last 14.17) - open/close 8003 (time 1788840972.65 total 343.52 last 20.38) - open/close 8163 (time 1788840982.66 total 353.53 last 15.98) - open/close 8393 (time 1788840993.20 total 364.07 last 21.82) - open/close 8642 (time 1788841003.63 total 374.51 last 23.86) - open/close 8893 (time 1788841013.64 total 384.52 last 25.08) - open/close 9084 (time 1788841023.65 total 394.52 last 19.09) - open/close 9300 (time 1788841034.01 total 404.88 last 20.84) - open/close 9503 (time 1788841044.02 total 414.89 last 20.29) - open/close 9733 (time 1788841054.36 total 425.24 last 22.23) - open/close 9943 (time 1788841064.37 total 435.24 last 20.99) - open/close 10176 (time 1788841074.78 total 445.65 last 22.39) - open/close 10405 (time 1788841085.45 total 456.33 last 21.45) - open/close 10631 (time 1788841095.46 total 466.34 last 22.58) - open/close 10895 (time 1788841106.13 total 477.00 last 24.76) - open/close 11154 (time 1788841117.08 total 487.96 last 23.64) - open/close 11425 (time 1788841128.02 total 498.89 last 24.79) - open/close 11671 (time 1788841138.04 total 508.92 last 24.54) - open/close 11962 (time 1788841149.01 total 519.89 last 26.53) - open/close 12218 (time 1788841159.02 total 529.90 last 25.57) - open/close 12424 (time 1788841169.55 total 540.43 last 19.56) - open/close 12688 (time 1788841179.56 total 550.43 last 26.38) - open/close 12939 (time 1788841189.72 total 560.59 last 24.72) - open/close 13193 (time 1788841200.02 total 570.89 last 24.65) - open/close 13415 (time 1788841210.58 total 581.45 last 21.02) - open/close 13639 (time 1788841220.88 total 591.75 last 21.75) - open/close 13856 (time 1788841230.93 total 601.80 last 21.60) - open/close 14062 (time 1788841241.17 total 612.04 last 20.11) - open/close 14303 (time 1788841252.06 total 622.93 last 22.14) - open/close 14541 (time 1788841262.80 total 633.67 last 22.15) - open/close 14780 (time 1788841273.82 total 644.69 last 21.70) - open/close 15003 (time 1788841283.83 total 654.70 last 22.27) - open/close 15295 (time 1788841294.74 total 665.62 last 26.76) - open/close 15549 (time 1788841304.75 total 675.62 last 25.39) - open/close 15734 (time 1788841315.73 total 686.61 last 16.84) - open/close 15913 (time 1788841326.69 total 697.56 last 16.33) - open/close 16178 (time 1788841336.70 total 707.57 last 26.48) - open/close 16461 (time 1788841346.71 total 717.58 last 28.27) - open/close 16692 (time 1788841357.46 total 728.33 last 21.48) - open/close 16962 (time 1788841368.03 total 738.91 last 25.53) - open/close 17243 (time 1788841378.91 total 749.78 last 25.84) - open/close 17586 (time 1788841388.91 total 759.79 last 34.29) - open/close 17825 (time 1788841398.92 total 769.79 last 23.88) - open/close 18055 (time 1788841409.13 total 780.01 last 22.52) - open/close 18308 (time 1788841419.14 total 790.01 last 25.28) - open/close 18471 (time 1788841429.34 total 800.21 last 15.98) - open/close 18743 (time 1788841440.04 total 810.91 last 25.43) - open/close 19617 (time 1788841450.04 total 820.91 last 87.36) - open/close 20000 (time 1788841454.24 total 825.11 last 91.18) - open/close 20911 (time 1788841464.25 total 835.12 last 91.01) - open/close 21787 (time 1788841474.26 total 845.13 last 87.54) - open/close 22688 (time 1788841484.26 total 855.13 last 90.07) - open/close 23550 (time 1788841494.26 total 865.14 last 86.17) - open/close 24410 (time 1788841504.28 total 875.15 last 85.89) - open/close 25259 (time 1788841514.28 total 885.15 last 84.85) - open/close 26056 (time 1788841524.29 total 895.16 last 79.67) - open/close 26882 (time 1788841534.29 total 905.16 last 82.54) - open/close 27752 (time 1788841544.29 total 915.17 last 87.00) - open/close 28556 (time 1788841554.30 total 925.17 last 80.38) - open/close 29390 (time 1788841564.30 total 935.17 last 83.38) - open/close 30000 (time 1788841571.59 total 942.47 last 83.59) - open/close 30867 (time 1788841581.60 total 952.47 last 86.66) - open/close 31770 (time 1788841591.60 total 962.47 last 90.29) - open/close 32648 (time 1788841601.61 total 972.48 last 87.73) - open/close 33505 (time 1788841611.61 total 982.49 last 85.66) - open/close 34354 (time 1788841621.62 total 992.49 last 84.82) - open/close 35177 (time 1788841631.64 total 1002.51 last 82.17) - open/close 36056 (time 1788841641.64 total 1012.51 last 87.89) - open/close 36922 (time 1788841651.64 total 1022.51 last 86.57) - open/close 37786 (time 1788841661.65 total 1032.52 last 86.35) - open/close 38613 (time 1788841671.65 total 1042.52 last 82.69) - open/close 39548 (time 1788841681.65 total 1052.52 last 93.48) - open/close 40000 (time 1788841686.62 total 1057.49 last 91.05) - open/close 40881 (time 1788841696.62 total 1067.49 last 88.06) - open/close 41635 (time 1788841706.65 total 1077.53 last 75.15) - open/close 42382 (time 1788841716.66 total 1087.53 last 74.67) - open/close 43102 (time 1788841726.66 total 1097.53 last 71.99) - open/close 43863 (time 1788841736.67 total 1107.54 last 76.04) - open/close 44699 (time 1788841746.67 total 1117.55 last 83.53) - open/close 45548 (time 1788841756.68 total 1127.55 last 84.85) - open/close 46416 (time 1788841766.69 total 1137.56 last 86.75) - open/close 47244 (time 1788841776.69 total 1147.56 last 82.79) - open/close 48075 (time 1788841786.69 total 1157.56 last 83.09) - open/close 48938 (time 1788841796.70 total 1167.57 last 86.23) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1175.74 seconds: 42.21 ops/second - unlinked 0 (time 1788841808 ; total 0 ; last 0) - unlinked 10000 (time 1788841933 ; total 125 ; last 125) - unlinked 20000 (time 1788842045 ; total 237 ; last 112) - unlinked 30000 (time 1788842109 ; total 301 ; last 64) - unlinked 40000 (time 1788842175 ; total 367 ; last 66) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 430 seconds: 115.420929 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg647-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg647-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg647-server stop mds service on oleg647-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg647-server stop mds service on oleg647-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg647-server unloading modules via unload_modules_local on: 'oleg647-server' oleg647-server: oleg647-server.virtnet: executing unload_modules_local oleg647-server: kmemleak disabled modules unloaded. pdsh@oleg647-client: oleg647-client: ssh exited with exit code 2 pdsh@oleg647-client: oleg647-server: ssh exited with exit code 2 pdsh@oleg647-client: oleg647-client: ssh exited with exit code 2 pdsh@oleg647-client: oleg647-server: ssh exited with exit code 2