== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:29:15 (1788442155) start mds service on oleg209-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing load_modules_local oleg209-server: kmemleak disabled oleg209-server: Loading modules from /home/green/git/lustre-release/lustre oleg209-server: detected 4 online CPUs by sysfs oleg209-server: MODOPTS_LIBCFS= oleg209-server: Force libcfs to create 2 CPU partitions oleg209-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg209-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg209-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg209-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg209-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg209-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 281 (time 1788442267.74 total 10.03 last 28.02) - open/close 521 (time 1788442277.75 total 20.03 last 23.99) - open/close 696 (time 1788442287.75 total 30.03 last 17.50) - open/close 929 (time 1788442297.77 total 40.05 last 23.26) - open/close 1221 (time 1788442308.45 total 50.73 last 27.34) - open/close 1405 (time 1788442318.46 total 60.74 last 18.38) - open/close 1570 (time 1788442328.76 total 71.04 last 16.02) - open/close 1868 (time 1788442338.78 total 81.06 last 29.73) - open/close 2101 (time 1788442348.79 total 91.07 last 23.28) - open/close 2395 (time 1788442359.38 total 101.67 last 27.75) - open/close 2615 (time 1788442370.21 total 112.49 last 20.32) - open/close 2804 (time 1788442380.23 total 122.52 last 18.86) - open/close 3073 (time 1788442390.24 total 132.53 last 26.87) - open/close 3362 (time 1788442400.26 total 142.54 last 28.86) - open/close 3630 (time 1788442410.98 total 153.27 last 24.99) - open/close 3839 (time 1788442421.01 total 163.29 last 20.85) - open/close 4040 (time 1788442431.01 total 173.29 last 20.10) - open/close 4167 (time 1788442441.04 total 183.33 last 12.65) - open/close 4382 (time 1788442451.12 total 193.40 last 21.34) - open/close 4650 (time 1788442461.12 total 203.41 last 26.78) - open/close 4963 (time 1788442471.12 total 213.41 last 31.30) - open/close 5252 (time 1788442481.68 total 223.97 last 27.37) - open/close 5542 (time 1788442491.82 total 234.10 last 28.62) - open/close 5785 (time 1788442501.83 total 244.12 last 24.26) - open/close 6035 (time 1788442512.03 total 254.31 last 24.53) - open/close 6280 (time 1788442522.26 total 264.54 last 23.95) - open/close 6501 (time 1788442532.53 total 274.81 last 21.52) - open/close 6746 (time 1788442543.58 total 285.87 last 22.16) - open/close 6985 (time 1788442553.59 total 295.87 last 23.89) - open/close 7226 (time 1788442564.44 total 306.72 last 22.21) - open/close 7453 (time 1788442574.86 total 317.15 last 21.78) - open/close 7672 (time 1788442584.87 total 327.15 last 21.89) - open/close 7908 (time 1788442594.88 total 337.17 last 23.56) - open/close 8057 (time 1788442605.99 total 348.28 last 13.41) - open/close 8181 (time 1788442616.00 total 358.29 last 12.39) - open/close 8374 (time 1788442626.77 total 369.05 last 17.92) - open/close 8610 (time 1788442636.77 total 379.06 last 23.60) - open/close 8733 (time 1788442652.82 total 395.11 last 7.66) - open/close 8973 (time 1788442662.95 total 405.23 last 23.71) - open/close 9235 (time 1788442673.35 total 415.63 last 25.19) - open/close 9441 (time 1788442684.33 total 426.61 last 18.76) - open/close 9630 (time 1788442695.00 total 437.29 last 17.70) - open/close 9751 (time 1788442705.06 total 447.34 last 12.03) - open/close 9986 (time 1788442715.07 total 457.36 last 23.47) - open/close 10000 (time 1788442718.42 total 460.70 last 4.18) - open/close 10243 (time 1788442729.19 total 471.47 last 22.56) - open/close 10495 (time 1788442739.76 total 482.04 last 23.85) - open/close 10766 (time 1788442750.05 total 492.33 last 26.33) - open/close 11041 (time 1788442760.05 total 502.34 last 27.48) - open/close 11288 (time 1788442770.71 total 512.99 last 23.19) - open/close 11478 (time 1788442781.41 total 523.70 last 17.74) - open/close 11664 (time 1788442792.10 total 534.38 last 17.41) - open/close 11912 (time 1788442802.11 total 544.39 last 24.77) - open/close 12128 (time 1788442812.39 total 554.68 last 21.01) - open/close 12288 (time 1788442822.88 total 565.16 last 15.25) - open/close 12570 (time 1788442833.84 total 576.12 last 25.74) - open/close 12839 (time 1788442843.84 total 586.13 last 26.89) - open/close 13078 (time 1788442853.86 total 596.15 last 23.85) - open/close 13354 (time 1788442863.89 total 606.17 last 27.52) - open/close 13647 (time 1788442874.21 total 616.50 last 28.38) - open/close 13895 (time 1788442884.64 total 626.92 last 23.79) - open/close 14142 (time 1788442894.89 total 637.18 last 24.08) - open/close 14373 (time 1788442906.65 total 648.93 last 19.65) - open/close 14653 (time 1788442917.66 total 659.95 last 25.42) - open/close 14929 (time 1788442927.83 total 670.12 last 27.14) - open/close 15226 (time 1788442937.84 total 680.12 last 29.68) - open/close 15459 (time 1788442948.08 total 690.36 last 22.76) - open/close 15737 (time 1788442958.91 total 701.19 last 25.67) - open/close 15983 (time 1788442969.20 total 711.49 last 23.89) - open/close 16196 (time 1788442979.22 total 721.50 last 21.26) - open/close 16509 (time 1788442990.12 total 732.40 last 28.73) - open/close 16762 (time 1788443000.41 total 742.70 last 24.57) - open/close 17068 (time 1788443011.12 total 753.40 last 28.59) - open/close 17283 (time 1788443021.17 total 763.45 last 21.39) - open/close 17351 (time 1788443031.99 total 774.27 last 6.28) - open/close 17492 (time 1788443042.44 total 784.73 last 13.49) - open/close 17630 (time 1788443052.47 total 794.75 last 13.77) - open/close 17900 (time 1788443063.28 total 805.57 last 24.97) - open/close 18172 (time 1788443074.66 total 816.95 last 23.90) - open/close 18252 (time 1788443085.42 total 827.71 last 7.44) - open/close 18370 (time 1788443095.45 total 837.73 last 11.77) - open/close 18644 (time 1788443105.46 total 847.74 last 27.37) - open/close 18855 (time 1788443116.33 total 858.61 last 19.41) - open/close 19145 (time 1788443127.27 total 869.56 last 26.50) - open/close 19326 (time 1788443137.63 total 879.91 last 17.48) - open/close 19558 (time 1788443147.66 total 889.94 last 23.13) - open/close 19801 (time 1788443157.67 total 899.95 last 24.27) - open/close 20000 (time 1788443166.35 total 908.63 last 22.93) - open/close 20185 (time 1788443176.73 total 919.01 last 17.82) - open/close 20334 (time 1788443186.77 total 929.06 last 14.83) - open/close 20432 (time 1788443196.79 total 939.07 last 9.79) - open/close 20655 (time 1788443207.86 total 950.14 last 20.14) - open/close 21233 (time 1788443217.86 total 960.14 last 57.79) - open/close 22292 (time 1788443227.86 total 970.15 last 105.86) - open/close 23266 (time 1788443237.88 total 980.17 last 97.22) - open/close 24091 (time 1788443247.89 total 990.18 last 82.43) - open/close 24978 (time 1788443257.90 total 1000.18 last 88.65) - open/close 25809 (time 1788443267.90 total 1010.18 last 83.09) - open/close 26676 (time 1788443277.91 total 1020.19 last 86.63) - open/close 27559 (time 1788443287.91 total 1030.20 last 88.26) - open/close 28263 (time 1788443297.91 total 1040.20 last 70.38) - open/close 28968 (time 1788443307.94 total 1050.23 last 70.30) - open/close 29562 (time 1788443317.96 total 1060.24 last 59.32) - open/close 30000 (time 1788443324.75 total 1067.04 last 64.45) - open/close 30842 (time 1788443334.76 total 1077.05 last 84.12) - open/close 31664 (time 1788443344.77 total 1087.05 last 82.14) - open/close 32589 (time 1788443354.77 total 1097.06 last 92.45) - open/close 33394 (time 1788443364.78 total 1107.06 last 80.49) - open/close 34319 (time 1788443374.79 total 1117.07 last 92.37) - open/close 35147 (time 1788443384.80 total 1127.08 last 82.76) - open/close 35952 (time 1788443394.81 total 1137.09 last 80.41) - open/close 36662 (time 1788443404.81 total 1147.09 last 71.00) - open/close 37631 (time 1788443414.81 total 1157.09 last 96.90) - open/close 38621 (time 1788443424.81 total 1167.09 last 99.00) - open/close 39510 (time 1788443434.82 total 1177.10 last 88.81) - open/close 40000 (time 1788443440.50 total 1182.78 last 86.26) - open/close 40707 (time 1788443450.51 total 1192.80 last 70.59) - open/close 41596 (time 1788443460.52 total 1202.80 last 88.89) - open/close 42450 (time 1788443470.52 total 1212.80 last 85.40) - open/close 43313 (time 1788443480.52 total 1222.80 last 86.28) - open/close 44048 (time 1788443490.52 total 1232.81 last 73.48) - open/close 44763 (time 1788443500.53 total 1242.81 last 71.45) - open/close 45513 (time 1788443510.54 total 1252.82 last 74.94) - open/close 46327 (time 1788443520.54 total 1262.82 last 81.38) - open/close 47343 (time 1788443530.55 total 1272.84 last 101.44) - open/close 48307 (time 1788443540.56 total 1282.84 last 96.37) - open/close 49191 (time 1788443550.57 total 1292.85 last 88.30) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1297.77 seconds: 38.24 ops/second - unlinked 0 (time 1788443560 ; total 0 ; last 0) - unlinked 10000 (time 1788443679 ; total 119 ; last 119) - unlinked 20000 (time 1788443805 ; total 245 ; last 126) - unlinked 30000 (time 1788443935 ; total 375 ; last 130) - unlinked 40000 (time 1788444047 ; total 487 ; last 112) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 587 seconds: 84.550255 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg209-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server stop mds service on oleg209-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server stop mds service on oleg209-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg209-server unloading modules via unload_modules_local on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing unload_modules_local oleg209-server: kmemleak disabled modules unloaded. pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2