== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:26:15 (1788647175) start mds service on oleg317-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 4 18:25 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: kmemleak disabled oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg317-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg317-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg317-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg317-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 199 (time 1788647281.92 total 10.49 last 18.98) - open/close 390 (time 1788647294.36 total 22.93 last 15.35) - open/close 622 (time 1788647304.75 total 33.32 last 22.33) - open/close 837 (time 1788647315.01 total 43.58 last 20.96) - open/close 1088 (time 1788647325.01 total 53.58 last 25.09) - open/close 1261 (time 1788647337.97 total 66.54 last 13.35) - open/close 1345 (time 1788647348.00 total 76.57 last 8.38) - open/close 1584 (time 1788647358.02 total 86.59 last 23.84) - open/close 1812 (time 1788647368.30 total 96.87 last 22.18) - open/close 2045 (time 1788647378.32 total 106.88 last 23.26) - open/close 2275 (time 1788647388.32 total 116.89 last 22.98) - open/close 2497 (time 1788647398.32 total 126.89 last 22.20) - open/close 2750 (time 1788647408.70 total 137.27 last 24.38) - open/close 2855 (time 1788647419.32 total 147.89 last 9.89) - open/close 3082 (time 1788647429.33 total 157.90 last 22.68) - open/close 3297 (time 1788647439.54 total 168.11 last 21.06) - open/close 3431 (time 1788647449.60 total 178.17 last 13.32) - open/close 3605 (time 1788647461.43 total 190.00 last 14.71) - open/close 3781 (time 1788647472.13 total 200.70 last 16.44) - open/close 4047 (time 1788647482.17 total 210.74 last 26.50) - open/close 4277 (time 1788647492.24 total 220.81 last 22.85) - open/close 4572 (time 1788647502.83 total 231.40 last 27.85) - open/close 4765 (time 1788647512.83 total 241.40 last 19.30) - open/close 4947 (time 1788647523.45 total 252.02 last 17.14) - open/close 5069 (time 1788647533.46 total 262.03 last 12.19) - open/close 5240 (time 1788647544.31 total 272.88 last 15.77) - open/close 5483 (time 1788647554.32 total 282.89 last 24.26) - open/close 5585 (time 1788647564.96 total 293.53 last 9.59) - open/close 5646 (time 1788647575.68 total 304.25 last 5.69) - open/close 5874 (time 1788647586.78 total 315.35 last 20.55) - open/close 6114 (time 1788647597.42 total 325.99 last 22.55) - open/close 6272 (time 1788647608.15 total 336.71 last 14.73) - open/close 6444 (time 1788647619.19 total 347.76 last 15.57) - open/close 6605 (time 1788647629.64 total 358.21 last 15.41) - open/close 6871 (time 1788647640.58 total 369.15 last 24.31) - open/close 7089 (time 1788647651.05 total 379.62 last 20.81) - open/close 7366 (time 1788647661.62 total 390.19 last 26.22) - open/close 7613 (time 1788647672.64 total 401.21 last 22.41) - open/close 7795 (time 1788647683.13 total 411.70 last 17.35) - open/close 7849 (time 1788647693.65 total 422.22 last 5.14) - open/close 8092 (time 1788647703.66 total 432.23 last 24.28) - open/close 8291 (time 1788647713.71 total 442.28 last 19.79) - open/close 8475 (time 1788647723.72 total 452.29 last 18.38) - open/close 8693 (time 1788647733.75 total 462.32 last 21.74) - open/close 8830 (time 1788647743.80 total 472.37 last 13.63) - open/close 8990 (time 1788647754.37 total 482.94 last 15.14) - open/close 9114 (time 1788647766.35 total 494.92 last 10.35) - open/close 9220 (time 1788647777.21 total 505.77 last 9.76) - open/close 9493 (time 1788647787.21 total 515.78 last 27.28) - open/close 9773 (time 1788647797.22 total 525.79 last 27.99) - open/close 9958 (time 1788647807.53 total 536.10 last 17.94) - open/close 10206 (time 1788647817.78 total 546.35 last 24.20) - open/close 10464 (time 1788647828.53 total 557.10 last 24.00) - open/close 10663 (time 1788647838.55 total 567.12 last 19.87) - open/close 10857 (time 1788647849.08 total 577.65 last 18.41) - open/close 11087 (time 1788647859.09 total 587.66 last 22.98) - open/close 11170 (time 1788647869.12 total 597.69 last 8.28) - open/close 11384 (time 1788647879.14 total 607.71 last 21.36) - open/close 11493 (time 1788647889.86 total 618.43 last 10.17) - open/close 11693 (time 1788647900.14 total 628.71 last 19.44) - open/close 11920 (time 1788647911.09 total 639.66 last 20.74) - open/close 12151 (time 1788647921.52 total 650.09 last 22.14) - open/close 12365 (time 1788647931.53 total 660.10 last 21.37) - open/close 12553 (time 1788647942.40 total 670.96 last 17.31) - open/close 12635 (time 1788647952.83 total 681.40 last 7.86) - open/close 12802 (time 1788647963.19 total 691.76 last 16.12) - open/close 13106 (time 1788647973.19 total 701.76 last 30.40) - open/close 13200 (time 1788647983.49 total 712.06 last 9.12) - open/close 13293 (time 1788647993.51 total 722.08 last 9.29) - open/close 13516 (time 1788648003.51 total 732.08 last 22.30) - open/close 13741 (time 1788648013.63 total 742.20 last 22.23) - open/close 13969 (time 1788648024.06 total 752.63 last 21.86) - open/close 14264 (time 1788648034.08 total 762.65 last 29.46) - open/close 14526 (time 1788648044.09 total 772.66 last 26.16) - open/close 14645 (time 1788648054.10 total 782.67 last 11.88) - open/close 14839 (time 1788648064.69 total 793.26 last 18.33) - open/close 14938 (time 1788648075.09 total 803.66 last 9.52) - open/close 14978 (time 1788648085.38 total 813.95 last 3.89) - open/close 15245 (time 1788648095.39 total 823.96 last 26.65) - open/close 15473 (time 1788648105.60 total 834.17 last 22.33) - open/close 15694 (time 1788648115.62 total 844.18 last 22.07) - open/close 15910 (time 1788648125.63 total 854.20 last 21.56) - open/close 16123 (time 1788648136.29 total 864.86 last 19.98) - open/close 16258 (time 1788648146.61 total 875.18 last 13.08) - open/close 16374 (time 1788648156.89 total 885.46 last 11.29) - open/close 16501 (time 1788648166.90 total 895.46 last 12.69) - open/close 16705 (time 1788648176.90 total 905.47 last 20.39) - open/close 16966 (time 1788648187.37 total 915.94 last 24.93) - open/close 17306 (time 1788648198.15 total 926.71 last 31.56) - open/close 17526 (time 1788648208.17 total 936.73 last 21.96) - open/close 17755 (time 1788648218.95 total 947.52 last 21.23) - open/close 17927 (time 1788648228.96 total 957.53 last 17.19) - open/close 18036 (time 1788648238.97 total 967.54 last 10.89) - open/close 18180 (time 1788648248.99 total 977.56 last 14.36) - open/close 18281 (time 1788648259.63 total 988.20 last 9.50) - open/close 18762 (time 1788648269.63 total 998.20 last 48.10) - open/close 19493 (time 1788648279.63 total 1008.20 last 73.06) - open/close 20000 (time 1788648286.20 total 1014.77 last 77.16) - open/close 20709 (time 1788648296.21 total 1024.78 last 70.84) - open/close 21465 (time 1788648306.21 total 1034.78 last 75.60) - open/close 22137 (time 1788648316.22 total 1044.79 last 67.17) - open/close 22856 (time 1788648326.22 total 1054.79 last 71.85) - open/close 23547 (time 1788648336.23 total 1064.80 last 69.04) - open/close 24273 (time 1788648346.24 total 1074.81 last 72.52) - open/close 24976 (time 1788648356.25 total 1084.82 last 70.23) - open/close 25700 (time 1788648366.26 total 1094.83 last 72.35) - open/close 26399 (time 1788648376.26 total 1104.83 last 69.88) - open/close 27086 (time 1788648386.26 total 1114.83 last 68.70) - open/close 27830 (time 1788648396.28 total 1124.85 last 74.28) - open/close 28569 (time 1788648406.29 total 1134.85 last 73.87) - open/close 29340 (time 1788648416.29 total 1144.86 last 77.06) - open/close 30000 (time 1788648424.72 total 1153.28 last 78.34) - open/close 30784 (time 1788648434.72 total 1163.29 last 78.36) - open/close 31541 (time 1788648444.72 total 1173.29 last 75.70) - open/close 32338 (time 1788648454.73 total 1183.30 last 79.62) - open/close 33084 (time 1788648464.74 total 1193.31 last 74.51) - open/close 33776 (time 1788648474.75 total 1203.32 last 69.14) - open/close 34490 (time 1788648484.76 total 1213.33 last 71.32) - open/close 35239 (time 1788648494.76 total 1223.33 last 74.89) - open/close 35959 (time 1788648504.78 total 1233.34 last 71.92) - open/close 36732 (time 1788648514.79 total 1243.36 last 77.21) - open/close 37485 (time 1788648524.80 total 1253.37 last 75.22) - open/close 38269 (time 1788648534.81 total 1263.37 last 78.33) - open/close 39050 (time 1788648544.82 total 1273.39 last 78.00) - open/close 39811 (time 1788648554.82 total 1283.39 last 76.08) - open/close 40000 (time 1788648557.17 total 1285.74 last 80.32) - open/close 40796 (time 1788648567.18 total 1295.75 last 79.55) - open/close 41514 (time 1788648577.18 total 1305.75 last 71.79) - open/close 42256 (time 1788648587.19 total 1315.76 last 74.14) - open/close 42943 (time 1788648597.20 total 1325.77 last 68.64) - open/close 43504 (time 1788648607.23 total 1335.80 last 55.95) - open/close 44117 (time 1788648617.23 total 1345.80 last 61.29) - open/close 44662 (time 1788648627.23 total 1355.80 last 54.49) - open/close 45313 (time 1788648637.23 total 1365.80 last 65.10) - open/close 46042 (time 1788648647.25 total 1375.81 last 72.78) - open/close 46694 (time 1788648657.25 total 1385.82 last 65.19) - open/close 47303 (time 1788648667.26 total 1395.83 last 60.85) - open/close 47984 (time 1788648677.27 total 1405.83 last 68.04) - open/close 48618 (time 1788648687.27 total 1415.84 last 63.37) - open/close 49261 (time 1788648697.28 total 1425.84 last 64.26) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1430.10 seconds: 34.70 ops/second - unlinked 0 (time 1788648706 ; total 0 ; last 0) - unlinked 10000 (time 1788648847 ; total 141 ; last 141) - unlinked 20000 (time 1788648980 ; total 274 ; last 133) - unlinked 30000 (time 1788649133 ; total 427 ; last 153) - unlinked 40000 (time 1788649269 ; total 563 ; last 136) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 689 seconds: 72.033379 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local oleg317-server: kmemleak disabled modules unloaded. pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2