== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:52:31 (1789588351) start mds service on oleg104-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1932672 Sep 16 08:21 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing load_modules_local oleg104-server: kmemleak disabled oleg104-server: Loading modules from /home/green/git/lustre-release/lustre oleg104-server: detected 4 online CPUs by sysfs oleg104-server: MODOPTS_LIBCFS= oleg104-server: Force libcfs to create 2 CPU partitions oleg104-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg104-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg104-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg104-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg104-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg104-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Started lustre-OST0000 oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 118 (time 1789588478.19 total 10.58 last 11.15) - open/close 224 (time 1789588488.20 total 20.59 last 10.59) - open/close 299 (time 1789588498.27 total 30.67 last 7.45) - open/close 448 (time 1789588508.29 total 40.68 last 14.88) - open/close 562 (time 1789588518.53 total 50.93 last 11.13) - open/close 739 (time 1789588530.37 total 62.77 last 14.95) - open/close 895 (time 1789588540.58 total 72.98 last 15.28) - open/close 1079 (time 1789588551.47 total 83.87 last 16.90) - open/close 1320 (time 1789588561.47 total 93.87 last 24.09) - open/close 1490 (time 1789588571.48 total 103.87 last 16.99) - open/close 1662 (time 1789588582.37 total 114.76 last 15.79) - open/close 1914 (time 1789588592.38 total 124.78 last 25.16) - open/close 2030 (time 1789588602.74 total 135.14 last 11.20) - open/close 2170 (time 1789588613.29 total 145.69 last 13.27) - open/close 2330 (time 1789588624.31 total 156.71 last 14.52) - open/close 2457 (time 1789588635.05 total 167.45 last 11.83) - open/close 2699 (time 1789588645.10 total 177.50 last 24.07) - open/close 2749 (time 1789588655.72 total 188.12 last 4.71) - open/close 2817 (time 1789588665.73 total 198.12 last 6.79) - open/close 2914 (time 1789588675.76 total 208.15 last 9.67) - open/close 3036 (time 1789588688.43 total 220.83 last 9.63) - open/close 3169 (time 1789588699.24 total 231.63 last 12.31) - open/close 3317 (time 1789588710.76 total 243.16 last 12.84) - open/close 3464 (time 1789588721.06 total 253.46 last 14.28) - open/close 3662 (time 1789588731.14 total 263.54 last 19.64) - open/close 3848 (time 1789588741.22 total 273.61 last 18.46) - open/close 4002 (time 1789588751.78 total 284.18 last 14.58) - open/close 4129 (time 1789588766.05 total 298.45 last 8.90) - open/close 4310 (time 1789588776.07 total 308.46 last 18.07) - open/close 4394 (time 1789588786.28 total 318.67 last 8.23) - open/close 4472 (time 1789588796.28 total 328.68 last 7.79) - open/close 4513 (time 1789588807.59 total 339.98 last 3.63) - open/close 4614 (time 1789588818.15 total 350.54 last 9.57) - open/close 4841 (time 1789588828.39 total 360.78 last 22.17) - open/close 5033 (time 1789588838.56 total 370.96 last 18.86) - open/close 5269 (time 1789588848.73 total 381.13 last 23.21) - open/close 5514 (time 1789588858.74 total 391.14 last 24.48) - open/close 5650 (time 1789588869.08 total 401.48 last 13.15) - open/close 5712 (time 1789588879.09 total 411.49 last 6.20) - open/close 5767 (time 1789588889.97 total 422.36 last 5.06) - open/close 5943 (time 1789588899.97 total 432.36 last 17.59) - open/close 6105 (time 1789588912.52 total 444.91 last 12.91) - open/close 6332 (time 1789588922.87 total 455.26 last 21.93) - open/close 6479 (time 1789588932.88 total 465.27 last 14.68) - open/close 6642 (time 1789588942.89 total 475.29 last 16.28) - open/close 6756 (time 1789588952.91 total 485.31 last 11.38) - open/close 6871 (time 1789588963.96 total 496.36 last 10.41) - open/close 7100 (time 1789588975.98 total 508.38 last 19.05) - open/close 7200 (time 1789588988.45 total 520.84 last 8.02) - open/close 7322 (time 1789588998.46 total 530.85 last 12.19) - open/close 7489 (time 1789589009.30 total 541.70 last 15.40) - open/close 7687 (time 1789589019.31 total 551.71 last 19.77) - open/close 7901 (time 1789589029.75 total 562.15 last 20.50) - open/close 8007 (time 1789589040.67 total 573.06 last 9.71) - open/close 8255 (time 1789589051.23 total 583.62 last 23.49) - open/close 8440 (time 1789589061.23 total 593.63 last 18.49) - open/close 8553 (time 1789589072.19 total 604.58 last 10.31) - open/close 8670 (time 1789589082.19 total 614.59 last 11.69) - open/close 8807 (time 1789589093.16 total 625.55 last 12.49) - open/close 8983 (time 1789589104.02 total 636.41 last 16.21) - open/close 9222 (time 1789589114.79 total 647.18 last 22.19) - open/close 9512 (time 1789589125.12 total 657.52 last 28.05) - open/close 9721 (time 1789589135.15 total 667.55 last 20.84) - open/close 9911 (time 1789589145.41 total 677.80 last 18.52) - open/close 10000 (time 1789589151.33 total 683.73 last 15.02) - open/close 10199 (time 1789589161.34 total 693.73 last 19.89) - open/close 10335 (time 1789589171.79 total 704.19 last 13.01) - open/close 10457 (time 1789589181.80 total 714.20 last 12.18) - open/close 10555 (time 1789589192.06 total 724.45 last 9.56) - open/close 10807 (time 1789589202.33 total 734.73 last 24.53) - open/close 10930 (time 1789589212.65 total 745.05 last 11.92) - open/close 11027 (time 1789589223.08 total 755.47 last 9.31) - open/close 11203 (time 1789589233.08 total 765.47 last 17.60) - open/close 11396 (time 1789589243.32 total 775.72 last 18.84) - open/close 11587 (time 1789589253.33 total 785.72 last 19.09) - open/close 11786 (time 1789589263.65 total 796.05 last 19.27) - open/close 12043 (time 1789589273.91 total 806.30 last 25.07) - open/close 12299 (time 1789589284.52 total 816.91 last 24.13) - open/close 12557 (time 1789589295.21 total 827.60 last 24.13) - open/close 12736 (time 1789589305.76 total 838.15 last 16.97) - open/close 12951 (time 1789589316.07 total 848.46 last 20.86) - open/close 13073 (time 1789589326.59 total 858.98 last 11.60) - open/close 13149 (time 1789589337.32 total 869.71 last 7.08) - open/close 13295 (time 1789589348.34 total 880.73 last 13.25) - open/close 13532 (time 1789589358.35 total 890.74 last 23.68) - open/close 13751 (time 1789589368.95 total 901.34 last 20.66) - open/close 13922 (time 1789589378.95 total 911.35 last 17.09) - open/close 14073 (time 1789589388.96 total 921.35 last 15.09) - open/close 14144 (time 1789589401.27 total 933.67 last 5.77) - open/close 14358 (time 1789589411.76 total 944.16 last 20.40) - open/close 14518 (time 1789589421.77 total 954.17 last 15.98) - open/close 14653 (time 1789589431.77 total 964.17 last 13.50) - open/close 14890 (time 1789589442.09 total 974.49 last 22.96) - open/close 15078 (time 1789589452.11 total 984.51 last 18.76) - open/close 15302 (time 1789589462.31 total 994.70 last 21.98) - open/close 15502 (time 1789589472.33 total 1004.73 last 19.95) - open/close 15598 (time 1789589484.70 total 1017.09 last 7.76) - open/close 15696 (time 1789589495.27 total 1027.67 last 9.27) - open/close 15981 (time 1789589505.70 total 1038.10 last 27.32) - open/close 16229 (time 1789589515.74 total 1048.14 last 24.70) - open/close 16376 (time 1789589526.68 total 1059.08 last 13.44) - open/close 16512 (time 1789589537.21 total 1069.61 last 12.91) - open/close 16652 (time 1789589547.25 total 1079.64 last 13.96) - open/close 16740 (time 1789589557.87 total 1090.27 last 8.28) - open/close 16840 (time 1789589567.89 total 1100.29 last 9.98) - open/close 17056 (time 1789589577.94 total 1110.34 last 21.50) - open/close 17567 (time 1789589587.97 total 1120.37 last 50.94) - open/close 18216 (time 1789589597.98 total 1130.38 last 64.85) - open/close 18784 (time 1789589607.98 total 1140.38 last 56.80) - open/close 19362 (time 1789589617.98 total 1150.38 last 57.78) - open/close 19984 (time 1789589627.99 total 1160.38 last 62.19) - open/close 20655 (time 1789589637.99 total 1170.39 last 67.06) - open/close 21318 (time 1789589648.01 total 1180.41 last 66.15) - open/close 22064 (time 1789589658.02 total 1190.41 last 74.56) - open/close 22751 (time 1789589668.02 total 1200.42 last 68.69) - open/close 23455 (time 1789589678.02 total 1210.42 last 70.39) - open/close 24254 (time 1789589688.04 total 1220.44 last 79.75) - open/close 25007 (time 1789589698.04 total 1230.44 last 75.29) - open/close 25732 (time 1789589708.06 total 1240.45 last 72.38) - open/close 26415 (time 1789589718.07 total 1250.46 last 68.23) - open/close 27178 (time 1789589728.08 total 1260.47 last 76.24) - open/close 28058 (time 1789589738.08 total 1270.48 last 87.95) - open/close 28891 (time 1789589748.09 total 1280.48 last 83.25) - open/close 29728 (time 1789589758.09 total 1290.49 last 83.66) - open/close 30000 (time 1789589762.32 total 1294.72 last 64.30) - open/close 30683 (time 1789589772.34 total 1304.73 last 68.20) - open/close 31369 (time 1789589782.37 total 1314.76 last 68.41) - open/close 32187 (time 1789589792.37 total 1324.77 last 81.77) - open/close 33009 (time 1789589802.37 total 1334.77 last 82.19) - open/close 33830 (time 1789589812.38 total 1344.78 last 82.02) - open/close 34648 (time 1789589822.38 total 1354.78 last 81.78) - open/close 35187 (time 1789589832.39 total 1364.79 last 53.85) - open/close 35748 (time 1789589842.42 total 1374.81 last 55.97) - open/close 36460 (time 1789589852.44 total 1384.83 last 71.04) - open/close 37192 (time 1789589862.46 total 1394.85 last 73.08) - open/close 38040 (time 1789589872.46 total 1404.86 last 84.72) - open/close 38883 (time 1789589882.48 total 1414.87 last 84.19) - open/close 39670 (time 1789589892.49 total 1424.89 last 78.58) - open/close 40000 (time 1789589896.50 total 1428.89 last 82.35) - open/close 40729 (time 1789589906.51 total 1438.90 last 72.84) - open/close 41439 (time 1789589916.51 total 1448.91 last 70.98) - open/close 42304 (time 1789589926.52 total 1458.91 last 86.45) - open/close 43123 (time 1789589936.53 total 1468.93 last 81.77) - open/close 43807 (time 1789589946.54 total 1478.93 last 68.36) - open/close 44482 (time 1789589956.54 total 1488.94 last 67.47) - open/close 45389 (time 1789589966.55 total 1498.95 last 90.62) - open/close 46259 (time 1789589976.56 total 1508.95 last 86.95) - open/close 46862 (time 1789589986.58 total 1518.97 last 60.18) - open/close 47371 (time 1789589996.58 total 1528.97 last 50.90) - open/close 47881 (time 1789590006.58 total 1538.98 last 50.98) - open/close 48522 (time 1789590016.58 total 1548.98 last 64.09) - open/close 49212 (time 1789590026.59 total 1558.99 last 68.96) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1565.33 seconds: 31.71 ops/second - unlinked 0 (time 1789590038 ; total 0 ; last 0) - unlinked 10000 (time 1789590164 ; total 126 ; last 126) - unlinked 20000 (time 1789590296 ; total 258 ; last 132) - unlinked 30000 (time 1789590414 ; total 376 ; last 118) - unlinked 40000 (time 1789590553 ; total 515 ; last 139) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 635 seconds: 78.159058 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg104-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg104-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg104-server stop mds service on oleg104-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg104-server stop mds service on oleg104-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg104-server unloading modules via unload_modules_local on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing unload_modules_local oleg104-server: kmemleak disabled modules unloaded. pdsh@oleg104-client: oleg104-client: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-client: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 2