== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:19:49 (1789453189) start mds service on oleg124-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1933472 Sep 14 13:54 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing load_modules_local oleg124-server: kmemleak disabled oleg124-server: Loading modules from /home/green/git/lustre-release/lustre oleg124-server: detected 4 online CPUs by sysfs oleg124-server: MODOPTS_LIBCFS= oleg124-server: Force libcfs to create 2 CPU partitions oleg124-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg124-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg124-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg124-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg124-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg124-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0000 oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 225 (time 1789453311.02 total 10.01 last 22.49) - open/close 467 (time 1789453321.04 total 20.03 last 24.15) - open/close 732 (time 1789453331.62 total 30.61 last 25.04) - open/close 992 (time 1789453341.64 total 40.63 last 25.95) - open/close 1095 (time 1789453353.09 total 52.07 last 9.00) - open/close 1261 (time 1789453363.10 total 62.08 last 16.59) - open/close 1338 (time 1789453373.36 total 72.34 last 7.50) - open/close 1478 (time 1789453383.48 total 82.47 last 13.83) - open/close 1658 (time 1789453395.05 total 94.04 last 15.55) - open/close 1862 (time 1789453405.07 total 104.05 last 20.37) - open/close 1996 (time 1789453415.49 total 114.47 last 12.86) - open/close 2170 (time 1789453427.85 total 126.84 last 14.07) - open/close 2364 (time 1789453437.87 total 136.85 last 19.37) - open/close 2487 (time 1789453448.71 total 147.70 last 11.34) - open/close 2572 (time 1789453459.08 total 158.06 last 8.20) - open/close 2660 (time 1789453469.56 total 168.55 last 8.39) - open/close 2813 (time 1789453480.44 total 179.42 last 14.07) - open/close 2970 (time 1789453491.06 total 190.05 last 14.78) - open/close 3103 (time 1789453501.10 total 200.09 last 13.25) - open/close 3290 (time 1789453511.61 total 210.60 last 17.78) - open/close 3372 (time 1789453521.70 total 220.68 last 8.13) - open/close 3577 (time 1789453533.32 total 232.30 last 17.64) - open/close 3715 (time 1789453543.33 total 242.31 last 13.78) - open/close 3889 (time 1789453554.24 total 253.23 last 15.95) - open/close 4052 (time 1789453564.59 total 263.57 last 15.75) - open/close 4261 (time 1789453575.42 total 274.41 last 19.29) - open/close 4367 (time 1789453585.43 total 284.42 last 10.59) - open/close 4529 (time 1789453595.69 total 294.68 last 15.79) - open/close 4678 (time 1789453606.70 total 305.69 last 13.53) - open/close 4798 (time 1789453616.73 total 315.72 last 11.96) - open/close 4955 (time 1789453627.51 total 326.50 last 14.57) - open/close 5138 (time 1789453637.51 total 336.50 last 18.30) - open/close 5400 (time 1789453648.21 total 347.19 last 24.50) - open/close 5578 (time 1789453659.82 total 358.81 last 15.32) - open/close 5747 (time 1789453670.39 total 369.38 last 15.99) - open/close 5843 (time 1789453681.10 total 380.08 last 8.97) - open/close 5925 (time 1789453691.43 total 390.42 last 7.93) - open/close 5989 (time 1789453701.55 total 400.54 last 6.32) - open/close 6079 (time 1789453711.99 total 410.98 last 8.62) - open/close 6224 (time 1789453724.77 total 423.76 last 11.34) - open/close 6384 (time 1789453734.79 total 433.77 last 15.98) - open/close 6566 (time 1789453745.77 total 444.75 last 16.58) - open/close 6708 (time 1789453756.20 total 455.19 last 13.61) - open/close 6799 (time 1789453766.22 total 465.21 last 9.08) - open/close 6940 (time 1789453776.75 total 475.73 last 13.40) - open/close 7089 (time 1789453787.40 total 486.38 last 13.99) - open/close 7181 (time 1789453798.02 total 497.00 last 8.66) - open/close 7381 (time 1789453808.02 total 507.01 last 19.98) - open/close 7528 (time 1789453818.03 total 517.02 last 14.69) - open/close 7683 (time 1789453828.03 total 527.02 last 15.50) - open/close 7895 (time 1789453838.04 total 537.02 last 21.18) - open/close 7981 (time 1789453848.37 total 547.36 last 8.32) - open/close 8141 (time 1789453859.14 total 558.12 last 14.86) - open/close 8298 (time 1789453869.14 total 568.13 last 15.69) - open/close 8413 (time 1789453881.02 total 580.01 last 9.68) - open/close 8633 (time 1789453891.18 total 590.17 last 21.65) - open/close 8712 (time 1789453901.19 total 600.18 last 7.89) - open/close 8920 (time 1789453911.61 total 610.60 last 19.96) - open/close 9077 (time 1789453922.55 total 621.53 last 14.36) - open/close 9136 (time 1789453933.29 total 632.27 last 5.49) - open/close 9316 (time 1789453945.98 total 644.97 last 14.18) - open/close 9467 (time 1789453956.27 total 655.26 last 14.67) - open/close 9567 (time 1789453966.50 total 665.49 last 9.78) - open/close 9691 (time 1789453978.87 total 677.86 last 10.02) - open/close 9846 (time 1789453989.74 total 688.72 last 14.26) - open/close 10000 (time 1789453998.25 total 697.24 last 18.09) - open/close 10146 (time 1789454008.87 total 707.86 last 13.75) - open/close 10238 (time 1789454019.52 total 718.51 last 8.64) - open/close 10351 (time 1789454029.52 total 728.51 last 11.30) - open/close 10482 (time 1789454040.24 total 739.22 last 12.23) - open/close 10661 (time 1789454050.24 total 749.23 last 17.88) - open/close 10781 (time 1789454060.26 total 759.24 last 11.98) - open/close 10905 (time 1789454070.83 total 769.82 last 11.73) - open/close 11018 (time 1789454080.84 total 779.82 last 11.29) - open/close 11085 (time 1789454090.85 total 789.84 last 6.69) - open/close 11189 (time 1789454100.87 total 799.85 last 10.39) - open/close 11293 (time 1789454110.95 total 809.94 last 10.31) - open/close 11397 (time 1789454120.95 total 819.94 last 10.40) - open/close 11503 (time 1789454131.17 total 830.15 last 10.38) - open/close 11567 (time 1789454143.39 total 842.38 last 5.23) - open/close 11684 (time 1789454154.28 total 853.27 last 10.74) - open/close 11806 (time 1789454164.30 total 863.29 last 12.18) - open/close 11864 (time 1789454175.16 total 874.15 last 5.34) - open/close 12003 (time 1789454185.71 total 884.70 last 13.18) - open/close 12092 (time 1789454196.20 total 895.19 last 8.48) - open/close 12205 (time 1789454206.38 total 905.36 last 11.10) - open/close 12360 (time 1789454219.52 total 918.51 last 11.79) - open/close 12468 (time 1789454230.77 total 929.76 last 9.60) - open/close 12702 (time 1789454240.78 total 939.77 last 23.37) - open/close 12947 (time 1789454250.79 total 949.77 last 24.49) - open/close 13172 (time 1789454261.36 total 960.34 last 21.29) - open/close 13416 (time 1789454271.72 total 970.70 last 23.55) - open/close 13576 (time 1789454281.72 total 980.71 last 16.00) - open/close 13721 (time 1789454293.87 total 992.86 last 11.93) - open/close 13840 (time 1789454304.11 total 1003.09 last 11.63) - open/close 14042 (time 1789454315.32 total 1014.31 last 18.01) - open/close 14302 (time 1789454325.33 total 1024.32 last 25.97) - open/close 14585 (time 1789454335.35 total 1034.34 last 28.25) - open/close 14790 (time 1789454346.41 total 1045.40 last 18.54) - open/close 14941 (time 1789454358.51 total 1057.49 last 12.48) - open/close 15092 (time 1789454368.52 total 1067.50 last 15.08) - open/close 15317 (time 1789454378.87 total 1077.86 last 21.73) - open/close 15509 (time 1789454388.88 total 1087.87 last 19.18) - open/close 15720 (time 1789454400.30 total 1099.29 last 18.47) - open/close 15889 (time 1789454410.37 total 1109.35 last 16.79) - open/close 16113 (time 1789454420.37 total 1119.36 last 22.39) - open/close 16909 (time 1789454430.38 total 1129.37 last 79.49) - open/close 17540 (time 1789454440.40 total 1139.39 last 62.99) - open/close 18133 (time 1789454450.43 total 1149.41 last 59.15) - open/close 18807 (time 1789454460.44 total 1159.42 last 67.34) - open/close 19604 (time 1789454470.45 total 1169.43 last 79.61) - open/close 20000 (time 1789454475.50 total 1174.49 last 78.33) - open/close 20676 (time 1789454485.51 total 1184.49 last 67.58) - open/close 21570 (time 1789454495.51 total 1194.49 last 89.39) - open/close 22176 (time 1789454505.52 total 1204.51 last 60.53) - open/close 22898 (time 1789454515.52 total 1214.51 last 72.17) - open/close 23681 (time 1789454525.54 total 1224.52 last 78.19) - open/close 24546 (time 1789454535.54 total 1234.53 last 86.46) - open/close 25229 (time 1789454545.54 total 1244.53 last 68.29) - open/close 25853 (time 1789454555.56 total 1254.55 last 62.28) - open/close 26636 (time 1789454565.56 total 1264.55 last 78.29) - open/close 27530 (time 1789454575.57 total 1274.56 last 89.34) - open/close 28245 (time 1789454585.57 total 1284.56 last 71.47) - open/close 28912 (time 1789454595.59 total 1294.57 last 66.62) - open/close 29727 (time 1789454605.59 total 1304.58 last 81.47) - open/close 30000 (time 1789454608.60 total 1307.59 last 90.72) - open/close 30677 (time 1789454618.61 total 1317.60 last 67.63) - open/close 31270 (time 1789454628.62 total 1327.61 last 59.24) - open/close 32054 (time 1789454638.63 total 1337.61 last 78.34) - open/close 32797 (time 1789454648.63 total 1347.62 last 74.29) - open/close 33642 (time 1789454658.64 total 1357.63 last 84.38) - open/close 34435 (time 1789454668.65 total 1367.64 last 79.26) - open/close 35181 (time 1789454678.67 total 1377.66 last 74.45) - open/close 36003 (time 1789454688.68 total 1387.66 last 82.13) - open/close 36653 (time 1789454698.69 total 1397.67 last 64.95) - open/close 37304 (time 1789454708.70 total 1407.68 last 65.02) - open/close 38000 (time 1789454718.71 total 1417.70 last 69.48) - open/close 38700 (time 1789454728.72 total 1427.70 last 69.98) - open/close 39439 (time 1789454738.72 total 1437.71 last 73.87) - open/close 40000 (time 1789454747.93 total 1446.91 last 60.94) - open/close 40615 (time 1789454757.93 total 1456.92 last 61.46) - open/close 41362 (time 1789454767.94 total 1466.93 last 74.66) - open/close 42033 (time 1789454777.95 total 1476.94 last 67.01) - open/close 42715 (time 1789454787.96 total 1486.94 last 68.18) - open/close 43399 (time 1789454797.98 total 1496.96 last 68.27) - open/close 43997 (time 1789454807.99 total 1506.97 last 59.73) - open/close 44570 (time 1789454818.01 total 1516.99 last 57.18) - open/close 45261 (time 1789454828.03 total 1527.01 last 68.98) - open/close 45999 (time 1789454838.03 total 1537.02 last 73.76) - open/close 46809 (time 1789454848.03 total 1547.02 last 80.97) - open/close 47664 (time 1789454858.03 total 1557.02 last 85.50) - open/close 48389 (time 1789454868.04 total 1567.02 last 72.48) - open/close 49076 (time 1789454878.05 total 1577.04 last 68.60) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1586.11 seconds: 31.29 ops/second - unlinked 0 (time 1789454891 ; total 0 ; last 0) - unlinked 10000 (time 1789455022 ; total 131 ; last 131) - unlinked 20000 (time 1789455150 ; total 259 ; last 128) - unlinked 30000 (time 1789455289 ; total 398 ; last 139) - unlinked 40000 (time 1789455431 ; total 540 ; last 142) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 665 seconds: 74.633080 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg124-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg124-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg124-server stop mds service on oleg124-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg124-server stop mds service on oleg124-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg124-server unloading modules via unload_modules_local on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing unload_modules_local oleg124-server: kmemleak disabled modules unloaded. pdsh@oleg124-client: oleg124-server: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-client: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-client: ssh exited with exit code 2