== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:57:12 (1788443832) start mds service on oleg358-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing load_modules_local oleg358-server: kmemleak disabled oleg358-server: Loading modules from /home/green/git/lustre-release/lustre oleg358-server: detected 4 online CPUs by sysfs oleg358-server: MODOPTS_LIBCFS= oleg358-server: Force libcfs to create 2 CPU partitions oleg358-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg358-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg358-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg358-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg358-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg358-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-OST0000 oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg358-client.virtnet: -o user_xattr,flock 192.168.203.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 186 (time 1788443946.07 total 10.01 last 18.59) - open/close 289 (time 1788443956.95 total 20.89 last 9.47) - open/close 422 (time 1788443969.60 total 33.54 last 10.51) - open/close 551 (time 1788443979.64 total 43.58 last 12.84) - open/close 746 (time 1788443989.67 total 53.61 last 19.45) - open/close 934 (time 1788443999.68 total 63.62 last 18.78) - open/close 1189 (time 1788444010.24 total 74.18 last 24.14) - open/close 1398 (time 1788444020.24 total 84.18 last 20.89) - open/close 1562 (time 1788444030.24 total 94.18 last 16.40) - open/close 1718 (time 1788444041.05 total 104.99 last 14.44) - open/close 1827 (time 1788444051.42 total 115.36 last 10.51) - open/close 2046 (time 1788444061.43 total 125.37 last 21.87) - open/close 2158 (time 1788444071.90 total 135.84 last 10.70) - open/close 2430 (time 1788444081.91 total 145.85 last 27.17) - open/close 2566 (time 1788444092.84 total 156.78 last 12.44) - open/close 2743 (time 1788444103.24 total 167.18 last 17.03) - open/close 3000 (time 1788444113.52 total 177.46 last 24.99) - open/close 3108 (time 1788444124.51 total 188.45 last 9.83) - open/close 3206 (time 1788444135.04 total 198.98 last 9.30) - open/close 3317 (time 1788444145.88 total 209.83 last 10.24) - open/close 3459 (time 1788444156.15 total 220.09 last 13.84) - open/close 3680 (time 1788444166.96 total 230.90 last 20.45) - open/close 3816 (time 1788444178.02 total 241.96 last 12.29) - open/close 4078 (time 1788444188.65 total 252.59 last 24.65) - open/close 4223 (time 1788444198.81 total 262.75 last 14.27) - open/close 4335 (time 1788444208.83 total 272.78 last 11.17) - open/close 4441 (time 1788444219.49 total 283.44 last 9.94) - open/close 4686 (time 1788444230.38 total 294.32 last 22.51) - open/close 4954 (time 1788444240.55 total 304.49 last 26.34) - open/close 5212 (time 1788444250.56 total 314.50 last 25.79) - open/close 5498 (time 1788444260.58 total 324.52 last 28.54) - open/close 5635 (time 1788444270.59 total 334.53 last 13.69) - open/close 5905 (time 1788444281.47 total 345.41 last 24.81) - open/close 6033 (time 1788444291.47 total 355.42 last 12.79) - open/close 6185 (time 1788444301.73 total 365.67 last 14.83) - open/close 6374 (time 1788444311.73 total 375.67 last 18.89) - open/close 6550 (time 1788444322.80 total 386.74 last 15.91) - open/close 6712 (time 1788444333.24 total 397.18 last 15.52) - open/close 6943 (time 1788444344.15 total 408.09 last 21.16) - open/close 7111 (time 1788444355.05 total 418.99 last 15.42) - open/close 7242 (time 1788444365.05 total 428.99 last 13.09) - open/close 7426 (time 1788444375.96 total 439.90 last 16.86) - open/close 7554 (time 1788444386.09 total 450.03 last 12.64) - open/close 7638 (time 1788444400.41 total 464.35 last 5.87) - open/close 7783 (time 1788444410.75 total 474.69 last 14.02) - open/close 7977 (time 1788444420.76 total 484.70 last 19.38) - open/close 8113 (time 1788444430.77 total 494.71 last 13.59) - open/close 8251 (time 1788444442.92 total 506.86 last 11.36) - open/close 8383 (time 1788444453.85 total 517.79 last 12.07) - open/close 8449 (time 1788444464.86 total 528.80 last 6.00) - open/close 8653 (time 1788444475.45 total 539.39 last 19.27) - open/close 8797 (time 1788444485.81 total 549.75 last 13.89) - open/close 8979 (time 1788444495.87 total 559.81 last 18.10) - open/close 9078 (time 1788444506.34 total 570.28 last 9.46) - open/close 9269 (time 1788444516.35 total 580.29 last 19.07) - open/close 9356 (time 1788444528.54 total 592.48 last 7.14) - open/close 9604 (time 1788444539.56 total 603.50 last 22.50) - open/close 9753 (time 1788444549.59 total 613.53 last 14.86) - open/close 9991 (time 1788444559.78 total 623.73 last 23.34) - open/close 10197 (time 1788444570.86 total 634.80 last 18.59) - open/close 10256 (time 1788444580.89 total 644.83 last 5.88) - open/close 10359 (time 1788444592.88 total 656.82 last 8.59) - open/close 10560 (time 1788444603.71 total 667.65 last 18.56) - open/close 10709 (time 1788444614.19 total 678.13 last 14.22) - open/close 10910 (time 1788444626.39 total 690.33 last 16.48) - open/close 11027 (time 1788444637.23 total 701.17 last 10.79) - open/close 11273 (time 1788444647.34 total 711.28 last 24.32) - open/close 11409 (time 1788444659.35 total 723.30 last 11.32) - open/close 11638 (time 1788444670.31 total 734.25 last 20.90) - open/close 11845 (time 1788444680.33 total 744.27 last 20.67) - open/close 12046 (time 1788444690.60 total 754.54 last 19.56) - open/close 12300 (time 1788444702.25 total 766.19 last 21.81) - open/close 12507 (time 1788444712.25 total 776.19 last 20.70) - open/close 12713 (time 1788444723.23 total 787.17 last 18.76) - open/close 12796 (time 1788444733.59 total 797.53 last 8.01) - open/close 12878 (time 1788444743.90 total 807.85 last 7.95) - open/close 12976 (time 1788444754.54 total 818.48 last 9.21) - open/close 13114 (time 1788444764.90 total 828.84 last 13.32) - open/close 13294 (time 1788444775.35 total 839.29 last 17.23) - open/close 13421 (time 1788444785.39 total 849.33 last 12.65) - open/close 13563 (time 1788444795.61 total 859.55 last 13.89) - open/close 13734 (time 1788444806.24 total 870.18 last 16.08) - open/close 13920 (time 1788444816.25 total 880.19 last 18.58) - open/close 14176 (time 1788444826.27 total 890.21 last 25.56) - open/close 14443 (time 1788444836.29 total 900.23 last 26.65) - open/close 14631 (time 1788444847.22 total 911.16 last 17.20) - open/close 14888 (time 1788444857.23 total 921.17 last 25.68) - open/close 15182 (time 1788444867.25 total 931.19 last 29.35) - open/close 15412 (time 1788444877.26 total 941.20 last 22.96) - open/close 15564 (time 1788444887.28 total 951.22 last 15.18) - open/close 15813 (time 1788444897.40 total 961.34 last 24.60) - open/close 16013 (time 1788444907.41 total 971.35 last 19.99) - open/close 16089 (time 1788444917.79 total 981.74 last 7.32) - open/close 16289 (time 1788444927.85 total 991.79 last 19.89) - open/close 16435 (time 1788444939.45 total 1003.39 last 12.59) - open/close 16609 (time 1788444950.32 total 1014.26 last 16.01) - open/close 16730 (time 1788444961.18 total 1025.12 last 11.14) - open/close 16840 (time 1788444972.07 total 1036.01 last 10.10) - open/close 16930 (time 1788444982.33 total 1046.27 last 8.77) - open/close 17037 (time 1788444992.82 total 1056.76 last 10.20) - open/close 17231 (time 1788445002.84 total 1066.78 last 19.35) - open/close 17926 (time 1788445012.85 total 1076.79 last 69.44) - open/close 18649 (time 1788445022.86 total 1086.80 last 72.24) - open/close 19348 (time 1788445032.87 total 1096.81 last 69.80) - open/close 19820 (time 1788445042.91 total 1106.85 last 47.04) - open/close 20000 (time 1788445046.90 total 1110.84 last 45.09) - open/close 20668 (time 1788445056.91 total 1120.85 last 66.70) - open/close 21325 (time 1788445066.92 total 1130.86 last 65.64) - open/close 22061 (time 1788445076.94 total 1140.88 last 73.51) - open/close 22813 (time 1788445086.95 total 1150.89 last 75.08) - open/close 23535 (time 1788445096.96 total 1160.90 last 72.17) - open/close 24222 (time 1788445106.96 total 1170.90 last 68.69) - open/close 24831 (time 1788445116.98 total 1180.92 last 60.77) - open/close 25435 (time 1788445126.98 total 1190.92 last 60.37) - open/close 26036 (time 1788445136.99 total 1200.93 last 60.07) - open/close 26704 (time 1788445146.99 total 1210.93 last 66.77) - open/close 27403 (time 1788445157.00 total 1220.94 last 69.88) - open/close 28065 (time 1788445167.01 total 1230.95 last 66.12) - open/close 28691 (time 1788445177.02 total 1240.96 last 62.53) - open/close 29297 (time 1788445187.04 total 1250.98 last 60.48) - open/close 29921 (time 1788445197.05 total 1260.99 last 62.32) - open/close 30582 (time 1788445207.07 total 1271.01 last 65.96) - open/close 31280 (time 1788445217.08 total 1281.02 last 69.72) - open/close 31964 (time 1788445227.08 total 1291.02 last 68.39) - open/close 32710 (time 1788445237.10 total 1301.04 last 74.45) - open/close 33447 (time 1788445247.11 total 1311.05 last 73.64) - open/close 34163 (time 1788445257.11 total 1321.05 last 71.60) - open/close 34886 (time 1788445267.12 total 1331.06 last 72.28) - open/close 35592 (time 1788445277.13 total 1341.07 last 70.51) - open/close 36122 (time 1788445287.15 total 1351.09 last 52.89) - open/close 36709 (time 1788445297.17 total 1361.11 last 58.55) - open/close 37299 (time 1788445307.19 total 1371.13 last 58.90) - open/close 38028 (time 1788445317.19 total 1381.13 last 72.87) - open/close 38748 (time 1788445327.20 total 1391.14 last 71.96) - open/close 39480 (time 1788445337.23 total 1401.17 last 73.00) - open/close 40000 (time 1788445344.81 total 1408.75 last 68.52) - open/close 40725 (time 1788445354.84 total 1418.78 last 72.32) - open/close 41407 (time 1788445364.85 total 1428.79 last 68.12) - open/close 41981 (time 1788445374.85 total 1438.79 last 57.39) - open/close 42673 (time 1788445384.85 total 1448.79 last 69.18) - open/close 43371 (time 1788445394.86 total 1458.80 last 69.79) - open/close 44105 (time 1788445404.86 total 1468.80 last 73.40) - open/close 44742 (time 1788445414.86 total 1478.80 last 63.69) - open/close 45534 (time 1788445424.86 total 1488.80 last 79.19) - open/close 46112 (time 1788445434.87 total 1498.81 last 57.74) - open/close 46768 (time 1788445444.87 total 1508.81 last 65.59) - open/close 47443 (time 1788445454.87 total 1518.81 last 67.49) - open/close 48100 (time 1788445464.88 total 1528.82 last 65.68) - open/close 48759 (time 1788445474.90 total 1538.84 last 65.74) - open/close 49422 (time 1788445484.91 total 1548.85 last 66.24) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1552.27 seconds: 31.97 ops/second - unlinked 0 (time 1788445494 ; total 1 ; last 1) - unlinked 10000 (time 1788445650 ; total 157 ; last 156) - unlinked 20000 (time 1788445798 ; total 305 ; last 148) - unlinked 30000 (time 1788445946 ; total 453 ; last 148) - unlinked 40000 (time 1788446094 ; total 601 ; last 148) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 739 seconds: 67.159676 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg358-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg358-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg358-server unloading modules via unload_modules_local on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing unload_modules_local oleg358-server: kmemleak disabled modules unloaded. pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2