== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:37:55 (1788583075) start mds service on oleg107-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1754936 Sep 4 08:27 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing load_modules_local oleg107-server: kmemleak disabled oleg107-server: Loading modules from /home/green/git/lustre-release/lustre oleg107-server: detected 4 online CPUs by sysfs oleg107-server: MODOPTS_LIBCFS= oleg107-server: Force libcfs to create 2 CPU partitions oleg107-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg107-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg107-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg107-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg107-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock 192.168.201.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 135 (time 1788583158.15 total 10.36 last 13.03) - open/close 337 (time 1788583168.15 total 20.36 last 20.19) - open/close 518 (time 1788583178.85 total 31.06 last 16.92) - open/close 714 (time 1788583190.01 total 42.22 last 17.57) - open/close 856 (time 1788583200.88 total 53.09 last 13.07) - open/close 1074 (time 1788583211.95 total 64.16 last 19.68) - open/close 1237 (time 1788583221.95 total 74.16 last 16.30) - open/close 1317 (time 1788583232.24 total 84.45 last 7.78) - open/close 1444 (time 1788583242.91 total 95.12 last 11.91) - open/close 1611 (time 1788583253.00 total 105.21 last 16.55) - open/close 1641 (time 1788583263.21 total 115.42 last 2.94) - open/close 1838 (time 1788583273.22 total 125.43 last 19.68) - open/close 2087 (time 1788583284.30 total 136.51 last 22.48) - open/close 2330 (time 1788583295.21 total 147.42 last 22.27) - open/close 2454 (time 1788583305.26 total 157.47 last 12.34) - open/close 2648 (time 1788583315.75 total 167.96 last 18.50) - open/close 2786 (time 1788583325.75 total 177.96 last 13.80) - open/close 2889 (time 1788583336.22 total 188.43 last 9.83) - open/close 3143 (time 1788583346.23 total 198.44 last 25.38) - open/close 3338 (time 1788583357.55 total 209.76 last 17.22) - open/close 3547 (time 1788583368.30 total 220.51 last 19.45) - open/close 3760 (time 1788583378.70 total 230.91 last 20.48) - open/close 3925 (time 1788583390.13 total 242.34 last 14.44) - open/close 4053 (time 1788583400.13 total 252.34 last 12.79) - open/close 4164 (time 1788583410.86 total 263.07 last 10.34) - open/close 4218 (time 1788583421.58 total 273.79 last 5.04) - open/close 4377 (time 1788583432.63 total 284.84 last 14.39) - open/close 4544 (time 1788583443.19 total 295.40 last 15.82) - open/close 4762 (time 1788583453.47 total 305.68 last 21.20) - open/close 4952 (time 1788583463.68 total 315.89 last 18.62) - open/close 5130 (time 1788583474.29 total 326.50 last 16.78) - open/close 5168 (time 1788583485.12 total 337.33 last 3.51) - open/close 5406 (time 1788583495.23 total 347.44 last 23.54) - open/close 5503 (time 1788583505.53 total 357.74 last 9.42) - open/close 5738 (time 1788583516.52 total 368.73 last 21.38) - open/close 5972 (time 1788583527.53 total 379.74 last 21.24) - open/close 6189 (time 1788583537.59 total 389.80 last 21.59) - open/close 6331 (time 1788583548.22 total 400.43 last 13.36) - open/close 6522 (time 1788583558.71 total 410.92 last 18.20) - open/close 6702 (time 1788583568.73 total 420.94 last 17.96) - open/close 6875 (time 1788583579.69 total 431.90 last 15.79) - open/close 7009 (time 1788583589.94 total 442.15 last 13.08) - open/close 7221 (time 1788583602.75 total 454.96 last 16.55) - open/close 7478 (time 1788583612.79 total 465.00 last 25.61) - open/close 7613 (time 1788583623.03 total 475.24 last 13.17) - open/close 7783 (time 1788583633.09 total 485.30 last 16.90) - open/close 7881 (time 1788583643.09 total 495.30 last 9.80) - open/close 7966 (time 1788583653.48 total 505.69 last 8.18) - open/close 8145 (time 1788583664.37 total 516.58 last 16.43) - open/close 8291 (time 1788583674.38 total 526.59 last 14.59) - open/close 8360 (time 1788583686.73 total 538.94 last 5.59) - open/close 8425 (time 1788583697.40 total 549.61 last 6.09) - open/close 8659 (time 1788583707.45 total 559.66 last 23.29) - open/close 8785 (time 1788583717.47 total 569.68 last 12.57) - open/close 8880 (time 1788583727.94 total 580.15 last 9.08) - open/close 9057 (time 1788583738.30 total 590.51 last 17.08) - open/close 9224 (time 1788583748.69 total 600.90 last 16.07) - open/close 9444 (time 1788583759.03 total 611.24 last 21.28) - open/close 9681 (time 1788583769.64 total 621.85 last 22.34) - open/close 9944 (time 1788583779.88 total 632.09 last 25.68) - open/close 10000 (time 1788583782.10 total 634.31 last 25.20) - open/close 10187 (time 1788583793.18 total 645.39 last 16.89) - open/close 10413 (time 1788583803.90 total 656.11 last 21.09) - open/close 10604 (time 1788583813.90 total 666.11 last 19.10) - open/close 10793 (time 1788583824.67 total 676.88 last 17.54) - open/close 10991 (time 1788583834.92 total 687.13 last 19.31) - open/close 11223 (time 1788583844.93 total 697.14 last 23.18) - open/close 11364 (time 1788583856.10 total 708.32 last 12.62) - open/close 11583 (time 1788583866.19 total 718.40 last 21.72) - open/close 11831 (time 1788583876.20 total 728.41 last 24.77) - open/close 12047 (time 1788583886.20 total 738.41 last 21.59) - open/close 12191 (time 1788583896.29 total 748.50 last 14.27) - open/close 12437 (time 1788583906.30 total 758.51 last 24.57) - open/close 12665 (time 1788583917.00 total 769.21 last 21.31) - open/close 12911 (time 1788583927.00 total 779.21 last 24.60) - open/close 13101 (time 1788583937.08 total 789.29 last 18.84) - open/close 13273 (time 1788583947.60 total 799.81 last 16.35) - open/close 13376 (time 1788583957.63 total 809.84 last 10.27) - open/close 13445 (time 1788583967.72 total 819.93 last 6.84) - open/close 13592 (time 1788583978.28 total 830.49 last 13.92) - open/close 13829 (time 1788583988.73 total 840.94 last 22.68) - open/close 14015 (time 1788583998.80 total 851.01 last 18.47) - open/close 14219 (time 1788584009.28 total 861.49 last 19.47) - open/close 14390 (time 1788584019.33 total 871.54 last 17.02) - open/close 14576 (time 1788584029.36 total 881.57 last 18.55) - open/close 14718 (time 1788584039.36 total 891.57 last 14.19) - open/close 14902 (time 1788584049.60 total 901.81 last 17.97) - open/close 15164 (time 1788584059.62 total 911.83 last 26.14) - open/close 15421 (time 1788584069.63 total 921.84 last 25.69) - open/close 15550 (time 1788584079.68 total 931.89 last 12.83) - open/close 15842 (time 1788584090.50 total 942.71 last 27.00) - open/close 15993 (time 1788584100.65 total 952.86 last 14.87) - open/close 16187 (time 1788584110.78 total 962.99 last 19.14) - open/close 16430 (time 1788584121.62 total 973.83 last 22.43) - open/close 16630 (time 1788584132.16 total 984.37 last 18.97) - open/close 16814 (time 1788584142.52 total 994.73 last 17.76) - open/close 17042 (time 1788584152.70 total 1004.91 last 22.40) - open/close 17143 (time 1788584162.72 total 1014.93 last 10.08) - open/close 17296 (time 1788584173.63 total 1025.84 last 14.02) - open/close 17865 (time 1788584183.64 total 1035.85 last 56.86) - open/close 18637 (time 1788584193.64 total 1045.85 last 77.18) - open/close 19337 (time 1788584203.64 total 1055.86 last 69.95) - open/close 19907 (time 1788584213.65 total 1065.86 last 56.98) - open/close 20588 (time 1788584223.66 total 1075.87 last 68.05) - open/close 21330 (time 1788584233.66 total 1085.87 last 74.18) - open/close 21960 (time 1788584243.67 total 1095.88 last 62.95) - open/close 22641 (time 1788584253.68 total 1105.89 last 68.01) - open/close 23283 (time 1788584263.69 total 1115.90 last 64.15) - open/close 23958 (time 1788584273.69 total 1125.90 last 67.47) - open/close 24605 (time 1788584283.70 total 1135.91 last 64.67) - open/close 25377 (time 1788584293.70 total 1145.91 last 77.15) - open/close 26015 (time 1788584303.75 total 1155.96 last 63.53) - open/close 26699 (time 1788584313.75 total 1165.96 last 68.39) - open/close 27281 (time 1788584323.75 total 1175.96 last 58.20) - open/close 27864 (time 1788584333.76 total 1185.97 last 58.20) - open/close 28532 (time 1788584343.76 total 1195.97 last 66.80) - open/close 29204 (time 1788584353.77 total 1205.98 last 67.16) - open/close 30000 (time 1788584363.46 total 1215.67 last 82.16) - open/close 30696 (time 1788584373.47 total 1225.68 last 69.56) - open/close 31388 (time 1788584383.47 total 1235.68 last 69.15) - open/close 32128 (time 1788584393.48 total 1245.69 last 73.98) - open/close 32826 (time 1788584403.49 total 1255.70 last 69.73) - open/close 33516 (time 1788584413.49 total 1265.70 last 68.97) - open/close 34223 (time 1788584423.50 total 1275.71 last 70.67) - open/close 34966 (time 1788584433.51 total 1285.72 last 74.23) - open/close 35666 (time 1788584443.51 total 1295.72 last 69.98) - open/close 36184 (time 1788584453.52 total 1305.73 last 51.76) - open/close 36622 (time 1788584463.53 total 1315.74 last 43.76) - open/close 37105 (time 1788584473.54 total 1325.75 last 48.24) - open/close 37612 (time 1788584483.55 total 1335.76 last 50.62) - open/close 38184 (time 1788584493.56 total 1345.77 last 57.18) - open/close 38763 (time 1788584503.57 total 1355.78 last 57.85) - open/close 39293 (time 1788584513.61 total 1365.82 last 52.76) - open/close 39943 (time 1788584523.63 total 1375.84 last 64.88) - open/close 40491 (time 1788584533.64 total 1385.85 last 54.73) - open/close 41154 (time 1788584543.66 total 1395.87 last 66.19) - open/close 41789 (time 1788584553.67 total 1405.88 last 63.46) - open/close 42424 (time 1788584563.69 total 1415.90 last 63.36) - open/close 42950 (time 1788584573.69 total 1425.90 last 52.59) - open/close 43570 (time 1788584583.70 total 1435.91 last 61.94) - open/close 44170 (time 1788584593.71 total 1445.92 last 59.97) - open/close 44768 (time 1788584603.71 total 1455.92 last 59.77) - open/close 45308 (time 1788584613.72 total 1465.93 last 53.96) - open/close 45999 (time 1788584623.73 total 1475.94 last 69.02) - open/close 46734 (time 1788584633.73 total 1485.94 last 73.48) - open/close 47434 (time 1788584643.74 total 1495.95 last 69.95) - open/close 48077 (time 1788584653.75 total 1505.96 last 64.21) - open/close 48684 (time 1788584663.76 total 1515.97 last 60.66) - open/close 49314 (time 1788584673.77 total 1525.98 last 62.90) - open/close 49851 (time 1788584683.79 total 1536.00 last 53.64) total: 50000 open/close in 1538.68 seconds: 32.50 ops/second - unlinked 0 (time 1788584692 ; total 0 ; last 0) - unlinked 10000 (time 1788584854 ; total 162 ; last 162) - unlinked 20000 (time 1788585029 ; total 337 ; last 175) - unlinked 30000 (time 1788585189 ; total 497 ; last 160) - unlinked 40000 (time 1788585336 ; total 644 ; last 147) total: 50000 unlinks in 803 seconds: 62.266502 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server unloading modules via unload_modules_local on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing unload_modules_local oleg107-server: kmemleak disabled modules unloaded. pdsh@oleg107-client: oleg107-client: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-client: ssh exited with exit code 2