== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:17:54 (1788265074) start mds service on oleg107-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1756096 Sep 1 06:05 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing load_modules_local oleg107-server: kmemleak disabled oleg107-server: Loading modules from /home/green/git/lustre-release/lustre oleg107-server: detected 4 online CPUs by sysfs oleg107-server: MODOPTS_LIBCFS= oleg107-server: Force libcfs to create 2 CPU partitions oleg107-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg107-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg107-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg107-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg107-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock 192.168.201.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 178 (time 1788265167.91 total 10.01 last 17.77) - open/close 325 (time 1788265178.65 total 20.76 last 13.69) - open/close 449 (time 1788265189.77 total 31.87 last 11.16) - open/close 530 (time 1788265200.10 total 42.21 last 7.84) - open/close 578 (time 1788265210.79 total 52.90 last 4.49) - open/close 705 (time 1788265220.81 total 62.91 last 12.68) - open/close 909 (time 1788265231.54 total 73.64 last 19.02) - open/close 1060 (time 1788265241.72 total 83.82 last 14.83) - open/close 1212 (time 1788265251.81 total 93.91 last 15.07) - open/close 1426 (time 1788265262.06 total 104.16 last 20.87) - open/close 1584 (time 1788265272.95 total 115.05 last 14.51) - open/close 1642 (time 1788265283.43 total 125.53 last 5.54) - open/close 1747 (time 1788265293.44 total 135.54 last 10.49) - open/close 1814 (time 1788265303.47 total 145.57 last 6.68) - open/close 1968 (time 1788265316.09 total 158.19 last 12.20) - open/close 2082 (time 1788265326.51 total 168.61 last 10.94) - open/close 2266 (time 1788265336.56 total 178.67 last 18.30) - open/close 2428 (time 1788265346.85 total 188.96 last 15.74) - open/close 2547 (time 1788265356.87 total 198.97 last 11.88) - open/close 2708 (time 1788265367.59 total 209.69 last 15.03) - open/close 2887 (time 1788265377.59 total 219.69 last 17.89) - open/close 3060 (time 1788265388.52 total 230.62 last 15.83) - open/close 3239 (time 1788265398.64 total 240.74 last 17.70) - open/close 3398 (time 1788265408.68 total 250.78 last 15.84) - open/close 3517 (time 1788265418.74 total 260.85 last 11.82) - open/close 3682 (time 1788265429.48 total 271.58 last 15.37) - open/close 3861 (time 1788265439.79 total 281.89 last 17.37) - open/close 3975 (time 1788265450.68 total 292.78 last 10.46) - open/close 4137 (time 1788265461.73 total 303.83 last 14.67) - open/close 4317 (time 1788265471.73 total 313.83 last 17.99) - open/close 4395 (time 1788265482.98 total 325.08 last 6.94) - open/close 4461 (time 1788265493.01 total 335.11 last 6.58) - open/close 4601 (time 1788265503.19 total 345.30 last 13.74) - open/close 4770 (time 1788265513.20 total 355.30 last 16.90) - open/close 4820 (time 1788265523.96 total 366.06 last 4.64) - open/close 5086 (time 1788265535.03 total 377.13 last 24.03) - open/close 5260 (time 1788265545.03 total 387.13 last 17.40) - open/close 5388 (time 1788265555.11 total 397.21 last 12.70) - open/close 5513 (time 1788265565.35 total 407.46 last 12.20) - open/close 5634 (time 1788265575.47 total 417.57 last 11.96) - open/close 5706 (time 1788265586.55 total 428.65 last 6.50) - open/close 5905 (time 1788265596.55 total 438.66 last 19.90) - open/close 6041 (time 1788265606.56 total 448.66 last 13.60) - open/close 6107 (time 1788265616.69 total 458.79 last 6.51) - open/close 6285 (time 1788265627.40 total 469.50 last 16.63) - open/close 6455 (time 1788265638.18 total 480.29 last 15.76) - open/close 6651 (time 1788265648.20 total 490.30 last 19.58) - open/close 6763 (time 1788265659.26 total 501.36 last 10.12) - open/close 7000 (time 1788265670.06 total 512.16 last 21.94) - open/close 7120 (time 1788265680.11 total 522.21 last 11.94) - open/close 7209 (time 1788265690.54 total 532.64 last 8.53) - open/close 7258 (time 1788265703.56 total 545.66 last 3.76) - open/close 7377 (time 1788265713.57 total 555.67 last 11.89) - open/close 7461 (time 1788265723.58 total 565.68 last 8.39) - open/close 7573 (time 1788265733.58 total 575.68 last 11.20) - open/close 7756 (time 1788265743.59 total 585.69 last 18.28) - open/close 7907 (time 1788265754.21 total 596.31 last 14.22) - open/close 8120 (time 1788265765.15 total 607.25 last 19.47) - open/close 8305 (time 1788265775.17 total 617.27 last 18.45) - open/close 8441 (time 1788265785.36 total 627.46 last 13.36) - open/close 8556 (time 1788265795.36 total 637.46 last 11.49) - open/close 8672 (time 1788265806.40 total 648.50 last 10.51) - open/close 8881 (time 1788265816.40 total 658.50 last 20.89) - open/close 9115 (time 1788265827.26 total 669.36 last 21.55) - open/close 9284 (time 1788265837.57 total 679.67 last 16.40) - open/close 9466 (time 1788265847.89 total 689.99 last 17.63) - open/close 9532 (time 1788265858.11 total 700.21 last 6.46) - open/close 9712 (time 1788265868.13 total 710.23 last 17.96) - open/close 9777 (time 1788265878.13 total 720.24 last 6.50) - open/close 10000 (time 1788265887.16 total 729.26 last 24.71) - open/close 10180 (time 1788265897.16 total 739.26 last 18.00) - open/close 10427 (time 1788265907.18 total 749.28 last 24.65) - open/close 10631 (time 1788265917.20 total 759.30 last 20.36) - open/close 10842 (time 1788265927.53 total 769.64 last 20.42) - open/close 10963 (time 1788265937.54 total 779.64 last 12.10) - open/close 11186 (time 1788265948.16 total 790.26 last 21.00) - open/close 11380 (time 1788265958.17 total 800.27 last 19.38) - open/close 11527 (time 1788265968.17 total 810.27 last 14.69) - open/close 11697 (time 1788265978.18 total 820.28 last 16.98) - open/close 11939 (time 1788265988.91 total 831.01 last 22.57) - open/close 12195 (time 1788265999.02 total 841.12 last 25.32) - open/close 12450 (time 1788266009.02 total 851.12 last 25.49) - open/close 12667 (time 1788266019.03 total 861.13 last 21.69) - open/close 12852 (time 1788266029.04 total 871.14 last 18.47) - open/close 13117 (time 1788266039.05 total 881.15 last 26.49) - open/close 13343 (time 1788266049.06 total 891.16 last 22.58) - open/close 13572 (time 1788266059.08 total 901.18 last 22.84) - open/close 13756 (time 1788266069.12 total 911.22 last 18.34) - open/close 13938 (time 1788266079.41 total 921.51 last 17.68) - open/close 14240 (time 1788266089.42 total 931.52 last 30.17) - open/close 14452 (time 1788266099.60 total 941.70 last 20.82) - open/close 14659 (time 1788266110.34 total 952.44 last 19.29) - open/close 14781 (time 1788266120.34 total 962.44 last 12.19) - open/close 14996 (time 1788266131.29 total 973.39 last 19.65) - open/close 15183 (time 1788266141.29 total 983.39 last 18.69) - open/close 15294 (time 1788266151.86 total 993.96 last 10.50) - open/close 15399 (time 1788266161.89 total 1003.99 last 10.46) - open/close 15546 (time 1788266172.20 total 1014.30 last 14.26) - open/close 15768 (time 1788266182.68 total 1024.78 last 21.18) - open/close 15964 (time 1788266192.69 total 1034.79 last 19.58) - open/close 16144 (time 1788266203.55 total 1045.65 last 16.57) - open/close 16396 (time 1788266213.56 total 1055.66 last 25.18) - open/close 16599 (time 1788266224.25 total 1066.35 last 18.99) - open/close 16861 (time 1788266234.25 total 1076.35 last 26.19) - open/close 17457 (time 1788266244.26 total 1086.36 last 59.52) - open/close 18100 (time 1788266254.27 total 1096.37 last 64.24) - open/close 18576 (time 1788266264.27 total 1106.38 last 47.59) - open/close 19098 (time 1788266274.28 total 1116.38 last 52.16) - open/close 19689 (time 1788266284.30 total 1126.40 last 59.01) - open/close 20000 (time 1788266288.48 total 1130.58 last 74.43) - open/close 20588 (time 1788266298.51 total 1140.61 last 58.61) - open/close 21202 (time 1788266308.51 total 1150.61 last 61.40) - open/close 21896 (time 1788266318.52 total 1160.62 last 69.34) - open/close 22536 (time 1788266328.52 total 1170.63 last 63.96) - open/close 23247 (time 1788266338.53 total 1180.63 last 71.07) - open/close 23903 (time 1788266348.53 total 1190.63 last 65.59) - open/close 24648 (time 1788266358.54 total 1200.64 last 74.43) - open/close 25300 (time 1788266368.55 total 1210.65 last 65.14) - open/close 26018 (time 1788266378.56 total 1220.66 last 71.73) - open/close 26796 (time 1788266388.56 total 1230.66 last 77.78) - open/close 27481 (time 1788266398.57 total 1240.67 last 68.47) - open/close 28159 (time 1788266408.57 total 1250.67 last 67.75) - open/close 28817 (time 1788266418.58 total 1260.68 last 65.73) - open/close 29411 (time 1788266428.60 total 1270.70 last 59.30) - open/close 30000 (time 1788266438.02 total 1280.12 last 62.51) - open/close 30721 (time 1788266448.03 total 1290.14 last 72.02) - open/close 31469 (time 1788266458.05 total 1300.15 last 74.70) - open/close 32210 (time 1788266468.05 total 1310.16 last 74.06) - open/close 32946 (time 1788266478.06 total 1320.16 last 73.57) - open/close 33620 (time 1788266488.06 total 1330.16 last 67.38) - open/close 34377 (time 1788266498.08 total 1340.18 last 75.60) - open/close 35048 (time 1788266508.09 total 1350.19 last 67.03) - open/close 35705 (time 1788266518.09 total 1360.19 last 65.68) - open/close 36445 (time 1788266528.09 total 1370.20 last 73.95) - open/close 37222 (time 1788266538.12 total 1380.22 last 77.49) - open/close 38093 (time 1788266548.15 total 1390.25 last 86.83) - open/close 38887 (time 1788266558.15 total 1400.26 last 79.38) - open/close 39689 (time 1788266568.16 total 1410.26 last 80.18) - open/close 40000 (time 1788266572.37 total 1414.47 last 73.90) - open/close 40752 (time 1788266582.37 total 1424.47 last 75.19) - open/close 41516 (time 1788266592.37 total 1434.48 last 76.35) - open/close 42293 (time 1788266602.39 total 1444.49 last 77.57) - open/close 43065 (time 1788266612.39 total 1454.50 last 77.18) - open/close 43684 (time 1788266622.40 total 1464.50 last 61.86) - open/close 44446 (time 1788266632.40 total 1474.51 last 76.17) - open/close 45266 (time 1788266642.41 total 1484.51 last 81.98) - open/close 45979 (time 1788266652.41 total 1494.51 last 71.29) - open/close 46814 (time 1788266662.41 total 1504.51 last 83.49) - open/close 47675 (time 1788266672.41 total 1514.51 last 86.06) - open/close 48551 (time 1788266682.42 total 1524.52 last 87.56) - open/close 49360 (time 1788266692.42 total 1534.52 last 80.89) total: 50000 open/close in 1542.92 seconds: 32.41 ops/second - unlinked 0 (time 1788266704 ; total 0 ; last 0) - unlinked 10000 (time 1788266834 ; total 130 ; last 130) - unlinked 20000 (time 1788266955 ; total 251 ; last 121) - unlinked 30000 (time 1788267082 ; total 378 ; last 127) - unlinked 40000 (time 1788267204 ; total 500 ; last 122) total: 50000 unlinks in 624 seconds: 80.128204 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server unloading modules via unload_modules_local on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing unload_modules_local oleg107-server: kmemleak disabled modules unloaded. pdsh@oleg107-client: oleg107-client: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-client: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2