== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:08:45 (1788440925) start mds service on oleg431-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing load_modules_local oleg431-server: kmemleak disabled oleg431-server: Loading modules from /home/green/git/lustre-release/lustre oleg431-server: detected 4 online CPUs by sysfs oleg431-server: MODOPTS_LIBCFS= oleg431-server: Force libcfs to create 2 CPU partitions oleg431-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg431-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg431-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg431-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg431-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg431-server: oleg431-server.virtnet: executing set_default_debug -1 all pdsh@oleg431-client: oleg431-server: ssh exited with exit code 1 Started lustre-OST0000 oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg431-client.virtnet: -o user_xattr,flock 192.168.204.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 175 (time 1788441021.07 total 10.11 last 17.30) - open/close 322 (time 1788441031.99 total 21.04 last 13.46) - open/close 446 (time 1788441041.99 total 31.04 last 12.40) - open/close 553 (time 1788441054.58 total 43.62 last 8.50) - open/close 780 (time 1788441065.47 total 54.52 last 20.83) - open/close 1012 (time 1788441076.53 total 65.58 last 20.98) - open/close 1162 (time 1788441086.97 total 76.02 last 14.36) - open/close 1374 (time 1788441096.99 total 86.04 last 21.17) - open/close 1598 (time 1788441107.00 total 96.05 last 22.38) - open/close 1734 (time 1788441117.02 total 106.07 last 13.57) - open/close 1927 (time 1788441132.48 total 121.53 last 12.48) - open/close 2133 (time 1788441142.94 total 131.99 last 19.69) - open/close 2252 (time 1788441152.94 total 141.99 last 11.90) - open/close 2434 (time 1788441163.86 total 152.91 last 16.66) - open/close 2619 (time 1788441174.47 total 163.52 last 17.45) - open/close 2856 (time 1788441184.47 total 173.52 last 23.68) - open/close 3112 (time 1788441194.47 total 183.52 last 25.60) - open/close 3216 (time 1788441205.94 total 194.99 last 9.07) - open/close 3418 (time 1788441216.72 total 205.76 last 18.75) - open/close 3549 (time 1788441226.75 total 215.79 last 13.06) - open/close 3722 (time 1788441236.76 total 225.81 last 17.28) - open/close 3837 (time 1788441246.76 total 235.81 last 11.50) - open/close 3996 (time 1788441256.77 total 245.82 last 15.88) - open/close 4148 (time 1788441266.94 total 255.99 last 14.95) - open/close 4264 (time 1788441277.08 total 266.13 last 11.44) - open/close 4430 (time 1788441287.81 total 276.86 last 15.48) - open/close 4497 (time 1788441298.14 total 287.19 last 6.48) - open/close 4622 (time 1788441308.89 total 297.93 last 11.63) - open/close 4723 (time 1788441319.60 total 308.65 last 9.43) - open/close 4833 (time 1788441329.60 total 318.65 last 11.00) - open/close 4910 (time 1788441339.60 total 328.65 last 7.70) - open/close 5040 (time 1788441351.94 total 340.99 last 10.54) - open/close 5238 (time 1788441362.81 total 351.85 last 18.22) - open/close 5285 (time 1788441373.56 total 362.61 last 4.37) - open/close 5463 (time 1788441385.71 total 374.76 last 14.65) - open/close 5540 (time 1788441395.73 total 384.78 last 7.69) - open/close 5619 (time 1788441405.95 total 395.00 last 7.73) - open/close 5731 (time 1788441415.96 total 405.01 last 11.19) - open/close 5837 (time 1788441426.48 total 415.53 last 10.08) - open/close 5954 (time 1788441436.49 total 425.54 last 11.68) - open/close 5998 (time 1788441447.04 total 436.09 last 4.17) - open/close 6187 (time 1788441457.05 total 446.10 last 18.88) - open/close 6249 (time 1788441468.00 total 457.05 last 5.66) - open/close 6473 (time 1788441478.40 total 467.45 last 21.54) - open/close 6657 (time 1788441490.05 total 479.10 last 15.79) - open/close 6738 (time 1788441500.49 total 489.54 last 7.76) - open/close 6905 (time 1788441511.55 total 500.60 last 15.10) - open/close 7027 (time 1788441521.86 total 510.91 last 11.83) - open/close 7226 (time 1788441532.60 total 521.65 last 18.52) - open/close 7452 (time 1788441543.53 total 532.58 last 20.68) - open/close 7693 (time 1788441553.54 total 542.59 last 24.07) - open/close 7769 (time 1788441563.65 total 552.70 last 7.52) - open/close 7931 (time 1788441574.40 total 563.45 last 15.07) - open/close 8090 (time 1788441585.45 total 574.50 last 14.39) - open/close 8195 (time 1788441596.15 total 585.20 last 9.81) - open/close 8307 (time 1788441607.16 total 596.21 last 10.17) - open/close 8415 (time 1788441617.79 total 606.83 last 10.17) - open/close 8555 (time 1788441627.80 total 616.85 last 13.97) - open/close 8733 (time 1788441638.79 total 627.84 last 16.20) - open/close 8983 (time 1788441648.82 total 637.87 last 24.93) - open/close 9212 (time 1788441659.71 total 648.76 last 21.02) - open/close 9338 (time 1788441669.72 total 658.76 last 12.60) - open/close 9479 (time 1788441680.52 total 669.57 last 13.05) - open/close 9560 (time 1788441694.01 total 683.05 last 6.01) - open/close 9731 (time 1788441704.07 total 693.12 last 16.98) - open/close 9901 (time 1788441716.41 total 705.45 last 13.79) - open/close 10000 (time 1788441720.64 total 709.69 last 23.38) - open/close 10143 (time 1788441731.38 total 720.43 last 13.32) - open/close 10268 (time 1788441741.51 total 730.55 last 12.34) - open/close 10372 (time 1788441754.21 total 743.26 last 8.19) - open/close 10503 (time 1788441764.24 total 753.29 last 13.06) - open/close 10637 (time 1788441775.11 total 764.16 last 12.33) - open/close 10787 (time 1788441785.61 total 774.65 last 14.29) - open/close 10963 (time 1788441796.24 total 785.29 last 16.55) - open/close 11240 (time 1788441806.25 total 795.29 last 27.68) - open/close 11449 (time 1788441816.76 total 805.81 last 19.87) - open/close 11646 (time 1788441827.52 total 816.57 last 18.31) - open/close 11815 (time 1788441838.27 total 827.32 last 15.73) - open/close 12065 (time 1788441848.27 total 837.32 last 25.00) - open/close 12293 (time 1788441858.30 total 847.35 last 22.73) - open/close 12573 (time 1788441869.19 total 858.23 last 25.72) - open/close 12673 (time 1788441879.35 total 868.40 last 9.83) - open/close 12854 (time 1788441889.59 total 878.63 last 17.69) - open/close 13117 (time 1788441900.28 total 889.33 last 24.60) - open/close 13312 (time 1788441910.30 total 899.35 last 19.47) - open/close 13482 (time 1788441921.22 total 910.27 last 15.57) - open/close 13645 (time 1788441931.22 total 920.27 last 16.29) - open/close 13843 (time 1788441941.50 total 930.55 last 19.26) - open/close 13943 (time 1788441952.18 total 941.23 last 9.37) - open/close 14069 (time 1788441962.64 total 951.69 last 12.05) - open/close 14215 (time 1788441973.42 total 962.47 last 13.54) - open/close 14329 (time 1788441986.24 total 975.29 last 8.89) - open/close 14440 (time 1788441996.26 total 985.31 last 11.08) - open/close 14629 (time 1788442006.28 total 995.32 last 18.86) - open/close 14783 (time 1788442017.87 total 1006.92 last 13.28) - open/close 14962 (time 1788442027.90 total 1016.95 last 17.84) - open/close 15178 (time 1788442037.91 total 1026.96 last 21.59) - open/close 15421 (time 1788442048.11 total 1037.16 last 23.82) - open/close 15532 (time 1788442059.16 total 1048.21 last 10.05) - open/close 15691 (time 1788442069.16 total 1058.21 last 15.90) - open/close 16242 (time 1788442079.19 total 1068.24 last 54.97) - open/close 16861 (time 1788442089.19 total 1078.24 last 61.88) - open/close 17478 (time 1788442099.20 total 1088.24 last 61.67) - open/close 18077 (time 1788442109.20 total 1098.25 last 59.85) - open/close 18703 (time 1788442119.21 total 1108.25 last 62.59) - open/close 19337 (time 1788442129.22 total 1118.27 last 63.29) - open/close 19929 (time 1788442139.23 total 1128.28 last 59.15) - open/close 20556 (time 1788442149.24 total 1138.29 last 62.62) - open/close 21122 (time 1788442159.25 total 1148.30 last 56.56) - open/close 21706 (time 1788442169.26 total 1158.31 last 58.35) - open/close 22246 (time 1788442179.29 total 1168.34 last 53.82) - open/close 22894 (time 1788442189.30 total 1178.35 last 64.74) - open/close 23532 (time 1788442199.31 total 1188.35 last 63.79) - open/close 24144 (time 1788442209.31 total 1198.36 last 61.16) - open/close 24751 (time 1788442219.32 total 1208.37 last 60.66) - open/close 25323 (time 1788442229.33 total 1218.38 last 57.16) - open/close 25918 (time 1788442239.34 total 1228.38 last 59.45) - open/close 26469 (time 1788442249.34 total 1238.39 last 55.06) - open/close 27044 (time 1788442259.35 total 1248.40 last 57.44) - open/close 27628 (time 1788442269.36 total 1258.41 last 58.36) - open/close 28240 (time 1788442279.37 total 1268.42 last 61.13) - open/close 28820 (time 1788442289.38 total 1278.43 last 57.96) - open/close 29379 (time 1788442299.43 total 1288.48 last 55.63) - open/close 29932 (time 1788442309.43 total 1298.48 last 55.26) - open/close 30511 (time 1788442319.44 total 1308.49 last 57.88) - open/close 31136 (time 1788442329.47 total 1318.52 last 62.32) - open/close 31678 (time 1788442339.48 total 1328.52 last 54.15) - open/close 32219 (time 1788442349.52 total 1338.56 last 53.88) - open/close 32748 (time 1788442359.53 total 1348.58 last 52.83) - open/close 33202 (time 1788442369.53 total 1358.58 last 45.39) - open/close 33766 (time 1788442379.53 total 1368.58 last 56.38) - open/close 34317 (time 1788442389.54 total 1378.59 last 55.07) - open/close 34928 (time 1788442399.56 total 1388.60 last 61.00) - open/close 35538 (time 1788442409.56 total 1398.61 last 60.95) - open/close 36168 (time 1788442419.57 total 1408.61 last 62.99) - open/close 36770 (time 1788442429.57 total 1418.62 last 60.18) - open/close 37378 (time 1788442439.58 total 1428.62 last 60.75) - open/close 38000 (time 1788442449.59 total 1438.64 last 62.13) - open/close 38610 (time 1788442459.59 total 1448.64 last 60.98) - open/close 39252 (time 1788442469.61 total 1458.66 last 64.06) - open/close 39859 (time 1788442479.62 total 1468.67 last 60.67) - open/close 40000 (time 1788442481.98 total 1471.03 last 59.78) - open/close 40613 (time 1788442491.98 total 1481.03 last 61.30) - open/close 41199 (time 1788442501.98 total 1491.03 last 58.58) - open/close 41779 (time 1788442511.98 total 1501.03 last 58.00) - open/close 42396 (time 1788442522.00 total 1511.05 last 61.57) - open/close 43033 (time 1788442532.02 total 1521.06 last 63.61) - open/close 43632 (time 1788442542.02 total 1531.07 last 59.88) - open/close 44184 (time 1788442552.02 total 1541.07 last 55.18) - open/close 44778 (time 1788442562.03 total 1551.07 last 59.39) - open/close 45339 (time 1788442572.04 total 1561.09 last 56.03) - open/close 45954 (time 1788442582.05 total 1571.10 last 61.44) - open/close 46537 (time 1788442592.09 total 1581.14 last 58.05) - open/close 47204 (time 1788442602.10 total 1591.15 last 66.63) - open/close 47843 (time 1788442612.11 total 1601.16 last 63.84) - open/close 48377 (time 1788442622.11 total 1611.16 last 53.38) - open/close 48957 (time 1788442632.14 total 1621.19 last 57.85) - open/close 49528 (time 1788442642.15 total 1631.20 last 57.06) total: 50000 open/close in 1639.78 seconds: 30.49 ops/second - unlinked 0 (time 1788442656 ; total 0 ; last 0) - unlinked 10000 (time 1788442832 ; total 176 ; last 176) - unlinked 20000 (time 1788443008 ; total 352 ; last 176) - unlinked 30000 (time 1788443189 ; total 533 ; last 181) - unlinked 40000 (time 1788443371 ; total 715 ; last 182) total: 50000 unlinks in 890 seconds: 56.179775 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg431-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg431-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg431-server stop mds service on oleg431-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg431-server unloading modules via unload_modules_local on: 'oleg431-server' oleg431-server: oleg431-server.virtnet: executing unload_modules_local oleg431-server: kmemleak disabled modules unloaded. pdsh@oleg431-client: oleg431-server: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-client: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-client: ssh exited with exit code 2 pdsh@oleg431-client: oleg431-server: ssh exited with exit code 2