== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:59:56 (1788872396) start mds service on oleg440-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: kmemleak disabled oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: detected 4 online CPUs by sysfs oleg440-server: MODOPTS_LIBCFS= oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg440-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg440-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg440-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg440-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 207 (time 1788872500.93 total 10.34 last 20.02) - open/close 388 (time 1788872511.62 total 21.03 last 16.93) - open/close 520 (time 1788872521.63 total 31.04 last 13.19) - open/close 716 (time 1788872531.73 total 41.14 last 19.40) - open/close 886 (time 1788872542.19 total 51.60 last 16.25) - open/close 978 (time 1788872552.33 total 61.74 last 9.08) - open/close 1145 (time 1788872563.11 total 72.52 last 15.49) - open/close 1239 (time 1788872573.32 total 82.73 last 9.21) - open/close 1312 (time 1788872583.32 total 92.73 last 7.30) - open/close 1419 (time 1788872593.33 total 102.74 last 10.69) - open/close 1518 (time 1788872604.19 total 113.60 last 9.12) - open/close 1664 (time 1788872614.97 total 124.38 last 13.54) - open/close 1780 (time 1788872625.09 total 134.50 last 11.46) - open/close 1946 (time 1788872635.11 total 144.52 last 16.57) - open/close 2065 (time 1788872645.85 total 155.26 last 11.08) - open/close 2219 (time 1788872656.01 total 165.42 last 15.16) - open/close 2310 (time 1788872666.49 total 175.90 last 8.68) - open/close 2407 (time 1788872676.50 total 185.91 last 9.70) - open/close 2533 (time 1788872686.72 total 196.13 last 12.32) - open/close 2692 (time 1788872697.06 total 206.47 last 15.38) - open/close 2790 (time 1788872707.14 total 216.55 last 9.73) - open/close 2890 (time 1788872717.44 total 226.85 last 9.71) - open/close 2997 (time 1788872728.26 total 237.67 last 9.89) - open/close 3245 (time 1788872738.58 total 247.99 last 24.02) - open/close 3422 (time 1788872748.58 total 257.99 last 17.70) - open/close 3556 (time 1788872758.68 total 268.09 last 13.27) - open/close 3692 (time 1788872770.20 total 279.61 last 11.81) - open/close 3822 (time 1788872780.50 total 289.91 last 12.62) - open/close 3988 (time 1788872790.50 total 299.91 last 16.60) - open/close 4194 (time 1788872801.03 total 310.44 last 19.58) - open/close 4424 (time 1788872811.04 total 320.45 last 22.96) - open/close 4570 (time 1788872821.34 total 330.75 last 14.18) - open/close 4786 (time 1788872832.29 total 341.70 last 19.72) - open/close 4825 (time 1788872842.63 total 352.04 last 3.77) - open/close 4963 (time 1788872853.32 total 362.73 last 12.91) - open/close 5068 (time 1788872863.76 total 373.17 last 10.05) - open/close 5150 (time 1788872874.24 total 383.65 last 7.83) - open/close 5329 (time 1788872884.25 total 393.66 last 17.89) - open/close 5500 (time 1788872894.25 total 403.66 last 17.09) - open/close 5652 (time 1788872906.07 total 415.48 last 12.86) - open/close 5849 (time 1788872917.01 total 426.42 last 18.00) - open/close 5912 (time 1788872927.04 total 436.45 last 6.28) - open/close 6065 (time 1788872942.02 total 451.43 last 10.21) - open/close 6258 (time 1788872953.09 total 462.50 last 17.44) - open/close 6345 (time 1788872966.01 total 475.42 last 6.74) - open/close 6480 (time 1788872976.44 total 485.85 last 12.94) - open/close 6625 (time 1788872987.50 total 496.91 last 13.11) - open/close 6848 (time 1788872998.33 total 507.74 last 20.59) - open/close 7125 (time 1788873008.65 total 518.06 last 26.85) - open/close 7313 (time 1788873020.67 total 530.08 last 15.64) - open/close 7468 (time 1788873031.63 total 541.04 last 14.14) - open/close 7660 (time 1788873043.53 total 552.94 last 16.14) - open/close 7980 (time 1788873053.53 total 562.94 last 31.99) - open/close 8223 (time 1788873063.55 total 572.96 last 24.25) - open/close 8420 (time 1788873073.56 total 582.97 last 19.69) - open/close 8526 (time 1788873084.49 total 593.90 last 9.69) - open/close 8763 (time 1788873094.56 total 603.97 last 23.55) - open/close 9015 (time 1788873104.57 total 613.98 last 25.17) - open/close 9211 (time 1788873114.58 total 623.99 last 19.58) - open/close 9436 (time 1788873124.59 total 634.00 last 22.48) - open/close 9576 (time 1788873134.60 total 644.01 last 13.99) - open/close 9651 (time 1788873145.42 total 654.83 last 6.93) - open/close 9839 (time 1788873155.42 total 664.83 last 18.80) - open/close 10000 (time 1788873160.81 total 670.22 last 29.88) - open/close 10111 (time 1788873170.82 total 680.23 last 11.09) - open/close 10281 (time 1788873181.70 total 691.11 last 15.62) - open/close 10565 (time 1788873191.74 total 701.15 last 28.30) - open/close 10843 (time 1788873202.75 total 712.16 last 25.23) - open/close 11114 (time 1788873212.76 total 722.17 last 27.09) - open/close 11258 (time 1788873223.11 total 732.52 last 13.91) - open/close 11461 (time 1788873234.07 total 743.48 last 18.53) - open/close 11691 (time 1788873244.33 total 753.74 last 22.41) - open/close 11871 (time 1788873257.35 total 766.76 last 13.83) - open/close 12095 (time 1788873267.91 total 777.32 last 21.21) - open/close 12381 (time 1788873278.88 total 788.29 last 26.07) - open/close 12581 (time 1788873289.87 total 799.27 last 18.20) - open/close 12752 (time 1788873300.37 total 809.78 last 16.28) - open/close 12838 (time 1788873310.79 total 820.20 last 8.25) - open/close 13019 (time 1788873320.79 total 830.20 last 18.10) - open/close 13153 (time 1788873331.79 total 841.20 last 12.18) - open/close 13431 (time 1788873342.54 total 851.95 last 25.87) - open/close 13649 (time 1788873353.35 total 862.75 last 20.17) - open/close 13806 (time 1788873364.35 total 873.76 last 14.27) - open/close 14028 (time 1788873374.39 total 883.80 last 22.11) - open/close 14269 (time 1788873384.62 total 894.02 last 23.57) - open/close 14499 (time 1788873394.82 total 904.23 last 22.55) - open/close 14717 (time 1788873405.87 total 915.28 last 19.73) - open/close 14989 (time 1788873415.89 total 925.30 last 27.15) - open/close 15229 (time 1788873425.89 total 935.30 last 23.98) - open/close 15446 (time 1788873436.16 total 945.57 last 21.14) - open/close 15546 (time 1788873448.53 total 957.94 last 8.09) - open/close 15748 (time 1788873458.53 total 967.94 last 20.19) - open/close 15920 (time 1788873469.38 total 978.79 last 15.86) - open/close 16192 (time 1788873480.28 total 989.69 last 24.94) - open/close 16434 (time 1788873490.82 total 1000.22 last 22.97) - open/close 16615 (time 1788873501.52 total 1010.93 last 16.90) - open/close 16893 (time 1788873511.53 total 1020.93 last 27.79) - open/close 17064 (time 1788873522.06 total 1031.47 last 16.24) - open/close 17149 (time 1788873532.36 total 1041.77 last 8.25) - open/close 17325 (time 1788873542.36 total 1051.77 last 17.59) - open/close 18068 (time 1788873552.37 total 1061.78 last 74.24) - open/close 18813 (time 1788873562.37 total 1071.78 last 74.48) - open/close 19563 (time 1788873572.38 total 1081.79 last 74.94) - open/close 20000 (time 1788873578.35 total 1087.76 last 73.17) - open/close 20720 (time 1788873588.37 total 1097.78 last 71.88) - open/close 21502 (time 1788873598.38 total 1107.79 last 78.12) - open/close 22245 (time 1788873608.39 total 1117.80 last 74.25) - open/close 22986 (time 1788873618.39 total 1127.80 last 74.09) - open/close 23755 (time 1788873628.39 total 1137.80 last 76.87) - open/close 24550 (time 1788873638.40 total 1147.81 last 79.41) - open/close 25279 (time 1788873648.42 total 1157.83 last 72.80) - open/close 26009 (time 1788873658.42 total 1167.83 last 72.96) - open/close 26774 (time 1788873668.42 total 1177.83 last 76.48) - open/close 27477 (time 1788873678.44 total 1187.85 last 70.18) - open/close 28171 (time 1788873688.44 total 1197.85 last 69.39) - open/close 28888 (time 1788873698.45 total 1207.86 last 71.67) - open/close 29641 (time 1788873708.45 total 1217.86 last 75.29) - open/close 30000 (time 1788873713.49 total 1222.90 last 71.21) - open/close 30683 (time 1788873723.50 total 1232.91 last 68.22) - open/close 31402 (time 1788873733.51 total 1242.92 last 71.81) - open/close 32132 (time 1788873743.52 total 1252.93 last 72.95) - open/close 32855 (time 1788873753.54 total 1262.94 last 72.19) - open/close 33613 (time 1788873763.54 total 1272.95 last 75.73) - open/close 34375 (time 1788873773.55 total 1282.95 last 76.19) - open/close 35105 (time 1788873783.56 total 1292.97 last 72.90) - open/close 35872 (time 1788873793.56 total 1302.97 last 76.69) - open/close 36612 (time 1788873803.59 total 1313.00 last 73.79) - open/close 37364 (time 1788873813.60 total 1323.01 last 75.14) - open/close 38133 (time 1788873823.60 total 1333.01 last 76.86) - open/close 38895 (time 1788873833.60 total 1343.01 last 76.19) - open/close 39653 (time 1788873843.61 total 1353.02 last 75.75) - open/close 40000 (time 1788873848.91 total 1358.32 last 65.51) - open/close 40754 (time 1788873858.91 total 1368.32 last 75.39) - open/close 41535 (time 1788873868.92 total 1378.33 last 77.99) - open/close 42207 (time 1788873878.93 total 1388.34 last 67.16) - open/close 42882 (time 1788873888.96 total 1398.37 last 67.32) - open/close 43537 (time 1788873898.97 total 1408.38 last 65.42) - open/close 44095 (time 1788873908.98 total 1418.39 last 55.76) - open/close 44815 (time 1788873918.99 total 1428.40 last 71.91) - open/close 45530 (time 1788873929.00 total 1438.41 last 71.44) - open/close 46144 (time 1788873939.00 total 1448.41 last 61.36) - open/close 46728 (time 1788873949.01 total 1458.42 last 58.34) - open/close 47377 (time 1788873959.02 total 1468.43 last 64.86) - open/close 47988 (time 1788873969.02 total 1478.43 last 61.08) - open/close 48735 (time 1788873979.03 total 1488.44 last 74.63) - open/close 49462 (time 1788873989.04 total 1498.45 last 72.65) total: 50000 open/close in 1505.49 seconds: 33.21 ops/second - unlinked 0 (time 1788874000 ; total 0 ; last 0) - unlinked 10000 (time 1788874146 ; total 146 ; last 146) - unlinked 20000 (time 1788874299 ; total 299 ; last 153) - unlinked 30000 (time 1788874450 ; total 450 ; last 151) - unlinked 40000 (time 1788874602 ; total 602 ; last 152) total: 50000 unlinks in 747 seconds: 66.934402 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server unloading modules via unload_modules_local on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local oleg440-server: kmemleak disabled modules unloaded. pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2