== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:07:30 (1788800850) start mds service on oleg234-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 7 09:50 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing load_modules_local oleg234-server: kmemleak disabled oleg234-server: Loading modules from /home/green/git/lustre-release/lustre oleg234-server: detected 4 online CPUs by sysfs oleg234-server: MODOPTS_LIBCFS= oleg234-server: Force libcfs to create 2 CPU partitions oleg234-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg234-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg234-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg234-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg234-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 183 (time 1788800936.58 total 10.00 last 18.30) - open/close 337 (time 1788800947.54 total 20.96 last 14.05) - open/close 575 (time 1788800957.54 total 30.96 last 23.79) - open/close 823 (time 1788800967.61 total 41.04 last 24.62) - open/close 1090 (time 1788800978.51 total 51.93 last 24.51) - open/close 1325 (time 1788800988.87 total 62.29 last 22.68) - open/close 1566 (time 1788800998.88 total 72.30 last 24.09) - open/close 1689 (time 1788801008.89 total 82.31 last 12.29) - open/close 1905 (time 1788801018.89 total 92.31 last 21.59) - open/close 2121 (time 1788801029.84 total 103.26 last 19.73) - open/close 2353 (time 1788801040.72 total 114.15 last 21.31) - open/close 2568 (time 1788801050.97 total 124.39 last 20.98) - open/close 2759 (time 1788801061.12 total 134.54 last 18.83) - open/close 2903 (time 1788801071.12 total 144.54 last 14.39) - open/close 3061 (time 1788801081.61 total 155.03 last 15.07) - open/close 3269 (time 1788801093.95 total 167.37 last 16.86) - open/close 3433 (time 1788801105.03 total 178.45 last 14.80) - open/close 3571 (time 1788801115.03 total 188.45 last 13.79) - open/close 3769 (time 1788801126.80 total 200.22 last 16.83) - open/close 3915 (time 1788801137.10 total 210.52 last 14.18) - open/close 4054 (time 1788801147.84 total 221.26 last 12.94) - open/close 4253 (time 1788801157.86 total 231.28 last 19.86) - open/close 4354 (time 1788801168.90 total 242.32 last 9.15) - open/close 4465 (time 1788801179.87 total 253.29 last 10.12) - open/close 4589 (time 1788801189.96 total 263.38 last 12.29) - open/close 4717 (time 1788801202.49 total 275.91 last 10.21) - open/close 4851 (time 1788801212.50 total 285.93 last 13.38) - open/close 5054 (time 1788801222.72 total 296.15 last 19.86) - open/close 5176 (time 1788801233.62 total 307.04 last 11.20) - open/close 5373 (time 1788801246.46 total 319.89 last 15.34) - open/close 5449 (time 1788801256.53 total 329.95 last 7.55) - open/close 5578 (time 1788801268.61 total 342.03 last 10.69) - open/close 5738 (time 1788801279.22 total 352.64 last 15.07) - open/close 5810 (time 1788801289.99 total 363.41 last 6.69) - open/close 5923 (time 1788801300.35 total 373.78 last 10.90) - open/close 6089 (time 1788801310.36 total 383.78 last 16.59) - open/close 6213 (time 1788801320.90 total 394.32 last 11.76) - open/close 6354 (time 1788801330.91 total 404.33 last 14.09) - open/close 6464 (time 1788801341.12 total 414.54 last 10.78) - open/close 6617 (time 1788801351.13 total 424.55 last 15.27) - open/close 6733 (time 1788801361.16 total 434.58 last 11.57) - open/close 6810 (time 1788801372.55 total 445.97 last 6.76) - open/close 6929 (time 1788801383.05 total 456.47 last 11.33) - open/close 7081 (time 1788801393.06 total 466.48 last 15.18) - open/close 7149 (time 1788801404.37 total 477.79 last 6.01) - open/close 7214 (time 1788801414.62 total 488.04 last 6.34) - open/close 7292 (time 1788801425.24 total 498.66 last 7.35) - open/close 7378 (time 1788801437.76 total 511.18 last 6.87) - open/close 7482 (time 1788801447.89 total 521.32 last 10.26) - open/close 7597 (time 1788801458.54 total 531.96 last 10.81) - open/close 7777 (time 1788801469.45 total 542.87 last 16.50) - open/close 7836 (time 1788801479.45 total 552.87 last 5.90) - open/close 7977 (time 1788801489.87 total 563.29 last 13.53) - open/close 8078 (time 1788801500.04 total 573.46 last 9.93) - open/close 8173 (time 1788801512.73 total 586.15 last 7.49) - open/close 8301 (time 1788801523.17 total 596.59 last 12.26) - open/close 8442 (time 1788801533.72 total 607.14 last 13.37) - open/close 8547 (time 1788801544.54 total 617.96 last 9.70) - open/close 8608 (time 1788801554.56 total 627.98 last 6.09) - open/close 8773 (time 1788801564.68 total 638.10 last 16.30) - open/close 8972 (time 1788801577.61 total 651.03 last 15.39) - open/close 9177 (time 1788801588.14 total 661.56 last 19.48) - open/close 9313 (time 1788801598.55 total 671.97 last 13.07) - open/close 9450 (time 1788801609.36 total 682.78 last 12.67) - open/close 9643 (time 1788801620.31 total 693.73 last 17.62) - open/close 9756 (time 1788801630.31 total 703.73 last 11.30) - open/close 9847 (time 1788801641.80 total 715.22 last 7.92) - open/close 10000 (time 1788801650.46 total 723.88 last 17.67) - open/close 10137 (time 1788801661.21 total 734.63 last 12.75) - open/close 10331 (time 1788801672.19 total 745.61 last 17.66) - open/close 10482 (time 1788801682.81 total 756.24 last 14.21) - open/close 10627 (time 1788801692.81 total 766.24 last 14.50) - open/close 10853 (time 1788801702.83 total 776.25 last 22.56) - open/close 10962 (time 1788801712.83 total 786.26 last 10.90) - open/close 11191 (time 1788801722.85 total 796.27 last 22.87) - open/close 11431 (time 1788801733.18 total 806.60 last 23.23) - open/close 11559 (time 1788801744.50 total 817.92 last 11.30) - open/close 11648 (time 1788801754.89 total 828.32 last 8.56) - open/close 11685 (time 1788801765.22 total 838.64 last 3.58) - open/close 11867 (time 1788801775.57 total 848.99 last 17.59) - open/close 11959 (time 1788801786.44 total 859.86 last 8.46) - open/close 12090 (time 1788801796.83 total 870.25 last 12.61) - open/close 12136 (time 1788801806.86 total 880.28 last 4.59) - open/close 12252 (time 1788801817.36 total 890.78 last 11.05) - open/close 12379 (time 1788801827.46 total 900.88 last 12.57) - open/close 12462 (time 1788801837.70 total 911.12 last 8.11) - open/close 12569 (time 1788801850.52 total 923.94 last 8.34) - open/close 12646 (time 1788801860.75 total 934.17 last 7.53) - open/close 12870 (time 1788801870.75 total 944.17 last 22.39) - open/close 13094 (time 1788801880.76 total 954.18 last 22.39) - open/close 13282 (time 1788801892.07 total 965.49 last 16.62) - open/close 13382 (time 1788801902.08 total 975.50 last 9.99) - open/close 13494 (time 1788801914.23 total 987.65 last 9.22) - open/close 13606 (time 1788801924.64 total 998.07 last 10.75) - open/close 13728 (time 1788801935.75 total 1009.17 last 10.99) - open/close 13809 (time 1788801945.78 total 1019.20 last 8.08) - open/close 13879 (time 1788801956.24 total 1029.66 last 6.69) - open/close 14024 (time 1788801966.54 total 1039.96 last 14.08) - open/close 14219 (time 1788801976.54 total 1049.96 last 19.49) - open/close 14316 (time 1788801986.91 total 1060.33 last 9.35) - open/close 14367 (time 1788801997.70 total 1071.12 last 4.73) - open/close 14545 (time 1788802008.20 total 1081.62 last 16.95) - open/close 14700 (time 1788802018.21 total 1091.63 last 15.49) - open/close 14833 (time 1788802028.23 total 1101.65 last 13.27) - open/close 15457 (time 1788802038.24 total 1111.66 last 62.35) - open/close 16111 (time 1788802048.24 total 1121.66 last 65.39) - open/close 16933 (time 1788802058.25 total 1131.67 last 82.14) - open/close 17733 (time 1788802068.25 total 1141.67 last 79.98) - open/close 18266 (time 1788802078.26 total 1151.69 last 53.21) - open/close 18990 (time 1788802088.27 total 1161.69 last 72.38) - open/close 19677 (time 1788802098.28 total 1171.70 last 68.60) - open/close 20000 (time 1788802104.96 total 1178.38 last 48.39) - open/close 20564 (time 1788802114.97 total 1188.39 last 56.33) - open/close 21157 (time 1788802124.98 total 1198.40 last 59.25) - open/close 21712 (time 1788802135.02 total 1208.44 last 55.27) - open/close 22388 (time 1788802145.02 total 1218.44 last 67.59) - open/close 23188 (time 1788802155.02 total 1228.44 last 79.99) - open/close 23950 (time 1788802165.03 total 1238.45 last 76.15) - open/close 24792 (time 1788802175.04 total 1248.46 last 84.08) - open/close 25557 (time 1788802185.06 total 1258.48 last 76.39) - open/close 26301 (time 1788802195.06 total 1268.48 last 74.39) - open/close 27061 (time 1788802205.06 total 1278.48 last 75.98) - open/close 27886 (time 1788802215.06 total 1288.48 last 82.49) - open/close 28660 (time 1788802225.08 total 1298.50 last 77.29) - open/close 29467 (time 1788802235.10 total 1308.52 last 80.51) - open/close 30000 (time 1788802242.17 total 1315.59 last 75.41) - open/close 30662 (time 1788802252.17 total 1325.59 last 66.18) - open/close 31278 (time 1788802262.18 total 1335.60 last 61.54) - open/close 31897 (time 1788802272.19 total 1345.61 last 61.87) - open/close 32645 (time 1788802282.19 total 1355.61 last 74.74) - open/close 33434 (time 1788802292.19 total 1365.62 last 78.89) - open/close 34204 (time 1788802302.20 total 1375.62 last 76.99) - open/close 34920 (time 1788802312.23 total 1385.65 last 71.36) - open/close 35730 (time 1788802322.25 total 1395.67 last 80.87) - open/close 36508 (time 1788802332.25 total 1405.67 last 77.78) - open/close 37220 (time 1788802342.25 total 1415.67 last 71.17) - open/close 37949 (time 1788802352.27 total 1425.69 last 72.80) - open/close 38695 (time 1788802362.27 total 1435.69 last 74.59) - open/close 39474 (time 1788802372.27 total 1445.70 last 77.84) - open/close 40000 (time 1788802379.00 total 1452.42 last 78.23) - open/close 40840 (time 1788802389.01 total 1462.43 last 83.92) - open/close 41665 (time 1788802399.02 total 1472.45 last 82.36) - open/close 42220 (time 1788802409.03 total 1482.45 last 55.50) - open/close 42717 (time 1788802419.03 total 1492.45 last 49.67) - open/close 43403 (time 1788802429.04 total 1502.46 last 68.51) - open/close 44092 (time 1788802439.05 total 1512.47 last 68.84) - open/close 44754 (time 1788802449.06 total 1522.48 last 66.17) - open/close 45541 (time 1788802459.06 total 1532.48 last 78.69) - open/close 46267 (time 1788802469.07 total 1542.49 last 72.55) - open/close 46976 (time 1788802479.08 total 1552.50 last 70.79) - open/close 47729 (time 1788802489.10 total 1562.52 last 75.14) - open/close 48465 (time 1788802499.10 total 1572.52 last 73.60) - open/close 49168 (time 1788802509.11 total 1582.53 last 70.28) - open/close 49902 (time 1788802519.12 total 1592.54 last 73.30) total: 50000 open/close in 1594.03 seconds: 31.37 ops/second - unlinked 0 (time 1788802526 ; total 0 ; last 0) - unlinked 10000 (time 1788802687 ; total 161 ; last 161) - unlinked 20000 (time 1788802827 ; total 301 ; last 140) - unlinked 30000 (time 1788802972 ; total 446 ; last 145) - unlinked 40000 (time 1788803110 ; total 584 ; last 138) total: 50000 unlinks in 719 seconds: 69.541031 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg234-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg234-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg234-server unloading modules via unload_modules_local on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing unload_modules_local oleg234-server: kmemleak disabled modules unloaded. pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2