== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:14:33 (1789265673) start mds service on oleg113-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 12 19:16 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing load_modules_local oleg113-server: kmemleak disabled oleg113-server: Loading modules from /home/green/git/lustre-release/lustre oleg113-server: detected 4 online CPUs by sysfs oleg113-server: MODOPTS_LIBCFS= oleg113-server: Force libcfs to create 2 CPU partitions oleg113-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg113-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg113-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg113-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg113-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug -1 all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 112 (time 1789265771.85 total 10.04 last 11.15) - open/close 231 (time 1789265782.76 total 20.94 last 10.92) - open/close 393 (time 1789265795.80 total 33.99 last 12.42) - open/close 482 (time 1789265805.80 total 43.99 last 8.90) - open/close 608 (time 1789265816.43 total 54.62 last 11.85) - open/close 796 (time 1789265827.81 total 66.00 last 16.53) - open/close 915 (time 1789265838.45 total 76.64 last 11.18) - open/close 1048 (time 1789265848.63 total 86.82 last 13.07) - open/close 1118 (time 1789265858.79 total 96.98 last 6.89) - open/close 1334 (time 1789265868.82 total 107.01 last 21.54) - open/close 1470 (time 1789265879.62 total 117.80 last 12.60) - open/close 1514 (time 1789265890.02 total 128.21 last 4.23) - open/close 1662 (time 1789265901.71 total 139.89 last 12.66) - open/close 1837 (time 1789265911.79 total 149.98 last 17.35) - open/close 1971 (time 1789265921.81 total 160.00 last 13.38) - open/close 2217 (time 1789265932.26 total 170.45 last 23.54) - open/close 2398 (time 1789265942.29 total 180.48 last 18.05) - open/close 2616 (time 1789265952.29 total 190.48 last 21.79) - open/close 2793 (time 1789265962.32 total 200.51 last 17.64) - open/close 2916 (time 1789265972.33 total 210.52 last 12.29) - open/close 3079 (time 1789265982.35 total 220.54 last 16.27) - open/close 3294 (time 1789265993.00 total 231.19 last 20.20) - open/close 3382 (time 1789266003.16 total 241.35 last 8.65) - open/close 3497 (time 1789266013.21 total 251.40 last 11.45) - open/close 3697 (time 1789266024.12 total 262.31 last 18.33) - open/close 3854 (time 1789266034.66 total 272.85 last 14.89) - open/close 4037 (time 1789266045.52 total 283.71 last 16.85) - open/close 4265 (time 1789266056.63 total 294.82 last 20.51) - open/close 4471 (time 1789266067.41 total 305.60 last 19.11) - open/close 4575 (time 1789266077.44 total 315.63 last 10.37) - open/close 4761 (time 1789266087.83 total 326.02 last 17.90) - open/close 4983 (time 1789266097.84 total 336.03 last 22.20) - open/close 5209 (time 1789266108.85 total 347.04 last 20.51) - open/close 5375 (time 1789266118.87 total 357.06 last 16.57) - open/close 5558 (time 1789266128.87 total 367.06 last 18.29) - open/close 5753 (time 1789266138.95 total 377.14 last 19.35) - open/close 5947 (time 1789266148.95 total 387.14 last 19.39) - open/close 6047 (time 1789266159.42 total 397.61 last 9.55) - open/close 6248 (time 1789266170.99 total 409.18 last 17.37) - open/close 6332 (time 1789266181.00 total 419.19 last 8.39) - open/close 6483 (time 1789266192.03 total 430.22 last 13.70) - open/close 6603 (time 1789266202.61 total 440.80 last 11.34) - open/close 6750 (time 1789266212.62 total 450.81 last 14.69) - open/close 6863 (time 1789266223.39 total 461.58 last 10.49) - open/close 7026 (time 1789266233.91 total 472.10 last 15.50) - open/close 7188 (time 1789266243.92 total 482.11 last 16.18) - open/close 7388 (time 1789266254.06 total 492.25 last 19.73) - open/close 7639 (time 1789266264.41 total 502.60 last 24.24) - open/close 7845 (time 1789266275.43 total 513.62 last 18.70) - open/close 8028 (time 1789266285.45 total 523.64 last 18.25) - open/close 8132 (time 1789266296.09 total 534.28 last 9.78) - open/close 8272 (time 1789266306.60 total 544.79 last 13.32) - open/close 8371 (time 1789266317.08 total 555.27 last 9.44) - open/close 8532 (time 1789266327.72 total 565.91 last 15.13) - open/close 8737 (time 1789266338.22 total 576.41 last 19.52) - open/close 8890 (time 1789266348.69 total 586.88 last 14.62) - open/close 8993 (time 1789266358.71 total 596.90 last 10.28) - open/close 9155 (time 1789266368.76 total 606.95 last 16.13) - open/close 9373 (time 1789266379.71 total 617.90 last 19.91) - open/close 9636 (time 1789266389.71 total 627.90 last 26.30) - open/close 9791 (time 1789266400.28 total 638.47 last 14.66) - open/close 9869 (time 1789266410.45 total 648.64 last 7.67) - open/close 10000 (time 1789266417.48 total 655.67 last 18.65) - open/close 10175 (time 1789266428.12 total 666.31 last 16.44) - open/close 10386 (time 1789266438.90 total 677.09 last 19.57) - open/close 10614 (time 1789266448.91 total 687.10 last 22.79) - open/close 10751 (time 1789266459.52 total 697.71 last 12.92) - open/close 10951 (time 1789266469.76 total 707.95 last 19.53) - open/close 11206 (time 1789266480.34 total 718.53 last 24.10) - open/close 11466 (time 1789266490.35 total 728.54 last 25.96) - open/close 11584 (time 1789266500.36 total 738.55 last 11.79) - open/close 11667 (time 1789266510.76 total 748.95 last 7.98) - open/close 11819 (time 1789266520.89 total 759.08 last 15.02) - open/close 11975 (time 1789266530.97 total 769.16 last 15.46) - open/close 12160 (time 1789266540.99 total 779.18 last 18.47) - open/close 12335 (time 1789266551.91 total 790.10 last 16.03) - open/close 12576 (time 1789266563.81 total 802.00 last 20.24) - open/close 12793 (time 1789266574.79 total 812.98 last 19.77) - open/close 12936 (time 1789266584.79 total 822.98 last 14.30) - open/close 13087 (time 1789266597.65 total 835.84 last 11.74) - open/close 13227 (time 1789266608.06 total 846.25 last 13.46) - open/close 13357 (time 1789266618.77 total 856.96 last 12.14) - open/close 13440 (time 1789266628.96 total 867.15 last 8.15) - open/close 13618 (time 1789266638.99 total 877.18 last 17.75) - open/close 13771 (time 1789266649.69 total 887.88 last 14.30) - open/close 13940 (time 1789266660.14 total 898.33 last 16.17) - open/close 14107 (time 1789266670.33 total 908.52 last 16.38) - open/close 14351 (time 1789266680.36 total 918.55 last 24.34) - open/close 14555 (time 1789266692.71 total 930.90 last 16.51) - open/close 14675 (time 1789266702.90 total 941.09 last 11.78) - open/close 14854 (time 1789266713.58 total 951.77 last 16.75) - open/close 15064 (time 1789266723.59 total 961.78 last 21.00) - open/close 15271 (time 1789266733.99 total 972.17 last 19.91) - open/close 15447 (time 1789266745.06 total 983.25 last 15.89) - open/close 15642 (time 1789266755.06 total 993.25 last 19.49) - open/close 15848 (time 1789266765.61 total 1003.80 last 19.53) - open/close 16024 (time 1789266776.11 total 1014.30 last 16.77) - open/close 16238 (time 1789266786.11 total 1024.30 last 21.40) - open/close 16402 (time 1789266796.12 total 1034.31 last 16.38) - open/close 16460 (time 1789266806.17 total 1044.36 last 5.77) - open/close 16592 (time 1789266816.18 total 1054.37 last 13.19) - open/close 16755 (time 1789266826.21 total 1064.40 last 16.25) - open/close 16939 (time 1789266842.15 total 1080.34 last 11.54) - open/close 17630 (time 1789266852.16 total 1090.35 last 69.07) - open/close 18238 (time 1789266862.19 total 1100.38 last 60.59) - open/close 18810 (time 1789266872.20 total 1110.39 last 57.18) - open/close 19394 (time 1789266882.20 total 1120.39 last 58.40) - open/close 19984 (time 1789266892.20 total 1130.39 last 58.97) - open/close 20610 (time 1789266902.22 total 1140.41 last 62.51) - open/close 21234 (time 1789266912.22 total 1150.41 last 62.39) - open/close 21835 (time 1789266922.22 total 1160.41 last 60.08) - open/close 22488 (time 1789266932.24 total 1170.43 last 65.15) - open/close 23215 (time 1789266942.24 total 1180.43 last 72.68) - open/close 23829 (time 1789266952.26 total 1190.45 last 61.32) - open/close 24456 (time 1789266962.27 total 1200.46 last 62.61) - open/close 25090 (time 1789266972.28 total 1210.47 last 63.36) - open/close 25767 (time 1789266982.29 total 1220.48 last 67.65) - open/close 26494 (time 1789266992.29 total 1230.48 last 72.66) - open/close 27304 (time 1789267002.31 total 1240.50 last 80.81) - open/close 28052 (time 1789267012.33 total 1250.52 last 74.70) - open/close 28735 (time 1789267022.36 total 1260.55 last 68.09) - open/close 29352 (time 1789267032.36 total 1270.55 last 61.66) - open/close 29918 (time 1789267042.38 total 1280.57 last 56.51) - open/close 30456 (time 1789267052.38 total 1290.57 last 53.79) - open/close 31081 (time 1789267062.38 total 1300.57 last 62.49) - open/close 31787 (time 1789267072.39 total 1310.58 last 70.55) - open/close 32592 (time 1789267082.41 total 1320.60 last 80.38) - open/close 33292 (time 1789267092.41 total 1330.60 last 69.97) - open/close 33991 (time 1789267102.42 total 1340.61 last 69.81) - open/close 34585 (time 1789267112.43 total 1350.62 last 59.39) - open/close 35205 (time 1789267122.44 total 1360.63 last 61.94) - open/close 35861 (time 1789267132.45 total 1370.64 last 65.52) - open/close 36582 (time 1789267142.45 total 1380.64 last 72.07) - open/close 37309 (time 1789267152.46 total 1390.65 last 72.67) - open/close 37980 (time 1789267162.46 total 1400.65 last 67.05) - open/close 38590 (time 1789267172.48 total 1410.67 last 60.91) - open/close 39223 (time 1789267182.48 total 1420.67 last 63.29) - open/close 39859 (time 1789267192.49 total 1430.68 last 63.56) - open/close 40000 (time 1789267194.75 total 1432.94 last 62.28) - open/close 40576 (time 1789267204.77 total 1442.96 last 57.51) - open/close 41238 (time 1789267214.78 total 1452.97 last 66.10) - open/close 41902 (time 1789267224.78 total 1462.97 last 66.38) - open/close 42566 (time 1789267234.79 total 1472.98 last 66.38) - open/close 43171 (time 1789267244.80 total 1482.99 last 60.44) - open/close 43765 (time 1789267254.81 total 1493.00 last 59.36) - open/close 44444 (time 1789267264.81 total 1503.00 last 67.84) - open/close 45117 (time 1789267274.82 total 1513.01 last 67.26) - open/close 45774 (time 1789267284.83 total 1523.02 last 65.65) - open/close 46481 (time 1789267294.83 total 1533.02 last 70.69) - open/close 47169 (time 1789267304.84 total 1543.03 last 68.75) - open/close 47792 (time 1789267314.84 total 1553.03 last 62.30) - open/close 48494 (time 1789267324.84 total 1563.03 last 70.16) - open/close 49121 (time 1789267334.88 total 1573.07 last 62.50) - open/close 49718 (time 1789267344.88 total 1583.07 last 59.70) total: 50000 open/close in 1588.07 seconds: 31.48 ops/second - unlinked 0 (time 1789267356 ; total 0 ; last 0) - unlinked 10000 (time 1789267515 ; total 159 ; last 159) - unlinked 20000 (time 1789267683 ; total 327 ; last 168) - unlinked 30000 (time 1789267849 ; total 493 ; last 166) - unlinked 40000 (time 1789268033 ; total 677 ; last 184) total: 50000 unlinks in 848 seconds: 58.962265 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg113-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg113-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg113-server stop mds service on oleg113-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg113-server unloading modules via unload_modules_local on: 'oleg113-server' oleg113-server: oleg113-server.virtnet: executing unload_modules_local oleg113-server: kmemleak disabled modules unloaded. pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-client: ssh exited with exit code 2 pdsh@oleg113-client: oleg113-server: ssh exited with exit code 2