== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:29:35 (1788442175) start mds service on oleg645-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg645-server' oleg645-server: oleg645-server.virtnet: executing load_modules_local oleg645-server: kmemleak disabled oleg645-server: Loading modules from /home/green/git/lustre-release/lustre oleg645-server: detected 4 online CPUs by sysfs oleg645-server: MODOPTS_LIBCFS= oleg645-server: Force libcfs to create 2 CPU partitions oleg645-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg645-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg645-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg645-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg645-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg645-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg645-server: oleg645-server.virtnet: executing set_default_debug -1 all pdsh@oleg645-client: oleg645-server: ssh exited with exit code 1 Started lustre-OST0000 oleg645-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg645-client.virtnet: -o user_xattr,flock 192.168.206.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 221 (time 1788442251.91 total 10.02 last 22.05) - open/close 457 (time 1788442261.91 total 20.03 last 23.59) - open/close 658 (time 1788442272.37 total 30.48 last 19.22) - open/close 918 (time 1788442282.62 total 40.73 last 25.37) - open/close 1153 (time 1788442292.66 total 50.78 last 23.40) - open/close 1409 (time 1788442303.09 total 61.21 last 24.54) - open/close 1619 (time 1788442313.65 total 71.77 last 19.88) - open/close 1823 (time 1788442324.60 total 82.71 last 18.64) - open/close 2050 (time 1788442334.60 total 92.72 last 22.70) - open/close 2263 (time 1788442346.94 total 105.05 last 17.27) - open/close 2458 (time 1788442356.95 total 115.07 last 19.47) - open/close 2734 (time 1788442366.95 total 125.07 last 27.59) - open/close 3010 (time 1788442376.96 total 135.08 last 27.58) - open/close 3227 (time 1788442387.05 total 145.17 last 21.50) - open/close 3397 (time 1788442397.37 total 155.48 last 16.48) - open/close 3652 (time 1788442409.91 total 168.02 last 20.33) - open/close 3826 (time 1788442420.15 total 178.27 last 16.98) - open/close 4015 (time 1788442432.95 total 191.06 last 14.77) - open/close 4255 (time 1788442443.31 total 201.43 last 23.15) - open/close 4491 (time 1788442454.85 total 212.97 last 20.45) - open/close 4721 (time 1788442465.60 total 223.72 last 21.40) - open/close 4951 (time 1788442475.62 total 233.73 last 22.97) - open/close 5081 (time 1788442485.64 total 243.76 last 12.96) - open/close 5324 (time 1788442496.30 total 254.42 last 22.80) - open/close 5507 (time 1788442507.12 total 265.23 last 16.92) - open/close 5738 (time 1788442518.34 total 276.45 last 20.59) - open/close 5957 (time 1788442528.84 total 286.96 last 20.85) - open/close 6115 (time 1788442540.08 total 298.20 last 14.05) - open/close 6333 (time 1788442550.35 total 308.47 last 21.23) - open/close 6613 (time 1788442561.40 total 319.52 last 25.35) - open/close 6874 (time 1788442572.35 total 330.46 last 23.84) - open/close 7047 (time 1788442582.84 total 340.95 last 16.49) - open/close 7321 (time 1788442593.08 total 351.20 last 26.75) - open/close 7547 (time 1788442603.27 total 361.38 last 22.18) - open/close 7764 (time 1788442613.28 total 371.40 last 21.67) - open/close 7972 (time 1788442623.71 total 381.83 last 19.95) - open/close 8189 (time 1788442633.71 total 391.83 last 21.69) - open/close 8436 (time 1788442643.73 total 401.85 last 24.66) - open/close 8468 (time 1788442655.99 total 414.10 last 2.61) - open/close 8640 (time 1788442667.01 total 425.13 last 15.60) - open/close 8689 (time 1788442677.17 total 435.28 last 4.82) - open/close 8904 (time 1788442688.18 total 446.30 last 19.52) - open/close 9063 (time 1788442699.59 total 457.70 last 13.94) - open/close 9252 (time 1788442709.82 total 467.94 last 18.47) - open/close 9535 (time 1788442719.82 total 477.94 last 28.30) - open/close 9720 (time 1788442729.83 total 487.94 last 18.49) - open/close 9923 (time 1788442739.84 total 497.96 last 20.28) - open/close 10000 (time 1788442745.65 total 503.77 last 13.25) - open/close 10243 (time 1788442756.27 total 514.39 last 22.87) - open/close 10459 (time 1788442766.29 total 524.40 last 21.57) - open/close 10684 (time 1788442776.68 total 534.80 last 21.65) - open/close 10908 (time 1788442786.71 total 544.82 last 22.34) - open/close 11112 (time 1788442796.93 total 555.05 last 19.96) - open/close 11369 (time 1788442806.97 total 565.08 last 25.60) - open/close 11641 (time 1788442817.35 total 575.47 last 26.20) - open/close 11902 (time 1788442828.34 total 586.45 last 23.75) - open/close 12099 (time 1788442838.34 total 596.46 last 19.69) - open/close 12327 (time 1788442849.01 total 607.13 last 21.38) - open/close 12457 (time 1788442859.84 total 617.96 last 12.00) - open/close 12681 (time 1788442869.84 total 627.96 last 22.40) - open/close 12810 (time 1788442879.85 total 637.97 last 12.89) - open/close 12933 (time 1788442889.86 total 647.97 last 12.29) - open/close 13158 (time 1788442899.88 total 657.99 last 22.46) - open/close 13367 (time 1788442909.88 total 668.00 last 20.89) - open/close 13517 (time 1788442919.89 total 678.01 last 14.98) - open/close 13732 (time 1788442929.91 total 688.02 last 21.47) - open/close 13985 (time 1788442939.91 total 698.03 last 25.30) - open/close 14229 (time 1788442950.32 total 708.43 last 23.44) - open/close 14534 (time 1788442960.32 total 718.43 last 30.50) - open/close 14762 (time 1788442971.32 total 729.44 last 20.72) - open/close 14928 (time 1788442981.37 total 739.49 last 16.52) - open/close 15116 (time 1788442991.37 total 749.49 last 18.79) - open/close 15356 (time 1788443001.87 total 759.98 last 22.87) - open/close 15576 (time 1788443011.88 total 769.99 last 21.98) - open/close 15860 (time 1788443022.78 total 780.90 last 26.04) - open/close 16116 (time 1788443035.88 total 794.00 last 19.54) - open/close 16243 (time 1788443045.89 total 804.01 last 12.69) - open/close 16489 (time 1788443056.73 total 814.85 last 22.69) - open/close 16730 (time 1788443069.71 total 827.83 last 18.57) - open/close 16900 (time 1788443080.00 total 838.12 last 16.52) - open/close 17053 (time 1788443092.36 total 850.48 last 12.38) - open/close 17206 (time 1788443102.92 total 861.04 last 14.49) - open/close 17365 (time 1788443113.92 total 872.04 last 14.45) - open/close 17453 (time 1788443124.15 total 882.27 last 8.60) - open/close 17617 (time 1788443134.16 total 892.27 last 16.39) - open/close 17762 (time 1788443144.91 total 903.03 last 13.48) - open/close 17967 (time 1788443155.39 total 913.51 last 19.57) - open/close 18180 (time 1788443165.94 total 924.06 last 20.19) - open/close 18267 (time 1788443175.94 total 934.06 last 8.70) - open/close 18575 (time 1788443185.95 total 944.07 last 30.77) - open/close 19042 (time 1788443195.96 total 954.08 last 46.67) - open/close 19745 (time 1788443205.96 total 964.08 last 70.25) - open/close 20000 (time 1788443209.42 total 967.54 last 73.73) - open/close 20742 (time 1788443219.43 total 977.54 last 74.19) - open/close 21485 (time 1788443229.44 total 987.56 last 74.20) - open/close 22226 (time 1788443239.44 total 997.56 last 74.09) - open/close 22955 (time 1788443249.45 total 1007.56 last 72.84) - open/close 23684 (time 1788443259.46 total 1017.58 last 72.78) - open/close 24404 (time 1788443269.48 total 1027.60 last 71.85) - open/close 25100 (time 1788443279.49 total 1037.61 last 69.55) - open/close 25832 (time 1788443289.52 total 1047.64 last 73.01) - open/close 26587 (time 1788443299.53 total 1057.64 last 75.44) - open/close 27284 (time 1788443309.53 total 1067.64 last 69.69) - open/close 27986 (time 1788443319.53 total 1077.65 last 70.17) - open/close 28683 (time 1788443329.54 total 1087.66 last 69.62) - open/close 29328 (time 1788443339.55 total 1097.67 last 64.46) - open/close 29984 (time 1788443349.56 total 1107.67 last 65.56) - open/close 30673 (time 1788443359.57 total 1117.69 last 68.81) - open/close 31356 (time 1788443369.58 total 1127.70 last 68.21) - open/close 32089 (time 1788443379.59 total 1137.71 last 73.24) - open/close 32781 (time 1788443389.59 total 1147.71 last 69.17) - open/close 33560 (time 1788443399.60 total 1157.72 last 77.82) - open/close 34269 (time 1788443409.63 total 1167.75 last 70.69) - open/close 34980 (time 1788443419.63 total 1177.75 last 71.09) - open/close 35654 (time 1788443429.64 total 1187.76 last 67.37) - open/close 36368 (time 1788443439.65 total 1197.77 last 71.30) - open/close 37096 (time 1788443449.66 total 1207.78 last 72.75) - open/close 37808 (time 1788443459.66 total 1217.78 last 71.17) - open/close 38427 (time 1788443469.67 total 1227.79 last 61.86) - open/close 39074 (time 1788443479.67 total 1237.79 last 64.67) - open/close 39744 (time 1788443489.68 total 1247.80 last 66.96) - open/close 40000 (time 1788443493.30 total 1251.42 last 70.72) - open/close 40693 (time 1788443503.31 total 1261.43 last 69.23) - open/close 41419 (time 1788443513.33 total 1271.45 last 72.44) - open/close 42093 (time 1788443523.35 total 1281.46 last 67.30) - open/close 42741 (time 1788443533.35 total 1291.46 last 64.79) - open/close 43447 (time 1788443543.36 total 1301.47 last 70.54) - open/close 44061 (time 1788443553.36 total 1311.47 last 61.39) - open/close 44743 (time 1788443563.36 total 1321.48 last 68.17) - open/close 45400 (time 1788443573.37 total 1331.49 last 65.65) - open/close 46086 (time 1788443583.38 total 1341.49 last 68.54) - open/close 46760 (time 1788443593.39 total 1351.50 last 67.33) - open/close 47459 (time 1788443603.39 total 1361.51 last 69.85) - open/close 48095 (time 1788443613.41 total 1371.53 last 63.48) - open/close 48671 (time 1788443623.42 total 1381.54 last 57.56) - open/close 49213 (time 1788443633.42 total 1391.54 last 54.18) - open/close 49849 (time 1788443643.43 total 1401.54 last 63.58) total: 50000 open/close in 1403.82 seconds: 35.62 ops/second - unlinked 0 (time 1788443649 ; total 0 ; last 0) - unlinked 10000 (time 1788443805 ; total 156 ; last 156) - unlinked 20000 (time 1788443951 ; total 302 ; last 146) - unlinked 30000 (time 1788444114 ; total 465 ; last 163) - unlinked 40000 (time 1788444269 ; total 620 ; last 155) total: 50000 unlinks in 778 seconds: 64.267349 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg645-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg645-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg645-server stop mds service on oleg645-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg645-server unloading modules via unload_modules_local on: 'oleg645-server' oleg645-server: oleg645-server.virtnet: executing unload_modules_local oleg645-server: kmemleak disabled modules unloaded. pdsh@oleg645-client: oleg645-client: ssh exited with exit code 2 pdsh@oleg645-client: oleg645-server: ssh exited with exit code 2 pdsh@oleg645-client: oleg645-client: ssh exited with exit code 2 pdsh@oleg645-client: oleg645-server: ssh exited with exit code 2