== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:29:46 (1788589786) start mds service on oleg652-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 4 11:05 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg652-server' oleg652-server: oleg652-server.virtnet: executing load_modules_local oleg652-server: kmemleak disabled oleg652-server: Loading modules from /home/green/git/lustre-release/lustre oleg652-server: detected 4 online CPUs by sysfs oleg652-server: MODOPTS_LIBCFS= oleg652-server: Force libcfs to create 2 CPU partitions oleg652-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg652-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg652-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg652-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg652-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-OST0000 oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg652-client.virtnet: -o user_xattr,flock 192.168.206.152@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 237 (time 1788589868.58 total 10.70 last 22.15) - open/close 438 (time 1788589878.98 total 21.10 last 19.32) - open/close 690 (time 1788589890.03 total 32.15 last 22.81) - open/close 805 (time 1788589900.77 total 42.90 last 10.70) - open/close 1072 (time 1788589910.78 total 52.90 last 26.69) - open/close 1304 (time 1788589921.64 total 63.76 last 21.36) - open/close 1519 (time 1788589931.64 total 73.77 last 21.49) - open/close 1745 (time 1788589943.19 total 85.31 last 19.58) - open/close 2007 (time 1788589953.19 total 95.32 last 26.18) - open/close 2217 (time 1788589963.31 total 105.43 last 20.76) - open/close 2461 (time 1788589973.32 total 115.44 last 24.38) - open/close 2773 (time 1788589983.34 total 125.46 last 31.14) - open/close 2918 (time 1788589993.34 total 135.46 last 14.50) - open/close 3143 (time 1788590003.34 total 145.46 last 22.50) - open/close 3399 (time 1788590013.73 total 155.85 last 24.64) - open/close 3602 (time 1788590024.49 total 166.61 last 18.87) - open/close 3769 (time 1788590034.63 total 176.75 last 16.47) - open/close 3964 (time 1788590044.80 total 186.92 last 19.18) - open/close 4208 (time 1788590055.18 total 197.30 last 23.50) - open/close 4513 (time 1788590065.70 total 207.83 last 28.98) - open/close 4759 (time 1788590075.71 total 217.83 last 24.58) - open/close 4935 (time 1788590086.08 total 228.20 last 16.97) - open/close 5250 (time 1788590096.63 total 238.76 last 29.85) - open/close 5509 (time 1788590106.65 total 248.77 last 25.85) - open/close 5753 (time 1788590116.90 total 259.03 last 23.80) - open/close 5842 (time 1788590127.48 total 269.60 last 8.42) - open/close 5989 (time 1788590138.16 total 280.28 last 13.76) - open/close 6186 (time 1788590148.17 total 290.29 last 19.68) - open/close 6404 (time 1788590158.18 total 300.31 last 21.76) - open/close 6635 (time 1788590168.37 total 310.50 last 22.67) - open/close 6912 (time 1788590179.40 total 321.52 last 25.12) - open/close 7173 (time 1788590189.93 total 332.05 last 24.80) - open/close 7450 (time 1788590200.65 total 342.78 last 25.83) - open/close 7619 (time 1788590211.12 total 353.25 last 16.14) - open/close 7722 (time 1788590221.67 total 363.79 last 9.77) - open/close 7900 (time 1788590231.69 total 373.81 last 17.77) - open/close 8059 (time 1788590241.97 total 384.10 last 15.46) - open/close 8267 (time 1788590251.98 total 394.11 last 20.78) - open/close 8548 (time 1788590262.77 total 404.90 last 26.04) - open/close 8771 (time 1788590273.01 total 415.13 last 21.79) - open/close 9087 (time 1788590283.96 total 426.09 last 28.84) - open/close 9354 (time 1788590294.63 total 436.75 last 25.03) - open/close 9598 (time 1788590304.64 total 446.77 last 24.36) - open/close 9888 (time 1788590315.31 total 457.43 last 27.19) - open/close 10000 (time 1788590319.18 total 461.30 last 28.94) - open/close 10220 (time 1788590329.76 total 471.89 last 20.78) - open/close 10460 (time 1788590340.59 total 482.71 last 22.18) - open/close 10704 (time 1788590350.60 total 492.72 last 24.37) - open/close 10976 (time 1788590361.00 total 503.12 last 26.16) - open/close 11222 (time 1788590371.05 total 513.18 last 24.46) - open/close 11486 (time 1788590381.50 total 523.62 last 25.28) - open/close 11732 (time 1788590391.79 total 533.92 last 23.88) - open/close 12009 (time 1788590404.54 total 546.66 last 21.73) - open/close 12185 (time 1788590414.79 total 556.91 last 17.18) - open/close 12448 (time 1788590427.71 total 569.84 last 20.35) - open/close 12730 (time 1788590437.74 total 579.86 last 28.12) - open/close 12901 (time 1788590447.97 total 590.10 last 16.71) - open/close 13194 (time 1788590457.98 total 600.11 last 29.28) - open/close 13452 (time 1788590468.86 total 610.98 last 23.73) - open/close 13752 (time 1788590478.86 total 620.99 last 29.98) - open/close 13978 (time 1788590488.89 total 631.01 last 22.54) - open/close 14183 (time 1788590498.92 total 641.04 last 20.44) - open/close 14407 (time 1788590509.54 total 651.67 last 21.08) - open/close 14645 (time 1788590519.57 total 661.69 last 23.75) - open/close 14834 (time 1788590529.57 total 671.69 last 18.90) - open/close 14985 (time 1788590539.57 total 681.70 last 15.09) - open/close 15164 (time 1788590549.93 total 692.06 last 17.27) - open/close 15346 (time 1788590559.94 total 702.06 last 18.19) - open/close 15594 (time 1788590570.28 total 712.40 last 23.99) - open/close 15881 (time 1788590581.20 total 723.33 last 26.28) - open/close 16093 (time 1788590591.21 total 733.33 last 21.19) - open/close 16269 (time 1788590601.24 total 743.36 last 17.55) - open/close 16443 (time 1788590612.02 total 754.14 last 16.14) - open/close 16641 (time 1788590622.03 total 764.15 last 19.78) - open/close 16850 (time 1788590632.19 total 774.31 last 20.58) - open/close 17118 (time 1788590642.36 total 784.48 last 26.34) - open/close 17380 (time 1788590653.12 total 795.25 last 24.34) - open/close 17619 (time 1788590663.54 total 805.66 last 22.94) - open/close 17658 (time 1788590673.78 total 815.90 last 3.81) - open/close 17740 (time 1788590683.82 total 825.94 last 8.17) - open/close 17900 (time 1788590693.83 total 835.95 last 15.99) - open/close 18026 (time 1788590703.89 total 846.01 last 12.52) - open/close 18268 (time 1788590714.24 total 856.36 last 23.38) - open/close 18517 (time 1788590725.29 total 867.41 last 22.53) - open/close 18797 (time 1788590735.30 total 877.43 last 27.97) - open/close 18994 (time 1788590745.80 total 887.93 last 18.76) - open/close 19236 (time 1788590755.83 total 897.95 last 24.14) - open/close 19474 (time 1788590765.83 total 907.96 last 23.78) - open/close 19763 (time 1788590776.68 total 918.80 last 26.66) - open/close 20000 (time 1788590785.09 total 927.22 last 28.16) - open/close 20634 (time 1788590795.10 total 937.22 last 63.38) - open/close 21429 (time 1788590805.10 total 947.22 last 79.47) - open/close 22287 (time 1788590815.11 total 957.23 last 85.73) - open/close 23125 (time 1788590825.11 total 967.24 last 83.76) - open/close 23950 (time 1788590835.12 total 977.24 last 82.48) - open/close 24674 (time 1788590845.12 total 987.24 last 72.40) - open/close 25420 (time 1788590855.12 total 997.25 last 74.53) - open/close 26172 (time 1788590865.13 total 1007.26 last 75.13) - open/close 26994 (time 1788590875.15 total 1017.27 last 82.11) - open/close 27671 (time 1788590885.16 total 1027.28 last 67.63) - open/close 28373 (time 1788590895.16 total 1037.28 last 70.17) - open/close 29137 (time 1788590905.17 total 1047.29 last 76.33) - open/close 29925 (time 1788590915.17 total 1057.29 last 78.79) - open/close 30692 (time 1788590925.18 total 1067.30 last 76.65) - open/close 31420 (time 1788590935.18 total 1077.30 last 72.77) - open/close 32242 (time 1788590945.18 total 1087.31 last 82.18) - open/close 33094 (time 1788590955.19 total 1097.31 last 85.15) - open/close 33828 (time 1788590965.20 total 1107.32 last 73.32) - open/close 34556 (time 1788590975.20 total 1117.32 last 72.80) - open/close 35321 (time 1788590985.21 total 1127.33 last 76.43) - open/close 36072 (time 1788590995.22 total 1137.34 last 75.03) - open/close 36861 (time 1788591005.22 total 1147.34 last 78.90) - open/close 37559 (time 1788591015.23 total 1157.35 last 69.73) - open/close 38383 (time 1788591025.24 total 1167.36 last 82.34) - open/close 39083 (time 1788591035.24 total 1177.36 last 69.99) - open/close 39666 (time 1788591045.24 total 1187.36 last 58.29) - open/close 40000 (time 1788591049.54 total 1191.66 last 77.72) - open/close 40797 (time 1788591059.55 total 1201.67 last 79.59) - open/close 41607 (time 1788591069.56 total 1211.68 last 80.96) - open/close 42345 (time 1788591079.56 total 1221.69 last 73.74) - open/close 43171 (time 1788591089.57 total 1231.69 last 82.57) - open/close 43987 (time 1788591099.57 total 1241.69 last 81.59) - open/close 44778 (time 1788591109.58 total 1251.71 last 78.99) - open/close 45609 (time 1788591119.59 total 1261.71 last 83.06) - open/close 46375 (time 1788591129.60 total 1271.72 last 76.53) - open/close 47182 (time 1788591139.60 total 1281.72 last 80.68) - open/close 47960 (time 1788591149.60 total 1291.72 last 77.80) - open/close 48752 (time 1788591159.61 total 1301.73 last 79.15) - open/close 49515 (time 1788591169.62 total 1311.74 last 76.22) total: 50000 open/close in 1317.94 seconds: 37.94 ops/second - unlinked 0 (time 1788591180 ; total 0 ; last 0) - unlinked 10000 (time 1788591315 ; total 135 ; last 135) - unlinked 20000 (time 1788591456 ; total 276 ; last 141) - unlinked 30000 (time 1788591589 ; total 409 ; last 133) - unlinked 40000 (time 1788591717 ; total 537 ; last 128) total: 50000 unlinks in 665 seconds: 75.187973 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg652-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg652-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg652-server stop mds service on oleg652-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg652-server unloading modules via unload_modules_local on: 'oleg652-server' oleg652-server: oleg652-server.virtnet: executing unload_modules_local oleg652-server: kmemleak disabled modules unloaded. pdsh@oleg652-client: oleg652-server: ssh exited with exit code 2 pdsh@oleg652-client: oleg652-client: ssh exited with exit code 2 pdsh@oleg652-client: oleg652-client: ssh exited with exit code 2 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 2