== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:54:43 (1789268083) start mds service on oleg260-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 12 19:15 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: kmemleak disabled oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions oleg260-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg260-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg260-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg260-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg260-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 267 (time 1789268163.79 total 10.01 last 26.68) - open/close 551 (time 1789268174.26 total 20.48 last 27.12) - open/close 785 (time 1789268184.55 total 30.77 last 22.74) - open/close 1024 (time 1789268195.57 total 41.79 last 21.69) - open/close 1251 (time 1789268205.68 total 51.89 last 22.47) - open/close 1477 (time 1789268215.69 total 61.91 last 22.57) - open/close 1578 (time 1789268225.76 total 71.98 last 10.03) - open/close 1810 (time 1789268236.04 total 82.26 last 22.58) - open/close 1876 (time 1789268247.94 total 94.15 last 5.55) - open/close 1963 (time 1789268258.50 total 104.72 last 8.23) - open/close 2060 (time 1789268269.51 total 115.73 last 8.81) - open/close 2149 (time 1789268279.86 total 126.08 last 8.59) - open/close 2239 (time 1789268289.90 total 136.12 last 8.96) - open/close 2478 (time 1789268300.91 total 147.12 last 21.72) - open/close 2764 (time 1789268313.43 total 159.65 last 22.84) - open/close 2902 (time 1789268323.43 total 169.65 last 13.80) - open/close 3200 (time 1789268334.26 total 180.48 last 27.51) - open/close 3438 (time 1789268344.29 total 190.51 last 23.72) - open/close 3706 (time 1789268354.93 total 201.15 last 25.20) - open/close 3934 (time 1789268365.03 total 211.25 last 22.56) - open/close 4149 (time 1789268375.05 total 221.27 last 21.47) - open/close 4442 (time 1789268385.51 total 231.73 last 28.00) - open/close 4658 (time 1789268395.53 total 241.75 last 21.55) - open/close 4815 (time 1789268406.03 total 252.25 last 14.96) - open/close 5042 (time 1789268416.03 total 262.25 last 22.70) - open/close 5210 (time 1789268426.88 total 273.10 last 15.48) - open/close 5440 (time 1789268437.26 total 283.48 last 22.17) - open/close 5633 (time 1789268447.80 total 294.02 last 18.30) - open/close 5815 (time 1789268459.62 total 305.84 last 15.40) - open/close 6068 (time 1789268470.07 total 316.29 last 24.22) - open/close 6257 (time 1789268480.64 total 326.86 last 17.88) - open/close 6465 (time 1789268491.89 total 338.11 last 18.49) - open/close 6660 (time 1789268501.89 total 348.11 last 19.49) - open/close 6836 (time 1789268512.10 total 358.32 last 17.24) - open/close 7083 (time 1789268522.36 total 368.58 last 24.09) - open/close 7333 (time 1789268532.41 total 378.63 last 24.86) - open/close 7526 (time 1789268542.53 total 388.74 last 19.08) - open/close 7613 (time 1789268552.54 total 398.75 last 8.69) - open/close 7643 (time 1789268562.62 total 408.84 last 2.98) - open/close 7676 (time 1789268572.93 total 419.15 last 3.20) - open/close 7908 (time 1789268583.96 total 430.18 last 21.05) - open/close 8100 (time 1789268594.78 total 441.00 last 17.74) - open/close 8312 (time 1789268605.71 total 451.93 last 19.39) - open/close 8387 (time 1789268616.17 total 462.39 last 7.17) - open/close 8577 (time 1789268627.14 total 473.36 last 17.32) - open/close 8825 (time 1789268638.07 total 484.29 last 22.68) - open/close 9067 (time 1789268648.08 total 494.30 last 24.18) - open/close 9300 (time 1789268658.12 total 504.34 last 23.20) - open/close 9577 (time 1789268668.13 total 514.35 last 27.68) - open/close 9769 (time 1789268678.13 total 524.35 last 19.20) - open/close 9931 (time 1789268688.25 total 534.47 last 16.01) - open/close 10000 (time 1789268692.94 total 539.16 last 14.71) - open/close 10233 (time 1789268703.60 total 549.82 last 21.86) - open/close 10407 (time 1789268713.61 total 559.83 last 17.38) - open/close 10586 (time 1789268723.62 total 569.84 last 17.89) - open/close 10830 (time 1789268734.32 total 580.54 last 22.80) - open/close 10949 (time 1789268744.33 total 590.55 last 11.89) - open/close 11085 (time 1789268754.74 total 600.96 last 13.06) - open/close 11251 (time 1789268764.99 total 611.21 last 16.19) - open/close 11387 (time 1789268775.00 total 621.22 last 13.59) - open/close 11595 (time 1789268785.02 total 631.24 last 20.77) - open/close 11721 (time 1789268795.83 total 642.05 last 11.65) - open/close 11942 (time 1789268806.07 total 652.29 last 21.59) - open/close 12215 (time 1789268816.09 total 662.31 last 27.25) - open/close 12331 (time 1789268826.83 total 673.05 last 10.80) - open/close 12524 (time 1789268837.58 total 683.80 last 17.96) - open/close 12726 (time 1789268848.65 total 694.87 last 18.24) - open/close 12916 (time 1789268858.66 total 704.88 last 18.99) - open/close 13074 (time 1789268869.24 total 715.46 last 14.93) - open/close 13310 (time 1789268879.47 total 725.69 last 23.07) - open/close 13479 (time 1789268889.84 total 736.06 last 16.29) - open/close 13648 (time 1789268900.03 total 746.25 last 16.59) - open/close 13874 (time 1789268910.45 total 756.67 last 21.69) - open/close 14066 (time 1789268921.26 total 767.48 last 17.76) - open/close 14261 (time 1789268931.46 total 777.68 last 19.11) - open/close 14440 (time 1789268942.46 total 788.67 last 16.28) - open/close 14682 (time 1789268953.39 total 799.61 last 22.12) - open/close 14831 (time 1789268963.83 total 810.05 last 14.27) - open/close 14969 (time 1789268975.09 total 821.31 last 12.26) - open/close 15085 (time 1789268985.14 total 831.36 last 11.54) - open/close 15273 (time 1789268995.16 total 841.38 last 18.77) - open/close 15451 (time 1789269005.49 total 851.71 last 17.23) - open/close 15638 (time 1789269015.51 total 861.73 last 18.66) - open/close 15945 (time 1789269026.23 total 872.45 last 28.65) - open/close 16196 (time 1789269037.21 total 883.43 last 22.85) - open/close 16467 (time 1789269047.22 total 893.44 last 27.08) - open/close 16623 (time 1789269057.22 total 903.44 last 15.59) - open/close 16874 (time 1789269068.15 total 914.37 last 22.96) - open/close 17025 (time 1789269078.32 total 924.54 last 14.85) - open/close 17279 (time 1789269089.34 total 935.56 last 23.06) - open/close 17508 (time 1789269099.97 total 946.19 last 21.54) - open/close 17781 (time 1789269109.98 total 956.20 last 27.28) - open/close 18020 (time 1789269121.94 total 968.16 last 19.98) - open/close 18184 (time 1789269132.82 total 979.04 last 15.07) - open/close 18439 (time 1789269142.83 total 989.05 last 25.48) - open/close 19168 (time 1789269152.83 total 999.05 last 72.88) - open/close 19760 (time 1789269162.85 total 1009.07 last 59.12) - open/close 20000 (time 1789269166.80 total 1013.02 last 60.76) - open/close 20689 (time 1789269176.81 total 1023.03 last 68.80) - open/close 21417 (time 1789269186.81 total 1033.03 last 72.79) - open/close 22175 (time 1789269196.82 total 1043.04 last 75.74) - open/close 23011 (time 1789269206.83 total 1053.04 last 83.57) - open/close 23875 (time 1789269216.85 total 1063.06 last 86.23) - open/close 24749 (time 1789269226.85 total 1073.07 last 87.32) - open/close 25693 (time 1789269236.86 total 1083.07 last 94.39) - open/close 26581 (time 1789269246.88 total 1093.10 last 88.59) - open/close 27325 (time 1789269256.89 total 1103.11 last 74.32) - open/close 27932 (time 1789269266.90 total 1113.12 last 60.66) - open/close 28680 (time 1789269276.91 total 1123.13 last 74.71) - open/close 29481 (time 1789269286.91 total 1133.13 last 80.06) - open/close 30000 (time 1789269293.75 total 1139.97 last 75.87) - open/close 30861 (time 1789269303.76 total 1149.98 last 86.01) - open/close 31697 (time 1789269313.77 total 1159.99 last 83.58) - open/close 32512 (time 1789269323.77 total 1169.99 last 81.48) - open/close 33430 (time 1789269333.77 total 1179.99 last 91.76) - open/close 34675 (time 1789269343.80 total 1190.02 last 124.15) - open/close 35493 (time 1789269353.81 total 1200.02 last 81.77) - open/close 36352 (time 1789269363.81 total 1210.03 last 85.85) - open/close 37200 (time 1789269373.82 total 1220.04 last 84.73) - open/close 38107 (time 1789269383.83 total 1230.05 last 90.62) - open/close 38929 (time 1789269393.84 total 1240.06 last 82.13) - open/close 39757 (time 1789269403.84 total 1250.06 last 82.78) - open/close 40000 (time 1789269407.16 total 1253.38 last 73.24) - open/close 40871 (time 1789269417.17 total 1263.39 last 86.99) - open/close 41717 (time 1789269427.22 total 1273.44 last 84.21) - open/close 42421 (time 1789269437.23 total 1283.45 last 70.32) - open/close 43124 (time 1789269447.25 total 1293.47 last 70.16) - open/close 43898 (time 1789269457.26 total 1303.48 last 77.29) - open/close 44708 (time 1789269467.27 total 1313.48 last 80.96) - open/close 45499 (time 1789269477.27 total 1323.49 last 79.05) - open/close 46235 (time 1789269487.28 total 1333.50 last 73.56) - open/close 46980 (time 1789269497.29 total 1343.51 last 74.43) - open/close 47800 (time 1789269507.32 total 1353.54 last 81.72) - open/close 48645 (time 1789269517.33 total 1363.55 last 84.44) - open/close 49471 (time 1789269527.34 total 1373.56 last 82.50) total: 50000 open/close in 1380.66 seconds: 36.21 ops/second - unlinked 0 (time 1789269538 ; total 0 ; last 0) - unlinked 10000 (time 1789269657 ; total 119 ; last 119) - unlinked 20000 (time 1789269783 ; total 245 ; last 126) - unlinked 30000 (time 1789269906 ; total 368 ; last 123) - unlinked 40000 (time 1789270029 ; total 491 ; last 123) total: 50000 unlinks in 617 seconds: 81.037277 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg260-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server unloading modules via unload_modules_local on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing unload_modules_local oleg260-server: kmemleak disabled modules unloaded. pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2