== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:01:27 (1789610487) start mds service on oleg329-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931640 Sep 16 13:21 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg329-server' oleg329-server: oleg329-server.virtnet: executing load_modules_local oleg329-server: kmemleak disabled oleg329-server: Loading modules from /home/green/git/lustre-release/lustre oleg329-server: detected 4 online CPUs by sysfs oleg329-server: MODOPTS_LIBCFS= oleg329-server: Force libcfs to create 2 CPU partitions oleg329-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg329-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg329-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg329-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg329-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 145 (time 1789610576.59 total 10.70 last 13.55) - open/close 275 (time 1789610586.74 total 20.85 last 12.81) - open/close 537 (time 1789610596.74 total 30.85 last 26.20) - open/close 695 (time 1789610607.02 total 41.13 last 15.37) - open/close 896 (time 1789610619.01 total 53.12 last 16.76) - open/close 1085 (time 1789610629.02 total 63.13 last 18.89) - open/close 1346 (time 1789610639.99 total 74.10 last 23.79) - open/close 1589 (time 1789610649.99 total 84.10 last 24.29) - open/close 1852 (time 1789610661.26 total 95.37 last 23.33) - open/close 2076 (time 1789610671.80 total 105.91 last 21.25) - open/close 2296 (time 1789610682.22 total 116.33 last 21.11) - open/close 2571 (time 1789610693.23 total 127.34 last 24.99) - open/close 2775 (time 1789610703.36 total 137.47 last 20.13) - open/close 3011 (time 1789610714.24 total 148.35 last 21.68) - open/close 3218 (time 1789610724.25 total 158.36 last 20.69) - open/close 3449 (time 1789610735.11 total 169.21 last 21.28) - open/close 3665 (time 1789610745.52 total 179.63 last 20.74) - open/close 3778 (time 1789610756.07 total 190.18 last 10.71) - open/close 3919 (time 1789610766.59 total 200.69 last 13.41) - open/close 4018 (time 1789610776.59 total 210.69 last 9.90) - open/close 4166 (time 1789610786.80 total 220.91 last 14.49) - open/close 4432 (time 1789610797.81 total 231.92 last 24.17) - open/close 4649 (time 1789610807.83 total 241.93 last 21.66) - open/close 4751 (time 1789610818.23 total 252.33 last 9.81) - open/close 4951 (time 1789610828.23 total 262.33 last 20.00) - open/close 5142 (time 1789610838.89 total 273.00 last 17.91) - open/close 5348 (time 1789610849.86 total 283.97 last 18.77) - open/close 5547 (time 1789610860.15 total 294.26 last 19.35) - open/close 5758 (time 1789610870.15 total 304.26 last 21.09) - open/close 5927 (time 1789610880.82 total 314.93 last 15.84) - open/close 6175 (time 1789610890.84 total 324.94 last 24.76) - open/close 6425 (time 1789610900.85 total 334.96 last 24.97) - open/close 6669 (time 1789610911.60 total 345.71 last 22.69) - open/close 6728 (time 1789610922.10 total 356.20 last 5.62) - open/close 6835 (time 1789610932.89 total 366.99 last 9.92) - open/close 7048 (time 1789610943.72 total 377.83 last 19.66) - open/close 7244 (time 1789610954.64 total 388.74 last 17.96) - open/close 7423 (time 1789610964.90 total 399.01 last 17.44) - open/close 7596 (time 1789610975.74 total 409.85 last 15.96) - open/close 7803 (time 1789610986.00 total 420.11 last 20.17) - open/close 8012 (time 1789610996.01 total 430.11 last 20.89) - open/close 8230 (time 1789611006.47 total 440.57 last 20.84) - open/close 8463 (time 1789611017.20 total 451.31 last 21.70) - open/close 8640 (time 1789611027.21 total 461.32 last 17.68) - open/close 8815 (time 1789611037.23 total 471.34 last 17.47) - open/close 8999 (time 1789611047.43 total 481.54 last 18.04) - open/close 9075 (time 1789611059.59 total 493.70 last 6.25) - open/close 9239 (time 1789611069.60 total 503.71 last 16.39) - open/close 9415 (time 1789611079.64 total 513.75 last 17.53) - open/close 9588 (time 1789611089.65 total 523.76 last 17.28) - open/close 9753 (time 1789611102.39 total 536.49 last 12.96) - open/close 9867 (time 1789611112.43 total 546.54 last 11.35) - open/close 10000 (time 1789611120.52 total 554.62 last 16.45) - open/close 10147 (time 1789611130.81 total 564.92 last 14.28) - open/close 10323 (time 1789611141.81 total 575.91 last 16.01) - open/close 10445 (time 1789611152.07 total 586.17 last 11.89) - open/close 10623 (time 1789611162.08 total 596.18 last 17.78) - open/close 10800 (time 1789611172.09 total 606.19 last 17.68) - open/close 11019 (time 1789611182.98 total 617.09 last 20.10) - open/close 11219 (time 1789611192.98 total 627.09 last 20.00) - open/close 11391 (time 1789611202.99 total 637.10 last 17.18) - open/close 11583 (time 1789611213.33 total 647.44 last 18.56) - open/close 11769 (time 1789611223.53 total 657.64 last 18.24) - open/close 11867 (time 1789611233.85 total 667.95 last 9.50) - open/close 12083 (time 1789611243.85 total 677.96 last 21.59) - open/close 12318 (time 1789611253.86 total 687.97 last 23.48) - open/close 12482 (time 1789611264.50 total 698.61 last 15.41) - open/close 12687 (time 1789611275.08 total 709.19 last 19.38) - open/close 12919 (time 1789611285.09 total 719.20 last 23.16) - open/close 13139 (time 1789611295.35 total 729.46 last 21.45) - open/close 13357 (time 1789611305.64 total 739.75 last 21.18) - open/close 13620 (time 1789611315.65 total 749.75 last 26.29) - open/close 13882 (time 1789611325.66 total 759.77 last 26.16) - open/close 14091 (time 1789611335.94 total 770.05 last 20.34) - open/close 14318 (time 1789611345.94 total 780.05 last 22.69) - open/close 14442 (time 1789611355.95 total 790.06 last 12.39) - open/close 14623 (time 1789611366.13 total 800.24 last 17.78) - open/close 14863 (time 1789611376.13 total 810.24 last 23.99) - open/close 15094 (time 1789611386.14 total 820.25 last 23.09) - open/close 15273 (time 1789611396.22 total 830.32 last 17.76) - open/close 15458 (time 1789611406.45 total 840.56 last 18.08) - open/close 15621 (time 1789611417.19 total 851.30 last 15.18) - open/close 15746 (time 1789611427.41 total 861.51 last 12.23) - open/close 15845 (time 1789611438.19 total 872.30 last 9.18) - open/close 16060 (time 1789611451.32 total 885.43 last 16.38) - open/close 16282 (time 1789611461.40 total 895.50 last 22.04) - open/close 16408 (time 1789611472.45 total 906.56 last 11.40) - open/close 16552 (time 1789611483.03 total 917.14 last 13.61) - open/close 16790 (time 1789611493.04 total 927.15 last 23.78) - open/close 17021 (time 1789611504.04 total 938.15 last 21.00) - open/close 17216 (time 1789611514.07 total 948.17 last 19.46) - open/close 17415 (time 1789611524.17 total 958.28 last 19.70) - open/close 17670 (time 1789611534.18 total 968.29 last 25.48) - open/close 17862 (time 1789611546.35 total 980.46 last 15.77) - open/close 18084 (time 1789611556.36 total 990.47 last 22.19) - open/close 18594 (time 1789611566.36 total 1000.47 last 50.98) - open/close 19226 (time 1789611576.36 total 1010.47 last 63.19) - open/close 19787 (time 1789611586.37 total 1020.48 last 56.06) - open/close 20000 (time 1789611589.86 total 1023.97 last 61.04) - open/close 20617 (time 1789611599.86 total 1033.97 last 61.69) - open/close 21216 (time 1789611609.88 total 1043.98 last 59.82) - open/close 21892 (time 1789611619.89 total 1053.99 last 67.52) - open/close 22521 (time 1789611629.89 total 1064.00 last 62.88) - open/close 23121 (time 1789611639.89 total 1074.00 last 59.98) - open/close 23748 (time 1789611649.90 total 1084.01 last 62.64) - open/close 24417 (time 1789611659.90 total 1094.01 last 66.88) - open/close 25013 (time 1789611669.93 total 1104.03 last 59.48) - open/close 25606 (time 1789611679.93 total 1114.03 last 59.30) - open/close 26169 (time 1789611689.95 total 1124.06 last 56.18) - open/close 26750 (time 1789611699.97 total 1134.08 last 57.95) - open/close 27341 (time 1789611709.99 total 1144.10 last 59.01) - open/close 27989 (time 1789611719.99 total 1154.10 last 64.76) - open/close 28636 (time 1789611730.00 total 1164.11 last 64.64) - open/close 29286 (time 1789611740.03 total 1174.14 last 64.84) - open/close 29883 (time 1789611750.05 total 1184.16 last 59.55) - open/close 30512 (time 1789611760.06 total 1194.16 last 62.87) - open/close 31155 (time 1789611770.07 total 1204.18 last 64.22) - open/close 31802 (time 1789611780.07 total 1214.18 last 64.70) - open/close 32454 (time 1789611790.07 total 1224.18 last 65.18) - open/close 33106 (time 1789611800.10 total 1234.20 last 65.05) - open/close 33758 (time 1789611810.11 total 1244.21 last 65.14) - open/close 34380 (time 1789611820.12 total 1254.22 last 62.13) - open/close 35046 (time 1789611830.12 total 1264.22 last 66.60) - open/close 35667 (time 1789611840.13 total 1274.23 last 62.04) - open/close 36315 (time 1789611850.13 total 1284.24 last 64.78) - open/close 36934 (time 1789611860.13 total 1294.24 last 61.88) - open/close 37610 (time 1789611870.14 total 1304.25 last 67.52) - open/close 38257 (time 1789611880.15 total 1314.26 last 64.67) - open/close 38918 (time 1789611890.15 total 1324.26 last 66.06) - open/close 39595 (time 1789611900.18 total 1334.29 last 67.52) - open/close 40000 (time 1789611905.81 total 1339.91 last 71.99) - open/close 40667 (time 1789611915.81 total 1349.92 last 66.69) - open/close 41360 (time 1789611925.86 total 1359.97 last 68.92) - open/close 41887 (time 1789611935.86 total 1369.97 last 52.69) - open/close 42385 (time 1789611945.88 total 1379.99 last 49.72) - open/close 42899 (time 1789611955.89 total 1389.99 last 51.37) - open/close 43501 (time 1789611965.89 total 1400.00 last 60.18) - open/close 44077 (time 1789611975.91 total 1410.02 last 57.46) - open/close 44665 (time 1789611985.92 total 1420.03 last 58.78) - open/close 45179 (time 1789611995.92 total 1430.03 last 51.39) - open/close 45776 (time 1789612005.93 total 1440.04 last 59.64) - open/close 46422 (time 1789612015.93 total 1450.04 last 64.58) - open/close 47026 (time 1789612025.95 total 1460.06 last 60.28) - open/close 47684 (time 1789612035.96 total 1470.07 last 65.72) - open/close 48288 (time 1789612045.98 total 1480.09 last 60.29) - open/close 48939 (time 1789612055.99 total 1490.09 last 65.07) - open/close 49601 (time 1789612065.99 total 1500.10 last 66.18) total: 50000 open/close in 1506.42 seconds: 33.19 ops/second - unlinked 0 (time 1789612076 ; total 0 ; last 0) - unlinked 10000 (time 1789612235 ; total 159 ; last 159) - unlinked 20000 (time 1789612399 ; total 323 ; last 164) - unlinked 30000 (time 1789612568 ; total 492 ; last 169) - unlinked 40000 (time 1789612730 ; total 654 ; last 162) total: 50000 unlinks in 820 seconds: 60.975609 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg329-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg329-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg329-server stop mds service on oleg329-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg329-server unloading modules via unload_modules_local on: 'oleg329-server' oleg329-server: oleg329-server.virtnet: executing unload_modules_local oleg329-server: kmemleak disabled modules unloaded. pdsh@oleg329-client: oleg329-client: ssh exited with exit code 2 pdsh@oleg329-client: oleg329-server: ssh exited with exit code 2 pdsh@oleg329-client: oleg329-client: ssh exited with exit code 2 pdsh@oleg329-client: oleg329-server: ssh exited with exit code 2