== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:45:23 (1788846323) start mds service on oleg135-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 7 17:42 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: kmemleak disabled oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions oleg135-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg135-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg135-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg135-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg135-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 169 (time 1788846420.76 total 10.00 last 16.90) - open/close 289 (time 1788846430.78 total 20.02 last 11.98) - open/close 509 (time 1788846441.50 total 30.74 last 20.52) - open/close 551 (time 1788846457.07 total 46.32 last 2.70) - open/close 771 (time 1788846467.11 total 56.35 last 21.92) - open/close 941 (time 1788846477.12 total 66.36 last 16.98) - open/close 1082 (time 1788846487.60 total 76.84 last 13.46) - open/close 1286 (time 1788846498.49 total 87.73 last 18.74) - open/close 1577 (time 1788846508.49 total 97.74 last 29.08) - open/close 1632 (time 1788846519.55 total 108.79 last 4.98) - open/close 1837 (time 1788846530.23 total 119.47 last 19.19) - open/close 1942 (time 1788846540.23 total 129.47 last 10.50) - open/close 2128 (time 1788846550.24 total 139.48 last 18.58) - open/close 2350 (time 1788846560.71 total 149.95 last 21.21) - open/close 2457 (time 1788846571.39 total 160.63 last 10.02) - open/close 2542 (time 1788846581.39 total 170.63 last 8.50) - open/close 2682 (time 1788846591.79 total 181.03 last 13.47) - open/close 2899 (time 1788846601.81 total 191.05 last 21.66) - open/close 2960 (time 1788846611.85 total 201.09 last 6.07) - open/close 3160 (time 1788846622.85 total 212.09 last 18.18) - open/close 3234 (time 1788846633.10 total 222.35 last 7.22) - open/close 3434 (time 1788846643.13 total 232.37 last 19.95) - open/close 3587 (time 1788846655.31 total 244.55 last 12.56) - open/close 3744 (time 1788846665.37 total 254.61 last 15.60) - open/close 3844 (time 1788846676.16 total 265.41 last 9.26) - open/close 3960 (time 1788846686.91 total 276.15 last 10.79) - open/close 4173 (time 1788846698.74 total 287.98 last 18.01) - open/close 4284 (time 1788846709.16 total 298.40 last 10.65) - open/close 4326 (time 1788846719.35 total 308.60 last 4.12) - open/close 4424 (time 1788846729.48 total 318.72 last 9.68) - open/close 4525 (time 1788846739.48 total 328.73 last 10.10) - open/close 4682 (time 1788846749.48 total 338.73 last 15.70) - open/close 4879 (time 1788846759.60 total 348.84 last 19.48) - open/close 4949 (time 1788846769.61 total 358.85 last 6.99) - open/close 5198 (time 1788846780.08 total 369.32 last 23.77) - open/close 5321 (time 1788846790.22 total 379.46 last 12.13) - open/close 5552 (time 1788846800.71 total 389.95 last 22.03) - open/close 5728 (time 1788846810.72 total 399.96 last 17.57) - open/close 5823 (time 1788846821.39 total 410.63 last 8.91) - open/close 5995 (time 1788846831.40 total 420.64 last 17.18) - open/close 6205 (time 1788846842.37 total 431.61 last 19.15) - open/close 6314 (time 1788846853.45 total 442.69 last 9.84) - open/close 6494 (time 1788846863.45 total 452.69 last 18.00) - open/close 6612 (time 1788846873.46 total 462.70 last 11.79) - open/close 6825 (time 1788846884.35 total 473.59 last 19.56) - open/close 6961 (time 1788846895.34 total 484.59 last 12.37) - open/close 7043 (time 1788846906.81 total 496.05 last 7.15) - open/close 7221 (time 1788846916.82 total 506.06 last 17.79) - open/close 7342 (time 1788846928.94 total 518.19 last 9.98) - open/close 7561 (time 1788846938.96 total 528.20 last 21.87) - open/close 7708 (time 1788846948.99 total 538.23 last 14.65) - open/close 7904 (time 1788846959.10 total 548.34 last 19.38) - open/close 8026 (time 1788846969.14 total 558.39 last 12.15) - open/close 8200 (time 1788846979.15 total 568.39 last 17.40) - open/close 8400 (time 1788846989.70 total 578.94 last 18.95) - open/close 8522 (time 1788847000.44 total 589.68 last 11.36) - open/close 8714 (time 1788847010.44 total 599.68 last 19.20) - open/close 8925 (time 1788847020.57 total 609.81 last 20.82) - open/close 9047 (time 1788847030.96 total 620.20 last 11.75) - open/close 9249 (time 1788847043.25 total 632.50 last 16.42) - open/close 9457 (time 1788847053.90 total 643.14 last 19.53) - open/close 9598 (time 1788847064.58 total 653.82 last 13.21) - open/close 9798 (time 1788847074.60 total 663.84 last 19.96) - open/close 9893 (time 1788847084.61 total 673.85 last 9.49) - open/close 9919 (time 1788847094.62 total 683.86 last 2.60) - open/close 10000 (time 1788847101.60 total 690.85 last 11.59) - open/close 10131 (time 1788847111.61 total 700.86 last 13.09) - open/close 10294 (time 1788847122.63 total 711.87 last 14.80) - open/close 10521 (time 1788847133.26 total 722.51 last 21.35) - open/close 10777 (time 1788847143.30 total 732.54 last 25.52) - open/close 11000 (time 1788847154.11 total 743.36 last 20.62) - open/close 11176 (time 1788847164.16 total 753.40 last 17.52) - open/close 11379 (time 1788847174.38 total 763.62 last 19.86) - open/close 11591 (time 1788847184.90 total 774.14 last 20.16) - open/close 11707 (time 1788847194.93 total 784.17 last 11.56) - open/close 11876 (time 1788847204.93 total 794.18 last 16.89) - open/close 12008 (time 1788847215.05 total 804.29 last 13.05) - open/close 12235 (time 1788847225.33 total 814.57 last 22.07) - open/close 12400 (time 1788847235.35 total 824.59 last 16.47) - open/close 12528 (time 1788847245.36 total 834.61 last 12.78) - open/close 12637 (time 1788847255.37 total 844.61 last 10.89) - open/close 12739 (time 1788847265.57 total 854.81 last 10.00) - open/close 12946 (time 1788847275.58 total 864.83 last 20.67) - open/close 13153 (time 1788847285.70 total 874.94 last 20.47) - open/close 13288 (time 1788847295.89 total 885.13 last 13.24) - open/close 13414 (time 1788847306.14 total 895.39 last 12.29) - open/close 13495 (time 1788847316.15 total 905.39 last 8.10) - open/close 13661 (time 1788847326.84 total 916.08 last 15.52) - open/close 13863 (time 1788847336.90 total 926.14 last 20.08) - open/close 14087 (time 1788847347.63 total 936.88 last 20.87) - open/close 14238 (time 1788847357.64 total 946.88 last 15.10) - open/close 14435 (time 1788847367.76 total 957.00 last 19.47) - open/close 14522 (time 1788847378.37 total 967.62 last 8.19) - open/close 14717 (time 1788847389.05 total 978.29 last 18.26) - open/close 14732 (time 1788847401.71 total 990.95 last 1.19) - open/close 14801 (time 1788847415.78 total 1005.03 last 4.90) - open/close 14971 (time 1788847426.39 total 1015.64 last 16.02) - open/close 15205 (time 1788847436.41 total 1025.65 last 23.36) - open/close 15366 (time 1788847452.50 total 1041.74 last 10.01) - open/close 15576 (time 1788847462.51 total 1051.75 last 20.98) - open/close 15712 (time 1788847473.10 total 1062.34 last 12.85) - open/close 16226 (time 1788847483.10 total 1072.34 last 51.39) - open/close 17110 (time 1788847493.10 total 1082.34 last 88.37) - open/close 17748 (time 1788847503.11 total 1092.35 last 63.73) - open/close 18590 (time 1788847513.12 total 1102.36 last 84.15) - open/close 19231 (time 1788847523.13 total 1112.37 last 64.03) - open/close 19803 (time 1788847533.13 total 1122.38 last 57.17) - open/close 20603 (time 1788847543.14 total 1132.38 last 79.98) - open/close 21290 (time 1788847553.16 total 1142.40 last 68.57) - open/close 21782 (time 1788847563.16 total 1152.41 last 49.16) - open/close 22352 (time 1788847573.18 total 1162.42 last 56.90) - open/close 23067 (time 1788847583.19 total 1172.43 last 71.44) - open/close 23862 (time 1788847593.20 total 1182.44 last 79.43) - open/close 24607 (time 1788847603.20 total 1192.44 last 74.46) - open/close 25326 (time 1788847613.21 total 1202.45 last 71.84) - open/close 26171 (time 1788847623.21 total 1212.45 last 84.50) - open/close 26878 (time 1788847633.22 total 1222.47 last 70.61) - open/close 27599 (time 1788847643.23 total 1232.47 last 72.08) - open/close 28346 (time 1788847653.23 total 1242.47 last 74.66) - open/close 28987 (time 1788847663.25 total 1252.49 last 64.02) - open/close 29520 (time 1788847673.25 total 1262.49 last 53.27) - open/close 30000 (time 1788847681.09 total 1270.33 last 61.22) - open/close 30467 (time 1788847691.20 total 1280.45 last 46.19) - open/close 31184 (time 1788847701.22 total 1290.46 last 71.57) - open/close 31865 (time 1788847711.23 total 1300.48 last 68.02) - open/close 32437 (time 1788847721.24 total 1310.48 last 57.16) - open/close 32968 (time 1788847731.26 total 1320.50 last 53.02) - open/close 33575 (time 1788847741.26 total 1330.50 last 60.69) - open/close 34169 (time 1788847751.28 total 1340.52 last 59.25) - open/close 34823 (time 1788847761.30 total 1350.54 last 65.27) - open/close 35435 (time 1788847771.32 total 1360.56 last 61.10) - open/close 36338 (time 1788847781.32 total 1370.56 last 90.27) - open/close 37057 (time 1788847791.33 total 1380.57 last 71.88) - open/close 37955 (time 1788847801.33 total 1390.57 last 89.76) - open/close 38741 (time 1788847811.34 total 1400.58 last 78.55) - open/close 39474 (time 1788847821.35 total 1410.59 last 73.22) - open/close 40000 (time 1788847828.64 total 1417.88 last 72.15) - open/close 40618 (time 1788847838.64 total 1427.88 last 61.79) - open/close 41188 (time 1788847848.65 total 1437.89 last 56.94) - open/close 41633 (time 1788847858.66 total 1447.90 last 44.47) - open/close 42177 (time 1788847868.67 total 1457.92 last 54.30) - open/close 42798 (time 1788847878.68 total 1467.92 last 62.05) - open/close 43458 (time 1788847888.69 total 1477.93 last 65.95) - open/close 44182 (time 1788847898.70 total 1487.95 last 72.31) - open/close 44756 (time 1788847908.74 total 1497.98 last 57.21) - open/close 45318 (time 1788847918.75 total 1507.99 last 56.13) - open/close 45800 (time 1788847928.75 total 1517.99 last 48.18) - open/close 46462 (time 1788847938.76 total 1528.01 last 66.12) - open/close 47232 (time 1788847948.76 total 1538.01 last 76.99) - open/close 47919 (time 1788847958.78 total 1548.02 last 68.63) - open/close 48464 (time 1788847968.78 total 1558.02 last 54.47) - open/close 49062 (time 1788847978.78 total 1568.03 last 59.79) - open/close 49776 (time 1788847988.79 total 1578.03 last 71.38) total: 50000 open/close in 1581.04 seconds: 31.62 ops/second - unlinked 0 (time 1788847997 ; total 0 ; last 0) - unlinked 10000 (time 1788848168 ; total 171 ; last 171) - unlinked 20000 (time 1788848326 ; total 329 ; last 158) - unlinked 30000 (time 1788848478 ; total 481 ; last 152) - unlinked 40000 (time 1788848630 ; total 633 ; last 152) total: 50000 unlinks in 785 seconds: 63.694267 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg135-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server stop mds service on oleg135-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server unloading modules via unload_modules_local on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing unload_modules_local oleg135-server: kmemleak disabled modules unloaded. pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-client: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2