== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:15:29 (1710101729) start mds service on oleg134-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing load_modules_local oleg134-server: Loading modules from /home/green/git/lustre-release/lustre oleg134-server: detected 4 online CPUs by sysfs oleg134-server: Force libcfs to create 2 CPU partitions oleg134-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg134-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg134-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock oleg134-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 187 (time 1710101782.15 total 10.00 last 18.69) - open/close 407 (time 1710101792.16 total 20.01 last 21.99) - open/close 734 (time 1710101803.07 total 30.92 last 29.96) - open/close 927 (time 1710101813.87 total 41.72 last 17.87) - open/close 1148 (time 1710101824.82 total 52.67 last 20.18) - open/close 1370 (time 1710101835.41 total 63.25 last 20.98) - open/close 1490 (time 1710101845.58 total 73.43 last 11.79) - open/close 1645 (time 1710101856.24 total 84.09 last 14.55) - open/close 1879 (time 1710101866.24 total 94.09 last 23.38) - open/close 2067 (time 1710101876.95 total 104.80 last 17.56) - open/close 2277 (time 1710101887.79 total 115.63 last 19.38) - open/close 2498 (time 1710101898.51 total 126.36 last 20.61) - open/close 2627 (time 1710101909.18 total 137.03 last 12.09) - open/close 2830 (time 1710101920.13 total 147.98 last 18.53) - open/close 3053 (time 1710101930.24 total 158.09 last 22.07) - open/close 3238 (time 1710101941.08 total 168.93 last 17.06) - open/close 3422 (time 1710101951.09 total 178.94 last 18.38) - open/close 3438 (time 1710101962.54 total 190.39 last 1.40) - open/close 3618 (time 1710101972.54 total 200.39 last 17.99) - open/close 3788 (time 1710101983.25 total 211.10 last 15.88) - open/close 4001 (time 1710101993.57 total 221.42 last 20.64) - open/close 4195 (time 1710102003.57 total 231.42 last 19.40) - open/close 4384 (time 1710102013.60 total 241.45 last 18.84) - open/close 4592 (time 1710102023.60 total 251.45 last 20.80) - open/close 4708 (time 1710102033.88 total 261.73 last 11.29) - open/close 4857 (time 1710102044.62 total 272.47 last 13.87) - open/close 5039 (time 1710102054.62 total 282.47 last 18.19) - open/close 5188 (time 1710102064.62 total 292.47 last 14.90) - open/close 5360 (time 1710102075.56 total 303.41 last 15.72) - open/close 5613 (time 1710102085.93 total 313.78 last 24.38) - open/close 5768 (time 1710102095.93 total 323.78 last 15.50) - open/close 5981 (time 1710102105.98 total 333.82 last 21.21) - open/close 6148 (time 1710102115.98 total 343.83 last 16.69) - open/close 6301 (time 1710102126.60 total 354.45 last 14.41) - open/close 6470 (time 1710102137.43 total 365.28 last 15.60) - open/close 6620 (time 1710102147.79 total 375.64 last 14.48) - open/close 6767 (time 1710102158.34 total 386.19 last 13.93) - open/close 6899 (time 1710102168.90 total 396.75 last 12.50) - open/close 7062 (time 1710102178.91 total 406.76 last 16.28) - open/close 7199 (time 1710102189.22 total 417.07 last 13.28) - open/close 7374 (time 1710102199.24 total 427.09 last 17.47) - open/close 7605 (time 1710102209.25 total 437.10 last 23.08) - open/close 7799 (time 1710102219.27 total 447.12 last 19.35) - open/close 8023 (time 1710102230.19 total 458.04 last 20.51) - open/close 8199 (time 1710102240.24 total 468.09 last 17.52) - open/close 8344 (time 1710102250.47 total 478.32 last 14.17) - open/close 8523 (time 1710102261.14 total 488.99 last 16.77) - open/close 8653 (time 1710102271.67 total 499.52 last 12.34) - open/close 8829 (time 1710102282.35 total 510.20 last 16.49) - open/close 8989 (time 1710102292.35 total 520.20 last 15.99) - open/close 9088 (time 1710102302.99 total 530.84 last 9.30) - open/close 9264 (time 1710102313.91 total 541.76 last 16.11) - open/close 9441 (time 1710102324.80 total 552.65 last 16.25) - open/close 9680 (time 1710102334.82 total 562.67 last 23.86) - open/close 9829 (time 1710102344.84 total 572.69 last 14.88) - open/close 9951 (time 1710102354.97 total 582.82 last 12.04) - open/close 10000 (time 1710102359.37 total 587.22 last 11.13) - open/close 10120 (time 1710102369.38 total 597.23 last 11.99) - open/close 10284 (time 1710102380.10 total 607.95 last 15.29) - open/close 10464 (time 1710102390.13 total 617.98 last 17.96) - open/close 10657 (time 1710102400.15 total 628.00 last 19.25) - open/close 10821 (time 1710102410.92 total 638.77 last 15.23) - open/close 10967 (time 1710102421.74 total 649.59 last 13.50) - open/close 11125 (time 1710102431.74 total 659.59 last 15.79) - open/close 11293 (time 1710102441.75 total 669.60 last 16.78) - open/close 11733 (time 1710102451.77 total 679.62 last 43.91) - open/close 12106 (time 1710102461.78 total 689.63 last 37.27) - open/close 12544 (time 1710102471.79 total 699.64 last 43.76) - open/close 12933 (time 1710102481.80 total 709.65 last 38.85) - open/close 13347 (time 1710102491.83 total 719.68 last 41.29) - open/close 13718 (time 1710102501.87 total 729.71 last 36.96) - open/close 14061 (time 1710102511.88 total 739.73 last 34.25) - open/close 14400 (time 1710102521.88 total 749.73 last 33.88) - open/close 14781 (time 1710102531.89 total 759.74 last 38.08) - open/close 15136 (time 1710102541.99 total 769.84 last 35.16) - open/close 15476 (time 1710102552.05 total 779.90 last 33.78) - open/close 15676 (time 1710102570.51 total 798.36 last 10.83) - open/close 16048 (time 1710102580.53 total 808.38 last 37.15) - open/close 16385 (time 1710102590.56 total 818.41 last 33.60) - open/close 16723 (time 1710102600.57 total 828.42 last 33.76) - open/close 17066 (time 1710102610.58 total 838.43 last 34.26) - open/close 17332 (time 1710102620.61 total 848.46 last 26.51) - open/close 17588 (time 1710102630.62 total 858.47 last 25.59) - open/close 17896 (time 1710102640.64 total 868.49 last 30.73) - open/close 18190 (time 1710102650.64 total 878.49 last 29.40) - open/close 18446 (time 1710102660.67 total 888.52 last 25.52) - open/close 18724 (time 1710102670.70 total 898.55 last 27.72) - open/close 18964 (time 1710102680.70 total 908.55 last 24.00) - open/close 19222 (time 1710102690.76 total 918.61 last 25.66) - open/close 19464 (time 1710102700.78 total 928.63 last 24.15) - open/close 19716 (time 1710102710.81 total 938.66 last 25.12) - open/close 20000 (time 1710102720.40 total 948.25 last 29.61) - open/close 20274 (time 1710102730.43 total 958.28 last 27.31) - open/close 20536 (time 1710102740.44 total 968.29 last 26.18) - open/close 20828 (time 1710102750.46 total 978.31 last 29.16) - open/close 21161 (time 1710102760.46 total 988.31 last 33.29) - open/close 21394 (time 1710102770.47 total 998.32 last 23.28) - open/close 21638 (time 1710102780.47 total 1008.32 last 24.39) - open/close 21867 (time 1710102790.48 total 1018.33 last 22.89) - open/close 22145 (time 1710102800.49 total 1028.34 last 27.76) - open/close 22402 (time 1710102810.51 total 1038.36 last 25.67) - open/close 22661 (time 1710102820.55 total 1048.40 last 25.79) - open/close 22884 (time 1710102830.56 total 1058.41 last 22.27) - open/close 23111 (time 1710102840.58 total 1068.43 last 22.65) - open/close 23314 (time 1710102850.63 total 1078.48 last 20.21) - open/close 23383 (time 1710102865.10 total 1092.95 last 4.77) - open/close 23631 (time 1710102875.14 total 1102.99 last 24.70) - open/close 23854 (time 1710102885.15 total 1113.00 last 22.26) - open/close 24074 (time 1710102895.18 total 1123.02 last 21.95) - open/close 24299 (time 1710102905.25 total 1133.10 last 22.33) - open/close 24544 (time 1710102915.27 total 1143.12 last 24.45) - open/close 24786 (time 1710102925.28 total 1153.13 last 24.18) - open/close 25019 (time 1710102935.34 total 1163.19 last 23.15) - open/close 25235 (time 1710102945.35 total 1173.20 last 21.58) - open/close 25449 (time 1710102955.39 total 1183.23 last 21.33) - open/close 25676 (time 1710102965.46 total 1193.31 last 22.52) - open/close 25878 (time 1710102975.50 total 1203.35 last 20.13) - open/close 26065 (time 1710102995.25 total 1223.10 last 9.47) - open/close 26287 (time 1710103005.26 total 1233.11 last 22.17) - open/close 26499 (time 1710103015.29 total 1243.14 last 21.15) - open/close 26745 (time 1710103025.29 total 1253.14 last 24.59) - open/close 26947 (time 1710103035.30 total 1263.15 last 20.18) - open/close 27145 (time 1710103045.34 total 1273.19 last 19.72) - open/close 27357 (time 1710103055.39 total 1283.24 last 21.11) - open/close 27590 (time 1710103065.42 total 1293.27 last 23.22) - open/close 27821 (time 1710103075.51 total 1303.36 last 22.90) - open/close 28053 (time 1710103085.54 total 1313.39 last 23.13) - open/close 28291 (time 1710103095.56 total 1323.41 last 23.74) - open/close 28503 (time 1710103105.64 total 1333.49 last 21.05) - open/close 28712 (time 1710103115.66 total 1343.51 last 20.85) - open/close 28935 (time 1710103125.70 total 1353.54 last 22.22) - open/close 29138 (time 1710103135.72 total 1363.57 last 20.24) - open/close 29364 (time 1710103145.75 total 1373.60 last 22.54) - open/close 29603 (time 1710103155.77 total 1383.62 last 23.85) - open/close 29827 (time 1710103165.81 total 1393.66 last 22.31) - open/close 30000 (time 1710103175.07 total 1402.92 last 18.69) - open/close 30266 (time 1710103185.08 total 1412.93 last 26.56) - open/close 30510 (time 1710103195.11 total 1422.96 last 24.34) - open/close 30754 (time 1710103205.14 total 1432.99 last 24.31) - open/close 30983 (time 1710103215.16 total 1443.01 last 22.86) - open/close 31219 (time 1710103225.20 total 1453.05 last 23.50) - open/close 31434 (time 1710103235.23 total 1463.08 last 21.43) - open/close 31664 (time 1710103245.27 total 1473.12 last 22.92) - open/close 31893 (time 1710103255.31 total 1483.16 last 22.81) - open/close 32128 (time 1710103265.32 total 1493.17 last 23.47) - open/close 32373 (time 1710103275.34 total 1503.19 last 24.46) - open/close 32626 (time 1710103285.37 total 1513.22 last 25.21) - open/close 32840 (time 1710103295.38 total 1523.23 last 21.37) - open/close 33102 (time 1710103305.39 total 1533.24 last 26.18) - open/close 33369 (time 1710103315.42 total 1543.27 last 26.62) - open/close 33631 (time 1710103325.44 total 1553.29 last 26.14) - open/close 33904 (time 1710103335.45 total 1563.30 last 27.28) - open/close 34149 (time 1710103345.46 total 1573.31 last 24.48) - open/close 34406 (time 1710103355.47 total 1583.32 last 25.69) - open/close 34663 (time 1710103365.48 total 1593.33 last 25.66) - open/close 34904 (time 1710103375.49 total 1603.34 last 24.08) - open/close 35151 (time 1710103385.50 total 1613.35 last 24.67) - open/close 35421 (time 1710103395.51 total 1623.36 last 26.97) - open/close 35676 (time 1710103405.52 total 1633.37 last 25.49) - open/close 35918 (time 1710103415.53 total 1643.38 last 24.17) - open/close 36179 (time 1710103425.57 total 1653.42 last 26.01) - open/close 36443 (time 1710103435.59 total 1663.44 last 26.34) - open/close 36709 (time 1710103445.62 total 1673.47 last 26.51) - open/close 36979 (time 1710103455.67 total 1683.52 last 26.88) - open/close 37232 (time 1710103465.69 total 1693.54 last 25.24) - open/close 37503 (time 1710103475.71 total 1703.56 last 27.04) - open/close 37779 (time 1710103485.73 total 1713.58 last 27.55) - open/close 38085 (time 1710103495.75 total 1723.60 last 30.54) - open/close 38385 (time 1710103505.78 total 1733.63 last 29.92) - open/close 38706 (time 1710103515.80 total 1743.65 last 32.04) - open/close 39029 (time 1710103525.81 total 1753.66 last 32.27) - open/close 39365 (time 1710103535.84 total 1763.69 last 33.50) - open/close 39690 (time 1710103545.85 total 1773.70 last 32.47) - open/close 40000 (time 1710103555.35 total 1783.20 last 32.61) - open/close 40340 (time 1710103565.38 total 1793.23 last 33.92) - open/close 40694 (time 1710103575.39 total 1803.24 last 35.36) - open/close 41057 (time 1710103585.40 total 1813.25 last 36.26) - open/close 41429 (time 1710103595.42 total 1823.27 last 37.14) - open/close 41809 (time 1710103605.42 total 1833.27 last 38.00) - open/close 42196 (time 1710103615.43 total 1843.28 last 38.64) - open/close 42596 (time 1710103625.43 total 1853.28 last 40.00) - open/close 42988 (time 1710103635.45 total 1863.30 last 39.14) - open/close 43384 (time 1710103645.45 total 1873.30 last 39.58) - open/close 43782 (time 1710103655.47 total 1883.32 last 39.75) - open/close 44160 (time 1710103665.48 total 1893.33 last 37.73) - open/close 44565 (time 1710103675.49 total 1903.34 last 40.46) - open/close 44982 (time 1710103685.51 total 1913.36 last 41.63) - open/close 45402 (time 1710103695.53 total 1923.38 last 41.91) - open/close 45828 (time 1710103705.54 total 1933.39 last 42.55) - open/close 46270 (time 1710103715.55 total 1943.40 last 44.18) - open/close 46696 (time 1710103725.56 total 1953.41 last 42.53) - open/close 47155 (time 1710103735.57 total 1963.42 last 45.89) - open/close 47638 (time 1710103745.58 total 1973.43 last 48.21) - open/close 48203 (time 1710103755.59 total 1983.44 last 56.46) - open/close 48789 (time 1710103765.59 total 1993.44 last 58.59) - open/close 49393 (time 1710103775.61 total 2003.46 last 60.32) total: 50000 open/close in 2013.35 seconds: 24.83 ops/second - unlinked 0 (time 1710103786 ; total 0 ; last 0) - unlinked 10000 (time 1710103909 ; total 123 ; last 123) - unlinked 20000 (time 1710104011 ; total 225 ; last 102) - unlinked 30000 (time 1710104093 ; total 307 ; last 82) - unlinked 40000 (time 1710104152 ; total 366 ; last 59) total: 50000 unlinks in 402 seconds: 124.378113 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server unloading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2