== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:02:37 (1746126157) Stopping clients: oleg449-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg449-client.virtnet /mnt/lustre2 (opts:-f) oleg449-server: oleg449-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing load_modules_local oleg449-server: Loading modules from /home/green/git/lustre-release/lustre oleg449-server: detected 4 online CPUs by sysfs oleg449-server: MODOPTS_LIBCFS= oleg449-server: Force libcfs to create 2 CPU partitions oleg449-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg449-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg449-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg449-client.virtnet: -o user_xattr,flock 192.168.204.149@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1085 (time 1746126223.25 total 10.00 last 108.48) - open/close 2210 (time 1746126233.25 total 20.00 last 112.50) - open/close 3334 (time 1746126243.25 total 30.01 last 112.35) - open/close 4556 (time 1746126253.25 total 40.01 last 122.19) - open/close 5701 (time 1746126263.26 total 50.02 last 114.36) - open/close 6793 (time 1746126273.27 total 60.03 last 109.14) - open/close 7895 (time 1746126283.27 total 70.03 last 110.18) - open/close 9009 (time 1746126293.28 total 80.04 last 111.32) - open/close 10000 (time 1746126302.61 total 89.36 last 106.23) - open/close 11028 (time 1746126312.61 total 99.37 last 102.74) - open/close 12070 (time 1746126322.62 total 109.37 last 104.16) - open/close 13100 (time 1746126332.62 total 119.38 last 102.95) - open/close 14080 (time 1746126342.62 total 129.38 last 97.99) - open/close 15110 (time 1746126352.63 total 139.39 last 102.94) - open/close 16087 (time 1746126362.63 total 149.39 last 97.65) - open/close 17095 (time 1746126372.64 total 159.40 last 100.75) - open/close 18099 (time 1746126382.64 total 169.40 last 100.38) - open/close 19097 (time 1746126392.65 total 179.41 last 99.69) - open/close 19926 (time 1746126402.67 total 189.42 last 82.78) - open/close 20912 (time 1746126412.67 total 199.43 last 98.54) - open/close 21882 (time 1746126422.68 total 209.43 last 96.98) - open/close 22880 (time 1746126432.68 total 219.44 last 99.75) - open/close 23994 (time 1746126442.68 total 229.44 last 111.39) - open/close 25169 (time 1746126452.68 total 239.44 last 117.49) - open/close 26256 (time 1746126462.69 total 249.45 last 108.61) - open/close 27432 (time 1746126472.71 total 259.46 last 117.43) - open/close 28606 (time 1746126482.71 total 269.47 last 117.35) - open/close 29752 (time 1746126492.71 total 279.47 last 114.57) - open/close 30000 (time 1746126494.87 total 281.62 last 115.17) - open/close 31135 (time 1746126504.88 total 291.63 last 113.37) - open/close 32261 (time 1746126514.89 total 301.64 last 112.51) - open/close 33427 (time 1746126524.89 total 311.64 last 116.60) - open/close 34523 (time 1746126534.90 total 321.66 last 109.40) - open/close 35622 (time 1746126544.91 total 331.66 last 109.87) - open/close 36731 (time 1746126554.91 total 341.67 last 110.87) - open/close 37803 (time 1746126564.91 total 351.67 last 107.18) - open/close 38813 (time 1746126574.91 total 361.67 last 100.98) - open/close 39843 (time 1746126584.92 total 371.68 last 102.92) - open/close 40929 (time 1746126594.92 total 381.68 last 108.56) - open/close 42030 (time 1746126604.93 total 391.68 last 110.09) - open/close 43097 (time 1746126614.93 total 401.69 last 106.65) - open/close 44224 (time 1746126624.94 total 411.70 last 112.57) - open/close 45383 (time 1746126634.95 total 421.71 last 115.80) - open/close 46504 (time 1746126644.95 total 431.71 last 112.08) - open/close 47806 (time 1746126654.96 total 441.71 last 130.17) - open/close 49153 (time 1746126664.96 total 451.71 last 134.68) - open/close 50000 (time 1746126671.74 total 458.50 last 124.85) - open/close 51167 (time 1746126681.75 total 468.50 last 116.63) - open/close 52499 (time 1746126691.75 total 478.50 last 133.18) - open/close 53822 (time 1746126701.75 total 488.51 last 132.22) - open/close 55170 (time 1746126711.76 total 498.51 last 134.76) - open/close 56828 (time 1746126721.76 total 508.52 last 165.77) - open/close 58240 (time 1746126731.76 total 518.52 last 141.17) - open/close 59722 (time 1746126741.77 total 528.53 last 148.07) - open/close 61169 (time 1746126751.77 total 538.53 last 144.64) - open/close 62615 (time 1746126761.78 total 548.53 last 144.58) - open/close 64460 (time 1746126771.78 total 558.54 last 184.38) total: 64768 open/close in 560.46 seconds: 115.56 ops/second - /unlink 3715 (time 1746126784.94 total 10.00 last 371.44) - /unlink 6707 (time 1746126794.94 total 20.01 last 299.09) - /unlink 8959 (time 1746126804.94 total 30.01 last 225.20) - /unlink 10000 (time 1746126808.74 total 33.80 last 274.55) - /unlink 12968 (time 1746126818.74 total 43.80 last 296.78) - /unlink 16347 (time 1746126828.74 total 53.80 last 337.78) - /unlink 19052 (time 1746126838.74 total 63.80 last 270.49) - /unlink 20000 (time 1746126842.82 total 67.88 last 232.40) - /unlink 22958 (time 1746126852.82 total 77.88 last 295.72) - /unlink 26623 (time 1746126862.82 total 87.89 last 366.43) - /unlink 30000 (time 1746126872.81 total 97.87 last 338.21) - /unlink 32822 (time 1746126882.81 total 107.87 last 282.16) - /unlink 35405 (time 1746126892.81 total 117.87 last 258.26) - /unlink 37818 (time 1746126902.81 total 127.87 last 241.29) - /unlink 40000 (time 1746126911.61 total 136.67 last 248.01) - /unlink 42449 (time 1746126921.61 total 146.67 last 244.85) - /unlink 46010 (time 1746126931.62 total 156.68 last 355.92) - /unlink 50000 (time 1746126941.26 total 166.32 last 413.67) - /unlink 53904 (time 1746126951.26 total 176.33 last 390.37) - /unlink 57166 (time 1746126961.26 total 186.33 last 326.17) - /unlink 60000 (time 1746126970.95 total 196.01 last 292.50) - /unlink 62951 (time 1746126980.95 total 206.02 last 295.07) total: 64768 /unlink in 211.30 seconds: 306.52 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1545 (time 1746127011.26 total 10.00 last 154.49) - open/close 3495 (time 1746127021.26 total 20.00 last 194.95) - open/close 5576 (time 1746127031.27 total 30.01 last 208.02) - open/close 7362 (time 1746127041.27 total 40.01 last 178.56) - open/close 9175 (time 1746127051.27 total 50.01 last 181.29) - open/close 10000 (time 1746127056.08 total 54.82 last 171.36) - open/close 12053 (time 1746127066.08 total 64.83 last 205.26) - open/close 14027 (time 1746127076.09 total 74.83 last 197.33) - open/close 16088 (time 1746127086.09 total 84.83 last 206.10) - open/close 17779 (time 1746127096.09 total 94.83 last 169.08) - open/close 19988 (time 1746127106.09 total 104.83 last 220.81) - open/close 22283 (time 1746127116.10 total 114.84 last 229.41) - open/close 24221 (time 1746127126.10 total 124.84 last 193.78) - open/close 26216 (time 1746127136.10 total 134.84 last 199.45) - open/close 28282 (time 1746127146.10 total 144.84 last 206.57) - open/close 30000 (time 1746127154.54 total 153.28 last 203.68) - open/close 32246 (time 1746127164.54 total 163.28 last 224.51) - open/close 34412 (time 1746127174.55 total 173.29 last 216.52) - open/close 36368 (time 1746127184.55 total 183.29 last 195.48) - open/close 38332 (time 1746127194.57 total 193.31 last 196.10) - open/close 39847 (time 1746127204.57 total 203.31 last 151.40) - open/close 41770 (time 1746127214.58 total 213.32 last 192.28) - open/close 43272 (time 1746127224.58 total 223.32 last 150.16) - open/close 45095 (time 1746127234.58 total 233.32 last 182.27) - open/close 46808 (time 1746127244.59 total 243.33 last 171.16) - open/close 49222 (time 1746127254.59 total 253.33 last 241.37) - open/close 50000 (time 1746127258.12 total 256.86 last 220.42) - open/close 52109 (time 1746127268.12 total 266.86 last 210.90) - open/close 54003 (time 1746127278.12 total 276.86 last 189.34) - open/close 56020 (time 1746127288.13 total 286.87 last 201.59) - open/close 57962 (time 1746127298.13 total 296.87 last 194.15) - open/close 60000 (time 1746127307.88 total 306.63 last 208.91) - open/close 61418 (time 1746127317.89 total 316.63 last 141.72) - open/close 62644 (time 1746127327.89 total 326.64 last 122.55) - open/close 64549 (time 1746127337.90 total 336.64 last 190.38) total: 64768 open/close in 337.59 seconds: 191.85 ops/second - /unlink 2854 (time 1746127349.87 total 10.00 last 285.35) - /unlink 6092 (time 1746127359.87 total 20.00 last 323.75) - /unlink 10000 (time 1746127369.68 total 29.81 last 398.36) - /unlink 13059 (time 1746127379.70 total 39.83 last 305.45) - /unlink 16422 (time 1746127389.71 total 49.84 last 336.03) - /unlink 19152 (time 1746127399.71 total 59.84 last 272.93) - /unlink 20000 (time 1746127403.02 total 63.16 last 255.69) - /unlink 23174 (time 1746127413.03 total 73.16 last 317.21) - /unlink 27100 (time 1746127423.03 total 83.16 last 392.57) - /unlink 29802 (time 1746127433.03 total 93.17 last 270.10) - /unlink 32779 (time 1746127443.04 total 103.17 last 297.63) - /unlink 35721 (time 1746127453.04 total 113.17 last 294.20) - /unlink 38214 (time 1746127463.04 total 123.18 last 249.10) - /unlink 40000 (time 1746127472.04 total 132.17 last 198.55) - /unlink 41891 (time 1746127482.04 total 142.17 last 189.08) - /unlink 43410 (time 1746127492.05 total 152.18 last 151.78) - /unlink 44994 (time 1746127502.05 total 162.18 last 158.38) - /unlink 46846 (time 1746127512.06 total 172.19 last 185.05) - /unlink 48726 (time 1746127522.06 total 182.19 last 187.97) - /unlink 49949 (time 1746127532.07 total 192.20 last 122.18) - /unlink 51179 (time 1746127542.07 total 202.21 last 122.94) - /unlink 52351 (time 1746127552.08 total 212.21 last 117.14) - /unlink 53539 (time 1746127562.09 total 222.22 last 118.72) - /unlink 54687 (time 1746127572.10 total 232.24 last 114.59) - /unlink 55898 (time 1746127582.11 total 242.24 last 121.08) - /unlink 58032 (time 1746127592.11 total 252.24 last 213.33) - /unlink 59969 (time 1746127602.11 total 262.24 last 193.68) - /unlink 61789 (time 1746127612.11 total 272.24 last 181.96) - /unlink 63682 (time 1746127622.11 total 282.24 last 189.26) total: 64768 /unlink in 287.77 seconds: 225.07 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1044 (time 1746127658.53 total 10.01 last 104.34) - open/close 2158 (time 1746127668.53 total 20.01 last 111.40) - open/close 3296 (time 1746127678.53 total 30.01 last 113.79) - open/close 4515 (time 1746127688.53 total 40.01 last 121.87) - open/close 5649 (time 1746127698.54 total 50.01 last 113.35) - open/close 6819 (time 1746127708.54 total 60.01 last 116.99) - open/close 7964 (time 1746127718.54 total 70.02 last 114.46) - open/close 9130 (time 1746127728.55 total 80.03 last 116.51) - open/close 10000 (time 1746127736.34 total 87.81 last 111.71) - open/close 11166 (time 1746127746.34 total 97.82 last 116.56) - open/close 12343 (time 1746127756.34 total 107.82 last 117.64) - open/close 13462 (time 1746127766.35 total 117.83 last 111.86) - open/close 14647 (time 1746127776.35 total 127.83 last 118.44) - open/close 15881 (time 1746127786.37 total 137.85 last 123.21) - open/close 17071 (time 1746127796.37 total 147.85 last 118.95) - open/close 18248 (time 1746127806.39 total 157.86 last 117.55) - open/close 19269 (time 1746127816.39 total 167.87 last 102.07) - open/close 20000 (time 1746127822.91 total 174.39 last 112.10) - open/close 21066 (time 1746127832.91 total 184.39 last 106.58) - open/close 22122 (time 1746127842.92 total 194.39 last 105.55) - open/close 23250 (time 1746127852.92 total 204.40 last 112.79) - open/close 24485 (time 1746127862.92 total 214.40 last 123.41) - open/close 25593 (time 1746127872.93 total 224.41 last 110.76) - open/close 26800 (time 1746127882.93 total 234.41 last 120.67) - open/close 27861 (time 1746127892.93 total 244.41 last 106.09) - open/close 28932 (time 1746127902.94 total 254.41 last 107.05) - open/close 29994 (time 1746127912.94 total 264.42 last 106.13) - open/close 31038 (time 1746127922.95 total 274.43 last 104.34) - open/close 32070 (time 1746127932.96 total 284.44 last 103.07) - open/close 33035 (time 1746127942.97 total 294.45 last 96.42) - open/close 34102 (time 1746127953.00 total 304.48 last 106.37) - open/close 35085 (time 1746127963.01 total 314.49 last 98.17) - open/close 36041 (time 1746127973.02 total 324.50 last 95.50) - open/close 37017 (time 1746127983.03 total 334.51 last 97.54) - open/close 37996 (time 1746127993.04 total 344.52 last 97.82) - open/close 39028 (time 1746128003.04 total 354.52 last 103.19) - open/close 40000 (time 1746128012.76 total 364.24 last 100.01) - open/close 41017 (time 1746128022.76 total 374.24 last 101.68) - open/close 42026 (time 1746128032.77 total 384.24 last 100.85) - open/close 43052 (time 1746128042.77 total 394.25 last 102.57) - open/close 44091 (time 1746128052.77 total 404.25 last 103.83) - open/close 45092 (time 1746128062.79 total 414.27 last 99.96) - open/close 46109 (time 1746128072.81 total 424.29 last 101.52) - open/close 47085 (time 1746128082.82 total 434.30 last 97.50) - open/close 48203 (time 1746128092.84 total 444.32 last 111.57) - open/close 49407 (time 1746128102.85 total 454.33 last 120.26) - open/close 50000 (time 1746128107.78 total 459.26 last 120.29) - open/close 51199 (time 1746128117.78 total 469.26 last 119.83) - open/close 52323 (time 1746128127.79 total 479.26 last 112.39) - open/close 53355 (time 1746128137.80 total 489.27 last 103.10) - open/close 54349 (time 1746128147.80 total 499.28 last 99.32) - open/close 55285 (time 1746128157.80 total 509.28 last 93.60) - open/close 56076 (time 1746128167.81 total 519.29 last 79.06) - open/close 56826 (time 1746128177.82 total 529.30 last 74.93) - open/close 57724 (time 1746128187.83 total 539.31 last 89.71) - open/close 58684 (time 1746128197.83 total 549.31 last 95.99) - open/close 59527 (time 1746128207.83 total 559.31 last 84.30) - open/close 60000 (time 1746128213.29 total 564.77 last 86.56) - open/close 60924 (time 1746128223.29 total 574.77 last 92.40) - open/close 61839 (time 1746128233.30 total 584.78 last 91.47) - open/close 62672 (time 1746128243.32 total 594.80 last 83.12) - open/close 63540 (time 1746128253.32 total 604.80 last 86.78) - open/close 64421 (time 1746128263.33 total 614.80 last 88.05) total: 64768 open/close in 618.75 seconds: 104.68 ops/second - /unlink 980 (time 1746128279.54 total 10.00 last 97.99) - /unlink 1908 (time 1746128289.55 total 20.01 last 92.71) - /unlink 2903 (time 1746128299.55 total 30.01 last 99.49) - /unlink 3962 (time 1746128309.56 total 40.02 last 105.83) - /unlink 5064 (time 1746128319.57 total 50.03 last 110.05) - /unlink 6125 (time 1746128329.57 total 60.03 last 106.07) - /unlink 7256 (time 1746128339.58 total 70.04 last 113.08) - /unlink 8457 (time 1746128349.59 total 80.05 last 119.97) - /unlink 9549 (time 1746128359.59 total 90.05 last 109.17) - /unlink 10000 (time 1746128363.79 total 94.25 last 107.33) - /unlink 11018 (time 1746128373.80 total 104.26 last 101.76) - /unlink 12116 (time 1746128383.81 total 114.27 last 109.69) - /unlink 13065 (time 1746128393.81 total 124.27 last 94.87) - /unlink 14084 (time 1746128403.81 total 134.27 last 101.86) - /unlink 15132 (time 1746128413.82 total 144.28 last 104.71) - /unlink 16231 (time 1746128423.82 total 154.28 last 109.90) - /unlink 17233 (time 1746128433.83 total 164.29 last 100.15) - /unlink 18287 (time 1746128443.84 total 174.30 last 105.30) - /unlink 19388 (time 1746128453.85 total 184.31 last 109.98) - /unlink 20000 (time 1746128459.59 total 190.05 last 106.58) - /unlink 21061 (time 1746128469.60 total 200.06 last 106.01) - /unlink 22051 (time 1746128479.60 total 210.06 last 98.94) - /unlink 22959 (time 1746128489.61 total 220.07 last 90.75) - /unlink 23896 (time 1746128499.62 total 230.08 last 93.56) - /unlink 24920 (time 1746128509.63 total 240.09 last 102.35) - /unlink 26049 (time 1746128519.63 total 250.09 last 112.90) - /unlink 27182 (time 1746128529.63 total 260.09 last 113.27) - /unlink 28252 (time 1746128539.63 total 270.09 last 106.99) - /unlink 29303 (time 1746128549.64 total 280.10 last 105.03) - /unlink 30000 (time 1746128556.30 total 286.76 last 104.64) - /unlink 30945 (time 1746128566.31 total 296.77 last 94.41) - /unlink 32087 (time 1746128576.32 total 306.78 last 114.13) - /unlink 33207 (time 1746128586.33 total 316.79 last 111.90) - /unlink 34236 (time 1746128596.33 total 326.79 last 102.82) - /unlink 35189 (time 1746128606.35 total 336.81 last 95.11) - /unlink 36179 (time 1746128616.37 total 346.83 last 98.81) - /unlink 37228 (time 1746128626.38 total 356.84 last 104.87) - /unlink 38229 (time 1746128636.38 total 366.84 last 100.04) - /unlink 39255 (time 1746128646.40 total 376.86 last 102.43) - /unlink 40000 (time 1746128653.50 total 383.96 last 104.87) - /unlink 40873 (time 1746128663.50 total 393.96 last 87.29) - /unlink 41771 (time 1746128673.52 total 403.98 last 89.68) - /unlink 42759 (time 1746128683.53 total 413.99 last 98.65) - /unlink 43714 (time 1746128693.53 total 423.99 last 95.47) - /unlink 44715 (time 1746128703.54 total 434.00 last 100.04) - /unlink 45712 (time 1746128713.55 total 444.01 last 99.65) - /unlink 46640 (time 1746128723.55 total 454.01 last 92.76) - /unlink 47592 (time 1746128733.55 total 464.01 last 95.16) - /unlink 48500 (time 1746128743.56 total 474.02 last 90.76) - /unlink 49323 (time 1746128753.57 total 484.03 last 82.23) - /unlink 50000 (time 1746128761.58 total 492.04 last 84.48) - /unlink 50881 (time 1746128771.58 total 502.04 last 88.08) - /unlink 51836 (time 1746128781.60 total 512.06 last 95.35) - /unlink 52696 (time 1746128791.61 total 522.07 last 85.93) - /unlink 53535 (time 1746128801.61 total 532.07 last 83.90) - /unlink 54393 (time 1746128811.62 total 542.08 last 85.67) - /unlink 55246 (time 1746128821.63 total 552.09 last 85.25) - /unlink 56054 (time 1746128831.64 total 562.10 last 80.73) - /unlink 56913 (time 1746128841.64 total 572.10 last 85.89) - /unlink 57880 (time 1746128851.64 total 582.10 last 96.66) - /unlink 58753 (time 1746128861.64 total 592.10 last 87.28) - /unlink 59539 (time 1746128871.65 total 602.11 last 78.55) - /unlink 60000 (time 1746128877.65 total 608.11 last 76.85) - /unlink 60899 (time 1746128887.65 total 618.11 last 89.87) - /unlink 61845 (time 1746128897.66 total 628.12 last 94.52) - /unlink 62770 (time 1746128907.66 total 638.12 last 92.49) - /unlink 63797 (time 1746128917.67 total 648.13 last 102.65) - /unlink 64713 (time 1746128927.67 total 658.13 last 91.55) total: 64768 /unlink in 658.61 seconds: 98.34 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1022 (time 1746128961.17 total 10.00 last 102.20) - open/close 1952 (time 1746128971.17 total 20.00 last 92.99) - open/close 2905 (time 1746128981.17 total 30.00 last 95.29) - open/close 3760 (time 1746128991.18 total 40.00 last 85.48) - open/close 4784 (time 1746129001.19 total 50.02 last 102.28) - open/close 5737 (time 1746129011.19 total 60.02 last 95.25) - open/close 6631 (time 1746129021.20 total 70.03 last 89.31) - open/close 7495 (time 1746129031.21 total 80.03 last 86.37) - open/close 8371 (time 1746129041.21 total 90.04 last 87.54) - open/close 9375 (time 1746129051.22 total 100.04 last 100.38) - open/close 10000 (time 1746129058.15 total 106.98 last 90.09) - open/close 11044 (time 1746129068.16 total 116.99 last 104.36) - open/close 12132 (time 1746129078.16 total 126.99 last 108.80) - open/close 13214 (time 1746129088.16 total 136.99 last 108.15) - open/close 14360 (time 1746129098.17 total 147.00 last 114.53) - open/close 15480 (time 1746129108.17 total 157.00 last 111.98) - open/close 16581 (time 1746129118.17 total 167.00 last 110.05) - open/close 17672 (time 1746129128.18 total 177.01 last 109.06) - open/close 18699 (time 1746129138.18 total 187.01 last 102.68) - open/close 19809 (time 1746129148.18 total 197.01 last 110.99) - open/close 20933 (time 1746129158.19 total 207.01 last 112.34) - open/close 22040 (time 1746129168.19 total 217.02 last 110.64) - open/close 23177 (time 1746129178.19 total 227.02 last 113.69) - open/close 24291 (time 1746129188.20 total 237.03 last 111.29) - open/close 25434 (time 1746129198.20 total 247.03 last 114.29) - open/close 26499 (time 1746129208.21 total 257.04 last 106.46) - open/close 27653 (time 1746129218.21 total 267.04 last 115.32) - open/close 28874 (time 1746129228.22 total 277.05 last 122.04) - open/close 30000 (time 1746129238.14 total 286.97 last 113.52) - open/close 31065 (time 1746129248.14 total 296.97 last 106.49) - open/close 32139 (time 1746129258.14 total 306.97 last 107.40) - open/close 33256 (time 1746129268.14 total 316.97 last 111.67) - open/close 34408 (time 1746129278.14 total 326.97 last 115.19) - open/close 35560 (time 1746129288.14 total 336.97 last 115.17) - open/close 36663 (time 1746129298.15 total 346.97 last 110.28) - open/close 37750 (time 1746129308.15 total 356.97 last 108.70) - open/close 38788 (time 1746129318.16 total 366.99 last 103.64) - open/close 39906 (time 1746129328.17 total 376.99 last 111.75) - open/close 41179 (time 1746129338.17 total 387.00 last 127.26) - open/close 42396 (time 1746129348.17 total 397.00 last 121.65) - open/close 43566 (time 1746129358.18 total 407.00 last 116.96) - open/close 44693 (time 1746129368.18 total 417.00 last 112.70) - open/close 45910 (time 1746129378.18 total 427.01 last 121.66) - open/close 47161 (time 1746129388.18 total 437.01 last 125.10) - open/close 48589 (time 1746129398.19 total 447.01 last 142.72) - open/close 49777 (time 1746129408.20 total 457.02 last 118.68) - open/close 50961 (time 1746129418.20 total 467.03 last 118.31) - open/close 52126 (time 1746129428.21 total 477.03 last 116.46) - open/close 53577 (time 1746129438.21 total 487.04 last 145.08) - open/close 55076 (time 1746129448.22 total 497.05 last 149.71) - open/close 56340 (time 1746129458.23 total 507.06 last 126.25) - open/close 57563 (time 1746129468.23 total 517.06 last 122.30) - open/close 58748 (time 1746129478.24 total 527.06 last 118.46) - open/close 60000 (time 1746129488.21 total 537.04 last 125.56) - open/close 61142 (time 1746129498.21 total 547.04 last 114.14) - open/close 62693 (time 1746129508.22 total 557.04 last 155.05) - open/close 64163 (time 1746129518.22 total 567.05 last 146.88) total: 64768 open/close in 571.24 seconds: 113.38 ops/second - /unlink 2528 (time 1746129533.80 total 10.00 last 252.72) - /unlink 5422 (time 1746129543.80 total 20.01 last 289.25) - /unlink 7505 (time 1746129553.80 total 30.01 last 208.23) - /unlink 9420 (time 1746129563.80 total 40.01 last 191.49) - /unlink 10000 (time 1746129566.68 total 42.89 last 201.49) - /unlink 12244 (time 1746129576.68 total 52.89 last 224.40) - /unlink 14214 (time 1746129586.69 total 62.89 last 196.94) - /unlink 16359 (time 1746129596.69 total 72.90 last 214.46) - /unlink 18436 (time 1746129606.69 total 82.90 last 207.61) - /unlink 20000 (time 1746129613.54 total 89.75 last 228.38) - /unlink 22049 (time 1746129623.55 total 99.75 last 204.79) - /unlink 24145 (time 1746129633.55 total 109.76 last 209.57) - /unlink 26118 (time 1746129643.56 total 119.77 last 197.06) - /unlink 28067 (time 1746129653.56 total 129.77 last 194.88) - /unlink 30000 (time 1746129663.46 total 139.67 last 195.24) - /unlink 31868 (time 1746129673.46 total 149.67 last 186.75) - /unlink 33860 (time 1746129683.47 total 159.68 last 199.09) - /unlink 35748 (time 1746129693.47 total 169.68 last 188.78) - /unlink 37570 (time 1746129703.47 total 179.68 last 182.13) - /unlink 39060 (time 1746129713.48 total 189.69 last 148.93) - /unlink 40000 (time 1746129719.79 total 196.00 last 148.96) - /unlink 41699 (time 1746129729.79 total 206.00 last 169.89) - /unlink 43497 (time 1746129739.80 total 216.00 last 179.71) - /unlink 45262 (time 1746129749.80 total 226.01 last 176.45) - /unlink 46850 (time 1746129759.81 total 236.02 last 158.57) - /unlink 48645 (time 1746129769.81 total 246.02 last 179.50) - /unlink 50000 (time 1746129777.51 total 253.71 last 176.10) - /unlink 52108 (time 1746129787.51 total 263.72 last 210.76) - /unlink 54291 (time 1746129797.51 total 273.72 last 218.26) - /unlink 56237 (time 1746129807.51 total 283.72 last 194.59) - /unlink 58203 (time 1746129817.51 total 293.72 last 196.57) - /unlink 60000 (time 1746129825.77 total 301.98 last 217.59) - /unlink 62006 (time 1746129835.77 total 311.98 last 200.55) - /unlink 64466 (time 1746129845.78 total 321.98 last 245.95) total: 64768 /unlink in 323.02 seconds: 200.51 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1927 (time 1746129871.57 total 10.00 last 192.68) - open/close 4333 (time 1746129881.57 total 20.00 last 240.57) - open/close 6388 (time 1746129891.57 total 30.01 last 205.44) - open/close 8327 (time 1746129901.57 total 40.01 last 193.87) - open/close 10000 (time 1746129909.97 total 48.41 last 199.11) - open/close 12206 (time 1746129919.97 total 58.41 last 220.57) - open/close 13890 (time 1746129929.98 total 68.42 last 168.29) - open/close 15434 (time 1746129939.99 total 78.42 last 154.33) - open/close 16755 (time 1746129949.99 total 88.43 last 132.03) - open/close 18421 (time 1746129959.99 total 98.43 last 166.59) - open/close 19971 (time 1746129969.99 total 108.43 last 154.96) - open/close 22482 (time 1746129979.99 total 118.43 last 251.07) - open/close 24867 (time 1746129990.00 total 128.43 last 238.44) - open/close 27224 (time 1746130000.00 total 138.44 last 235.57) - open/close 29747 (time 1746130010.01 total 148.44 last 252.22) - open/close 31687 (time 1746130020.01 total 158.45 last 193.94) - open/close 33829 (time 1746130030.01 total 168.45 last 214.18) - open/close 35522 (time 1746130040.01 total 178.45 last 169.27) - open/close 36950 (time 1746130050.02 total 188.45 last 142.73) - open/close 38702 (time 1746130060.02 total 198.46 last 175.14) - open/close 40000 (time 1746130069.28 total 207.71 last 140.25) - open/close 41417 (time 1746130079.28 total 217.71 last 141.67) - open/close 42733 (time 1746130089.28 total 227.72 last 131.56) - open/close 44464 (time 1746130099.29 total 237.72 last 173.00) - open/close 46071 (time 1746130109.29 total 247.72 last 160.66) - open/close 47721 (time 1746130119.30 total 257.74 last 164.78) - open/close 49579 (time 1746130129.31 total 267.74 last 185.68) - open/close 50000 (time 1746130131.72 total 270.15 last 174.69) - open/close 51629 (time 1746130141.73 total 280.16 last 162.77) - open/close 53365 (time 1746130151.73 total 290.16 last 173.60) - open/close 54589 (time 1746130161.73 total 300.17 last 122.35) - open/close 55864 (time 1746130171.73 total 310.17 last 127.47) - open/close 57521 (time 1746130181.74 total 320.17 last 165.63) - open/close 59113 (time 1746130191.75 total 330.18 last 159.07) - open/close 60000 (time 1746130196.76 total 335.19 last 176.92) - open/close 61541 (time 1746130206.76 total 345.20 last 154.06) - open/close 63214 (time 1746130216.77 total 355.20 last 167.22) total: 64768 open/close in 363.70 seconds: 178.08 ops/second - /unlink 2557 (time 1746130236.51 total 10.00 last 255.64) - /unlink 5504 (time 1746130246.52 total 20.00 last 294.66) - /unlink 9290 (time 1746130256.52 total 30.00 last 378.60) - /unlink 10000 (time 1746130259.37 total 32.86 last 248.43) - /unlink 12796 (time 1746130269.38 total 42.87 last 279.50) - /unlink 15628 (time 1746130279.38 total 52.87 last 283.13) - /unlink 18535 (time 1746130289.38 total 62.87 last 290.69) - /unlink 20000 (time 1746130295.32 total 68.81 last 246.57) - /unlink 22405 (time 1746130305.32 total 78.81 last 240.47) - /unlink 24828 (time 1746130315.33 total 88.82 last 242.09) - /unlink 26990 (time 1746130325.34 total 98.83 last 216.06) - /unlink 29044 (time 1746130335.34 total 108.83 last 205.28) - /unlink 30000 (time 1746130338.98 total 112.47 last 263.11) - /unlink 32631 (time 1746130348.98 total 122.47 last 263.08) - /unlink 35035 (time 1746130358.98 total 132.47 last 240.39) - /unlink 38145 (time 1746130368.98 total 142.47 last 310.92) - /unlink 40000 (time 1746130374.21 total 147.69 last 355.08) - /unlink 43317 (time 1746130384.21 total 157.70 last 331.57) - /unlink 46127 (time 1746130394.21 total 167.70 last 280.98) - /unlink 48473 (time 1746130404.21 total 177.70 last 234.53) - /unlink 50000 (time 1746130410.06 total 183.55 last 261.17) - /unlink 52288 (time 1746130420.07 total 193.55 last 228.67) - /unlink 54574 (time 1746130430.07 total 203.56 last 228.54) - /unlink 56796 (time 1746130440.08 total 213.56 last 222.03) - /unlink 58642 (time 1746130450.08 total 223.56 last 184.58) - /unlink 60000 (time 1746130454.89 total 228.38 last 282.29) - /unlink 63855 (time 1746130464.89 total 238.38 last 385.50) total: 64768 /unlink in 240.20 seconds: 269.64 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg449-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg449-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg449-server stop mds service on oleg449-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg449-server unloading modules via unload_modules_local on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg449-client: oleg449-client: ssh exited with exit code 2 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 2 pdsh@oleg449-client: oleg449-client: ssh exited with exit code 2 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 2