== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:01:34 (1746147694) Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg339-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1452 (time 1746147747.36 total 10.00 last 145.14) - open/close 2950 (time 1746147757.37 total 20.01 last 149.75) - open/close 4525 (time 1746147767.37 total 30.01 last 157.43) - open/close 6446 (time 1746147777.37 total 40.02 last 192.00) - open/close 7807 (time 1746147787.38 total 50.02 last 136.01) - open/close 9100 (time 1746147797.38 total 60.02 last 129.28) - open/close 10000 (time 1746147805.67 total 68.31 last 108.65) - open/close 11070 (time 1746147815.67 total 78.31 last 106.96) - open/close 12397 (time 1746147825.67 total 88.32 last 132.62) - open/close 13578 (time 1746147835.68 total 98.32 last 118.06) - open/close 14912 (time 1746147845.69 total 108.33 last 133.27) - open/close 16136 (time 1746147855.70 total 118.34 last 122.30) - open/close 17393 (time 1746147865.70 total 128.35 last 125.61) - open/close 18661 (time 1746147875.70 total 138.35 last 126.78) - open/close 19850 (time 1746147885.71 total 148.35 last 118.87) - open/close 21018 (time 1746147895.71 total 158.36 last 116.73) - open/close 22362 (time 1746147905.72 total 168.36 last 134.37) - open/close 23731 (time 1746147915.72 total 178.36 last 136.86) - open/close 25079 (time 1746147925.72 total 188.36 last 134.76) - open/close 26534 (time 1746147935.73 total 198.37 last 145.35) - open/close 27986 (time 1746147945.74 total 208.38 last 145.09) - open/close 29249 (time 1746147955.75 total 218.39 last 126.20) - open/close 30000 (time 1746147961.03 total 223.67 last 142.17) - open/close 31503 (time 1746147971.03 total 233.67 last 150.26) - open/close 32635 (time 1746147981.05 total 243.69 last 113.03) - open/close 33897 (time 1746147991.05 total 253.69 last 126.15) - open/close 35145 (time 1746148001.06 total 263.70 last 124.74) - open/close 36552 (time 1746148011.06 total 273.71 last 140.58) - open/close 37671 (time 1746148021.07 total 283.71 last 111.84) - open/close 38889 (time 1746148031.07 total 293.72 last 121.75) - open/close 40000 (time 1746148040.12 total 302.77 last 122.75) - open/close 41402 (time 1746148050.13 total 312.77 last 140.17) - open/close 42746 (time 1746148060.13 total 322.77 last 134.34) - open/close 43917 (time 1746148070.14 total 332.78 last 117.05) - open/close 45015 (time 1746148080.14 total 342.78 last 109.78) - open/close 46175 (time 1746148090.14 total 352.78 last 116.00) - open/close 47248 (time 1746148100.14 total 362.78 last 107.29) - open/close 48445 (time 1746148110.14 total 372.78 last 119.70) - open/close 49709 (time 1746148120.14 total 382.78 last 126.36) - open/close 50000 (time 1746148123.47 total 386.11 last 87.51) - open/close 51102 (time 1746148133.47 total 396.11 last 110.14) - open/close 52194 (time 1746148143.47 total 406.12 last 109.18) - open/close 53543 (time 1746148153.48 total 416.12 last 134.82) - open/close 54963 (time 1746148163.49 total 426.13 last 141.88) - open/close 56377 (time 1746148173.50 total 436.14 last 141.26) - open/close 57790 (time 1746148183.50 total 446.15 last 141.25) - open/close 59014 (time 1746148193.50 total 456.15 last 122.39) - open/close 59994 (time 1746148203.51 total 466.15 last 97.98) - open/close 61254 (time 1746148213.51 total 476.15 last 125.94) - open/close 62547 (time 1746148223.52 total 486.16 last 129.23) - open/close 63530 (time 1746148233.53 total 496.17 last 98.17) - open/close 64538 (time 1746148243.54 total 506.18 last 100.73) total: 64768 open/close in 508.33 seconds: 127.41 ops/second - /unlink 1722 (time 1746148257.47 total 10.00 last 172.14) - /unlink 3489 (time 1746148267.47 total 20.00 last 176.68) - /unlink 5249 (time 1746148277.47 total 30.01 last 175.94) - /unlink 6829 (time 1746148287.47 total 40.01 last 157.95) - /unlink 8390 (time 1746148297.48 total 50.02 last 156.02) - /unlink 9797 (time 1746148307.48 total 60.02 last 140.68) - /unlink 11243 (time 1746148317.49 total 70.02 last 144.53) - /unlink 12997 (time 1746148327.49 total 80.02 last 175.38) - /unlink 14859 (time 1746148337.49 total 90.03 last 186.14) - /unlink 16465 (time 1746148347.49 total 100.03 last 160.55) - /unlink 18236 (time 1746148357.50 total 110.04 last 177.01) - /unlink 20000 (time 1746148366.95 total 119.49 last 186.68) - /unlink 21888 (time 1746148376.95 total 129.49 last 188.73) - /unlink 23745 (time 1746148386.96 total 139.49 last 185.63) - /unlink 25651 (time 1746148396.96 total 149.50 last 190.55) - /unlink 27921 (time 1746148406.96 total 159.50 last 226.90) - /unlink 29806 (time 1746148416.97 total 169.51 last 188.35) - /unlink 31396 (time 1746148426.98 total 179.52 last 158.80) - /unlink 33014 (time 1746148436.98 total 189.52 last 161.78) - /unlink 34618 (time 1746148446.99 total 199.53 last 160.27) - /unlink 36354 (time 1746148456.99 total 209.53 last 173.57) - /unlink 38205 (time 1746148467.00 total 219.53 last 185.07) - /unlink 40000 (time 1746148475.97 total 228.51 last 199.98) - /unlink 41668 (time 1746148485.98 total 238.52 last 166.69) - /unlink 43275 (time 1746148495.99 total 248.52 last 160.59) - /unlink 45399 (time 1746148505.99 total 258.52 last 212.37) - /unlink 47375 (time 1746148515.99 total 268.53 last 197.49) - /unlink 49097 (time 1746148525.99 total 278.53 last 172.16) - /unlink 50000 (time 1746148531.32 total 283.85 last 169.66) - /unlink 51984 (time 1746148541.32 total 293.86 last 198.35) - /unlink 53853 (time 1746148551.33 total 303.86 last 186.78) - /unlink 55422 (time 1746148561.33 total 313.86 last 156.88) - /unlink 57087 (time 1746148571.33 total 323.87 last 166.43) - /unlink 58886 (time 1746148581.33 total 333.87 last 179.88) - /unlink 60000 (time 1746148586.65 total 339.19 last 209.29) - /unlink 61902 (time 1746148596.66 total 349.20 last 190.11) - /unlink 63726 (time 1746148606.66 total 359.20 last 182.32) total: 64768 /unlink in 364.83 seconds: 177.53 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 986 (time 1746148638.92 total 10.00 last 98.56) - open/close 2038 (time 1746148648.92 total 20.01 last 105.17) - open/close 3222 (time 1746148658.92 total 30.01 last 118.36) - open/close 4200 (time 1746148668.93 total 40.02 last 97.71) - open/close 5301 (time 1746148678.94 total 50.02 last 110.07) - open/close 6407 (time 1746148688.94 total 60.02 last 110.60) - open/close 7574 (time 1746148698.94 total 70.03 last 116.66) - open/close 8589 (time 1746148708.95 total 80.04 last 101.40) - open/close 9652 (time 1746148718.96 total 90.04 last 106.24) - open/close 10000 (time 1746148722.50 total 93.58 last 98.28) - open/close 11020 (time 1746148732.51 total 103.59 last 101.89) - open/close 11973 (time 1746148742.53 total 113.61 last 95.13) - open/close 12927 (time 1746148752.53 total 123.62 last 95.32) - open/close 14056 (time 1746148762.53 total 133.62 last 112.89) - open/close 15244 (time 1746148772.54 total 143.62 last 118.79) - open/close 16268 (time 1746148782.55 total 153.64 last 102.25) - open/close 17306 (time 1746148792.55 total 163.64 last 103.79) - open/close 18353 (time 1746148802.56 total 173.64 last 104.64) - open/close 19516 (time 1746148812.56 total 183.64 last 116.29) - open/close 20000 (time 1746148817.69 total 188.77 last 94.37) - open/close 21064 (time 1746148827.69 total 198.78 last 106.36) - open/close 22506 (time 1746148837.69 total 208.78 last 144.16) - open/close 24140 (time 1746148847.69 total 218.78 last 163.37) - open/close 25563 (time 1746148857.70 total 228.79 last 142.18) - open/close 26867 (time 1746148867.71 total 238.80 last 130.32) - open/close 27993 (time 1746148877.73 total 248.82 last 112.35) - open/close 28906 (time 1746148887.73 total 258.82 last 91.30) - open/close 29834 (time 1746148897.74 total 268.82 last 92.74) - open/close 30998 (time 1746148907.74 total 278.83 last 116.33) - open/close 31979 (time 1746148917.75 total 288.83 last 98.09) - open/close 32989 (time 1746148927.75 total 298.83 last 100.98) - open/close 34127 (time 1746148937.75 total 308.83 last 113.78) - open/close 35265 (time 1746148947.75 total 318.84 last 113.72) - open/close 36406 (time 1746148957.76 total 328.85 last 114.02) - open/close 37601 (time 1746148967.76 total 338.85 last 119.50) - open/close 38848 (time 1746148977.77 total 348.85 last 124.66) - open/close 40000 (time 1746148987.21 total 358.30 last 121.97) - open/close 41449 (time 1746148997.21 total 368.30 last 144.89) - open/close 42715 (time 1746149007.21 total 378.30 last 126.60) - open/close 43892 (time 1746149017.21 total 388.30 last 117.69) - open/close 44966 (time 1746149027.22 total 398.30 last 107.33) - open/close 46140 (time 1746149037.22 total 408.31 last 117.33) - open/close 47206 (time 1746149047.23 total 418.31 last 106.58) - open/close 48201 (time 1746149057.23 total 428.31 last 99.48) - open/close 49243 (time 1746149067.23 total 438.32 last 104.13) - open/close 50000 (time 1746149074.61 total 445.69 last 102.67) - open/close 50920 (time 1746149084.62 total 455.71 last 91.85) - open/close 51829 (time 1746149094.65 total 465.73 last 90.68) - open/close 52815 (time 1746149104.68 total 475.77 last 98.28) - open/close 53802 (time 1746149114.69 total 485.78 last 98.57) - open/close 54807 (time 1746149124.71 total 495.79 last 100.38) - open/close 55828 (time 1746149134.71 total 505.80 last 102.02) - open/close 56835 (time 1746149144.73 total 515.82 last 100.50) - open/close 57865 (time 1746149154.74 total 525.82 last 102.96) - open/close 58814 (time 1746149164.74 total 535.83 last 94.84) - open/close 59790 (time 1746149174.75 total 545.84 last 97.49) - open/close 60000 (time 1746149176.89 total 547.98 last 98.18) - open/close 61069 (time 1746149186.90 total 557.98 last 106.88) - open/close 61993 (time 1746149196.90 total 567.99 last 92.35) - open/close 62971 (time 1746149206.91 total 577.99 last 97.75) - open/close 63996 (time 1746149216.91 total 588.00 last 102.46) total: 64768 open/close in 594.73 seconds: 108.90 ops/second - /unlink 1546 (time 1746149235.50 total 10.00 last 154.57) - /unlink 3207 (time 1746149245.51 total 20.01 last 166.00) - /unlink 4912 (time 1746149255.51 total 30.01 last 170.44) - /unlink 6982 (time 1746149265.52 total 40.02 last 206.92) - /unlink 8852 (time 1746149275.52 total 50.02 last 186.91) - /unlink 10000 (time 1746149281.77 total 56.27 last 183.62) - /unlink 11809 (time 1746149291.78 total 66.28 last 180.73) - /unlink 13633 (time 1746149301.79 total 76.28 last 182.35) - /unlink 15701 (time 1746149311.79 total 86.28 last 206.80) - /unlink 17752 (time 1746149321.79 total 96.29 last 205.05) - /unlink 19795 (time 1746149331.79 total 106.29 last 204.21) - /unlink 21638 (time 1746149341.79 total 116.29 last 184.28) - /unlink 23400 (time 1746149351.80 total 126.30 last 176.14) - /unlink 25800 (time 1746149361.80 total 136.30 last 239.93) - /unlink 27806 (time 1746149371.80 total 146.30 last 200.56) - /unlink 29660 (time 1746149381.81 total 156.31 last 185.30) - /unlink 30000 (time 1746149383.92 total 158.42 last 161.03) - /unlink 32129 (time 1746149393.92 total 168.42 last 212.82) - /unlink 33744 (time 1746149403.92 total 178.42 last 161.47) - /unlink 35352 (time 1746149413.93 total 188.43 last 160.76) - /unlink 37249 (time 1746149423.93 total 198.43 last 189.65) - /unlink 38990 (time 1746149433.93 total 208.43 last 174.10) - /unlink 40000 (time 1746149440.52 total 215.01 last 153.33) - /unlink 41566 (time 1746149450.52 total 225.02 last 156.54) - /unlink 43269 (time 1746149460.52 total 235.02 last 170.22) - /unlink 45067 (time 1746149470.53 total 245.02 last 179.77) - /unlink 47015 (time 1746149480.53 total 255.03 last 194.78) - /unlink 48738 (time 1746149490.53 total 265.03 last 172.17) - /unlink 50000 (time 1746149496.65 total 271.15 last 206.33) - /unlink 51750 (time 1746149506.65 total 281.15 last 174.98) - /unlink 53629 (time 1746149516.65 total 291.15 last 187.85) - /unlink 56244 (time 1746149526.65 total 301.15 last 261.49) - /unlink 58151 (time 1746149536.65 total 311.15 last 190.70) - /unlink 60000 (time 1746149546.29 total 320.79 last 191.85) - /unlink 62374 (time 1746149556.29 total 330.79 last 237.39) - /unlink 64268 (time 1746149566.30 total 340.79 last 189.37) total: 64768 /unlink in 343.53 seconds: 188.54 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1010 (time 1746149600.10 total 10.00 last 100.97) - open/close 1990 (time 1746149610.10 total 20.01 last 97.96) - open/close 3117 (time 1746149620.11 total 30.01 last 112.64) - open/close 4154 (time 1746149630.11 total 40.02 last 103.62) - open/close 5139 (time 1746149640.11 total 50.02 last 98.50) - open/close 6116 (time 1746149650.12 total 60.02 last 97.67) - open/close 7197 (time 1746149660.13 total 70.03 last 107.99) - open/close 8246 (time 1746149670.13 total 80.04 last 104.84) - open/close 9236 (time 1746149680.14 total 90.05 last 98.91) - open/close 10000 (time 1746149687.33 total 97.23 last 106.32) - open/close 11185 (time 1746149697.33 total 107.24 last 118.42) - open/close 12148 (time 1746149707.35 total 117.25 last 96.19) - open/close 13088 (time 1746149717.35 total 127.26 last 93.95) - open/close 14256 (time 1746149727.35 total 137.26 last 116.78) - open/close 15333 (time 1746149737.35 total 147.26 last 107.68) - open/close 16385 (time 1746149747.37 total 157.27 last 105.08) - open/close 17530 (time 1746149757.37 total 167.28 last 114.42) - open/close 18733 (time 1746149767.38 total 177.29 last 120.16) - open/close 19719 (time 1746149777.39 total 187.29 last 98.56) - open/close 20000 (time 1746149780.35 total 190.26 last 94.89) - open/close 20982 (time 1746149790.35 total 200.26 last 98.19) - open/close 22370 (time 1746149800.36 total 210.27 last 138.63) - open/close 23580 (time 1746149810.37 total 220.27 last 120.93) - open/close 24680 (time 1746149820.37 total 230.28 last 109.96) - open/close 25849 (time 1746149830.37 total 240.28 last 116.87) - open/close 26808 (time 1746149840.39 total 250.30 last 95.73) - open/close 27824 (time 1746149850.40 total 260.30 last 101.56) - open/close 28967 (time 1746149860.40 total 270.31 last 114.25) - open/close 30000 (time 1746149868.85 total 278.76 last 122.22) - open/close 31079 (time 1746149878.86 total 288.76 last 107.85) - open/close 32217 (time 1746149888.86 total 298.77 last 113.77) - open/close 33707 (time 1746149898.86 total 308.77 last 148.94) - open/close 34968 (time 1746149908.87 total 318.78 last 126.04) - open/close 36296 (time 1746149918.87 total 328.78 last 132.75) - open/close 37486 (time 1746149928.88 total 338.78 last 118.94) - open/close 38526 (time 1746149938.89 total 348.79 last 103.93) - open/close 39584 (time 1746149948.89 total 358.80 last 105.72) - open/close 40000 (time 1746149951.55 total 361.46 last 156.44) - open/close 41177 (time 1746149961.56 total 371.47 last 117.60) - open/close 42130 (time 1746149971.57 total 381.47 last 95.22) - open/close 43288 (time 1746149981.58 total 391.48 last 115.72) - open/close 44425 (time 1746149991.59 total 401.50 last 113.54) - open/close 45399 (time 1746150001.60 total 411.50 last 97.31) - open/close 46516 (time 1746150011.60 total 421.51 last 111.65) - open/close 47506 (time 1746150021.62 total 431.52 last 98.84) - open/close 48421 (time 1746150031.62 total 441.53 last 91.46) - open/close 49246 (time 1746150041.64 total 451.55 last 82.35) - open/close 50000 (time 1746150049.69 total 459.60 last 93.67) - open/close 51114 (time 1746150059.70 total 469.60 last 111.36) - open/close 51928 (time 1746150069.70 total 479.60 last 81.37) - open/close 52847 (time 1746150079.70 total 489.61 last 91.89) - open/close 53803 (time 1746150089.71 total 499.62 last 95.50) - open/close 54729 (time 1746150099.73 total 509.63 last 92.46) - open/close 55545 (time 1746150109.73 total 519.64 last 81.54) - open/close 56434 (time 1746150119.74 total 529.64 last 88.88) - open/close 57320 (time 1746150129.75 total 539.65 last 88.48) - open/close 58373 (time 1746150139.76 total 549.66 last 105.24) - open/close 59349 (time 1746150149.76 total 559.67 last 97.55) - open/close 60000 (time 1746150156.93 total 566.84 last 90.79) - open/close 60911 (time 1746150166.94 total 576.84 last 91.04) - open/close 61975 (time 1746150176.94 total 586.84 last 106.38) - open/close 62942 (time 1746150186.94 total 596.85 last 96.68) - open/close 63860 (time 1746150196.95 total 606.86 last 91.72) total: 64768 open/close in 615.96 seconds: 105.15 ops/second - /unlink 1148 (time 1746150217.85 total 10.00 last 114.74) - /unlink 2036 (time 1746150227.86 total 20.01 last 88.77) - /unlink 2935 (time 1746150237.86 total 30.02 last 89.83) - /unlink 3940 (time 1746150247.87 total 40.02 last 100.44) - /unlink 4927 (time 1746150257.87 total 50.03 last 98.68) - /unlink 5798 (time 1746150267.88 total 60.03 last 87.03) - /unlink 6650 (time 1746150277.88 total 70.04 last 85.17) - /unlink 7572 (time 1746150287.89 total 80.04 last 92.19) - /unlink 8497 (time 1746150297.89 total 90.04 last 92.45) - /unlink 9415 (time 1746150307.90 total 100.05 last 91.75) - /unlink 10000 (time 1746150314.07 total 106.22 last 94.78) - /unlink 11185 (time 1746150324.07 total 116.23 last 118.43) - /unlink 12174 (time 1746150334.09 total 126.24 last 98.79) - /unlink 13141 (time 1746150344.11 total 136.26 last 96.45) - /unlink 14206 (time 1746150354.12 total 146.27 last 106.45) - /unlink 15272 (time 1746150364.12 total 156.27 last 106.60) - /unlink 16149 (time 1746150374.13 total 166.28 last 87.61) - /unlink 17186 (time 1746150384.13 total 176.28 last 103.67) - /unlink 18358 (time 1746150394.14 total 186.29 last 117.09) - /unlink 19221 (time 1746150404.14 total 196.29 last 86.29) - /unlink 20000 (time 1746150412.31 total 204.46 last 95.35) - /unlink 21178 (time 1746150422.31 total 214.47 last 117.77) - /unlink 22195 (time 1746150432.32 total 224.47 last 101.62) - /unlink 23014 (time 1746150442.32 total 234.48 last 81.89) - /unlink 23975 (time 1746150452.32 total 244.48 last 96.09) - /unlink 25052 (time 1746150462.33 total 254.48 last 107.66) - /unlink 25964 (time 1746150472.34 total 264.49 last 91.13) - /unlink 26937 (time 1746150482.34 total 274.49 last 97.25) - /unlink 28012 (time 1746150492.34 total 284.49 last 107.48) - /unlink 29103 (time 1746150502.36 total 294.51 last 108.94) - /unlink 30000 (time 1746150511.01 total 303.16 last 103.68) - /unlink 31237 (time 1746150521.01 total 313.17 last 123.63) - /unlink 32684 (time 1746150531.02 total 323.17 last 144.60) - /unlink 33751 (time 1746150541.03 total 333.18 last 106.64) - /unlink 34756 (time 1746150551.03 total 343.18 last 100.46) - /unlink 35937 (time 1746150561.03 total 353.18 last 118.09) - /unlink 36963 (time 1746150571.04 total 363.19 last 102.52) - /unlink 38070 (time 1746150581.04 total 373.20 last 110.63) - /unlink 39248 (time 1746150591.05 total 383.20 last 117.74) - /unlink 40000 (time 1746150597.57 total 389.72 last 115.29) - /unlink 41036 (time 1746150607.58 total 399.73 last 103.53) - /unlink 42124 (time 1746150617.59 total 409.74 last 108.68) - /unlink 43312 (time 1746150627.59 total 419.75 last 118.75) - /unlink 44347 (time 1746150637.60 total 429.75 last 103.46) - /unlink 45435 (time 1746150647.60 total 439.75 last 108.75) - /unlink 46458 (time 1746150657.61 total 449.76 last 102.26) - /unlink 47531 (time 1746150667.61 total 459.76 last 107.27) - /unlink 48629 (time 1746150677.61 total 469.76 last 109.78) - /unlink 49745 (time 1746150687.62 total 479.77 last 111.54) - /unlink 50000 (time 1746150690.42 total 482.57 last 90.92) - /unlink 51098 (time 1746150700.42 total 492.57 last 109.78) - /unlink 52047 (time 1746150710.43 total 502.58 last 94.82) - /unlink 53095 (time 1746150720.44 total 512.59 last 104.73) - /unlink 54287 (time 1746150730.44 total 522.59 last 119.17) - /unlink 55317 (time 1746150740.45 total 532.60 last 102.89) - /unlink 56424 (time 1746150750.46 total 542.61 last 110.64) - /unlink 57651 (time 1746150760.46 total 552.61 last 122.70) - /unlink 58660 (time 1746150770.46 total 562.62 last 100.83) - /unlink 59744 (time 1746150780.47 total 572.62 last 108.32) - /unlink 60000 (time 1746150782.52 total 574.67 last 124.86) - /unlink 61095 (time 1746150792.53 total 584.68 last 109.42) - /unlink 62078 (time 1746150802.53 total 594.69 last 98.25) - /unlink 63057 (time 1746150812.54 total 604.69 last 97.88) - /unlink 64154 (time 1746150822.54 total 614.69 last 109.68) total: 64768 /unlink in 621.10 seconds: 104.28 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1038 (time 1746150863.76 total 10.01 last 103.70) - open/close 2086 (time 1746150873.76 total 20.01 last 104.77) - open/close 3055 (time 1746150883.76 total 30.02 last 96.86) - open/close 4000 (time 1746150893.78 total 40.03 last 94.35) - open/close 4936 (time 1746150903.79 total 50.04 last 93.53) - open/close 5804 (time 1746150913.81 total 60.06 last 86.64) - open/close 6675 (time 1746150923.81 total 70.06 last 87.09) - open/close 7643 (time 1746150933.82 total 80.07 last 96.72) - open/close 8544 (time 1746150943.82 total 90.07 last 90.07) - open/close 9432 (time 1746150953.82 total 100.08 last 88.76) - open/close 10000 (time 1746150959.53 total 105.78 last 99.61) - open/close 11011 (time 1746150969.54 total 115.79 last 100.94) - open/close 12014 (time 1746150979.55 total 125.80 last 100.27) - open/close 13120 (time 1746150989.55 total 135.80 last 110.57) - open/close 14152 (time 1746150999.55 total 145.81 last 103.15) - open/close 15217 (time 1746151009.56 total 155.81 last 106.42) - open/close 16094 (time 1746151019.57 total 165.82 last 87.64) - open/close 17179 (time 1746151029.57 total 175.82 last 108.47) - open/close 18295 (time 1746151039.58 total 185.83 last 111.55) - open/close 19501 (time 1746151049.59 total 195.84 last 120.44) - open/close 20000 (time 1746151054.42 total 200.68 last 103.23) - open/close 20957 (time 1746151064.43 total 210.68 last 95.62) - open/close 21989 (time 1746151074.43 total 220.69 last 103.18) - open/close 23049 (time 1746151084.44 total 230.69 last 105.95) - open/close 24006 (time 1746151094.45 total 240.70 last 95.63) - open/close 25093 (time 1746151104.45 total 250.71 last 108.61) - open/close 26154 (time 1746151114.46 total 260.71 last 106.07) - open/close 27182 (time 1746151124.46 total 270.71 last 102.78) - open/close 28178 (time 1746151134.46 total 280.71 last 99.56) - open/close 29392 (time 1746151144.47 total 290.72 last 121.29) - open/close 30000 (time 1746151149.10 total 295.35 last 131.37) - open/close 31259 (time 1746151159.10 total 305.35 last 125.87) - open/close 32539 (time 1746151169.11 total 315.36 last 127.94) - open/close 33796 (time 1746151179.11 total 325.36 last 125.67) - open/close 35163 (time 1746151189.11 total 335.36 last 136.65) - open/close 36335 (time 1746151199.11 total 345.36 last 117.19) - open/close 37308 (time 1746151209.11 total 355.37 last 97.29) - open/close 38369 (time 1746151219.12 total 365.37 last 106.06) - open/close 39413 (time 1746151229.12 total 375.37 last 104.36) - open/close 40000 (time 1746151234.40 total 380.65 last 111.18) - open/close 41201 (time 1746151244.41 total 390.66 last 120.04) - open/close 42456 (time 1746151254.41 total 400.66 last 125.43) - open/close 43433 (time 1746151264.42 total 410.67 last 97.59) - open/close 44541 (time 1746151274.43 total 420.68 last 110.77) - open/close 45541 (time 1746151284.43 total 430.68 last 99.93) - open/close 46423 (time 1746151294.44 total 440.69 last 88.12) - open/close 47346 (time 1746151304.44 total 450.70 last 92.27) - open/close 48502 (time 1746151314.44 total 460.70 last 115.60) - open/close 49618 (time 1746151324.45 total 470.70 last 111.59) - open/close 50000 (time 1746151328.02 total 474.27 last 106.76) - open/close 51226 (time 1746151338.03 total 484.28 last 122.49) - open/close 52608 (time 1746151348.04 total 494.29 last 138.16) - open/close 53927 (time 1746151358.04 total 504.29 last 131.89) - open/close 55060 (time 1746151368.04 total 514.29 last 113.28) - open/close 56132 (time 1746151378.04 total 524.29 last 107.19) - open/close 57459 (time 1746151388.04 total 534.29 last 132.69) - open/close 58557 (time 1746151398.04 total 544.29 last 109.75) - open/close 59831 (time 1746151408.04 total 554.30 last 127.38) - open/close 61268 (time 1746151418.05 total 564.30 last 143.64) - open/close 62539 (time 1746151428.05 total 574.30 last 127.07) - open/close 63803 (time 1746151438.05 total 584.31 last 126.35) total: 64768 open/close in 591.15 seconds: 109.56 ops/second - /unlink 1951 (time 1746151456.52 total 10.00 last 195.07) - /unlink 3935 (time 1746151466.53 total 20.01 last 198.31) - /unlink 6158 (time 1746151476.53 total 30.01 last 222.23) - /unlink 8141 (time 1746151486.53 total 40.01 last 198.29) - /unlink 10000 (time 1746151496.49 total 49.97 last 186.64) - /unlink 12026 (time 1746151506.50 total 59.98 last 202.47) - /unlink 14421 (time 1746151516.50 total 69.98 last 239.45) - /unlink 16427 (time 1746151526.51 total 79.99 last 200.31) - /unlink 18249 (time 1746151536.51 total 89.99 last 182.18) - /unlink 20000 (time 1746151542.80 total 96.28 last 278.68) - /unlink 22417 (time 1746151552.80 total 106.28 last 241.68) - /unlink 24551 (time 1746151562.80 total 116.28 last 213.36) - /unlink 27011 (time 1746151572.81 total 126.28 last 245.89) - /unlink 29620 (time 1746151582.81 total 136.29 last 260.78) - /unlink 32307 (time 1746151592.81 total 146.29 last 268.66) - /unlink 34577 (time 1746151602.81 total 156.29 last 226.97) - /unlink 36735 (time 1746151612.81 total 166.29 last 215.75) - /unlink 38872 (time 1746151622.82 total 176.30 last 213.57) - /unlink 40000 (time 1746151629.04 total 182.52 last 181.48) - /unlink 42197 (time 1746151639.04 total 192.52 last 219.66) - /unlink 44443 (time 1746151649.04 total 202.52 last 224.53) - /unlink 47002 (time 1746151659.04 total 212.52 last 255.89) - /unlink 49040 (time 1746151669.05 total 222.53 last 203.68) - /unlink 50000 (time 1746151673.87 total 227.35 last 198.88) - /unlink 52190 (time 1746151683.88 total 237.36 last 218.92) - /unlink 54277 (time 1746151693.88 total 247.36 last 208.60) - /unlink 56076 (time 1746151703.89 total 257.36 last 179.86) - /unlink 58191 (time 1746151713.89 total 267.37 last 211.30) - /unlink 60000 (time 1746151722.17 total 275.65 last 218.66) - /unlink 62028 (time 1746151732.17 total 285.65 last 202.78) - /unlink 64078 (time 1746151742.17 total 295.65 last 204.90) total: 64768 /unlink in 298.21 seconds: 217.19 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1384 (time 1746151773.73 total 10.00 last 138.36) - open/close 2558 (time 1746151783.73 total 20.00 last 117.38) - open/close 3850 (time 1746151793.73 total 30.01 last 129.13) - open/close 5021 (time 1746151803.74 total 40.02 last 117.02) - open/close 6358 (time 1746151813.74 total 50.02 last 133.64) - open/close 7614 (time 1746151823.74 total 60.02 last 125.60) - open/close 8804 (time 1746151833.75 total 70.02 last 118.99) - open/close 10000 (time 1746151842.87 total 79.14 last 131.14) - open/close 11299 (time 1746151852.87 total 89.15 last 129.84) - open/close 12507 (time 1746151862.87 total 99.15 last 120.79) - open/close 13800 (time 1746151872.88 total 109.15 last 129.25) - open/close 14978 (time 1746151882.88 total 119.16 last 117.74) - open/close 16347 (time 1746151892.88 total 129.16 last 136.88) - open/close 17578 (time 1746151902.88 total 139.16 last 123.09) - open/close 18865 (time 1746151912.89 total 149.16 last 128.64) - open/close 20000 (time 1746151921.59 total 157.86 last 130.49) - open/close 21160 (time 1746151931.59 total 167.87 last 115.94) - open/close 22302 (time 1746151941.59 total 177.87 last 114.17) - open/close 23473 (time 1746151951.60 total 187.87 last 117.05) - open/close 24807 (time 1746151961.61 total 197.88 last 133.26) - open/close 26013 (time 1746151971.61 total 207.89 last 120.55) - open/close 27061 (time 1746151981.63 total 217.91 last 104.62) - open/close 28275 (time 1746151991.63 total 227.91 last 121.38) - open/close 29491 (time 1746152001.63 total 237.91 last 121.60) - open/close 30000 (time 1746152006.02 total 242.29 last 116.04) - open/close 31092 (time 1746152016.02 total 252.30 last 109.13) - open/close 32311 (time 1746152026.03 total 262.30 last 121.88) - open/close 33553 (time 1746152036.03 total 272.31 last 124.11) - open/close 34796 (time 1746152046.04 total 282.32 last 124.23) - open/close 36080 (time 1746152056.05 total 292.33 last 128.25) - open/close 37351 (time 1746152066.05 total 302.33 last 127.07) - open/close 38438 (time 1746152076.05 total 312.33 last 108.68) - open/close 39440 (time 1746152086.07 total 322.35 last 100.03) - open/close 40000 (time 1746152091.55 total 327.83 last 102.20) - open/close 41020 (time 1746152101.55 total 337.83 last 101.98) - open/close 41961 (time 1746152111.56 total 347.83 last 94.06) - open/close 43026 (time 1746152121.56 total 357.84 last 106.47) - open/close 44063 (time 1746152131.57 total 367.84 last 103.62) - open/close 45291 (time 1746152141.57 total 377.85 last 122.75) - open/close 46376 (time 1746152151.58 total 387.86 last 108.41) - open/close 47427 (time 1746152161.58 total 397.86 last 105.10) - open/close 48486 (time 1746152171.58 total 407.86 last 105.87) - open/close 49754 (time 1746152181.58 total 417.86 last 126.78) - open/close 50000 (time 1746152183.87 total 420.15 last 107.42) - open/close 51037 (time 1746152193.88 total 430.16 last 103.62) - open/close 52282 (time 1746152203.89 total 440.16 last 124.43) - open/close 53598 (time 1746152213.89 total 450.17 last 131.53) - open/close 54750 (time 1746152223.89 total 460.17 last 115.19) - open/close 55847 (time 1746152233.90 total 470.18 last 109.59) - open/close 57101 (time 1746152243.91 total 480.18 last 125.35) - open/close 58380 (time 1746152253.91 total 490.19 last 127.83) - open/close 59622 (time 1746152263.91 total 500.19 last 124.19) - open/close 60000 (time 1746152267.27 total 503.55 last 112.68) - open/close 61066 (time 1746152277.27 total 513.55 last 106.60) - open/close 62263 (time 1746152287.27 total 523.55 last 119.65) - open/close 63463 (time 1746152297.28 total 533.56 last 119.91) - open/close 64628 (time 1746152307.29 total 543.56 last 116.43) total: 64768 open/close in 544.91 seconds: 118.86 ops/second - /unlink 1620 (time 1746152320.67 total 10.00 last 161.99) - /unlink 3410 (time 1746152330.68 total 20.01 last 178.90) - /unlink 5214 (time 1746152340.68 total 30.01 last 180.35) - /unlink 6797 (time 1746152350.69 total 40.01 last 158.24) - /unlink 8405 (time 1746152360.69 total 50.01 last 160.78) - /unlink 10000 (time 1746152370.12 total 59.44 last 169.12) - /unlink 12035 (time 1746152380.12 total 69.45 last 203.48) - /unlink 13900 (time 1746152390.12 total 79.45 last 186.45) - /unlink 15506 (time 1746152400.13 total 89.46 last 160.44) - /unlink 17044 (time 1746152410.13 total 99.46 last 153.77) - /unlink 18552 (time 1746152420.13 total 109.46 last 150.80) - /unlink 20000 (time 1746152428.43 total 117.75 last 174.60) - /unlink 21815 (time 1746152438.43 total 127.76 last 181.39) - /unlink 23758 (time 1746152448.44 total 137.76 last 194.24) - /unlink 25821 (time 1746152458.44 total 147.77 last 206.21) - /unlink 27837 (time 1746152468.45 total 157.77 last 201.50) - /unlink 29814 (time 1746152478.45 total 167.77 last 197.70) - /unlink 31655 (time 1746152488.45 total 177.78 last 184.05) - /unlink 33566 (time 1746152498.45 total 187.78 last 191.09) - /unlink 35450 (time 1746152508.45 total 197.78 last 188.38) - /unlink 37176 (time 1746152518.45 total 207.78 last 172.59) - /unlink 38867 (time 1746152528.46 total 217.78 last 169.03) - /unlink 40000 (time 1746152535.72 total 225.05 last 155.91) - /unlink 41647 (time 1746152545.73 total 235.06 last 164.57) - /unlink 43275 (time 1746152555.73 total 245.06 last 162.74) - /unlink 45026 (time 1746152565.73 total 255.06 last 175.09) - /unlink 47104 (time 1746152575.74 total 265.06 last 207.73) - /unlink 48917 (time 1746152585.75 total 275.08 last 181.01) - /unlink 50000 (time 1746152591.20 total 280.52 last 198.99) - /unlink 51906 (time 1746152601.20 total 290.52 last 190.59) - /unlink 53882 (time 1746152611.20 total 300.52 last 197.55) - /unlink 55678 (time 1746152621.20 total 310.53 last 179.54) - /unlink 57270 (time 1746152631.21 total 320.54 last 159.08) - /unlink 58816 (time 1746152641.21 total 330.54 last 154.54) - /unlink 60000 (time 1746152647.96 total 337.29 last 175.49) - /unlink 61749 (time 1746152657.96 total 347.29 last 174.86) - /unlink 63571 (time 1746152667.97 total 357.29 last 182.14) total: 64768 /unlink in 362.93 seconds: 178.46 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:)