== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:46:04 (1709786764) Stopping clients: oleg124-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg124-client.virtnet /mnt/lustre2 (opts:-f) oleg124-server: oleg124-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing load_modules_local oleg124-server: Loading modules from /home/green/git/lustre-release/lustre oleg124-server: detected 4 online CPUs by sysfs oleg124-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg124-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg124-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg124-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg124-client.virtnet: -o user_xattr,flock oleg124-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1174 (time 1709786834.20 total 10.01 last 117.25) - open/close 2165 (time 1709786844.20 total 20.02 last 99.06) - open/close 3015 (time 1709786854.21 total 30.02 last 84.95) - open/close 3783 (time 1709786864.22 total 40.03 last 76.73) - open/close 4570 (time 1709786874.22 total 50.04 last 78.63) - open/close 5283 (time 1709786884.23 total 60.05 last 71.22) - open/close 5889 (time 1709786894.24 total 70.06 last 60.57) - open/close 6553 (time 1709786904.24 total 80.06 last 66.40) - open/close 7210 (time 1709786914.26 total 90.07 last 65.58) - open/close 7856 (time 1709786924.27 total 100.08 last 64.55) - open/close 8513 (time 1709786934.29 total 110.11 last 65.54) - open/close 9187 (time 1709786944.30 total 120.11 last 67.36) - open/close 9764 (time 1709786954.30 total 130.12 last 57.66) - open/close 10000 (time 1709786958.07 total 133.88 last 62.73) - open/close 10515 (time 1709786968.08 total 143.90 last 51.41) - open/close 10633 (time 1709786981.52 total 157.33 last 8.78) - open/close 11243 (time 1709786991.52 total 167.34 last 60.96) - open/close 11881 (time 1709787001.53 total 177.35 last 63.74) - open/close 12457 (time 1709787011.55 total 187.36 last 57.52) - open/close 12971 (time 1709787021.57 total 197.39 last 51.26) - open/close 13502 (time 1709787031.57 total 207.39 last 53.10) - open/close 14037 (time 1709787041.59 total 217.41 last 53.40) - open/close 14498 (time 1709787051.59 total 227.41 last 46.09) - open/close 14990 (time 1709787061.60 total 237.42 last 49.15) - open/close 15374 (time 1709787071.61 total 247.43 last 38.38) - open/close 15842 (time 1709787081.61 total 257.43 last 46.78) - open/close 16237 (time 1709787091.62 total 267.44 last 39.47) - open/close 16362 (time 1709787105.78 total 281.59 last 8.83) - open/close 16775 (time 1709787115.78 total 291.60 last 41.27) - open/close 17140 (time 1709787125.80 total 301.61 last 36.45) - open/close 17540 (time 1709787135.80 total 311.62 last 40.00) - open/close 17894 (time 1709787145.81 total 321.63 last 35.36) - open/close 18243 (time 1709787155.82 total 331.63 last 34.88) - open/close 18595 (time 1709787165.83 total 341.64 last 35.16) - open/close 18923 (time 1709787175.89 total 351.70 last 32.61) - open/close 19284 (time 1709787185.91 total 361.73 last 36.00) - open/close 19322 (time 1709787198.12 total 373.93 last 3.11) - open/close 19359 (time 1709787210.14 total 385.96 last 3.08) - open/close 19692 (time 1709787220.16 total 395.98 last 33.23) - open/close 19962 (time 1709787230.17 total 405.98 last 26.99) - open/close 20248 (time 1709787240.21 total 416.03 last 28.47) - open/close 20550 (time 1709787250.22 total 426.04 last 30.18) - open/close 20890 (time 1709787260.22 total 436.04 last 34.00) - open/close 21223 (time 1709787270.23 total 446.04 last 33.28) - open/close 21530 (time 1709787280.25 total 456.06 last 30.64) - open/close 21892 (time 1709787290.25 total 466.07 last 36.19) - open/close 21969 (time 1709787303.85 total 479.67 last 5.66) - open/close 22237 (time 1709787313.90 total 489.72 last 26.67) - open/close 22537 (time 1709787323.92 total 499.73 last 29.96) - open/close 22832 (time 1709787333.93 total 509.74 last 29.47) - open/close 23087 (time 1709787343.95 total 519.77 last 25.43) - open/close 23370 (time 1709787353.97 total 529.79 last 28.25) - open/close 23654 (time 1709787363.97 total 539.79 last 28.40) - open/close 23958 (time 1709787373.99 total 549.80 last 30.36) - open/close 24223 (time 1709787383.99 total 559.81 last 26.48) - open/close 24488 (time 1709787394.01 total 569.83 last 26.46) - open/close 24720 (time 1709787404.03 total 579.84 last 23.16) - open/close 24981 (time 1709787414.04 total 589.85 last 26.08) - open/close 25265 (time 1709787424.05 total 599.87 last 28.36) - open/close 25505 (time 1709787434.09 total 609.90 last 23.91) - open/close 25743 (time 1709787444.14 total 619.95 last 23.69) - open/close 25982 (time 1709787454.15 total 629.96 last 23.88) - open/close 26233 (time 1709787464.19 total 640.00 last 24.99) - open/close 26461 (time 1709787474.24 total 650.05 last 22.69) - open/close 26549 (time 1709787489.91 total 665.73 last 5.61) - open/close 26737 (time 1709787508.80 total 684.62 last 9.95) - open/close 26820 (time 1709787523.53 total 699.35 last 5.63) - open/close 27065 (time 1709787533.54 total 709.35 last 24.48) - open/close 27283 (time 1709787543.59 total 719.41 last 21.69) - open/close 27492 (time 1709787553.61 total 729.42 last 20.87) - open/close 27680 (time 1709787563.64 total 739.46 last 18.74) - open/close 27885 (time 1709787573.67 total 749.48 last 20.45) - open/close 28096 (time 1709787583.69 total 759.51 last 21.05) - open/close 28333 (time 1709787593.74 total 769.55 last 23.59) - open/close 28560 (time 1709787603.74 total 779.56 last 22.68) - open/close 28789 (time 1709787613.74 total 789.56 last 22.90) - open/close 29029 (time 1709787623.75 total 799.56 last 23.99) - open/close 29244 (time 1709787633.75 total 809.57 last 21.49) - open/close 29494 (time 1709787643.77 total 819.59 last 24.95) - open/close 29722 (time 1709787653.81 total 829.63 last 22.72) - open/close 29953 (time 1709787663.87 total 839.69 last 22.96) - open/close 30186 (time 1709787673.89 total 849.70 last 23.26) - open/close 30420 (time 1709787683.93 total 859.74 last 23.31) - open/close 30653 (time 1709787693.96 total 869.77 last 23.23) - open/close 30893 (time 1709787703.98 total 879.79 last 23.95) - open/close 31133 (time 1709787714.01 total 889.82 last 23.93) - open/close 31393 (time 1709787724.03 total 899.84 last 25.94) - open/close 31644 (time 1709787734.05 total 909.87 last 25.04) - open/close 31862 (time 1709787744.09 total 919.90 last 21.72) - open/close 32083 (time 1709787754.09 total 929.90 last 22.10) - open/close 32290 (time 1709787764.12 total 939.94 last 20.63) - open/close 32524 (time 1709787774.13 total 949.94 last 23.39) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 952.48 seconds: 34.18 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 210 (time 1709787788.68 total 10.00 last 21.00) - /unlink 417 (time 1709787798.69 total 20.01 last 20.67) - /unlink 647 (time 1709787808.72 total 30.04 last 22.94) - /unlink 892 (time 1709787818.75 total 40.07 last 24.43) - /unlink 1116 (time 1709787828.75 total 50.07 last 22.39) - /unlink 1325 (time 1709787838.79 total 60.11 last 20.81) - /unlink 1502 (time 1709787848.90 total 70.22 last 17.52) - /unlink 1730 (time 1709787858.92 total 80.24 last 22.75) - /unlink 1957 (time 1709787868.94 total 90.27 last 22.64) - /unlink 2187 (time 1709787878.98 total 100.30 last 22.91) - /unlink 2403 (time 1709787889.02 total 110.34 last 21.51) - /unlink 2641 (time 1709787899.03 total 120.35 last 23.78) - /unlink 2873 (time 1709787909.07 total 130.39 last 23.10) - /unlink 3070 (time 1709787919.15 total 140.47 last 19.55) - /unlink 3291 (time 1709787929.20 total 150.52 last 21.99) - /unlink 3516 (time 1709787939.22 total 160.54 last 22.46) - /unlink 3733 (time 1709787949.25 total 170.57 last 21.64) - /unlink 3982 (time 1709787959.26 total 180.58 last 24.86) - /unlink 4209 (time 1709787969.30 total 190.62 last 22.62) - /unlink 4449 (time 1709787979.33 total 200.65 last 23.93) - /unlink 4700 (time 1709787989.33 total 210.65 last 25.09) - /unlink 4949 (time 1709787999.35 total 220.67 last 24.86) - /unlink 5196 (time 1709788009.36 total 230.68 last 24.67) - /unlink 5446 (time 1709788019.38 total 240.70 last 24.94) - /unlink 5692 (time 1709788029.39 total 250.71 last 24.59) - /unlink 5945 (time 1709788039.40 total 260.72 last 25.28) - /unlink 6201 (time 1709788049.43 total 270.75 last 25.52) - /unlink 6463 (time 1709788059.44 total 280.76 last 26.19) - /unlink 6729 (time 1709788069.46 total 290.78 last 26.53) - /unlink 7000 (time 1709788079.47 total 300.79 last 27.07) - /unlink 7277 (time 1709788089.50 total 310.82 last 27.63) - /unlink 7549 (time 1709788099.52 total 320.84 last 27.14) - /unlink 7813 (time 1709788109.53 total 330.85 last 26.40) - /unlink 8081 (time 1709788119.55 total 340.87 last 26.74) - /unlink 8352 (time 1709788129.58 total 350.90 last 27.02) - /unlink 8643 (time 1709788139.60 total 360.92 last 29.03) - /unlink 8899 (time 1709788149.62 total 370.94 last 25.54) - /unlink 9162 (time 1709788159.63 total 380.95 last 26.28) - /unlink 9445 (time 1709788169.63 total 390.95 last 28.30) - /unlink 9736 (time 1709788179.63 total 400.95 last 29.10) - /unlink 10000 (time 1709788189.06 total 410.38 last 28.00) - /unlink 10302 (time 1709788199.08 total 420.41 last 30.13) - /unlink 10627 (time 1709788209.10 total 430.42 last 32.44) - /unlink 10948 (time 1709788219.12 total 440.44 last 32.03) - /unlink 11264 (time 1709788229.13 total 450.45 last 31.58) - /unlink 11609 (time 1709788239.16 total 460.48 last 34.40) - /unlink 11956 (time 1709788249.17 total 470.49 last 34.67) - /unlink 12275 (time 1709788259.18 total 480.50 last 31.87) - /unlink 12615 (time 1709788269.20 total 490.52 last 33.91) - /unlink 12967 (time 1709788279.22 total 500.54 last 35.14) - /unlink 13348 (time 1709788289.24 total 510.56 last 38.00) - /unlink 13737 (time 1709788299.25 total 520.57 last 38.88) - /unlink 14112 (time 1709788309.27 total 530.59 last 37.42) - /unlink 14501 (time 1709788319.29 total 540.61 last 38.82) - /unlink 14921 (time 1709788329.29 total 550.62 last 41.97) - /unlink 15313 (time 1709788339.30 total 560.62 last 39.18) - /unlink 15683 (time 1709788349.31 total 570.63 last 36.98) - /unlink 16108 (time 1709788359.31 total 580.63 last 42.48) - /unlink 16550 (time 1709788369.34 total 590.66 last 44.08) - /unlink 16968 (time 1709788379.35 total 600.67 last 41.75) - /unlink 17405 (time 1709788389.36 total 610.68 last 43.64) - /unlink 17866 (time 1709788399.38 total 620.70 last 46.03) - /unlink 18378 (time 1709788409.39 total 630.71 last 51.14) - /unlink 18883 (time 1709788419.42 total 640.74 last 50.35) - /unlink 19420 (time 1709788429.45 total 650.77 last 53.54) - /unlink 19991 (time 1709788439.46 total 660.78 last 57.02) - /unlink 20596 (time 1709788449.48 total 670.80 last 60.41) - /unlink 21237 (time 1709788459.48 total 680.80 last 64.07) - /unlink 21889 (time 1709788469.49 total 690.81 last 65.15) - /unlink 22640 (time 1709788479.52 total 700.84 last 74.90) - /unlink 23397 (time 1709788489.53 total 710.85 last 75.63) - /unlink 24317 (time 1709788499.53 total 720.85 last 91.98) - /unlink 25314 (time 1709788509.54 total 730.86 last 99.63) - /unlink 26488 (time 1709788519.54 total 740.86 last 117.36) - /unlink 27749 (time 1709788529.54 total 750.86 last 126.05) - /unlink 29274 (time 1709788539.54 total 760.87 last 152.48) - /unlink 30000 (time 1709788543.87 total 765.19 last 167.76) - /unlink 31807 (time 1709788553.88 total 775.20 last 180.64) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 778.54 seconds: 41.82 ops/second Waiting for MDT destroys to complete - open/close 1225 (time 1709788572.18 total 10.00 last 122.44) - open/close 2175 (time 1709788582.18 total 20.01 last 94.96) - open/close 3159 (time 1709788592.19 total 30.02 last 98.33) - open/close 4121 (time 1709788602.20 total 40.03 last 96.09) - open/close 5062 (time 1709788612.20 total 50.03 last 94.08) - open/close 6021 (time 1709788622.21 total 60.04 last 95.85) - open/close 6996 (time 1709788632.21 total 70.04 last 97.49) - open/close 7741 (time 1709788642.21 total 80.04 last 74.47) - open/close 8434 (time 1709788652.22 total 90.04 last 69.27) - open/close 9097 (time 1709788662.22 total 100.05 last 66.29) - open/close 9715 (time 1709788672.22 total 110.05 last 61.77) - open/close 10000 (time 1709788677.35 total 115.18 last 55.56) - open/close 10542 (time 1709788687.36 total 125.19 last 54.16) - open/close 11099 (time 1709788697.37 total 135.19 last 55.67) - open/close 11658 (time 1709788707.37 total 145.20 last 55.86) - open/close 12198 (time 1709788717.39 total 155.21 last 53.93) - open/close 12750 (time 1709788727.39 total 165.21 last 55.20) - open/close 13251 (time 1709788737.40 total 175.23 last 50.03) - open/close 13708 (time 1709788747.41 total 185.24 last 45.64) - open/close 14158 (time 1709788757.42 total 195.24 last 44.99) - open/close 14598 (time 1709788767.42 total 205.25 last 43.99) - open/close 15005 (time 1709788777.42 total 215.25 last 40.69) - open/close 15437 (time 1709788787.43 total 225.25 last 43.18) - open/close 15906 (time 1709788797.45 total 235.27 last 46.81) - open/close 16338 (time 1709788807.45 total 245.28 last 43.17) - open/close 16797 (time 1709788817.46 total 255.29 last 45.85) - open/close 17236 (time 1709788827.47 total 265.30 last 43.86) - open/close 17685 (time 1709788837.47 total 275.30 last 44.89) - open/close 18123 (time 1709788847.49 total 285.31 last 43.74) - open/close 18542 (time 1709788857.50 total 295.32 last 41.87) - open/close 18945 (time 1709788867.50 total 305.33 last 40.28) - open/close 19370 (time 1709788877.51 total 315.34 last 42.44) - open/close 19760 (time 1709788887.53 total 325.36 last 38.93) - open/close 20000 (time 1709788893.51 total 331.34 last 40.12) - open/close 20402 (time 1709788903.53 total 341.36 last 40.12) - open/close 20785 (time 1709788913.55 total 351.38 last 38.23) - open/close 21159 (time 1709788923.58 total 361.40 last 37.30) - open/close 21539 (time 1709788933.60 total 371.43 last 37.90) - open/close 21906 (time 1709788943.61 total 381.44 last 36.67) - open/close 22249 (time 1709788953.62 total 391.45 last 34.26) - open/close 22593 (time 1709788963.63 total 401.46 last 34.36) - open/close 22940 (time 1709788973.66 total 411.48 last 34.62) - open/close 23280 (time 1709788983.67 total 421.50 last 33.95) - open/close 23616 (time 1709788993.68 total 431.50 last 33.59) - open/close 23953 (time 1709789003.69 total 441.52 last 33.65) - open/close 24278 (time 1709789013.72 total 451.54 last 32.41) - open/close 24632 (time 1709789023.72 total 461.55 last 35.40) - open/close 24988 (time 1709789033.73 total 471.56 last 35.54) - open/close 25314 (time 1709789043.73 total 481.56 last 32.60) - open/close 25635 (time 1709789053.76 total 491.58 last 32.03) - open/close 25946 (time 1709789063.77 total 501.60 last 31.05) - open/close 26262 (time 1709789073.78 total 511.61 last 31.57) - open/close 26547 (time 1709789083.79 total 521.61 last 28.48) - open/close 26835 (time 1709789093.79 total 531.62 last 28.78) - open/close 27126 (time 1709789103.80 total 541.62 last 29.08) - open/close 27409 (time 1709789113.82 total 551.65 last 28.24) - open/close 27690 (time 1709789123.85 total 561.68 last 28.01) - open/close 27960 (time 1709789133.87 total 571.70 last 26.95) - open/close 28234 (time 1709789143.87 total 581.70 last 27.40) - open/close 28508 (time 1709789153.91 total 591.73 last 27.30) - open/close 28776 (time 1709789163.92 total 601.75 last 26.77) - open/close 29030 (time 1709789173.95 total 611.77 last 25.33) - open/close 29280 (time 1709789183.98 total 621.80 last 24.93) - open/close 29539 (time 1709789194.01 total 631.83 last 25.82) - open/close 29817 (time 1709789204.03 total 641.86 last 27.73) - open/close 30000 (time 1709789210.99 total 648.81 last 26.30) - open/close 30273 (time 1709789221.01 total 658.84 last 27.24) - open/close 30552 (time 1709789231.03 total 668.86 last 27.84) - open/close 30813 (time 1709789241.05 total 678.88 last 26.05) - open/close 31075 (time 1709789251.06 total 688.89 last 26.17) - open/close 31340 (time 1709789261.10 total 698.93 last 26.41) - open/close 31605 (time 1709789271.11 total 708.94 last 26.47) - open/close 31865 (time 1709789281.14 total 718.96 last 25.93) - open/close 32104 (time 1709789291.15 total 728.98 last 23.85) - open/close 32340 (time 1709789301.19 total 739.02 last 23.51) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 748.21 seconds: 43.52 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 207 (time 1709789322.12 total 10.04 last 20.61) - /unlink 416 (time 1709789332.13 total 20.05 last 20.88) - /unlink 635 (time 1709789342.15 total 30.07 last 21.87) - /unlink 866 (time 1709789352.15 total 40.07 last 23.09) - /unlink 1108 (time 1709789362.17 total 50.09 last 24.15) - /unlink 1341 (time 1709789372.20 total 60.12 last 23.24) - /unlink 1577 (time 1709789382.23 total 70.15 last 23.52) - /unlink 1808 (time 1709789392.28 total 80.20 last 22.99) - /unlink 2040 (time 1709789402.30 total 90.22 last 23.17) - /unlink 2280 (time 1709789412.30 total 100.22 last 24.00) - /unlink 2510 (time 1709789422.34 total 110.26 last 22.91) - /unlink 2752 (time 1709789432.35 total 120.27 last 24.17) - /unlink 2986 (time 1709789442.36 total 130.28 last 23.37) - /unlink 3227 (time 1709789452.38 total 140.30 last 24.06) - /unlink 3474 (time 1709789462.42 total 150.34 last 24.60) - /unlink 3684 (time 1709789472.43 total 160.35 last 20.98) - /unlink 3901 (time 1709789482.44 total 170.36 last 21.67) - /unlink 4123 (time 1709789492.45 total 180.37 last 22.18) - /unlink 4377 (time 1709789502.48 total 190.40 last 25.34) - /unlink 4631 (time 1709789512.50 total 200.41 last 25.36) - /unlink 4891 (time 1709789522.51 total 210.43 last 25.97) - /unlink 5134 (time 1709789532.55 total 220.47 last 24.21) - /unlink 5386 (time 1709789542.56 total 230.48 last 25.17) - /unlink 5623 (time 1709789552.58 total 240.49 last 23.66) - /unlink 5872 (time 1709789562.59 total 250.51 last 24.87) - /unlink 6116 (time 1709789572.59 total 260.51 last 24.40) - /unlink 6384 (time 1709789582.61 total 270.53 last 26.75) - /unlink 6645 (time 1709789592.64 total 280.56 last 26.01) - /unlink 6899 (time 1709789602.66 total 290.58 last 25.36) - /unlink 7158 (time 1709789612.69 total 300.61 last 25.82) - /unlink 7408 (time 1709789622.73 total 310.65 last 24.91) - /unlink 7681 (time 1709789632.75 total 320.67 last 27.24) - /unlink 7943 (time 1709789642.77 total 330.69 last 26.15) - /unlink 8201 (time 1709789652.78 total 340.70 last 25.76) - /unlink 8465 (time 1709789662.83 total 350.75 last 26.29) - /unlink 8735 (time 1709789672.84 total 360.76 last 26.96) - /unlink 9007 (time 1709789682.88 total 370.80 last 27.10) - /unlink 9290 (time 1709789692.90 total 380.82 last 28.25) - /unlink 9597 (time 1709789702.93 total 390.85 last 30.60) - /unlink 9916 (time 1709789712.93 total 400.85 last 31.90) - /unlink 10000 (time 1709789715.59 total 403.51 last 31.62) - /unlink 10293 (time 1709789725.61 total 413.53 last 29.23) - /unlink 10599 (time 1709789735.62 total 423.54 last 30.59) - /unlink 10924 (time 1709789745.63 total 433.54 last 32.47) - /unlink 11230 (time 1709789755.63 total 443.55 last 30.60) - /unlink 11525 (time 1709789765.63 total 453.55 last 29.49) - /unlink 11828 (time 1709789775.65 total 463.57 last 30.24) - /unlink 12149 (time 1709789785.67 total 473.59 last 32.04) - /unlink 12463 (time 1709789795.69 total 483.61 last 31.34) - /unlink 12767 (time 1709789805.71 total 493.63 last 30.33) - /unlink 13090 (time 1709789815.73 total 503.65 last 32.26) - /unlink 13408 (time 1709789825.76 total 513.68 last 31.69) - /unlink 13737 (time 1709789835.77 total 523.69 last 32.86) - /unlink 14070 (time 1709789845.80 total 533.72 last 33.21) - /unlink 14403 (time 1709789855.80 total 543.72 last 33.29) - /unlink 14736 (time 1709789865.83 total 553.75 last 33.21) - /unlink 15082 (time 1709789875.85 total 563.77 last 34.52) - /unlink 15429 (time 1709789885.87 total 573.79 last 34.64) - /unlink 15795 (time 1709789895.87 total 583.79 last 36.59) - /unlink 16186 (time 1709789905.89 total 593.80 last 39.04) - /unlink 16556 (time 1709789915.90 total 603.82 last 36.94) - /unlink 16940 (time 1709789925.92 total 613.84 last 38.34) - /unlink 17355 (time 1709789935.92 total 623.84 last 41.47) - /unlink 17783 (time 1709789945.95 total 633.87 last 42.68) - /unlink 18230 (time 1709789955.96 total 643.88 last 44.67) - /unlink 18682 (time 1709789965.96 total 653.88 last 45.19) - /unlink 19145 (time 1709789975.97 total 663.89 last 46.25) - /unlink 19626 (time 1709789985.97 total 673.89 last 48.08) - /unlink 20000 (time 1709789992.96 total 680.88 last 53.50) - /unlink 20550 (time 1709790002.97 total 690.89 last 54.97) - /unlink 21194 (time 1709790012.97 total 700.89 last 64.38) - /unlink 21971 (time 1709790022.98 total 710.90 last 77.66) - /unlink 22905 (time 1709790032.98 total 720.90 last 93.37) - /unlink 23904 (time 1709790042.98 total 730.90 last 99.89) - /unlink 25002 (time 1709790052.98 total 740.90 last 109.78) - /unlink 26115 (time 1709790062.99 total 750.90 last 111.27) - /unlink 27360 (time 1709790072.99 total 760.91 last 124.49) - /unlink 28944 (time 1709790082.99 total 770.91 last 158.39) - /unlink 30000 (time 1709790089.16 total 777.08 last 170.99) - /unlink 32005 (time 1709790099.17 total 787.08 last 200.44) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 789.52 seconds: 41.24 ops/second Waiting for MDT destroys to complete - open/close 1163 (time 1709790122.17 total 10.01 last 116.19) - open/close 2210 (time 1709790132.18 total 20.02 last 104.63) - open/close 3174 (time 1709790142.18 total 30.03 last 96.32) - open/close 4166 (time 1709790152.19 total 40.03 last 99.11) - open/close 5099 (time 1709790162.20 total 50.04 last 93.21) - open/close 5948 (time 1709790172.20 total 60.05 last 84.89) - open/close 6690 (time 1709790182.22 total 70.06 last 74.11) - open/close 7429 (time 1709790192.23 total 80.07 last 73.82) - open/close 8188 (time 1709790202.24 total 90.08 last 75.83) - open/close 8876 (time 1709790212.24 total 100.08 last 68.80) - open/close 9505 (time 1709790222.24 total 110.08 last 62.86) - open/close 10000 (time 1709790229.68 total 117.52 last 66.60) - open/close 10713 (time 1709790239.69 total 127.53 last 71.22) - open/close 11379 (time 1709790249.70 total 137.54 last 66.52) - open/close 12021 (time 1709790259.71 total 147.55 last 64.13) - open/close 12612 (time 1709790269.72 total 157.56 last 59.02) - open/close 13195 (time 1709790279.73 total 167.57 last 58.25) - open/close 13798 (time 1709790289.74 total 177.59 last 60.22) - open/close 14347 (time 1709790299.75 total 187.59 last 54.85) - open/close 14917 (time 1709790309.76 total 197.60 last 56.98) - open/close 15488 (time 1709790319.77 total 207.61 last 57.02) - open/close 16015 (time 1709790329.78 total 217.63 last 52.63) - open/close 16572 (time 1709790339.80 total 227.64 last 55.63) - open/close 17146 (time 1709790349.80 total 237.64 last 57.37) - open/close 17708 (time 1709790359.81 total 247.65 last 56.14) - open/close 18273 (time 1709790369.81 total 257.66 last 56.49) - open/close 18839 (time 1709790379.82 total 267.66 last 56.56) - open/close 19383 (time 1709790389.82 total 277.67 last 54.38) - open/close 19952 (time 1709790399.84 total 287.68 last 56.80) - open/close 20526 (time 1709790409.86 total 297.70 last 57.32) - open/close 21075 (time 1709790419.87 total 307.71 last 54.82) - open/close 21648 (time 1709790429.89 total 317.73 last 57.20) - open/close 22263 (time 1709790439.90 total 327.74 last 61.44) - open/close 22874 (time 1709790449.91 total 337.75 last 61.04) - open/close 23184 (time 1709790459.92 total 347.76 last 30.97) - open/close 23472 (time 1709790469.95 total 357.79 last 28.72) - open/close 23910 (time 1709790479.96 total 367.80 last 43.75) - open/close 24366 (time 1709790489.97 total 377.81 last 45.56) - open/close 24869 (time 1709790499.97 total 387.81 last 50.30) - open/close 25310 (time 1709790509.99 total 397.84 last 43.99) - open/close 25612 (time 1709790520.02 total 407.86 last 30.13) - open/close 26043 (time 1709790530.03 total 417.87 last 43.05) - open/close 26508 (time 1709790540.04 total 427.88 last 46.48) - open/close 26988 (time 1709790550.04 total 437.88 last 47.99) - open/close 27442 (time 1709790560.05 total 447.89 last 45.34) - open/close 27936 (time 1709790570.06 total 457.90 last 49.37) - open/close 28443 (time 1709790580.06 total 467.91 last 50.65) - open/close 28999 (time 1709790590.07 total 477.91 last 55.55) - open/close 29472 (time 1709790600.08 total 487.92 last 47.28) - open/close 30000 (time 1709790610.02 total 497.86 last 53.12) - open/close 30544 (time 1709790620.03 total 507.87 last 54.35) - open/close 31055 (time 1709790630.03 total 517.87 last 51.08) - open/close 31582 (time 1709790640.04 total 527.88 last 52.67) - open/close 32110 (time 1709790650.05 total 537.89 last 52.73) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 546.59 seconds: 59.57 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 467 (time 1709790669.70 total 10.01 last 46.67) - /unlink 891 (time 1709790679.71 total 20.01 last 42.37) - /unlink 1383 (time 1709790689.73 total 30.03 last 49.12) - /unlink 1824 (time 1709790699.74 total 40.04 last 44.06) - /unlink 2348 (time 1709790709.74 total 50.04 last 52.39) - /unlink 2845 (time 1709790719.74 total 60.05 last 49.68) - /unlink 3401 (time 1709790729.76 total 70.07 last 55.49) - /unlink 3998 (time 1709790739.77 total 80.07 last 59.68) - /unlink 4616 (time 1709790749.77 total 90.07 last 61.79) - /unlink 5258 (time 1709790759.78 total 100.08 last 64.15) - /unlink 6020 (time 1709790769.78 total 110.09 last 76.14) - /unlink 6679 (time 1709790779.80 total 120.10 last 65.81) - /unlink 7521 (time 1709790789.81 total 130.11 last 84.11) - /unlink 8239 (time 1709790799.82 total 140.12 last 71.72) - /unlink 9041 (time 1709790809.82 total 150.13 last 80.15) - /unlink 9884 (time 1709790819.83 total 160.13 last 84.28) - /unlink 10804 (time 1709790829.84 total 170.14 last 91.91) - /unlink 11760 (time 1709790839.84 total 180.14 last 95.60) - /unlink 12745 (time 1709790849.85 total 190.15 last 98.41) - /unlink 13759 (time 1709790859.85 total 200.15 last 101.36) - /unlink 14818 (time 1709790869.85 total 210.16 last 105.86) - /unlink 16085 (time 1709790879.86 total 220.16 last 126.66) - /unlink 17227 (time 1709790889.86 total 230.17 last 114.16) - /unlink 18171 (time 1709790899.87 total 240.17 last 94.36) - /unlink 19247 (time 1709790909.87 total 250.17 last 107.56) - /unlink 20000 (time 1709790915.55 total 255.85 last 132.67) - /unlink 21217 (time 1709790925.55 total 265.85 last 121.64) - /unlink 22554 (time 1709790935.55 total 275.86 last 133.65) - /unlink 24556 (time 1709790945.56 total 285.86 last 200.16) - /unlink 26692 (time 1709790955.56 total 295.86 last 213.58) - /unlink 29419 (time 1709790965.56 total 305.86 last 272.63) - /unlink 30000 (time 1709790967.90 total 308.20 last 248.21) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 316.19 seconds: 102.98 ops/second Waiting for MDT destroys to complete - open/close 2035 (time 1709790996.35 total 10.00 last 203.42) - open/close 3586 (time 1709791006.35 total 20.00 last 155.08) - open/close 4848 (time 1709791016.35 total 30.01 last 126.15) - open/close 6334 (time 1709791026.35 total 40.01 last 148.57) - open/close 7715 (time 1709791036.36 total 50.02 last 137.97) - open/close 8881 (time 1709791046.36 total 60.02 last 116.59) - open/close 9897 (time 1709791056.37 total 70.03 last 101.53) - open/close 10776 (time 1709791066.37 total 80.03 last 87.86) - open/close 11580 (time 1709791076.38 total 90.03 last 80.37) - open/close 12605 (time 1709791086.38 total 100.04 last 102.45) - open/close 13570 (time 1709791096.39 total 110.05 last 96.39) - open/close 14376 (time 1709791106.40 total 120.06 last 80.57) - open/close 15400 (time 1709791116.40 total 130.06 last 102.38) - open/close 16392 (time 1709791126.40 total 140.06 last 99.16) - open/close 17132 (time 1709791136.41 total 150.07 last 73.93) - open/close 17918 (time 1709791146.41 total 160.07 last 78.60) - open/close 18763 (time 1709791156.41 total 170.07 last 84.50) - open/close 19728 (time 1709791166.41 total 180.07 last 96.50) - open/close 20000 (time 1709791169.28 total 182.93 last 95.00) - open/close 20743 (time 1709791179.28 total 192.94 last 74.26) - open/close 21478 (time 1709791189.28 total 202.94 last 73.49) - open/close 22163 (time 1709791199.29 total 212.94 last 68.49) - open/close 22874 (time 1709791209.29 total 222.94 last 71.10) - open/close 23606 (time 1709791219.29 total 232.95 last 73.17) - open/close 24347 (time 1709791229.30 total 242.96 last 74.00) - open/close 25048 (time 1709791239.32 total 252.97 last 70.01) - open/close 25733 (time 1709791249.33 total 262.99 last 68.42) - open/close 26450 (time 1709791259.33 total 272.99 last 71.67) - open/close 27167 (time 1709791269.34 total 282.99 last 71.66) - open/close 27884 (time 1709791279.35 total 293.01 last 71.60) - open/close 28561 (time 1709791289.35 total 303.01 last 67.69) - open/close 29220 (time 1709791299.36 total 313.02 last 65.82) - open/close 29815 (time 1709791309.37 total 323.03 last 59.47) - open/close 30000 (time 1709791312.05 total 325.71 last 68.98) - open/close 30618 (time 1709791322.07 total 335.72 last 61.71) - open/close 31198 (time 1709791332.08 total 345.74 last 57.90) - open/close 31782 (time 1709791342.10 total 355.76 last 58.30) - open/close 32320 (time 1709791352.11 total 365.77 last 53.75) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 369.83 seconds: 88.04 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 504 (time 1709791367.21 total 10.02 last 50.30) - /unlink 1022 (time 1709791377.23 total 20.03 last 51.73) - /unlink 1605 (time 1709791387.23 total 30.04 last 58.29) - /unlink 2208 (time 1709791397.23 total 40.04 last 60.29) - /unlink 2832 (time 1709791407.24 total 50.04 last 62.36) - /unlink 3482 (time 1709791417.25 total 60.06 last 64.91) - /unlink 4184 (time 1709791427.27 total 70.08 last 70.05) - /unlink 4989 (time 1709791437.28 total 80.09 last 80.43) - /unlink 5759 (time 1709791447.29 total 90.09 last 76.96) - /unlink 6593 (time 1709791457.30 total 100.11 last 83.30) - /unlink 7401 (time 1709791467.30 total 110.11 last 80.76) - /unlink 8133 (time 1709791477.31 total 120.11 last 73.18) - /unlink 8936 (time 1709791487.31 total 130.12 last 80.25) - /unlink 9663 (time 1709791497.32 total 140.13 last 72.61) - /unlink 10000 (time 1709791501.24 total 144.05 last 86.06) - /unlink 11133 (time 1709791511.24 total 154.05 last 113.26) - /unlink 12165 (time 1709791521.24 total 164.05 last 103.20) - /unlink 13318 (time 1709791531.25 total 174.05 last 115.27) - /unlink 14661 (time 1709791541.25 total 184.06 last 134.22) - /unlink 16204 (time 1709791551.26 total 194.06 last 154.22) - /unlink 17841 (time 1709791561.26 total 204.07 last 163.64) - /unlink 19501 (time 1709791571.26 total 214.07 last 165.99) - /unlink 20000 (time 1709791574.45 total 217.25 last 156.62) - /unlink 22026 (time 1709791584.45 total 227.25 last 202.59) - /unlink 24293 (time 1709791594.45 total 237.26 last 226.66) - /unlink 27218 (time 1709791604.45 total 247.26 last 292.45) - /unlink 30000 (time 1709791612.21 total 255.02 last 358.49) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 258.57 seconds: 125.92 ops/second Waiting for MDT destroys to complete