-----============= acceptance-small: conf-sanity ============----- Tue Feb 27 12:09:39 EST 2024 excepting tests: 32newtarball 110 Stopping clients: oleg118-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg118-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg118-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg118-server oleg118-server: oleg118-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing load_modules_local oleg118-server: Loading modules from /home/green/git/lustre-release/lustre oleg118-server: detected 4 online CPUs by sysfs oleg118-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg118-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg118-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg118-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg118-server: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg118-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg118-server: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg118-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg118-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:11:00 (1709053860) Stopping clients: oleg118-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg118-client.virtnet /mnt/lustre2 (opts:-f) oleg118-server: oleg118-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing load_modules_local oleg118-server: Loading modules from /home/green/git/lustre-release/lustre oleg118-server: detected 4 online CPUs by sysfs oleg118-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg118-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg118-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg118-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock oleg118-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4578 (time 1709053898.00 total 10.00 last 457.76) - open/close 5945 (time 1709053908.00 total 20.01 last 136.63) - open/close 6694 (time 1709053918.02 total 30.02 last 74.78) - open/close 6753 (time 1709053929.92 total 41.92 last 4.96) - open/close 7914 (time 1709053939.92 total 51.92 last 116.04) - open/close 9197 (time 1709053949.92 total 61.93 last 128.27) - open/close 9991 (time 1709053959.93 total 71.93 last 79.36) - open/close 10830 (time 1709053969.93 total 81.94 last 83.86) - open/close 11661 (time 1709053979.94 total 91.95 last 83.03) - open/close 12461 (time 1709053989.98 total 101.99 last 79.66) - open/close 13317 (time 1709053999.99 total 111.99 last 85.57) - open/close 14124 (time 1709054009.99 total 122.00 last 80.65) - open/close 14743 (time 1709054020.00 total 132.00 last 61.86) - open/close 15320 (time 1709054030.00 total 142.01 last 57.69) - open/close 15926 (time 1709054040.01 total 152.01 last 60.58) - open/close 16492 (time 1709054050.01 total 162.01 last 56.57) - open/close 17125 (time 1709054060.01 total 172.01 last 63.30) - open/close 17858 (time 1709054070.01 total 182.01 last 73.30) - open/close 18479 (time 1709054080.02 total 192.02 last 62.06) - open/close 19236 (time 1709054090.02 total 202.02 last 75.70) - open/close 19935 (time 1709054100.04 total 212.04 last 69.78) - open/close 20000 (time 1709054102.09 total 214.09 last 31.67) - open/close 20376 (time 1709054112.11 total 224.11 last 37.51) - open/close 20688 (time 1709054122.14 total 234.15 last 31.10) - open/close 21240 (time 1709054132.15 total 244.15 last 55.18) - open/close 21815 (time 1709054142.15 total 254.16 last 57.46) - open/close 22342 (time 1709054152.17 total 264.18 last 52.59) - open/close 22698 (time 1709054162.17 total 274.18 last 35.60) - open/close 23045 (time 1709054172.21 total 284.21 last 34.58) - open/close 23435 (time 1709054182.22 total 294.23 last 38.94) - open/close 23863 (time 1709054192.24 total 304.24 last 42.73) - open/close 24337 (time 1709054202.24 total 314.25 last 47.39) - open/close 24740 (time 1709054212.29 total 324.29 last 40.10) - open/close 24864 (time 1709054226.19 total 338.19 last 8.92) - open/close 25152 (time 1709054236.21 total 348.21 last 28.75) - open/close 25407 (time 1709054246.23 total 358.23 last 25.45) - open/close 25731 (time 1709054256.26 total 368.26 last 32.30) - open/close 26051 (time 1709054266.28 total 378.28 last 31.93) - open/close 26308 (time 1709054276.28 total 388.28 last 25.70) - open/close 26378 (time 1709054289.85 total 401.85 last 5.16) - open/close 26614 (time 1709054299.85 total 411.85 last 23.59) - open/close 26861 (time 1709054309.89 total 421.89 last 24.61) - open/close 27101 (time 1709054319.89 total 431.89 last 24.00) - open/close 27356 (time 1709054329.91 total 441.91 last 25.45) - open/close 27614 (time 1709054339.92 total 451.92 last 25.78) - open/close 27851 (time 1709054360.04 total 472.04 last 11.78) - open/close 28095 (time 1709054370.06 total 482.07 last 24.35) - open/close 28395 (time 1709054380.07 total 492.07 last 29.98) - open/close 28679 (time 1709054390.08 total 502.08 last 28.38) - open/close 28956 (time 1709054400.11 total 512.12 last 27.61) - open/close 29116 (time 1709054416.36 total 528.36 last 9.85) - open/close 29346 (time 1709054426.36 total 538.37 last 23.00) - open/close 29562 (time 1709054436.38 total 548.39 last 21.56) - open/close 29799 (time 1709054446.39 total 558.40 last 23.67) - open/close 30000 (time 1709054455.20 total 567.20 last 22.82) - open/close 30259 (time 1709054465.20 total 577.20 last 25.90) - open/close 30505 (time 1709054475.23 total 587.24 last 24.53) - open/close 30760 (time 1709054485.24 total 597.25 last 25.47) - open/close 31031 (time 1709054495.26 total 607.27 last 27.04) - open/close 31295 (time 1709054505.27 total 617.27 last 26.39) - open/close 31527 (time 1709054515.33 total 627.33 last 23.06) - open/close 31744 (time 1709054525.33 total 637.33 last 21.69) - open/close 32001 (time 1709054535.35 total 647.35 last 25.66) - open/close 32269 (time 1709054545.38 total 657.39 last 26.70) - open/close 32499 (time 1709054555.39 total 667.39 last 22.99) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 669.86 seconds: 48.61 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 214 (time 1709054569.95 total 10.03 last 21.34) - /unlink 460 (time 1709054579.99 total 20.06 last 24.51) - /unlink 681 (time 1709054589.99 total 30.07 last 22.09) - /unlink 904 (time 1709054599.99 total 40.07 last 22.29) - /unlink 1134 (time 1709054610.00 total 50.07 last 23.00) - /unlink 1373 (time 1709054620.04 total 60.12 last 23.79) - /unlink 1654 (time 1709054630.04 total 70.12 last 28.10) - /unlink 1882 (time 1709054640.08 total 80.16 last 22.71) - /unlink 2110 (time 1709054650.14 total 90.22 last 22.67) - /unlink 2404 (time 1709054660.20 total 100.27 last 29.24) - /unlink 2660 (time 1709054670.22 total 110.30 last 25.55) - /unlink 2918 (time 1709054680.26 total 120.34 last 25.68) - /unlink 3193 (time 1709054690.27 total 130.34 last 27.49) - /unlink 3453 (time 1709054700.28 total 140.35 last 25.98) - /unlink 3497 (time 1709054712.74 total 152.82 last 3.53) - /unlink 3771 (time 1709054722.76 total 162.84 last 27.34) - /unlink 4076 (time 1709054732.76 total 172.84 last 30.49) - /unlink 4350 (time 1709054742.79 total 182.87 last 27.32) - /unlink 4653 (time 1709054752.80 total 192.88 last 30.27) - /unlink 4972 (time 1709054762.80 total 202.88 last 31.90) - /unlink 5252 (time 1709054772.81 total 212.89 last 27.97) - /unlink 5577 (time 1709054782.82 total 222.90 last 32.46) - /unlink 5939 (time 1709054792.82 total 232.90 last 36.19) - /unlink 6278 (time 1709054802.83 total 242.91 last 33.89) - /unlink 6577 (time 1709054812.84 total 252.92 last 29.86) - /unlink 6895 (time 1709054822.85 total 262.92 last 31.78) - /unlink 7222 (time 1709054832.87 total 272.95 last 32.63) - /unlink 7519 (time 1709054842.88 total 282.96 last 29.65) - /unlink 7811 (time 1709054852.88 total 292.96 last 29.20) - /unlink 8107 (time 1709054862.90 total 302.98 last 29.54) - /unlink 8454 (time 1709054872.92 total 312.99 last 34.66) - /unlink 8779 (time 1709054882.92 total 323.00 last 32.48) - /unlink 9166 (time 1709054892.93 total 333.01 last 38.67) - /unlink 9629 (time 1709054902.95 total 343.03 last 46.21) - /unlink 10000 (time 1709054910.08 total 350.16 last 52.05) - /unlink 10517 (time 1709054920.08 total 360.16 last 51.66) - /unlink 11060 (time 1709054930.09 total 370.17 last 54.26) - /unlink 11552 (time 1709054940.11 total 380.19 last 49.12) - /unlink 12310 (time 1709054950.12 total 390.20 last 75.73) - /unlink 13098 (time 1709054960.13 total 400.21 last 78.72) - /unlink 13788 (time 1709054970.13 total 410.21 last 68.98) - /unlink 14391 (time 1709054980.14 total 420.22 last 60.23) - /unlink 14932 (time 1709054990.14 total 430.22 last 54.09) - /unlink 15388 (time 1709055000.16 total 440.24 last 45.52) - /unlink 15872 (time 1709055010.16 total 450.24 last 48.39) - /unlink 16355 (time 1709055020.17 total 460.25 last 48.25) - /unlink 16927 (time 1709055030.18 total 470.26 last 57.15) - /unlink 17546 (time 1709055040.21 total 480.29 last 61.75) - /unlink 18058 (time 1709055050.21 total 490.29 last 51.17) - /unlink 18610 (time 1709055060.23 total 500.30 last 55.13) - /unlink 19221 (time 1709055070.24 total 510.32 last 61.04) - /unlink 19934 (time 1709055080.24 total 520.32 last 71.27) - /unlink 20620 (time 1709055090.25 total 530.33 last 68.53) - /unlink 21341 (time 1709055100.27 total 540.35 last 71.94) - /unlink 22268 (time 1709055110.28 total 550.35 last 92.68) - /unlink 22775 (time 1709055126.36 total 566.44 last 31.52) - /unlink 24105 (time 1709055136.37 total 576.44 last 132.96) - /unlink 25301 (time 1709055146.38 total 586.45 last 119.47) - /unlink 26578 (time 1709055156.39 total 596.47 last 127.54) - /unlink 27965 (time 1709055166.39 total 606.47 last 138.67) - /unlink 29471 (time 1709055176.39 total 616.47 last 150.58) - /unlink 30000 (time 1709055179.39 total 619.47 last 176.68) - /unlink 31729 (time 1709055189.39 total 629.47 last 172.82) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 633.58 seconds: 51.39 ops/second Waiting for MDT destroys to complete - open/close 1172 (time 1709055209.34 total 10.00 last 117.17) - open/close 2345 (time 1709055219.34 total 20.01 last 117.24) - open/close 3261 (time 1709055229.35 total 30.01 last 91.55) - open/close 4145 (time 1709055239.35 total 40.02 last 88.39) - open/close 5000 (time 1709055249.36 total 50.03 last 85.40) - open/close 5821 (time 1709055259.36 total 60.03 last 82.09) - open/close 6521 (time 1709055269.37 total 70.04 last 69.93) - open/close 7203 (time 1709055279.38 total 80.04 last 68.17) - open/close 7777 (time 1709055289.38 total 90.05 last 57.38) - open/close 8330 (time 1709055299.41 total 100.07 last 55.14) - open/close 8907 (time 1709055309.42 total 110.09 last 57.62) - open/close 9402 (time 1709055319.44 total 120.11 last 49.41) - open/close 9941 (time 1709055329.44 total 130.11 last 53.87) - open/close 10484 (time 1709055339.47 total 140.14 last 54.15) - open/close 10997 (time 1709055349.48 total 150.15 last 51.26) - open/close 11535 (time 1709055359.49 total 160.15 last 53.77) - open/close 12063 (time 1709055369.50 total 170.17 last 52.70) - open/close 12540 (time 1709055379.51 total 180.17 last 47.69) - open/close 13009 (time 1709055389.51 total 190.18 last 46.88) - open/close 13571 (time 1709055399.51 total 200.18 last 56.20) - open/close 14053 (time 1709055409.53 total 210.20 last 48.11) - open/close 14499 (time 1709055419.54 total 220.20 last 44.57) - open/close 14972 (time 1709055429.55 total 230.21 last 47.25) - open/close 15464 (time 1709055439.55 total 240.22 last 49.17) - open/close 15961 (time 1709055449.56 total 250.23 last 49.65) - open/close 16408 (time 1709055459.57 total 260.23 last 44.67) - open/close 16820 (time 1709055469.59 total 270.25 last 41.12) - open/close 17268 (time 1709055479.59 total 280.26 last 44.79) - open/close 17777 (time 1709055489.59 total 290.26 last 50.86) - open/close 18280 (time 1709055499.60 total 300.27 last 50.25) - open/close 18720 (time 1709055509.60 total 310.27 last 44.00) - open/close 19221 (time 1709055519.62 total 320.28 last 50.04) - open/close 19782 (time 1709055529.62 total 330.29 last 56.06) - open/close 20000 (time 1709055534.31 total 334.98 last 46.51) - open/close 20462 (time 1709055544.33 total 345.00 last 46.10) - open/close 20942 (time 1709055554.34 total 355.01 last 47.95) - open/close 21463 (time 1709055564.35 total 365.01 last 52.08) - open/close 21971 (time 1709055574.35 total 375.02 last 50.76) - open/close 22533 (time 1709055584.36 total 385.03 last 56.16) - open/close 22992 (time 1709055594.38 total 395.04 last 45.84) - open/close 23447 (time 1709055604.39 total 405.06 last 45.43) - open/close 23946 (time 1709055614.41 total 415.08 last 49.81) - open/close 24404 (time 1709055624.42 total 425.08 last 45.76) - open/close 24765 (time 1709055634.46 total 435.12 last 35.96) - open/close 25140 (time 1709055644.46 total 445.12 last 37.50) - open/close 25541 (time 1709055654.50 total 455.17 last 39.92) - open/close 25975 (time 1709055664.51 total 465.18 last 43.36) - open/close 26436 (time 1709055674.51 total 475.18 last 46.10) - open/close 26783 (time 1709055684.51 total 485.18 last 34.69) - open/close 27108 (time 1709055694.56 total 495.22 last 32.36) - open/close 27429 (time 1709055704.57 total 505.24 last 32.05) - open/close 27951 (time 1709055714.59 total 515.26 last 52.12) - open/close 28433 (time 1709055724.59 total 525.26 last 48.19) - open/close 28896 (time 1709055734.61 total 535.27 last 46.23) - open/close 29307 (time 1709055744.61 total 545.28 last 41.10) - open/close 29728 (time 1709055754.62 total 555.29 last 42.03) - open/close 30000 (time 1709055760.78 total 561.44 last 44.21) - open/close 30458 (time 1709055770.79 total 571.46 last 45.74) - open/close 30884 (time 1709055780.82 total 581.49 last 42.45) - open/close 31246 (time 1709055790.84 total 591.51 last 36.13) - open/close 31604 (time 1709055800.86 total 601.53 last 35.73) - open/close 31901 (time 1709055810.89 total 611.55 last 29.63) - open/close 32236 (time 1709055820.89 total 621.56 last 33.48) - open/close 32513 (time 1709055830.92 total 631.59 last 27.63) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 633.18 seconds: 51.42 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 251 (time 1709055844.16 total 10.00 last 25.10) - /unlink 571 (time 1709055854.17 total 20.01 last 31.96) - /unlink 876 (time 1709055864.19 total 30.03 last 30.45) - /unlink 1195 (time 1709055874.20 total 40.03 last 31.88) - /unlink 1466 (time 1709055884.22 total 50.06 last 27.04) - /unlink 1745 (time 1709055894.23 total 60.07 last 27.85) - /unlink 2024 (time 1709055904.23 total 70.07 last 27.90) - /unlink 2396 (time 1709055914.29 total 80.12 last 37.01) - /unlink 2760 (time 1709055924.30 total 90.14 last 36.33) - /unlink 3053 (time 1709055934.34 total 100.18 last 29.21) - /unlink 3369 (time 1709055944.35 total 110.19 last 31.55) - /unlink 3692 (time 1709055954.38 total 120.22 last 32.20) - /unlink 4070 (time 1709055964.40 total 130.24 last 37.72) - /unlink 4372 (time 1709055974.43 total 140.27 last 30.13) - /unlink 4749 (time 1709055984.44 total 150.28 last 37.67) - /unlink 5085 (time 1709055994.46 total 160.29 last 33.54) - /unlink 5458 (time 1709056004.48 total 170.32 last 37.21) - /unlink 5826 (time 1709056014.49 total 180.33 last 36.75) - /unlink 6228 (time 1709056024.50 total 190.34 last 40.16) - /unlink 6625 (time 1709056034.51 total 200.35 last 39.65) - /unlink 7128 (time 1709056044.52 total 210.36 last 50.29) - /unlink 7538 (time 1709056054.52 total 220.36 last 40.99) - /unlink 8029 (time 1709056064.54 total 230.38 last 49.02) - /unlink 8415 (time 1709056074.54 total 240.38 last 38.58) - /unlink 8842 (time 1709056084.55 total 250.38 last 42.68) - /unlink 9265 (time 1709056094.56 total 260.40 last 42.22) - /unlink 9742 (time 1709056104.58 total 270.42 last 47.62) - /unlink 10000 (time 1709056109.81 total 275.65 last 49.31) - /unlink 10463 (time 1709056119.82 total 285.66 last 46.25) - /unlink 10961 (time 1709056129.84 total 295.68 last 49.73) - /unlink 11503 (time 1709056139.84 total 305.68 last 54.19) - /unlink 12043 (time 1709056149.85 total 315.69 last 53.96) - /unlink 12476 (time 1709056159.87 total 325.71 last 43.22) - /unlink 13029 (time 1709056169.89 total 335.73 last 55.17) - /unlink 13467 (time 1709056179.89 total 345.73 last 43.79) - /unlink 13994 (time 1709056189.90 total 355.73 last 52.69) - /unlink 14565 (time 1709056199.90 total 365.74 last 57.09) - /unlink 15126 (time 1709056209.90 total 375.74 last 56.07) - /unlink 15672 (time 1709056219.91 total 385.75 last 54.57) - /unlink 16326 (time 1709056229.92 total 395.75 last 65.35) - /unlink 16969 (time 1709056239.92 total 405.76 last 64.24) - /unlink 17759 (time 1709056249.93 total 415.77 last 78.97) - /unlink 18541 (time 1709056259.94 total 425.78 last 78.11) - /unlink 19316 (time 1709056269.94 total 435.78 last 77.48) - /unlink 20000 (time 1709056278.64 total 444.48 last 78.63) - /unlink 20909 (time 1709056288.64 total 454.48 last 90.88) - /unlink 21887 (time 1709056298.65 total 464.48 last 97.78) - /unlink 23020 (time 1709056308.65 total 474.49 last 113.28) - /unlink 24199 (time 1709056318.65 total 484.49 last 117.82) - /unlink 25472 (time 1709056328.65 total 494.49 last 127.28) - /unlink 26974 (time 1709056338.66 total 504.50 last 150.13) - /unlink 28581 (time 1709056348.66 total 514.50 last 160.65) - /unlink 30000 (time 1709056356.68 total 522.52 last 177.04) - /unlink 32267 (time 1709056366.68 total 532.52 last 226.64) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 533.71 seconds: 61.01 ops/second Waiting for MDT destroys to complete - open/close 1669 (time 1709056388.86 total 10.00 last 166.83) - open/close 3087 (time 1709056398.86 total 20.01 last 141.78) - open/close 4129 (time 1709056408.87 total 30.01 last 104.15) - open/close 5285 (time 1709056418.87 total 40.01 last 115.55) - open/close 6360 (time 1709056428.88 total 50.02 last 107.45) - open/close 7329 (time 1709056438.88 total 60.02 last 96.85) - open/close 8258 (time 1709056448.89 total 70.03 last 92.85) - open/close 9122 (time 1709056458.90 total 80.05 last 86.25) - open/close 9927 (time 1709056468.92 total 90.06 last 80.40) - open/close 10688 (time 1709056478.92 total 100.06 last 76.09) - open/close 11400 (time 1709056488.92 total 110.06 last 71.18) - open/close 12266 (time 1709056498.93 total 120.07 last 86.52) - open/close 12960 (time 1709056508.93 total 130.08 last 69.38) - open/close 13652 (time 1709056518.95 total 140.09 last 69.11) - open/close 14335 (time 1709056528.96 total 150.11 last 68.19) - open/close 14976 (time 1709056538.97 total 160.11 last 64.04) - open/close 15564 (time 1709056548.98 total 170.12 last 58.76) - open/close 16210 (time 1709056558.99 total 180.13 last 64.51) - open/close 16769 (time 1709056569.00 total 190.14 last 55.84) - open/close 17345 (time 1709056579.01 total 200.16 last 57.54) - open/close 17862 (time 1709056589.03 total 210.17 last 51.63) - open/close 18375 (time 1709056599.04 total 220.18 last 51.26) - open/close 18901 (time 1709056609.06 total 230.20 last 52.46) - open/close 19353 (time 1709056619.08 total 240.22 last 45.13) - open/close 19797 (time 1709056629.08 total 250.22 last 44.40) - open/close 20000 (time 1709056633.46 total 254.60 last 46.32) - open/close 20423 (time 1709056643.47 total 264.61 last 42.25) - open/close 20912 (time 1709056653.47 total 274.61 last 48.90) - open/close 21418 (time 1709056663.48 total 284.62 last 50.55) - open/close 21776 (time 1709056673.51 total 294.65 last 35.69) - open/close 22156 (time 1709056683.52 total 304.67 last 37.95) - open/close 22549 (time 1709056693.54 total 314.68 last 39.24) - open/close 22957 (time 1709056703.55 total 324.69 last 40.77) - open/close 23292 (time 1709056713.56 total 334.70 last 33.45) - open/close 23679 (time 1709056723.58 total 344.72 last 38.63) - open/close 24100 (time 1709056733.58 total 354.73 last 42.08) - open/close 24456 (time 1709056743.62 total 364.76 last 35.49) - open/close 24901 (time 1709056753.62 total 374.76 last 44.47) - open/close 25260 (time 1709056763.63 total 384.77 last 35.88) - open/close 25675 (time 1709056773.63 total 394.77 last 41.48) - open/close 26045 (time 1709056783.65 total 404.80 last 36.92) - open/close 26360 (time 1709056793.66 total 414.81 last 31.47) - open/close 26692 (time 1709056803.67 total 424.82 last 33.17) - open/close 27056 (time 1709056813.70 total 434.84 last 36.32) - open/close 27398 (time 1709056823.70 total 444.84 last 34.19) - open/close 27742 (time 1709056833.71 total 454.85 last 34.38) - open/close 28076 (time 1709056843.73 total 464.88 last 33.30) - open/close 28433 (time 1709056853.75 total 474.89 last 35.65) - open/close 28773 (time 1709056863.76 total 484.90 last 33.96) - open/close 29080 (time 1709056873.77 total 494.91 last 30.68) - open/close 29473 (time 1709056883.78 total 504.93 last 39.24) - open/close 29880 (time 1709056893.79 total 514.93 last 40.67) - open/close 30000 (time 1709056897.21 total 518.36 last 35.05) - open/close 30365 (time 1709056907.22 total 528.36 last 36.47) - open/close 30788 (time 1709056917.24 total 538.38 last 42.22) - open/close 31206 (time 1709056927.26 total 548.40 last 41.74) - open/close 31600 (time 1709056937.28 total 558.42 last 39.32) - open/close 31936 (time 1709056947.28 total 568.42 last 33.58) - open/close 32349 (time 1709056957.30 total 578.44 last 41.23) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 585.81 seconds: 55.58 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 336 (time 1709056976.56 total 10.02 last 33.55) - /unlink 646 (time 1709056986.57 total 20.03 last 30.97) - /unlink 995 (time 1709056996.57 total 30.03 last 34.90) - /unlink 1353 (time 1709057006.58 total 40.04 last 35.77) - /unlink 1706 (time 1709057016.60 total 50.05 last 35.23) - /unlink 2114 (time 1709057026.60 total 60.06 last 40.78) - /unlink 2469 (time 1709057036.65 total 70.10 last 35.35) - /unlink 2845 (time 1709057046.66 total 80.12 last 37.54) - /unlink 3166 (time 1709057056.69 total 90.15 last 32.00) - /unlink 3472 (time 1709057066.71 total 100.17 last 30.55) - /unlink 3768 (time 1709057076.71 total 110.17 last 29.59) - /unlink 4194 (time 1709057086.72 total 120.18 last 42.57) - /unlink 4627 (time 1709057096.73 total 130.19 last 43.26) - /unlink 5008 (time 1709057106.76 total 140.21 last 37.99) - /unlink 5324 (time 1709057116.76 total 150.22 last 31.59) - /unlink 5741 (time 1709057126.77 total 160.23 last 41.66) - /unlink 6096 (time 1709057136.78 total 170.24 last 35.46) - /unlink 6495 (time 1709057146.80 total 180.26 last 39.82) - /unlink 6934 (time 1709057156.81 total 190.26 last 43.88) - /unlink 7396 (time 1709057166.81 total 200.27 last 46.18) - /unlink 7794 (time 1709057176.81 total 210.27 last 39.79) - /unlink 8279 (time 1709057186.81 total 220.27 last 48.48) - /unlink 8896 (time 1709057196.82 total 230.28 last 61.65) - /unlink 9476 (time 1709057206.84 total 240.29 last 57.92) - /unlink 10000 (time 1709057215.25 total 248.70 last 62.30) - /unlink 10596 (time 1709057225.25 total 258.70 last 59.59) - /unlink 11060 (time 1709057235.27 total 268.72 last 46.31) - /unlink 11536 (time 1709057245.29 total 278.75 last 47.49) - /unlink 12045 (time 1709057255.31 total 288.76 last 50.81) - /unlink 12614 (time 1709057265.32 total 298.78 last 56.81) - /unlink 13261 (time 1709057275.34 total 308.79 last 64.63) - /unlink 13958 (time 1709057285.34 total 318.80 last 69.68) - /unlink 14828 (time 1709057295.34 total 328.80 last 86.97) - /unlink 15918 (time 1709057305.35 total 338.81 last 108.88) - /unlink 16817 (time 1709057315.36 total 348.82 last 89.81) - /unlink 17817 (time 1709057325.37 total 358.82 last 99.95) - /unlink 18921 (time 1709057335.37 total 368.83 last 110.35) - /unlink 20000 (time 1709057344.84 total 378.30 last 113.92) - /unlink 21348 (time 1709057354.85 total 388.31 last 134.70) - /unlink 22795 (time 1709057364.86 total 398.31 last 144.60) - /unlink 24074 (time 1709057374.86 total 408.32 last 127.89) - /unlink 25676 (time 1709057384.86 total 418.32 last 160.19) - /unlink 27364 (time 1709057394.86 total 428.32 last 168.74) - /unlink 29567 (time 1709057404.86 total 438.32 last 220.30) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 448.03 seconds: 72.67 ops/second Waiting for MDT destroys to complete - open/close 1584 (time 1709057435.40 total 10.00 last 158.36) - open/close 3108 (time 1709057445.41 total 20.01 last 152.35) - open/close 4410 (time 1709057455.41 total 30.01 last 130.12) - open/close 5650 (time 1709057465.42 total 40.02 last 123.95) - open/close 6781 (time 1709057475.42 total 50.02 last 113.08) - open/close 7687 (time 1709057485.43 total 60.03 last 90.48) - open/close 8584 (time 1709057495.45 total 70.05 last 89.56) - open/close 9451 (time 1709057505.45 total 80.05 last 86.65) - open/close 10000 (time 1709057511.91 total 86.50 last 85.08) - open/close 10790 (time 1709057521.92 total 96.52 last 78.89) - open/close 11550 (time 1709057531.92 total 106.52 last 75.99) - open/close 12350 (time 1709057541.92 total 116.52 last 79.97) - open/close 13119 (time 1709057551.93 total 126.53 last 76.84) - open/close 13754 (time 1709057561.94 total 136.54 last 63.45) - open/close 14613 (time 1709057571.95 total 146.55 last 85.83) - open/close 15378 (time 1709057581.96 total 156.55 last 76.43) - open/close 15990 (time 1709057591.97 total 166.56 last 61.14) - open/close 16538 (time 1709057601.97 total 176.57 last 54.76) - open/close 17031 (time 1709057611.98 total 186.58 last 49.28) - open/close 17597 (time 1709057621.99 total 196.59 last 56.52) - open/close 18089 (time 1709057631.99 total 206.59 last 49.19) - open/close 18545 (time 1709057642.01 total 216.61 last 45.54) - open/close 19039 (time 1709057652.01 total 226.61 last 49.40) - open/close 19516 (time 1709057662.01 total 236.61 last 47.67) - open/close 19974 (time 1709057672.02 total 246.62 last 45.78) - open/close 20529 (time 1709057682.03 total 256.63 last 55.42) - open/close 21044 (time 1709057692.04 total 266.64 last 51.44) - open/close 21541 (time 1709057702.05 total 276.65 last 49.65) - open/close 22021 (time 1709057712.07 total 286.67 last 47.93) - open/close 22473 (time 1709057722.09 total 296.69 last 45.09) - open/close 22851 (time 1709057732.10 total 306.70 last 37.78) - open/close 23230 (time 1709057742.12 total 316.71 last 37.84) - open/close 23602 (time 1709057752.13 total 326.73 last 37.15) - open/close 24022 (time 1709057762.14 total 336.74 last 41.95) - open/close 24420 (time 1709057772.16 total 346.76 last 39.72) - open/close 24840 (time 1709057782.18 total 356.78 last 41.92) - open/close 25390 (time 1709057792.19 total 366.79 last 54.93) - open/close 25958 (time 1709057802.20 total 376.80 last 56.76) - open/close 26540 (time 1709057812.20 total 386.80 last 58.18) - open/close 27090 (time 1709057822.23 total 396.83 last 54.86) - open/close 27552 (time 1709057832.23 total 406.83 last 46.19) - open/close 27950 (time 1709057842.23 total 416.83 last 39.79) - open/close 28424 (time 1709057852.24 total 426.84 last 47.37) - open/close 28822 (time 1709057862.26 total 436.86 last 39.72) - open/close 29280 (time 1709057872.27 total 446.87 last 45.74) - open/close 29809 (time 1709057882.28 total 456.87 last 52.89) - open/close 30000 (time 1709057885.83 total 460.43 last 53.69) - open/close 30569 (time 1709057895.84 total 470.44 last 56.86) - open/close 31108 (time 1709057905.84 total 480.44 last 53.90) - open/close 31630 (time 1709057915.86 total 490.46 last 52.09) - open/close 32027 (time 1709057925.87 total 500.47 last 39.65) - open/close 32464 (time 1709057935.88 total 510.48 last 43.66) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 512.54 seconds: 63.53 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 392 (time 1709057949.40 total 10.01 last 39.16) - /unlink 798 (time 1709057959.42 total 20.03 last 40.52) - /unlink 1179 (time 1709057969.42 total 30.03 last 38.09) - /unlink 1577 (time 1709057979.42 total 40.03 last 39.79) - /unlink 1892 (time 1709057989.45 total 50.06 last 31.41) - /unlink 2210 (time 1709057999.47 total 60.08 last 31.73) - /unlink 2586 (time 1709058009.50 total 70.11 last 37.52) - /unlink 2961 (time 1709058019.52 total 80.13 last 37.42) - /unlink 3337 (time 1709058029.53 total 90.14 last 37.55) - /unlink 3719 (time 1709058039.54 total 100.15 last 38.16) - /unlink 4070 (time 1709058049.55 total 110.16 last 35.06) - /unlink 4502 (time 1709058059.56 total 120.17 last 43.17) - /unlink 4865 (time 1709058069.57 total 130.18 last 36.25) - /unlink 5246 (time 1709058079.58 total 140.19 last 38.07) - /unlink 5621 (time 1709058089.60 total 150.21 last 37.44) - /unlink 5972 (time 1709058099.64 total 160.25 last 34.95) - /unlink 6352 (time 1709058109.68 total 170.29 last 37.87) - /unlink 6804 (time 1709058119.68 total 180.29 last 45.16) - /unlink 7155 (time 1709058129.71 total 190.32 last 35.01) - /unlink 7527 (time 1709058139.72 total 200.33 last 37.15) - /unlink 7954 (time 1709058149.73 total 210.34 last 42.70) - /unlink 8442 (time 1709058159.73 total 220.34 last 48.78) - /unlink 9060 (time 1709058169.74 total 230.35 last 61.76) - /unlink 9746 (time 1709058179.75 total 240.36 last 68.53) - /unlink 10000 (time 1709058185.89 total 246.50 last 41.34) - /unlink 10499 (time 1709058195.89 total 256.50 last 49.89) - /unlink 11038 (time 1709058205.90 total 266.51 last 53.85) - /unlink 11638 (time 1709058215.91 total 276.52 last 59.96) - /unlink 12188 (time 1709058225.92 total 286.53 last 54.97) - /unlink 12790 (time 1709058235.93 total 296.54 last 60.13) - /unlink 13428 (time 1709058245.93 total 306.54 last 63.79) - /unlink 13966 (time 1709058255.94 total 316.55 last 53.74) - /unlink 14398 (time 1709058265.96 total 326.57 last 43.13) - /unlink 14912 (time 1709058275.98 total 336.59 last 51.27) - /unlink 15463 (time 1709058285.99 total 346.60 last 55.07) - /unlink 16118 (time 1709058295.99 total 356.60 last 65.45) - /unlink 16806 (time 1709058306.00 total 366.61 last 68.73) - /unlink 17613 (time 1709058316.01 total 376.62 last 80.63) - /unlink 18388 (time 1709058326.02 total 386.63 last 77.48) - /unlink 19154 (time 1709058336.03 total 396.64 last 76.49) - /unlink 19942 (time 1709058346.04 total 406.65 last 78.69) - /unlink 20802 (time 1709058356.05 total 416.66 last 85.93) - /unlink 21674 (time 1709058366.06 total 426.67 last 87.18) - /unlink 22472 (time 1709058376.07 total 436.68 last 79.71) - /unlink 23408 (time 1709058386.07 total 446.68 last 93.58) - /unlink 23496 (time 1709058398.15 total 458.76 last 7.29) - /unlink 24581 (time 1709058408.15 total 468.76 last 108.45) - /unlink 25713 (time 1709058418.15 total 478.76 last 113.15) - /unlink 26916 (time 1709058428.16 total 488.77 last 120.23) - /unlink 28144 (time 1709058438.17 total 498.78 last 122.74) - /unlink 29595 (time 1709058448.17 total 508.78 last 145.03) - /unlink 30000 (time 1709058450.54 total 511.15 last 171.11) - /unlink 32040 (time 1709058460.54 total 521.15 last 203.96) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 523.30 seconds: 62.22 ops/second Waiting for MDT destroys to complete