== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:51:24 (1776196284) Stopping clients: oleg242-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg242-client.virtnet /mnt/lustre2 (opts:-f) oleg242-server: oleg242-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg242-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg242-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg242-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg242-client.virtnet: -o user_xattr,flock oleg242-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 892 (time 1776196429.05 total 10.00 last 89.18) - open/close 1923 (time 1776196439.06 total 20.01 last 103.05) - open/close 2987 (time 1776196449.06 total 30.01 last 106.37) - open/close 4004 (time 1776196459.07 total 40.01 last 101.66) - open/close 5059 (time 1776196469.07 total 50.02 last 105.49) - open/close 6096 (time 1776196479.07 total 60.02 last 103.70) - open/close 7107 (time 1776196489.08 total 70.02 last 101.01) - open/close 8055 (time 1776196499.09 total 80.04 last 94.69) - open/close 8902 (time 1776196509.10 total 90.04 last 84.64) - open/close 9717 (time 1776196519.10 total 100.05 last 81.48) - open/close 10000 (time 1776196523.31 total 104.26 last 67.20) - open/close 10726 (time 1776196533.31 total 114.26 last 72.59) - open/close 11475 (time 1776196543.32 total 124.27 last 74.85) - open/close 12511 (time 1776196553.34 total 134.29 last 103.34) - open/close 13301 (time 1776196563.35 total 144.29 last 78.97) - open/close 14256 (time 1776196573.35 total 154.30 last 95.43) - open/close 15248 (time 1776196583.36 total 164.30 last 99.18) - open/close 16191 (time 1776196593.37 total 174.31 last 94.21) - open/close 17133 (time 1776196603.37 total 184.32 last 94.18) - open/close 17953 (time 1776196613.38 total 194.33 last 81.91) - open/close 18870 (time 1776196623.38 total 204.33 last 91.67) - open/close 19439 (time 1776196633.40 total 214.35 last 56.80) - open/close 20000 (time 1776196640.24 total 221.19 last 81.96) - open/close 20989 (time 1776196650.25 total 231.19 last 98.88) - open/close 21910 (time 1776196660.25 total 241.20 last 92.07) - open/close 22859 (time 1776196670.25 total 251.20 last 94.86) - open/close 23760 (time 1776196680.26 total 261.21 last 90.06) - open/close 24588 (time 1776196690.26 total 271.21 last 82.75) - open/close 25527 (time 1776196700.27 total 281.22 last 93.81) - open/close 26224 (time 1776196710.28 total 291.23 last 69.65) - open/close 27095 (time 1776196720.28 total 301.23 last 87.09) - open/close 27852 (time 1776196730.30 total 311.24 last 75.58) - open/close 28694 (time 1776196740.30 total 321.25 last 84.12) - open/close 29325 (time 1776196750.31 total 331.26 last 63.04) - open/close 29878 (time 1776196760.34 total 341.29 last 55.14) - open/close 30765 (time 1776196770.35 total 351.29 last 88.68) - open/close 31460 (time 1776196780.36 total 361.30 last 69.44) - open/close 32371 (time 1776196790.36 total 371.30 last 91.08) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 374.19 seconds: 87.11 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1334 (time 1776196808.07 total 10.00 last 133.36) - /unlink 2672 (time 1776196818.08 total 20.01 last 133.75) - /unlink 3929 (time 1776196828.08 total 30.01 last 125.65) - /unlink 5383 (time 1776196838.09 total 40.02 last 145.26) - /unlink 6833 (time 1776196848.10 total 50.03 last 144.92) - /unlink 8396 (time 1776196858.10 total 60.03 last 156.27) - /unlink 9668 (time 1776196868.10 total 70.03 last 127.19) - /unlink 10000 (time 1776196870.61 total 72.54 last 132.09) - /unlink 11471 (time 1776196880.62 total 82.55 last 147.03) - /unlink 12469 (time 1776196890.62 total 92.55 last 99.75) - /unlink 13736 (time 1776196900.62 total 102.55 last 126.68) - /unlink 15107 (time 1776196910.64 total 112.57 last 136.88) - /unlink 16460 (time 1776196920.64 total 122.57 last 135.24) - /unlink 17918 (time 1776196930.65 total 132.58 last 145.72) - /unlink 19179 (time 1776196940.65 total 142.58 last 126.09) - /unlink 20000 (time 1776196946.62 total 148.55 last 137.44) - /unlink 21324 (time 1776196956.63 total 158.56 last 132.36) - /unlink 22834 (time 1776196966.63 total 168.56 last 150.93) - /unlink 24238 (time 1776196976.64 total 178.56 last 140.35) - /unlink 25488 (time 1776196986.64 total 188.57 last 124.97) - /unlink 26904 (time 1776196996.64 total 198.57 last 141.54) - /unlink 27961 (time 1776197006.64 total 208.57 last 105.69) - /unlink 29237 (time 1776197016.65 total 218.57 last 127.58) - /unlink 30000 (time 1776197023.23 total 225.15 last 115.96) - /unlink 31116 (time 1776197033.23 total 235.16 last 111.57) - /unlink 32495 (time 1776197043.23 total 245.16 last 137.83) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 245.77 seconds: 132.62 ops/second Waiting for MDT destroys to complete - open/close 899 (time 1776197070.54 total 10.01 last 89.85) - open/close 1887 (time 1776197080.55 total 20.01 last 98.70) - open/close 2796 (time 1776197090.56 total 30.02 last 90.85) - open/close 3648 (time 1776197100.56 total 40.03 last 85.15) - open/close 4563 (time 1776197110.56 total 50.03 last 91.48) - open/close 5407 (time 1776197120.57 total 60.03 last 84.36) - open/close 6314 (time 1776197130.58 total 70.04 last 90.60) - open/close 7138 (time 1776197140.59 total 80.05 last 82.34) - open/close 7917 (time 1776197150.59 total 90.06 last 77.83) - open/close 8728 (time 1776197160.60 total 100.06 last 81.07) - open/close 9408 (time 1776197170.63 total 110.10 last 67.75) - open/close 10000 (time 1776197178.33 total 117.79 last 76.95) - open/close 10691 (time 1776197188.33 total 127.80 last 69.06) - open/close 11571 (time 1776197198.34 total 137.81 last 87.92) - open/close 12420 (time 1776197208.36 total 147.82 last 84.79) - open/close 13308 (time 1776197218.36 total 157.83 last 88.75) - open/close 14249 (time 1776197228.37 total 167.83 last 94.05) - open/close 15079 (time 1776197238.38 total 177.85 last 82.87) - open/close 15734 (time 1776197248.39 total 187.86 last 65.43) - open/close 16499 (time 1776197258.40 total 197.86 last 76.45) - open/close 17405 (time 1776197268.40 total 207.87 last 90.58) - open/close 18214 (time 1776197278.41 total 217.87 last 80.85) - open/close 19057 (time 1776197288.41 total 227.88 last 84.26) - open/close 19899 (time 1776197298.41 total 237.88 last 84.20) - open/close 20699 (time 1776197308.42 total 247.88 last 79.97) - open/close 21501 (time 1776197318.42 total 257.89 last 80.15) - open/close 22352 (time 1776197328.43 total 267.90 last 85.02) - open/close 23177 (time 1776197338.44 total 277.90 last 82.45) - open/close 23914 (time 1776197348.44 total 287.91 last 73.65) - open/close 24686 (time 1776197358.45 total 297.92 last 77.15) - open/close 25614 (time 1776197368.46 total 307.93 last 92.71) - open/close 26499 (time 1776197378.47 total 317.93 last 88.44) - open/close 27448 (time 1776197388.47 total 327.93 last 94.89) - open/close 28404 (time 1776197398.47 total 337.94 last 95.56) - open/close 29352 (time 1776197408.48 total 347.95 last 94.72) - open/close 30000 (time 1776197416.91 total 356.38 last 76.84) - open/close 30744 (time 1776197426.92 total 366.38 last 74.37) - open/close 31504 (time 1776197436.93 total 376.39 last 75.94) - open/close 32211 (time 1776197446.94 total 386.41 last 70.59) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 391.87 seconds: 83.17 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1063 (time 1776197466.83 total 10.01 last 106.17) - /unlink 1899 (time 1776197476.84 total 20.02 last 83.56) - /unlink 3204 (time 1776197486.85 total 30.03 last 130.33) - /unlink 4433 (time 1776197496.85 total 40.03 last 122.86) - /unlink 6052 (time 1776197506.86 total 50.04 last 161.85) - /unlink 7754 (time 1776197516.86 total 60.04 last 170.15) - /unlink 9160 (time 1776197526.87 total 70.05 last 140.45) - /unlink 10000 (time 1776197536.70 total 79.88 last 85.47) - /unlink 11341 (time 1776197546.70 total 89.88 last 134.05) - /unlink 12732 (time 1776197556.71 total 99.89 last 139.00) - /unlink 14252 (time 1776197566.71 total 109.89 last 151.95) - /unlink 15737 (time 1776197576.71 total 119.89 last 148.47) - /unlink 17034 (time 1776197586.72 total 129.90 last 129.61) - /unlink 18099 (time 1776197596.72 total 139.90 last 106.46) - /unlink 19158 (time 1776197606.72 total 149.91 last 105.88) - /unlink 20000 (time 1776197612.19 total 155.37 last 154.19) - /unlink 21567 (time 1776197622.19 total 165.37 last 156.59) - /unlink 22962 (time 1776197632.20 total 175.38 last 139.43) - /unlink 24368 (time 1776197642.20 total 185.38 last 140.51) - /unlink 25699 (time 1776197652.21 total 195.39 last 133.02) - /unlink 27220 (time 1776197662.21 total 205.39 last 152.06) - /unlink 28667 (time 1776197672.22 total 215.40 last 144.66) - /unlink 30000 (time 1776197681.08 total 224.26 last 150.34) - /unlink 31414 (time 1776197691.09 total 234.27 last 141.34) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 242.42 seconds: 134.45 ops/second Waiting for MDT destroys to complete - open/close 842 (time 1776197724.84 total 10.01 last 84.12) - open/close 1803 (time 1776197734.85 total 20.01 last 96.06) - open/close 2761 (time 1776197744.85 total 30.02 last 95.77) - open/close 3584 (time 1776197754.85 total 40.02 last 82.28) - open/close 4601 (time 1776197764.86 total 50.03 last 101.60) - open/close 5622 (time 1776197774.87 total 60.04 last 101.99) - open/close 6565 (time 1776197784.88 total 70.05 last 94.25) - open/close 7464 (time 1776197794.89 total 80.06 last 89.79) - open/close 8285 (time 1776197804.89 total 90.06 last 82.09) - open/close 9192 (time 1776197814.90 total 100.06 last 90.64) - open/close 10000 (time 1776197824.25 total 109.41 last 86.44) - open/close 10729 (time 1776197834.25 total 119.41 last 72.89) - open/close 11430 (time 1776197844.25 total 129.42 last 70.07) - open/close 12180 (time 1776197854.26 total 139.43 last 74.93) - open/close 12964 (time 1776197864.27 total 149.43 last 78.35) - open/close 13779 (time 1776197874.27 total 159.44 last 81.44) - open/close 14575 (time 1776197884.28 total 169.45 last 79.52) - open/close 15348 (time 1776197894.29 total 179.45 last 77.26) - open/close 16270 (time 1776197904.30 total 189.46 last 92.15) - open/close 17048 (time 1776197914.30 total 199.47 last 77.73) - open/close 18022 (time 1776197924.31 total 209.47 last 97.35) - open/close 19028 (time 1776197934.32 total 219.49 last 100.49) - open/close 19696 (time 1776197944.34 total 229.51 last 66.66) - open/close 20000 (time 1776197948.84 total 234.01 last 67.50) - open/close 20578 (time 1776197958.85 total 244.01 last 57.78) - open/close 21491 (time 1776197968.85 total 254.02 last 91.24) - open/close 22448 (time 1776197978.87 total 264.04 last 95.52) - open/close 23088 (time 1776197988.89 total 274.05 last 63.91) - open/close 23698 (time 1776197998.90 total 284.06 last 60.93) - open/close 24280 (time 1776198008.91 total 294.07 last 58.14) - open/close 25158 (time 1776198018.91 total 304.08 last 87.79) - open/close 26252 (time 1776198028.92 total 314.08 last 109.33) - open/close 27227 (time 1776198038.92 total 324.08 last 97.47) - open/close 28122 (time 1776198048.92 total 334.09 last 89.49) - open/close 29167 (time 1776198058.92 total 344.09 last 104.46) - open/close 29998 (time 1776198068.93 total 354.09 last 83.09) - open/close 30855 (time 1776198078.93 total 364.09 last 85.68) - open/close 31312 (time 1776198088.94 total 374.11 last 45.63) - open/close 31956 (time 1776198098.95 total 384.12 last 64.35) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 390.78 seconds: 83.41 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1393 (time 1776198119.69 total 10.00 last 139.28) - /unlink 2769 (time 1776198129.69 total 20.01 last 137.54) - /unlink 4256 (time 1776198139.69 total 30.01 last 148.69) - /unlink 5939 (time 1776198149.69 total 40.01 last 168.29) - /unlink 7552 (time 1776198159.69 total 50.01 last 161.29) - /unlink 9142 (time 1776198169.70 total 60.01 last 158.91) - /unlink 10000 (time 1776198176.30 total 66.62 last 129.88) - /unlink 11798 (time 1776198186.30 total 76.62 last 179.80) - /unlink 13091 (time 1776198196.31 total 86.63 last 129.19) - /unlink 14324 (time 1776198206.32 total 96.64 last 123.19) - /unlink 15646 (time 1776198216.32 total 106.64 last 132.16) - /unlink 16937 (time 1776198226.32 total 116.64 last 129.09) - /unlink 18356 (time 1776198236.33 total 126.65 last 141.83) - /unlink 19963 (time 1776198246.33 total 136.65 last 160.69) - /unlink 21296 (time 1776198256.33 total 146.65 last 133.26) - /unlink 22651 (time 1776198266.34 total 156.66 last 135.42) - /unlink 23968 (time 1776198276.34 total 166.66 last 131.64) - /unlink 25446 (time 1776198286.35 total 176.66 last 147.74) - /unlink 26994 (time 1776198296.36 total 186.68 last 154.56) - /unlink 28703 (time 1776198306.36 total 196.68 last 170.89) - /unlink 30000 (time 1776198313.87 total 204.18 last 172.84) - /unlink 31345 (time 1776198323.87 total 214.19 last 134.41) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 224.16 seconds: 145.41 ops/second Waiting for MDT destroys to complete - open/close 929 (time 1776198359.62 total 10.01 last 92.79) - open/close 2016 (time 1776198369.62 total 20.02 last 108.65) - open/close 2990 (time 1776198379.63 total 30.03 last 97.27) - open/close 3871 (time 1776198389.64 total 40.04 last 88.04) - open/close 4886 (time 1776198399.65 total 50.04 last 101.44) - open/close 5940 (time 1776198409.65 total 60.05 last 105.33) - open/close 6829 (time 1776198419.66 total 70.05 last 88.87) - open/close 7780 (time 1776198429.66 total 80.06 last 95.07) - open/close 8738 (time 1776198439.67 total 90.07 last 95.66) - open/close 9797 (time 1776198449.67 total 100.07 last 105.89) - open/close 10000 (time 1776198452.58 total 102.98 last 69.92) - open/close 10808 (time 1776198462.58 total 112.98 last 80.75) - open/close 11688 (time 1776198472.59 total 122.99 last 87.92) - open/close 12620 (time 1776198482.59 total 132.99 last 93.19) - open/close 13366 (time 1776198492.61 total 143.01 last 74.49) - open/close 14146 (time 1776198502.63 total 153.02 last 77.86) - open/close 14982 (time 1776198512.63 total 163.03 last 83.57) - open/close 15755 (time 1776198522.64 total 173.03 last 77.25) - open/close 16615 (time 1776198532.64 total 183.04 last 85.94) - open/close 17491 (time 1776198542.66 total 193.05 last 87.48) - open/close 18367 (time 1776198552.66 total 203.05 last 87.58) - open/close 19284 (time 1776198562.66 total 213.06 last 91.65) - open/close 20000 (time 1776198572.42 total 222.82 last 73.35) - open/close 20928 (time 1776198582.42 total 232.82 last 92.80) - open/close 21885 (time 1776198592.43 total 242.83 last 95.61) - open/close 22768 (time 1776198602.44 total 252.84 last 88.20) - open/close 23733 (time 1776198612.46 total 262.85 last 96.38) - open/close 24584 (time 1776198622.47 total 272.86 last 85.01) - open/close 25532 (time 1776198632.47 total 282.87 last 94.74) - open/close 26494 (time 1776198642.48 total 292.88 last 96.12) - open/close 27479 (time 1776198652.49 total 302.89 last 98.42) - open/close 28415 (time 1776198662.50 total 312.90 last 93.52) - open/close 29306 (time 1776198672.50 total 322.90 last 89.08) - open/close 30000 (time 1776198680.21 total 330.61 last 90.00) - open/close 30960 (time 1776198690.22 total 340.62 last 95.92) - open/close 31913 (time 1776198700.22 total 350.62 last 95.30) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 359.18 seconds: 90.75 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1454 (time 1776198722.61 total 10.00 last 145.40) - /unlink 2915 (time 1776198732.62 total 20.00 last 146.05) - /unlink 4687 (time 1776198742.62 total 30.00 last 177.18) - /unlink 6308 (time 1776198752.62 total 40.01 last 162.09) - /unlink 7986 (time 1776198762.62 total 50.01 last 167.78) - /unlink 9764 (time 1776198772.63 total 60.01 last 177.71) - /unlink 11660 (time 1776198782.63 total 70.01 last 189.58) - /unlink 13190 (time 1776198792.63 total 80.01 last 152.98) - /unlink 14822 (time 1776198802.63 total 90.01 last 163.19) - /unlink 16433 (time 1776198812.63 total 100.02 last 161.03) - /unlink 18045 (time 1776198822.64 total 110.02 last 161.16) - /unlink 19513 (time 1776198832.64 total 120.03 last 146.70) - /unlink 20000 (time 1776198835.47 total 122.86 last 171.98) - /unlink 21309 (time 1776198845.48 total 132.87 last 130.80) - /unlink 22475 (time 1776198855.48 total 142.87 last 116.58) - /unlink 24003 (time 1776198865.49 total 152.87 last 152.75) - /unlink 25558 (time 1776198875.50 total 162.88 last 155.35) - /unlink 26994 (time 1776198885.51 total 172.89 last 143.48) - /unlink 28617 (time 1776198895.51 total 182.90 last 162.22) - /unlink 30000 (time 1776198903.33 total 190.72 last 176.81) - /unlink 31612 (time 1776198913.34 total 200.72 last 161.11) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 206.06 seconds: 158.18 ops/second Waiting for MDT destroys to complete - open/close 1003 (time 1776198942.39 total 10.00 last 100.27) - open/close 1966 (time 1776198952.41 total 20.02 last 96.15) - open/close 2894 (time 1776198962.41 total 30.03 last 92.72) - open/close 3935 (time 1776198972.42 total 40.03 last 104.06) - open/close 4699 (time 1776198982.43 total 50.05 last 76.28) - open/close 5645 (time 1776198992.44 total 60.05 last 94.54) - open/close 6671 (time 1776199002.44 total 70.06 last 102.56) - open/close 7654 (time 1776199012.44 total 80.06 last 98.30) - open/close 8642 (time 1776199022.45 total 90.06 last 98.76) - open/close 9584 (time 1776199032.45 total 100.07 last 94.14) - open/close 10000 (time 1776199036.91 total 104.52 last 93.37) - open/close 10949 (time 1776199046.91 total 114.52 last 94.90) - open/close 11923 (time 1776199056.92 total 124.53 last 97.34) - open/close 12823 (time 1776199066.93 total 134.54 last 89.91) - open/close 13712 (time 1776199076.93 total 144.54 last 88.87) - open/close 14545 (time 1776199086.93 total 154.54 last 83.28) - open/close 15436 (time 1776199096.93 total 164.55 last 89.08) - open/close 16253 (time 1776199106.94 total 174.55 last 81.64) - open/close 16917 (time 1776199116.97 total 184.58 last 66.24) - open/close 17473 (time 1776199126.97 total 194.59 last 55.55) - open/close 18252 (time 1776199136.98 total 204.59 last 77.88) - open/close 19042 (time 1776199146.98 total 214.59 last 78.95) - open/close 19822 (time 1776199156.99 total 224.60 last 77.92) - open/close 20000 (time 1776199158.99 total 226.60 last 88.98) - open/close 20867 (time 1776199169.00 total 236.61 last 86.66) - open/close 21691 (time 1776199179.00 total 246.61 last 82.39) - open/close 22541 (time 1776199189.00 total 256.61 last 84.99) - open/close 23429 (time 1776199199.00 total 266.61 last 88.77) - open/close 24360 (time 1776199209.01 total 276.63 last 93.00) - open/close 25269 (time 1776199219.02 total 286.63 last 90.82) - open/close 26296 (time 1776199229.03 total 296.64 last 102.62) - open/close 27250 (time 1776199239.04 total 306.66 last 95.28) - open/close 28024 (time 1776199249.05 total 316.66 last 77.36) - open/close 28919 (time 1776199259.05 total 326.66 last 89.49) - open/close 29584 (time 1776199269.05 total 336.66 last 66.48) - open/close 30000 (time 1776199274.71 total 342.33 last 73.47) - open/close 30923 (time 1776199284.72 total 352.33 last 92.28) - open/close 31899 (time 1776199294.73 total 362.34 last 97.47) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 369.82 seconds: 88.14 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1667 (time 1776199316.42 total 10.00 last 166.66) - /unlink 3326 (time 1776199326.43 total 20.01 last 165.81) - /unlink 4862 (time 1776199336.43 total 30.01 last 153.52) - /unlink 6445 (time 1776199346.44 total 40.02 last 158.23) - /unlink 8012 (time 1776199356.44 total 50.02 last 156.70) - /unlink 9562 (time 1776199366.44 total 60.02 last 154.98) - /unlink 10000 (time 1776199369.36 total 62.94 last 149.82) - /unlink 11516 (time 1776199379.37 total 72.95 last 151.55) - /unlink 12976 (time 1776199389.37 total 82.95 last 145.98) - /unlink 14471 (time 1776199399.37 total 92.95 last 149.48) - /unlink 16086 (time 1776199409.37 total 102.95 last 161.44) - /unlink 17533 (time 1776199419.38 total 112.96 last 144.55) - /unlink 18950 (time 1776199429.39 total 122.97 last 141.59) - /unlink 20000 (time 1776199435.46 total 129.04 last 172.93) - /unlink 21635 (time 1776199445.46 total 139.04 last 163.45) - /unlink 23034 (time 1776199455.47 total 149.05 last 139.85) - /unlink 24539 (time 1776199465.47 total 159.05 last 150.40) - /unlink 25992 (time 1776199475.48 total 169.05 last 145.29) - /unlink 27288 (time 1776199485.48 total 179.06 last 129.53) - /unlink 28848 (time 1776199495.48 total 189.06 last 155.98) - /unlink 30000 (time 1776199502.56 total 196.14 last 162.66) - /unlink 31818 (time 1776199512.57 total 206.15 last 181.75) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 210.32 seconds: 154.97 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg242-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg242-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg242-server: LNET ready to unload oleg242-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2