== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:14:44 (1761311684) Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:-f) oleg257-server: oleg257-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg257-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg257-server: detected 4 online CPUs by sysfs oleg257-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg257-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg257-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg257-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock oleg257-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 966 (time 1761311792.11 total 10.01 last 96.54) - open/close 1922 (time 1761311802.12 total 20.02 last 95.50) - open/close 2948 (time 1761311812.13 total 30.02 last 102.56) - open/close 3834 (time 1761311822.14 total 40.03 last 88.51) - open/close 4633 (time 1761311832.14 total 50.03 last 79.87) - open/close 5418 (time 1761311842.14 total 60.04 last 78.47) - open/close 6215 (time 1761311852.16 total 70.06 last 79.55) - open/close 7068 (time 1761311862.17 total 80.06 last 85.25) - open/close 7912 (time 1761311872.17 total 90.06 last 84.38) - open/close 8752 (time 1761311882.18 total 100.08 last 83.88) - open/close 9563 (time 1761311892.19 total 110.08 last 81.07) - open/close 10000 (time 1761311897.41 total 115.30 last 83.72) - open/close 10960 (time 1761311907.42 total 125.31 last 95.93) - open/close 11887 (time 1761311917.42 total 135.32 last 92.63) - open/close 12752 (time 1761311927.43 total 145.32 last 86.45) - open/close 13636 (time 1761311937.43 total 155.33 last 88.38) - open/close 14420 (time 1761311947.44 total 165.33 last 78.36) - open/close 15363 (time 1761311957.44 total 175.33 last 94.28) - open/close 16164 (time 1761311967.45 total 185.34 last 80.01) - open/close 16971 (time 1761311977.47 total 195.37 last 80.51) - open/close 17829 (time 1761311987.48 total 205.38 last 85.72) - open/close 18746 (time 1761311997.49 total 215.39 last 91.61) - open/close 19709 (time 1761312007.50 total 225.40 last 96.21) - open/close 20000 (time 1761312010.55 total 228.45 last 95.44) - open/close 20950 (time 1761312020.55 total 238.45 last 94.97) - open/close 21941 (time 1761312030.56 total 248.46 last 99.01) - open/close 22834 (time 1761312040.57 total 258.47 last 89.23) - open/close 23830 (time 1761312050.58 total 268.47 last 99.56) - open/close 24735 (time 1761312060.59 total 278.49 last 90.33) - open/close 25638 (time 1761312070.60 total 288.49 last 90.26) - open/close 26580 (time 1761312080.60 total 298.50 last 94.16) - open/close 27479 (time 1761312090.60 total 308.50 last 89.88) - open/close 28372 (time 1761312100.61 total 318.51 last 89.23) - open/close 29316 (time 1761312110.62 total 328.51 last 94.31) - open/close 30000 (time 1761312117.79 total 335.68 last 95.40) - open/close 31012 (time 1761312127.80 total 345.69 last 101.15) - open/close 31948 (time 1761312137.80 total 355.70 last 93.55) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 362.35 seconds: 89.95 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1669 (time 1761312157.38 total 10.00 last 166.89) - /unlink 3282 (time 1761312167.39 total 20.01 last 161.17) - /unlink 5053 (time 1761312177.39 total 30.01 last 176.99) - /unlink 6979 (time 1761312187.40 total 40.02 last 192.52) - /unlink 8594 (time 1761312197.40 total 50.02 last 161.47) - /unlink 10000 (time 1761312205.59 total 58.21 last 171.60) - /unlink 11678 (time 1761312215.60 total 68.22 last 167.74) - /unlink 13230 (time 1761312225.60 total 78.22 last 155.14) - /unlink 14911 (time 1761312235.60 total 88.22 last 168.06) - /unlink 16523 (time 1761312245.61 total 98.23 last 161.07) - /unlink 17994 (time 1761312255.61 total 108.23 last 147.08) - /unlink 19557 (time 1761312265.62 total 118.24 last 156.20) - /unlink 20000 (time 1761312268.47 total 121.09 last 155.45) - /unlink 21632 (time 1761312278.47 total 131.09 last 163.15) - /unlink 23555 (time 1761312288.48 total 141.10 last 192.18) - /unlink 25085 (time 1761312298.48 total 151.10 last 152.99) - /unlink 26744 (time 1761312308.48 total 161.11 last 165.79) - /unlink 28145 (time 1761312318.49 total 171.11 last 140.04) - /unlink 29505 (time 1761312328.50 total 181.12 last 135.86) - /unlink 30000 (time 1761312331.80 total 184.42 last 149.83) - /unlink 31770 (time 1761312341.80 total 194.42 last 177.00) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 198.85 seconds: 163.91 ops/second Waiting for MDT destroys to complete - open/close 992 (time 1761312372.40 total 10.01 last 99.13) - open/close 1864 (time 1761312382.41 total 20.02 last 87.12) - open/close 2711 (time 1761312392.42 total 30.03 last 84.56) - open/close 3544 (time 1761312402.43 total 40.04 last 83.25) - open/close 4405 (time 1761312412.44 total 50.05 last 86.05) - open/close 5230 (time 1761312422.44 total 60.05 last 82.47) - open/close 6289 (time 1761312432.44 total 70.05 last 105.87) - open/close 7244 (time 1761312442.45 total 80.06 last 95.41) - open/close 8118 (time 1761312452.46 total 90.07 last 87.35) - open/close 9027 (time 1761312462.46 total 100.07 last 90.88) - open/close 9962 (time 1761312472.47 total 110.08 last 93.42) - open/close 10844 (time 1761312482.47 total 120.08 last 88.14) - open/close 11738 (time 1761312492.48 total 130.09 last 89.38) - open/close 12633 (time 1761312502.48 total 140.09 last 89.49) - open/close 13594 (time 1761312512.48 total 150.09 last 96.08) - open/close 14523 (time 1761312522.49 total 160.10 last 92.77) - open/close 15461 (time 1761312532.50 total 170.11 last 93.72) - open/close 16418 (time 1761312542.51 total 180.12 last 95.62) - open/close 17295 (time 1761312552.51 total 190.12 last 87.70) - open/close 18202 (time 1761312562.52 total 200.13 last 90.65) - open/close 19096 (time 1761312572.52 total 210.13 last 89.38) - open/close 19815 (time 1761312582.53 total 220.14 last 71.85) - open/close 20000 (time 1761312584.64 total 222.25 last 87.72) - open/close 20872 (time 1761312594.64 total 232.25 last 87.18) - open/close 21693 (time 1761312604.64 total 242.25 last 82.09) - open/close 22548 (time 1761312614.65 total 252.26 last 85.41) - open/close 23453 (time 1761312624.65 total 262.26 last 90.46) - open/close 24457 (time 1761312634.66 total 272.27 last 100.37) - open/close 25315 (time 1761312644.66 total 282.27 last 85.77) - open/close 26175 (time 1761312654.67 total 292.28 last 85.93) - open/close 27058 (time 1761312664.67 total 302.28 last 88.29) - open/close 27976 (time 1761312674.67 total 312.28 last 91.77) - open/close 28897 (time 1761312684.68 total 322.29 last 92.06) - open/close 29625 (time 1761312694.68 total 332.29 last 72.79) - open/close 30000 (time 1761312699.47 total 337.08 last 78.29) - open/close 30855 (time 1761312709.47 total 347.08 last 85.46) - open/close 31808 (time 1761312719.48 total 357.09 last 95.24) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 365.67 seconds: 89.14 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1542 (time 1761312741.03 total 10.00 last 154.14) - /unlink 3165 (time 1761312751.03 total 20.01 last 162.24) - /unlink 4690 (time 1761312761.04 total 30.01 last 152.49) - /unlink 6270 (time 1761312771.05 total 40.02 last 157.81) - /unlink 8207 (time 1761312781.05 total 50.02 last 193.68) - /unlink 10000 (time 1761312790.96 total 59.94 last 180.83) - /unlink 11576 (time 1761312800.96 total 69.94 last 157.60) - /unlink 13165 (time 1761312810.97 total 79.94 last 158.88) - /unlink 15074 (time 1761312820.97 total 89.95 last 190.73) - /unlink 16711 (time 1761312830.98 total 99.95 last 163.68) - /unlink 18240 (time 1761312840.98 total 109.95 last 152.87) - /unlink 19712 (time 1761312850.98 total 119.95 last 147.15) - /unlink 20000 (time 1761312853.02 total 121.99 last 141.44) - /unlink 21494 (time 1761312863.02 total 131.99 last 149.40) - /unlink 22976 (time 1761312873.02 total 141.99 last 148.17) - /unlink 24466 (time 1761312883.02 total 152.00 last 148.96) - /unlink 25987 (time 1761312893.03 total 162.00 last 152.05) - /unlink 27744 (time 1761312903.03 total 172.00 last 175.67) - /unlink 29628 (time 1761312913.03 total 182.00 last 188.37) - /unlink 31727 (time 1761312923.03 total 192.00 last 209.84) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 196.71 seconds: 165.70 ops/second Waiting for MDT destroys to complete - open/close 992 (time 1761312949.78 total 10.00 last 99.18) - open/close 1978 (time 1761312959.78 total 20.00 last 98.58) - open/close 2936 (time 1761312969.79 total 30.02 last 95.69) - open/close 3803 (time 1761312979.81 total 40.03 last 86.57) - open/close 4681 (time 1761312989.82 total 50.04 last 87.73) - open/close 5577 (time 1761312999.82 total 60.05 last 89.52) - open/close 6543 (time 1761313009.83 total 70.06 last 96.51) - open/close 7494 (time 1761313019.84 total 80.07 last 95.00) - open/close 8457 (time 1761313029.85 total 90.07 last 96.26) - open/close 9525 (time 1761313039.85 total 100.07 last 106.75) - open/close 10000 (time 1761313044.95 total 105.18 last 93.10) - open/close 10813 (time 1761313054.96 total 115.18 last 81.27) - open/close 11651 (time 1761313064.96 total 125.18 last 83.79) - open/close 12636 (time 1761313074.96 total 135.19 last 98.46) - open/close 13637 (time 1761313084.97 total 145.19 last 100.05) - open/close 14596 (time 1761313094.97 total 155.19 last 95.90) - open/close 15670 (time 1761313104.99 total 165.21 last 107.22) - open/close 16565 (time 1761313114.99 total 175.21 last 89.48) - open/close 17391 (time 1761313125.00 total 185.22 last 82.51) - open/close 18257 (time 1761313135.00 total 195.23 last 86.55) - open/close 19100 (time 1761313145.01 total 205.23 last 84.25) - open/close 19941 (time 1761313155.02 total 215.24 last 84.05) - open/close 20846 (time 1761313165.02 total 225.24 last 90.49) - open/close 21680 (time 1761313175.02 total 235.24 last 83.39) - open/close 22406 (time 1761313185.02 total 245.24 last 72.57) - open/close 23198 (time 1761313195.03 total 255.25 last 79.16) - open/close 24029 (time 1761313205.03 total 265.26 last 83.04) - open/close 24882 (time 1761313215.04 total 275.26 last 85.27) - open/close 25802 (time 1761313225.05 total 285.27 last 91.91) - open/close 26732 (time 1761313235.05 total 295.27 last 92.97) - open/close 27600 (time 1761313245.05 total 305.28 last 86.77) - open/close 28437 (time 1761313255.06 total 315.29 last 83.62) - open/close 29198 (time 1761313265.07 total 325.29 last 76.09) - open/close 30000 (time 1761313274.60 total 334.82 last 84.12) - open/close 30766 (time 1761313284.60 total 344.83 last 76.56) - open/close 31574 (time 1761313294.61 total 354.84 last 80.73) - open/close 32410 (time 1761313304.62 total 364.84 last 83.58) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 366.92 seconds: 88.83 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1594 (time 1761313320.02 total 10.00 last 159.37) - /unlink 3107 (time 1761313330.02 total 20.01 last 151.25) - /unlink 4623 (time 1761313340.03 total 30.01 last 151.47) - /unlink 6287 (time 1761313350.03 total 40.01 last 166.39) - /unlink 7707 (time 1761313360.04 total 50.02 last 141.93) - /unlink 9189 (time 1761313370.05 total 60.03 last 148.05) - /unlink 10000 (time 1761313376.12 total 66.10 last 133.66) - /unlink 11527 (time 1761313386.12 total 76.10 last 152.66) - /unlink 13065 (time 1761313396.12 total 86.10 last 153.75) - /unlink 14598 (time 1761313406.12 total 96.11 last 153.27) - /unlink 16146 (time 1761313416.13 total 106.11 last 154.78) - /unlink 17597 (time 1761313426.14 total 116.12 last 144.97) - /unlink 19151 (time 1761313436.14 total 126.13 last 155.26) - /unlink 20000 (time 1761313441.53 total 131.51 last 157.54) - /unlink 21461 (time 1761313451.53 total 141.52 last 146.09) - /unlink 22937 (time 1761313461.54 total 151.52 last 147.58) - /unlink 24480 (time 1761313471.54 total 161.52 last 154.29) - /unlink 26134 (time 1761313481.54 total 171.52 last 165.31) - /unlink 27709 (time 1761313491.54 total 181.53 last 157.45) - /unlink 29196 (time 1761313501.55 total 191.53 last 148.63) - /unlink 30000 (time 1761313506.54 total 196.52 last 161.08) - /unlink 31518 (time 1761313516.54 total 206.52 last 151.80) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 214.62 seconds: 151.87 ops/second Waiting for MDT destroys to complete - open/close 883 (time 1761313550.43 total 10.00 last 88.30) - open/close 1788 (time 1761313560.43 total 20.00 last 90.50) - open/close 2691 (time 1761313570.43 total 30.00 last 90.28) - open/close 3615 (time 1761313580.44 total 40.01 last 92.32) - open/close 4479 (time 1761313590.44 total 50.01 last 86.38) - open/close 5307 (time 1761313600.45 total 60.02 last 82.73) - open/close 6172 (time 1761313610.46 total 70.03 last 86.44) - open/close 7038 (time 1761313620.46 total 80.03 last 86.58) - open/close 7951 (time 1761313630.47 total 90.04 last 91.24) - open/close 8786 (time 1761313640.47 total 100.04 last 83.49) - open/close 9432 (time 1761313650.49 total 110.06 last 64.48) - open/close 10000 (time 1761313658.17 total 117.74 last 73.89) - open/close 10779 (time 1761313668.18 total 127.75 last 77.89) - open/close 11582 (time 1761313678.18 total 137.75 last 80.29) - open/close 12504 (time 1761313688.18 total 147.75 last 92.19) - open/close 13403 (time 1761313698.18 total 157.75 last 89.89) - open/close 14424 (time 1761313708.19 total 167.76 last 102.04) - open/close 15466 (time 1761313718.19 total 177.76 last 104.12) - open/close 16421 (time 1761313728.20 total 187.78 last 95.40) - open/close 17323 (time 1761313738.22 total 197.79 last 90.11) - open/close 18200 (time 1761313748.22 total 207.79 last 87.70) - open/close 18872 (time 1761313758.22 total 217.79 last 67.14) - open/close 19695 (time 1761313768.23 total 227.80 last 82.24) - open/close 20000 (time 1761313771.80 total 231.37 last 85.38) - open/close 20729 (time 1761313781.81 total 241.38 last 72.83) - open/close 21406 (time 1761313791.82 total 251.39 last 67.63) - open/close 22085 (time 1761313801.83 total 261.40 last 67.86) - open/close 22935 (time 1761313811.83 total 271.40 last 84.97) - open/close 23764 (time 1761313821.84 total 281.41 last 82.84) - open/close 24597 (time 1761313831.85 total 291.42 last 83.17) - open/close 25424 (time 1761313841.86 total 301.43 last 82.61) - open/close 26265 (time 1761313851.88 total 311.45 last 83.99) - open/close 27132 (time 1761313861.88 total 321.45 last 86.64) - open/close 27754 (time 1761313871.90 total 331.47 last 62.09) - open/close 28396 (time 1761313881.91 total 341.48 last 64.15) - open/close 29199 (time 1761313891.92 total 351.49 last 80.23) - open/close 30000 (time 1761313901.48 total 361.05 last 83.74) - open/close 30849 (time 1761313911.49 total 371.06 last 84.86) - open/close 31693 (time 1761313921.49 total 381.06 last 84.36) - open/close 32465 (time 1761313931.49 total 391.06 last 77.20) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 392.67 seconds: 83.01 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1430 (time 1761313946.58 total 10.01 last 142.86) - /unlink 2790 (time 1761313956.58 total 20.01 last 135.93) - /unlink 4238 (time 1761313966.58 total 30.02 last 144.78) - /unlink 5651 (time 1761313976.58 total 40.02 last 141.30) - /unlink 7099 (time 1761313986.58 total 50.02 last 144.79) - /unlink 8471 (time 1761313996.59 total 60.02 last 137.14) - /unlink 9889 (time 1761314006.60 total 70.03 last 141.71) - /unlink 11290 (time 1761314016.60 total 80.03 last 140.08) - /unlink 12700 (time 1761314026.60 total 90.03 last 140.99) - /unlink 14173 (time 1761314036.60 total 100.03 last 147.29) - /unlink 15712 (time 1761314046.60 total 110.03 last 153.86) - /unlink 17140 (time 1761314056.61 total 120.04 last 142.70) - /unlink 18619 (time 1761314066.61 total 130.05 last 147.82) - /unlink 20000 (time 1761314076.41 total 139.84 last 141.02) - /unlink 21368 (time 1761314086.42 total 149.85 last 136.66) - /unlink 22662 (time 1761314096.42 total 159.85 last 129.35) - /unlink 24034 (time 1761314106.42 total 169.86 last 137.15) - /unlink 25461 (time 1761314116.43 total 179.86 last 142.68) - /unlink 26896 (time 1761314126.43 total 189.86 last 143.50) - /unlink 28286 (time 1761314136.43 total 199.86 last 138.98) - /unlink 29720 (time 1761314146.43 total 209.86 last 143.34) - /unlink 30000 (time 1761314148.79 total 212.22 last 118.84) - /unlink 31306 (time 1761314158.79 total 222.22 last 130.59) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 231.54 seconds: 140.77 ops/second Waiting for MDT destroys to complete - open/close 786 (time 1761314194.69 total 10.00 last 78.58) - open/close 1624 (time 1761314204.69 total 20.01 last 83.75) - open/close 2502 (time 1761314214.70 total 30.01 last 87.74) - open/close 3344 (time 1761314224.71 total 40.02 last 84.14) - open/close 4139 (time 1761314234.71 total 50.02 last 79.49) - open/close 4875 (time 1761314244.71 total 60.03 last 73.55) - open/close 5662 (time 1761314254.72 total 70.04 last 78.61) - open/close 6311 (time 1761314264.73 total 80.05 last 64.83) - open/close 6966 (time 1761314274.74 total 90.05 last 65.49) - open/close 7724 (time 1761314284.75 total 100.06 last 75.72) - open/close 8501 (time 1761314294.75 total 110.06 last 77.69) - open/close 9330 (time 1761314304.76 total 120.08 last 82.77) - open/close 10000 (time 1761314312.91 total 128.23 last 82.22) - open/close 10806 (time 1761314322.91 total 138.23 last 80.60) - open/close 11658 (time 1761314332.92 total 148.24 last 85.13) - open/close 12527 (time 1761314342.92 total 158.24 last 86.90) - open/close 13375 (time 1761314352.92 total 168.24 last 84.78) - open/close 14205 (time 1761314362.93 total 178.25 last 82.92) - open/close 15037 (time 1761314372.94 total 188.25 last 83.16) - open/close 15864 (time 1761314382.94 total 198.26 last 82.66) - open/close 16719 (time 1761314392.95 total 208.26 last 85.47) - open/close 17569 (time 1761314402.96 total 218.27 last 84.90) - open/close 18335 (time 1761314412.96 total 228.27 last 76.59) - open/close 19114 (time 1761314422.96 total 238.28 last 77.88) - open/close 19948 (time 1761314432.97 total 248.28 last 83.35) - open/close 20852 (time 1761314442.98 total 258.29 last 90.32) - open/close 21676 (time 1761314452.98 total 268.30 last 82.36) - open/close 22435 (time 1761314462.99 total 278.30 last 75.86) - open/close 23271 (time 1761314472.99 total 288.31 last 83.54) - open/close 24074 (time 1761314483.00 total 298.32 last 80.23) - open/close 24897 (time 1761314493.00 total 308.32 last 82.28) - open/close 25716 (time 1761314503.02 total 318.34 last 81.75) - open/close 26536 (time 1761314513.02 total 328.34 last 81.99) - open/close 27293 (time 1761314523.02 total 338.34 last 75.70) - open/close 28024 (time 1761314533.03 total 348.34 last 73.09) - open/close 28683 (time 1761314543.03 total 358.35 last 65.84) - open/close 29510 (time 1761314553.04 total 368.35 last 82.68) - open/close 30000 (time 1761314559.26 total 374.58 last 78.72) - open/close 30666 (time 1761314569.27 total 384.58 last 66.58) - open/close 31339 (time 1761314579.27 total 394.59 last 67.24) - open/close 32113 (time 1761314589.28 total 404.59 last 77.36) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 411.00 seconds: 79.30 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1635 (time 1761314609.18 total 10.00 last 163.43) - /unlink 3210 (time 1761314619.19 total 20.01 last 157.42) - /unlink 4704 (time 1761314629.19 total 30.01 last 149.37) - /unlink 6149 (time 1761314639.19 total 40.01 last 144.45) - /unlink 7552 (time 1761314649.20 total 50.02 last 140.28) - /unlink 8948 (time 1761314659.20 total 60.02 last 139.60) - /unlink 10000 (time 1761314666.67 total 67.49 last 140.71) - /unlink 11384 (time 1761314676.67 total 77.49 last 138.38) - /unlink 12818 (time 1761314686.68 total 87.50 last 143.36) - /unlink 14192 (time 1761314696.68 total 97.50 last 137.35) - /unlink 15559 (time 1761314706.68 total 107.50 last 136.65) - /unlink 16971 (time 1761314716.69 total 117.51 last 141.12) - /unlink 18337 (time 1761314726.69 total 127.51 last 136.58) - /unlink 19772 (time 1761314736.70 total 137.52 last 143.45) - /unlink 21333 (time 1761314746.70 total 147.52 last 156.07) - /unlink 22925 (time 1761314756.70 total 157.52 last 159.19) - /unlink 24401 (time 1761314766.70 total 167.52 last 147.60) - /unlink 25877 (time 1761314776.70 total 177.52 last 147.51) - /unlink 27347 (time 1761314786.71 total 187.53 last 146.92) - /unlink 28763 (time 1761314796.71 total 197.53 last 141.59) - /unlink 30000 (time 1761314805.38 total 206.20 last 142.65) - /unlink 31372 (time 1761314815.38 total 216.21 last 137.15) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 224.55 seconds: 145.15 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg257-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local oleg257-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg257-server: LNET ready to unload oleg257-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2