== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 08:57:17 (1761310637) Stopping clients: oleg313-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg313-client.virtnet /mnt/lustre2 (opts:-f) oleg313-server: oleg313-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing load_modules_local oleg313-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg313-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg313-server: Loading modules from /home/green/git/lustre-release/lustre oleg313-server: detected 4 online CPUs by sysfs oleg313-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg313-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg313-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg313-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg313-client.virtnet: -o user_xattr,flock oleg313-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 2175 (time 1761310699.72 total 10.00 last 217.47) - open/close 4480 (time 1761310709.72 total 20.00 last 230.47) - open/close 6638 (time 1761310719.73 total 30.01 last 215.72) - open/close 8257 (time 1761310729.73 total 40.01 last 161.88) - open/close 10000 (time 1761310739.43 total 49.71 last 179.66) - open/close 11937 (time 1761310749.43 total 59.71 last 193.63) - open/close 14266 (time 1761310759.44 total 69.71 last 232.86) - open/close 16734 (time 1761310769.44 total 79.72 last 246.71) - open/close 18774 (time 1761310779.44 total 89.72 last 203.90) - open/close 20000 (time 1761310785.46 total 95.73 last 203.94) - open/close 21935 (time 1761310795.46 total 105.74 last 193.33) - open/close 22792 (time 1761310805.47 total 115.75 last 85.68) - open/close 23510 (time 1761310815.47 total 125.75 last 71.79) - open/close 24252 (time 1761310825.47 total 135.75 last 74.17) - open/close 25029 (time 1761310835.47 total 145.75 last 77.70) - open/close 25800 (time 1761310845.47 total 155.75 last 77.10) - open/close 26614 (time 1761310855.48 total 165.76 last 81.33) - open/close 27404 (time 1761310865.49 total 175.77 last 78.91) - open/close 28231 (time 1761310875.50 total 185.78 last 82.65) - open/close 29148 (time 1761310885.50 total 195.78 last 91.68) - open/close 30000 (time 1761310894.67 total 204.94 last 92.98) - open/close 30897 (time 1761310904.68 total 214.96 last 89.53) - open/close 31840 (time 1761310914.69 total 224.97 last 94.20) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 233.62 seconds: 139.52 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1663 (time 1761310936.10 total 10.00 last 166.25) - /unlink 3274 (time 1761310946.10 total 20.01 last 161.05) - /unlink 5227 (time 1761310956.11 total 30.01 last 195.20) - /unlink 7055 (time 1761310966.11 total 40.01 last 182.76) - /unlink 8518 (time 1761310976.11 total 50.02 last 146.25) - /unlink 9861 (time 1761310986.12 total 60.02 last 134.28) - /unlink 11180 (time 1761310996.12 total 70.02 last 131.87) - /unlink 12476 (time 1761311006.12 total 80.02 last 129.57) - /unlink 13938 (time 1761311016.12 total 90.03 last 146.16) - /unlink 15470 (time 1761311026.13 total 100.03 last 153.15) - /unlink 17008 (time 1761311036.13 total 110.03 last 153.79) - /unlink 18554 (time 1761311046.13 total 120.03 last 154.55) - /unlink 20000 (time 1761311055.72 total 129.62 last 150.80) - /unlink 22042 (time 1761311065.72 total 139.63 last 204.15) - /unlink 23921 (time 1761311075.73 total 149.63 last 187.83) - /unlink 25756 (time 1761311085.73 total 159.63 last 183.47) - /unlink 27999 (time 1761311095.73 total 169.64 last 224.18) - /unlink 30000 (time 1761311104.71 total 178.62 last 222.84) - /unlink 32479 (time 1761311114.71 total 188.62 last 247.87) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 189.02 seconds: 172.44 ops/second Waiting for MDT destroys to complete - open/close 1732 (time 1761311137.58 total 10.00 last 173.17) - open/close 3255 (time 1761311147.58 total 20.00 last 152.26) - open/close 4709 (time 1761311157.59 total 30.01 last 145.34) - open/close 6059 (time 1761311167.59 total 40.01 last 134.98) - open/close 7409 (time 1761311177.59 total 50.01 last 134.99) - open/close 8915 (time 1761311187.59 total 60.02 last 150.53) - open/close 10000 (time 1761311195.54 total 67.96 last 136.55) - open/close 11468 (time 1761311205.54 total 77.96 last 146.76) - open/close 12984 (time 1761311215.54 total 87.96 last 151.59) - open/close 14251 (time 1761311225.55 total 97.97 last 126.57) - open/close 15347 (time 1761311235.56 total 107.98 last 109.54) - open/close 16356 (time 1761311245.56 total 117.98 last 100.90) - open/close 17372 (time 1761311255.56 total 127.98 last 101.58) - open/close 18541 (time 1761311265.57 total 137.99 last 116.82) - open/close 19849 (time 1761311275.57 total 148.00 last 130.73) - open/close 21045 (time 1761311285.58 total 158.00 last 119.52) - open/close 21748 (time 1761311295.59 total 168.01 last 70.25) - open/close 22333 (time 1761311305.61 total 178.03 last 58.38) - open/close 22859 (time 1761311315.61 total 188.03 last 52.60) - open/close 23382 (time 1761311325.63 total 198.05 last 52.17) - open/close 24094 (time 1761311335.64 total 208.07 last 71.11) - open/close 24974 (time 1761311345.65 total 218.07 last 87.93) - open/close 25937 (time 1761311355.66 total 228.08 last 96.23) - open/close 26820 (time 1761311365.66 total 238.08 last 88.30) - open/close 27662 (time 1761311375.67 total 248.09 last 84.12) - open/close 28516 (time 1761311385.68 total 258.10 last 85.31) - open/close 29327 (time 1761311395.69 total 268.11 last 81.01) - open/close 29831 (time 1761311405.70 total 278.12 last 50.35) - open/close 30000 (time 1761311410.15 total 282.57 last 37.96) - open/close 30433 (time 1761311420.18 total 292.60 last 43.19) - open/close 30831 (time 1761311430.18 total 302.60 last 39.79) - open/close 31365 (time 1761311440.18 total 312.61 last 53.38) - open/close 31900 (time 1761311450.20 total 322.63 last 53.40) - open/close 32558 (time 1761311460.21 total 332.63 last 65.76) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 333.23 seconds: 97.81 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1339 (time 1761311475.44 total 10.00 last 133.87) - /unlink 2590 (time 1761311485.45 total 20.01 last 125.01) - /unlink 3886 (time 1761311495.46 total 30.01 last 129.55) - /unlink 5179 (time 1761311505.46 total 40.02 last 129.24) - /unlink 6452 (time 1761311515.46 total 50.02 last 127.25) - /unlink 7810 (time 1761311525.47 total 60.02 last 135.76) - /unlink 9108 (time 1761311535.47 total 70.03 last 129.78) - /unlink 10000 (time 1761311544.15 total 78.70 last 102.80) - /unlink 11215 (time 1761311554.15 total 88.71 last 121.47) - /unlink 12507 (time 1761311564.16 total 98.72 last 129.06) - /unlink 13752 (time 1761311574.16 total 108.72 last 124.50) - /unlink 14918 (time 1761311584.16 total 118.72 last 116.54) - /unlink 16281 (time 1761311594.17 total 128.73 last 136.17) - /unlink 17474 (time 1761311604.18 total 138.74 last 119.24) - /unlink 18773 (time 1761311614.18 total 148.74 last 129.87) - /unlink 19963 (time 1761311624.19 total 158.75 last 118.92) - /unlink 21237 (time 1761311634.19 total 168.75 last 127.32) - /unlink 22564 (time 1761311644.20 total 178.76 last 132.63) - /unlink 23784 (time 1761311654.21 total 188.77 last 121.89) - /unlink 25044 (time 1761311664.22 total 198.77 last 125.90) - /unlink 26300 (time 1761311674.22 total 208.78 last 125.55) - /unlink 27537 (time 1761311684.22 total 218.78 last 123.67) - /unlink 28755 (time 1761311694.23 total 228.79 last 121.70) - /unlink 29937 (time 1761311704.24 total 238.80 last 118.11) - /unlink 31107 (time 1761311714.24 total 248.80 last 116.95) - /unlink 32458 (time 1761311724.25 total 258.81 last 135.03) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 259.78 seconds: 125.47 ops/second Waiting for MDT destroys to complete - open/close 658 (time 1761311749.74 total 10.01 last 65.72) - open/close 1318 (time 1761311759.75 total 20.02 last 65.97) - open/close 2104 (time 1761311769.75 total 30.03 last 78.53) - open/close 2821 (time 1761311779.76 total 40.03 last 71.63) - open/close 3538 (time 1761311789.77 total 50.04 last 71.66) - open/close 4278 (time 1761311799.77 total 60.04 last 73.99) - open/close 5071 (time 1761311809.78 total 70.05 last 79.20) - open/close 5815 (time 1761311819.79 total 80.06 last 74.37) - open/close 6612 (time 1761311829.79 total 90.06 last 79.68) - open/close 7366 (time 1761311839.79 total 100.06 last 75.38) - open/close 8137 (time 1761311849.80 total 110.07 last 77.02) - open/close 8877 (time 1761311859.80 total 120.07 last 74.00) - open/close 9518 (time 1761311869.81 total 130.09 last 64.03) - open/close 10000 (time 1761311876.00 total 136.27 last 77.98) - open/close 10804 (time 1761311886.00 total 146.27 last 80.33) - open/close 11611 (time 1761311896.00 total 156.27 last 80.70) - open/close 12362 (time 1761311906.01 total 166.28 last 75.04) - open/close 13144 (time 1761311916.01 total 176.28 last 78.18) - open/close 13880 (time 1761311926.02 total 186.29 last 73.56) - open/close 14614 (time 1761311936.02 total 196.29 last 73.38) - open/close 15387 (time 1761311946.03 total 206.30 last 77.26) - open/close 16182 (time 1761311956.03 total 216.30 last 79.49) - open/close 16981 (time 1761311966.04 total 226.31 last 79.80) - open/close 17804 (time 1761311976.04 total 236.31 last 82.29) - open/close 18606 (time 1761311986.05 total 246.32 last 80.12) - open/close 19382 (time 1761311996.06 total 256.33 last 77.55) - open/close 20000 (time 1761312004.44 total 264.71 last 73.75) - open/close 20764 (time 1761312014.45 total 274.72 last 76.34) - open/close 21545 (time 1761312024.45 total 284.72 last 78.09) - open/close 22272 (time 1761312034.46 total 294.73 last 72.63) - open/close 23049 (time 1761312044.46 total 304.73 last 77.68) - open/close 23849 (time 1761312054.46 total 314.73 last 79.98) - open/close 24579 (time 1761312064.46 total 324.73 last 72.98) - open/close 25336 (time 1761312074.47 total 334.74 last 75.67) - open/close 26090 (time 1761312084.47 total 344.74 last 75.37) - open/close 26893 (time 1761312094.48 total 354.75 last 80.23) - open/close 27697 (time 1761312104.48 total 364.75 last 80.40) - open/close 28483 (time 1761312114.49 total 374.76 last 78.50) - open/close 29269 (time 1761312124.50 total 384.77 last 78.54) - open/close 29994 (time 1761312134.51 total 394.78 last 72.46) - open/close 30806 (time 1761312144.52 total 404.79 last 81.12) - open/close 31575 (time 1761312154.53 total 414.80 last 76.82) - open/close 32412 (time 1761312164.53 total 424.80 last 83.66) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 427.15 seconds: 76.31 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1305 (time 1761312180.74 total 10.00 last 130.44) - /unlink 2634 (time 1761312190.74 total 20.01 last 132.88) - /unlink 3895 (time 1761312200.75 total 30.01 last 126.04) - /unlink 5132 (time 1761312210.75 total 40.02 last 123.64) - /unlink 6437 (time 1761312220.76 total 50.02 last 130.45) - /unlink 7837 (time 1761312230.76 total 60.03 last 139.91) - /unlink 9123 (time 1761312240.79 total 70.06 last 128.21) - /unlink 10000 (time 1761312247.42 total 76.68 last 132.32) - /unlink 11297 (time 1761312257.43 total 86.69 last 129.63) - /unlink 12618 (time 1761312267.44 total 96.70 last 131.97) - /unlink 13913 (time 1761312277.44 total 106.71 last 129.41) - /unlink 15166 (time 1761312287.45 total 116.71 last 125.28) - /unlink 16472 (time 1761312297.45 total 126.71 last 130.56) - /unlink 17906 (time 1761312307.45 total 136.71 last 143.34) - /unlink 19354 (time 1761312317.46 total 146.72 last 144.75) - /unlink 20000 (time 1761312322.30 total 151.56 last 133.37) - /unlink 21362 (time 1761312332.31 total 161.57 last 136.14) - /unlink 22722 (time 1761312342.31 total 171.57 last 135.92) - /unlink 24082 (time 1761312352.32 total 181.58 last 135.94) - /unlink 25288 (time 1761312362.32 total 191.58 last 120.58) - /unlink 26518 (time 1761312372.32 total 201.58 last 122.96) - /unlink 27832 (time 1761312382.32 total 211.58 last 131.39) - /unlink 29254 (time 1761312392.33 total 221.59 last 142.09) - /unlink 30000 (time 1761312397.90 total 227.16 last 133.98) - /unlink 31614 (time 1761312407.90 total 237.16 last 161.37) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 242.54 seconds: 134.38 ops/second Waiting for MDT destroys to complete - open/close 903 (time 1761312434.03 total 10.00 last 90.27) - open/close 1817 (time 1761312444.04 total 20.01 last 91.32) - open/close 2688 (time 1761312454.04 total 30.01 last 87.09) - open/close 3536 (time 1761312464.05 total 40.02 last 84.71) - open/close 4339 (time 1761312474.05 total 50.02 last 80.28) - open/close 5188 (time 1761312484.06 total 60.04 last 84.77) - open/close 6078 (time 1761312494.08 total 70.05 last 88.91) - open/close 6884 (time 1761312504.09 total 80.06 last 80.51) - open/close 7773 (time 1761312514.09 total 90.06 last 88.88) - open/close 8640 (time 1761312524.10 total 100.07 last 86.62) - open/close 9544 (time 1761312534.10 total 110.08 last 90.37) - open/close 10000 (time 1761312539.74 total 115.71 last 80.89) - open/close 10828 (time 1761312549.74 total 125.72 last 82.77) - open/close 11562 (time 1761312559.74 total 135.72 last 73.39) - open/close 12394 (time 1761312569.75 total 145.72 last 83.17) - open/close 13192 (time 1761312579.75 total 155.73 last 79.75) - open/close 13874 (time 1761312589.75 total 165.73 last 68.19) - open/close 14608 (time 1761312599.75 total 175.73 last 73.40) - open/close 15356 (time 1761312609.76 total 185.73 last 74.77) - open/close 16171 (time 1761312619.78 total 195.75 last 81.33) - open/close 16973 (time 1761312629.78 total 205.76 last 80.18) - open/close 17778 (time 1761312639.79 total 215.76 last 80.46) - open/close 18462 (time 1761312649.79 total 225.77 last 68.36) - open/close 19398 (time 1761312659.80 total 235.78 last 93.53) - open/close 20000 (time 1761312666.45 total 242.42 last 90.57) - open/close 20853 (time 1761312676.45 total 252.43 last 85.28) - open/close 21667 (time 1761312686.45 total 262.43 last 81.39) - open/close 22514 (time 1761312696.46 total 272.43 last 84.66) - open/close 23325 (time 1761312706.47 total 282.44 last 81.02) - open/close 24152 (time 1761312716.47 total 292.44 last 82.68) - open/close 25001 (time 1761312726.47 total 302.45 last 84.86) - open/close 25856 (time 1761312736.48 total 312.46 last 85.41) - open/close 26694 (time 1761312746.49 total 322.46 last 83.79) - open/close 27543 (time 1761312756.49 total 332.46 last 84.89) - open/close 28440 (time 1761312766.50 total 342.48 last 89.55) - open/close 29297 (time 1761312776.51 total 352.49 last 85.64) - open/close 30000 (time 1761312784.22 total 360.19 last 91.23) - open/close 30858 (time 1761312794.22 total 370.19 last 85.80) - open/close 31729 (time 1761312804.22 total 380.20 last 87.07) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 389.60 seconds: 83.66 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1286 (time 1761312827.32 total 10.01 last 128.46) - /unlink 2705 (time 1761312837.32 total 20.02 last 141.78) - /unlink 4076 (time 1761312847.33 total 30.03 last 136.97) - /unlink 5574 (time 1761312857.34 total 40.04 last 149.66) - /unlink 7241 (time 1761312867.34 total 50.04 last 166.70) - /unlink 9096 (time 1761312877.35 total 60.04 last 185.42) - /unlink 10000 (time 1761312881.28 total 63.97 last 230.07) - /unlink 11955 (time 1761312891.28 total 73.97 last 195.48) - /unlink 13657 (time 1761312901.28 total 83.98 last 170.16) - /unlink 15272 (time 1761312911.28 total 93.98 last 161.46) - /unlink 16944 (time 1761312921.29 total 103.98 last 167.15) - /unlink 18803 (time 1761312931.29 total 113.98 last 185.87) - /unlink 20000 (time 1761312939.01 total 121.70 last 155.08) - /unlink 21665 (time 1761312949.01 total 131.70 last 166.47) - /unlink 23191 (time 1761312959.01 total 141.70 last 152.57) - /unlink 24791 (time 1761312969.01 total 151.71 last 159.93) - /unlink 26551 (time 1761312979.02 total 161.71 last 175.93) - /unlink 28184 (time 1761312989.02 total 171.72 last 163.21) - /unlink 29665 (time 1761312999.03 total 181.72 last 148.07) - /unlink 30000 (time 1761313001.24 total 183.93 last 151.59) - /unlink 31588 (time 1761313011.24 total 193.93 last 158.77) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 199.94 seconds: 163.02 ops/second Waiting for MDT destroys to complete - open/close 811 (time 1761313042.93 total 10.00 last 81.07) - open/close 1672 (time 1761313052.93 total 20.01 last 86.02) - open/close 2571 (time 1761313062.94 total 30.02 last 89.85) - open/close 3398 (time 1761313072.94 total 40.02 last 82.69) - open/close 4194 (time 1761313082.96 total 50.03 last 79.47) - open/close 5050 (time 1761313092.96 total 60.03 last 85.60) - open/close 5917 (time 1761313102.96 total 70.04 last 86.65) - open/close 6793 (time 1761313112.97 total 80.05 last 87.56) - open/close 7677 (time 1761313122.97 total 90.05 last 88.36) - open/close 8494 (time 1761313132.98 total 100.06 last 81.60) - open/close 9319 (time 1761313142.99 total 110.06 last 82.48) - open/close 10000 (time 1761313150.77 total 117.85 last 87.47) - open/close 10782 (time 1761313160.78 total 127.85 last 78.17) - open/close 11602 (time 1761313170.78 total 137.85 last 82.00) - open/close 12430 (time 1761313180.78 total 147.86 last 82.74) - open/close 13349 (time 1761313190.80 total 157.88 last 91.77) - open/close 14162 (time 1761313200.80 total 167.88 last 81.27) - open/close 14919 (time 1761313210.81 total 177.89 last 75.64) - open/close 15727 (time 1761313220.81 total 187.89 last 80.77) - open/close 16569 (time 1761313230.81 total 197.89 last 84.19) - open/close 17400 (time 1761313240.81 total 207.89 last 83.09) - open/close 18248 (time 1761313250.82 total 217.90 last 84.77) - open/close 19048 (time 1761313260.83 total 227.91 last 79.91) - open/close 19964 (time 1761313270.83 total 237.91 last 91.60) - open/close 20866 (time 1761313280.84 total 247.91 last 90.14) - open/close 21910 (time 1761313290.84 total 257.92 last 104.38) - open/close 22775 (time 1761313300.84 total 267.92 last 86.47) - open/close 23636 (time 1761313310.85 total 277.93 last 86.05) - open/close 24534 (time 1761313320.85 total 287.93 last 89.75) - open/close 25383 (time 1761313330.86 total 297.94 last 84.84) - open/close 26156 (time 1761313340.86 total 307.94 last 77.28) - open/close 26934 (time 1761313350.88 total 317.96 last 77.69) - open/close 27692 (time 1761313360.89 total 327.97 last 75.73) - open/close 28453 (time 1761313370.89 total 337.97 last 76.08) - open/close 29257 (time 1761313380.90 total 347.98 last 80.31) - open/close 30000 (time 1761313390.22 total 357.30 last 79.68) - open/close 30806 (time 1761313400.23 total 367.31 last 80.57) - open/close 31572 (time 1761313410.24 total 377.31 last 76.54) - open/close 32349 (time 1761313420.25 total 387.33 last 77.59) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 390.51 seconds: 83.47 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1358 (time 1761313437.18 total 10.00 last 135.79) - /unlink 2794 (time 1761313447.19 total 20.01 last 143.52) - /unlink 4135 (time 1761313457.19 total 30.01 last 134.09) - /unlink 5524 (time 1761313467.19 total 40.01 last 138.86) - /unlink 6905 (time 1761313477.20 total 50.02 last 138.03) - /unlink 8327 (time 1761313487.20 total 60.02 last 142.17) - /unlink 9723 (time 1761313497.20 total 70.02 last 139.54) - /unlink 10000 (time 1761313499.22 total 72.04 last 137.16) - /unlink 11461 (time 1761313509.23 total 82.04 last 146.05) - /unlink 12844 (time 1761313519.24 total 92.06 last 138.14) - /unlink 14241 (time 1761313529.26 total 102.08 last 139.38) - /unlink 15615 (time 1761313539.26 total 112.08 last 137.40) - /unlink 17056 (time 1761313549.27 total 122.09 last 143.97) - /unlink 18420 (time 1761313559.28 total 132.10 last 136.28) - /unlink 19597 (time 1761313569.28 total 142.10 last 117.68) - /unlink 20000 (time 1761313572.22 total 145.03 last 137.29) - /unlink 21397 (time 1761313582.22 total 155.03 last 139.70) - /unlink 22826 (time 1761313592.22 total 165.04 last 142.81) - /unlink 24215 (time 1761313602.23 total 175.04 last 138.85) - /unlink 25509 (time 1761313612.23 total 185.05 last 129.39) - /unlink 26708 (time 1761313622.23 total 195.05 last 119.88) - /unlink 28079 (time 1761313632.23 total 205.05 last 137.02) - /unlink 29534 (time 1761313642.24 total 215.06 last 145.46) - /unlink 30000 (time 1761313645.25 total 218.07 last 154.53) - /unlink 31517 (time 1761313655.26 total 228.07 last 151.66) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 235.36 seconds: 138.49 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg313-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg313-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg313-server stop mds service on oleg313-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg313-server stop mds service on oleg313-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg313-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing unload_modules_local oleg313-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg313-server: LNET ready to unload oleg313-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg313-client: oleg313-client: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-client: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2