== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:39:02 (1761313142) Stopping clients: oleg356-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg356-client.virtnet /mnt/lustre2 (opts:-f) oleg356-server: oleg356-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing load_modules_local oleg356-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg356-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg356-server: Loading modules from /home/green/git/lustre-release/lustre oleg356-server: detected 4 online CPUs by sysfs oleg356-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg356-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg356-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg356-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock oleg356-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 844 (time 1761313247.93 total 10.01 last 84.29) - open/close 1676 (time 1761313257.94 total 20.03 last 83.09) - open/close 2570 (time 1761313267.95 total 30.03 last 89.33) - open/close 3451 (time 1761313277.96 total 40.04 last 88.01) - open/close 4555 (time 1761313287.97 total 50.05 last 110.32) - open/close 5373 (time 1761313297.97 total 60.06 last 81.77) - open/close 6233 (time 1761313307.98 total 70.06 last 85.93) - open/close 7131 (time 1761313317.99 total 80.07 last 89.75) - open/close 7901 (time 1761313327.99 total 90.07 last 76.99) - open/close 8659 (time 1761313338.00 total 100.08 last 75.70) - open/close 9460 (time 1761313348.01 total 110.09 last 80.05) - open/close 10000 (time 1761313354.54 total 116.62 last 82.64) - open/close 10845 (time 1761313364.55 total 126.63 last 84.45) - open/close 11657 (time 1761313374.55 total 136.63 last 81.20) - open/close 12478 (time 1761313384.56 total 146.64 last 82.04) - open/close 13293 (time 1761313394.57 total 156.65 last 81.42) - open/close 14104 (time 1761313404.57 total 166.66 last 81.03) - open/close 14865 (time 1761313414.59 total 176.67 last 75.98) - open/close 15648 (time 1761313424.61 total 186.69 last 78.18) - open/close 16407 (time 1761313434.61 total 196.69 last 75.86) - open/close 17252 (time 1761313444.61 total 206.70 last 84.46) - open/close 18090 (time 1761313454.63 total 216.71 last 83.70) - open/close 18899 (time 1761313464.63 total 226.71 last 80.87) - open/close 19693 (time 1761313474.64 total 236.72 last 79.33) - open/close 20000 (time 1761313478.45 total 240.53 last 80.51) - open/close 20809 (time 1761313488.46 total 250.55 last 80.82) - open/close 21596 (time 1761313498.47 total 260.55 last 78.67) - open/close 22387 (time 1761313508.47 total 270.55 last 79.07) - open/close 23167 (time 1761313518.47 total 280.55 last 77.98) - open/close 24003 (time 1761313528.49 total 290.57 last 83.49) - open/close 24809 (time 1761313538.49 total 300.57 last 80.56) - open/close 25580 (time 1761313548.51 total 310.59 last 76.96) - open/close 26332 (time 1761313558.52 total 320.60 last 75.11) - open/close 27043 (time 1761313568.53 total 330.61 last 71.05) - open/close 27845 (time 1761313578.53 total 340.61 last 80.19) - open/close 28661 (time 1761313588.54 total 350.62 last 81.55) - open/close 29428 (time 1761313598.54 total 360.62 last 76.68) - open/close 30000 (time 1761313605.90 total 367.98 last 77.70) - open/close 30795 (time 1761313615.92 total 378.00 last 79.38) - open/close 31500 (time 1761313625.93 total 388.01 last 70.41) - open/close 32283 (time 1761313635.94 total 398.02 last 78.24) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 401.72 seconds: 81.14 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1599 (time 1761313652.66 total 10.00 last 159.89) - /unlink 3050 (time 1761313662.66 total 20.00 last 145.07) - /unlink 4621 (time 1761313672.67 total 30.01 last 157.00) - /unlink 6393 (time 1761313682.67 total 40.01 last 177.10) - /unlink 8331 (time 1761313692.68 total 50.02 last 193.71) - /unlink 10000 (time 1761313702.43 total 59.77 last 171.25) - /unlink 11801 (time 1761313712.43 total 69.77 last 180.05) - /unlink 13293 (time 1761313722.43 total 79.77 last 149.19) - /unlink 14945 (time 1761313732.43 total 89.77 last 165.14) - /unlink 16620 (time 1761313742.44 total 99.78 last 167.43) - /unlink 18428 (time 1761313752.44 total 109.78 last 180.74) - /unlink 20000 (time 1761313759.98 total 117.32 last 208.36) - /unlink 21541 (time 1761313769.99 total 127.33 last 153.98) - /unlink 23005 (time 1761313779.99 total 137.33 last 146.39) - /unlink 24728 (time 1761313789.99 total 147.33 last 172.28) - /unlink 26454 (time 1761313799.99 total 157.33 last 172.58) - /unlink 27839 (time 1761313810.00 total 167.34 last 138.45) - /unlink 29251 (time 1761313820.00 total 177.34 last 141.20) - /unlink 30000 (time 1761313825.09 total 182.43 last 147.04) - /unlink 31652 (time 1761313835.10 total 192.44 last 165.13) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 197.80 seconds: 164.78 ops/second Waiting for MDT destroys to complete - open/close 836 (time 1761313864.88 total 10.01 last 83.51) - open/close 1719 (time 1761313874.89 total 20.02 last 88.23) - open/close 2612 (time 1761313884.89 total 30.02 last 89.28) - open/close 3483 (time 1761313894.89 total 40.02 last 87.10) - open/close 4381 (time 1761313904.90 total 50.02 last 89.79) - open/close 5337 (time 1761313914.90 total 60.03 last 95.55) - open/close 6238 (time 1761313924.91 total 70.03 last 90.05) - open/close 7131 (time 1761313934.91 total 80.04 last 89.28) - open/close 8063 (time 1761313944.91 total 90.04 last 93.15) - open/close 8908 (time 1761313954.92 total 100.04 last 84.49) - open/close 9723 (time 1761313964.92 total 110.05 last 81.44) - open/close 10000 (time 1761313968.36 total 113.49 last 80.51) - open/close 10831 (time 1761313978.37 total 123.50 last 83.07) - open/close 11699 (time 1761313988.37 total 133.50 last 86.77) - open/close 12490 (time 1761313998.38 total 143.50 last 79.06) - open/close 13322 (time 1761314008.39 total 153.52 last 83.09) - open/close 14123 (time 1761314018.39 total 163.52 last 80.08) - open/close 14897 (time 1761314028.39 total 173.52 last 77.38) - open/close 15760 (time 1761314038.41 total 183.53 last 86.20) - open/close 16621 (time 1761314048.41 total 193.54 last 86.04) - open/close 17511 (time 1761314058.42 total 203.54 last 88.98) - open/close 18425 (time 1761314068.42 total 213.55 last 91.33) - open/close 19362 (time 1761314078.43 total 223.56 last 93.62) - open/close 20000 (time 1761314085.66 total 230.79 last 88.26) - open/close 20898 (time 1761314095.67 total 240.79 last 89.75) - open/close 21727 (time 1761314105.67 total 250.79 last 82.90) - open/close 22617 (time 1761314115.67 total 260.80 last 88.97) - open/close 23501 (time 1761314125.67 total 270.80 last 88.38) - open/close 24287 (time 1761314135.68 total 280.81 last 78.54) - open/close 25204 (time 1761314145.68 total 290.81 last 91.66) - open/close 26144 (time 1761314155.70 total 300.82 last 93.89) - open/close 26973 (time 1761314165.70 total 310.82 last 82.89) - open/close 27776 (time 1761314175.70 total 320.83 last 80.28) - open/close 28547 (time 1761314185.71 total 330.83 last 77.04) - open/close 29284 (time 1761314195.72 total 340.84 last 73.63) - open/close 30000 (time 1761314204.77 total 349.90 last 79.07) - open/close 30866 (time 1761314214.78 total 359.91 last 86.52) - open/close 31665 (time 1761314224.78 total 369.91 last 79.89) - open/close 32414 (time 1761314234.79 total 379.92 last 74.86) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 382.17 seconds: 85.29 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1336 (time 1761314251.06 total 10.00 last 133.58) - /unlink 2680 (time 1761314261.06 total 20.01 last 134.31) - /unlink 4157 (time 1761314271.06 total 30.01 last 147.69) - /unlink 5514 (time 1761314281.07 total 40.01 last 135.67) - /unlink 6959 (time 1761314291.07 total 50.02 last 144.42) - /unlink 8434 (time 1761314301.07 total 60.02 last 147.49) - /unlink 9781 (time 1761314311.08 total 70.02 last 134.63) - /unlink 11169 (time 1761314321.09 total 80.03 last 138.65) - /unlink 12655 (time 1761314331.10 total 90.04 last 148.45) - /unlink 14124 (time 1761314341.10 total 100.04 last 146.90) - /unlink 15884 (time 1761314351.11 total 110.05 last 175.83) - /unlink 17397 (time 1761314361.12 total 120.06 last 151.21) - /unlink 18898 (time 1761314371.12 total 130.06 last 150.04) - /unlink 20000 (time 1761314378.70 total 137.64 last 145.40) - /unlink 21225 (time 1761314388.71 total 147.66 last 122.30) - /unlink 22536 (time 1761314398.72 total 157.66 last 131.06) - /unlink 23957 (time 1761314408.72 total 167.66 last 142.06) - /unlink 25614 (time 1761314418.73 total 177.67 last 165.60) - /unlink 27144 (time 1761314428.73 total 187.68 last 152.91) - /unlink 28643 (time 1761314438.74 total 197.69 last 149.77) - /unlink 30000 (time 1761314447.43 total 206.37 last 156.25) - /unlink 31663 (time 1761314457.43 total 216.38 last 166.21) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 222.20 seconds: 146.69 ops/second Waiting for MDT destroys to complete - open/close 810 (time 1761314484.71 total 10.00 last 80.98) - open/close 1663 (time 1761314494.72 total 20.01 last 85.27) - open/close 2577 (time 1761314504.73 total 30.02 last 91.29) - open/close 3394 (time 1761314514.73 total 40.02 last 81.66) - open/close 4205 (time 1761314524.74 total 50.03 last 81.07) - open/close 4971 (time 1761314534.75 total 60.04 last 76.51) - open/close 5775 (time 1761314544.76 total 70.05 last 80.35) - open/close 6612 (time 1761314554.77 total 80.07 last 83.54) - open/close 7366 (time 1761314564.79 total 90.08 last 75.32) - open/close 8128 (time 1761314574.80 total 100.09 last 76.09) - open/close 8913 (time 1761314584.81 total 110.10 last 78.44) - open/close 9663 (time 1761314594.81 total 120.11 last 74.95) - open/close 10000 (time 1761314598.57 total 123.86 last 89.64) - open/close 10886 (time 1761314608.58 total 133.87 last 88.58) - open/close 11741 (time 1761314618.58 total 143.87 last 85.49) - open/close 12555 (time 1761314628.58 total 153.87 last 81.38) - open/close 13355 (time 1761314638.58 total 163.87 last 79.97) - open/close 14165 (time 1761314648.59 total 173.88 last 80.97) - open/close 15065 (time 1761314658.59 total 183.88 last 89.97) - open/close 15935 (time 1761314668.59 total 193.88 last 86.99) - open/close 16875 (time 1761314678.60 total 203.89 last 93.89) - open/close 17698 (time 1761314688.62 total 213.91 last 82.16) - open/close 18488 (time 1761314698.62 total 223.91 last 78.98) - open/close 19328 (time 1761314708.62 total 233.91 last 83.99) - open/close 20000 (time 1761314716.88 total 242.17 last 81.38) - open/close 20858 (time 1761314726.89 total 252.18 last 85.74) - open/close 21652 (time 1761314736.90 total 262.19 last 79.32) - open/close 22550 (time 1761314746.90 total 272.19 last 89.80) - open/close 23476 (time 1761314756.90 total 282.19 last 92.59) - open/close 24399 (time 1761314766.90 total 292.19 last 92.29) - open/close 25232 (time 1761314776.90 total 302.19 last 83.30) - open/close 26127 (time 1761314786.90 total 312.19 last 89.49) - open/close 26945 (time 1761314796.91 total 322.20 last 81.71) - open/close 27695 (time 1761314806.92 total 332.21 last 74.97) - open/close 28449 (time 1761314816.93 total 342.22 last 75.29) - open/close 29167 (time 1761314826.94 total 352.23 last 71.77) - open/close 29935 (time 1761314836.94 total 362.23 last 76.77) - open/close 30759 (time 1761314846.95 total 372.24 last 82.34) - open/close 31593 (time 1761314856.96 total 382.25 last 83.33) - open/close 32485 (time 1761314866.96 total 392.25 last 89.18) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 393.52 seconds: 82.83 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1280 (time 1761314881.85 total 10.00 last 127.99) - /unlink 2674 (time 1761314891.85 total 20.00 last 139.37) - /unlink 4228 (time 1761314901.86 total 30.01 last 155.29) - /unlink 5645 (time 1761314911.86 total 40.01 last 141.67) - /unlink 7194 (time 1761314921.86 total 50.01 last 154.88) - /unlink 8668 (time 1761314931.87 total 60.02 last 147.37) - /unlink 10000 (time 1761314940.82 total 68.97 last 148.74) - /unlink 11687 (time 1761314950.83 total 78.98 last 168.60) - /unlink 13555 (time 1761314960.83 total 88.98 last 186.78) - /unlink 15195 (time 1761314970.83 total 98.98 last 163.97) - /unlink 17005 (time 1761314980.83 total 108.98 last 180.95) - /unlink 19241 (time 1761314990.84 total 118.99 last 223.52) - /unlink 20000 (time 1761314994.18 total 122.33 last 227.02) - /unlink 21645 (time 1761315004.19 total 132.34 last 164.34) - /unlink 23303 (time 1761315014.19 total 142.34 last 165.79) - /unlink 24727 (time 1761315024.19 total 152.34 last 142.35) - /unlink 26130 (time 1761315034.19 total 162.34 last 140.27) - /unlink 27550 (time 1761315044.20 total 172.35 last 141.98) - /unlink 28997 (time 1761315054.20 total 182.35 last 144.64) - /unlink 30000 (time 1761315061.13 total 189.28 last 144.81) - /unlink 31359 (time 1761315071.14 total 199.29 last 135.75) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 208.22 seconds: 156.53 ops/second Waiting for MDT destroys to complete - open/close 804 (time 1761315103.84 total 10.00 last 80.39) - open/close 1649 (time 1761315113.84 total 20.00 last 84.47) - open/close 2608 (time 1761315123.85 total 30.01 last 95.82) - open/close 3490 (time 1761315133.86 total 40.02 last 88.13) - open/close 4370 (time 1761315143.86 total 50.03 last 87.94) - open/close 5372 (time 1761315153.87 total 60.04 last 100.12) - open/close 6351 (time 1761315163.87 total 70.04 last 97.89) - open/close 7329 (time 1761315173.87 total 80.04 last 97.79) - open/close 8231 (time 1761315183.89 total 90.05 last 90.07) - open/close 9147 (time 1761315193.90 total 100.06 last 91.48) - open/close 10000 (time 1761315203.27 total 109.44 last 91.01) - open/close 10827 (time 1761315213.28 total 119.44 last 82.66) - open/close 11695 (time 1761315223.28 total 129.44 last 86.80) - open/close 12844 (time 1761315233.28 total 139.45 last 114.86) - open/close 13738 (time 1761315243.30 total 149.46 last 89.29) - open/close 14670 (time 1761315253.30 total 159.46 last 93.15) - open/close 15551 (time 1761315263.30 total 169.47 last 88.08) - open/close 16385 (time 1761315273.31 total 179.47 last 83.33) - open/close 17260 (time 1761315283.32 total 189.48 last 87.46) - open/close 18082 (time 1761315293.32 total 199.48 last 82.17) - open/close 18919 (time 1761315303.33 total 209.49 last 83.64) - open/close 19763 (time 1761315313.33 total 219.49 last 84.37) - open/close 20000 (time 1761315316.14 total 222.30 last 84.44) - open/close 20816 (time 1761315326.15 total 232.31 last 81.48) - open/close 21631 (time 1761315336.15 total 242.32 last 81.49) - open/close 22469 (time 1761315346.16 total 252.32 last 83.79) - open/close 23303 (time 1761315356.16 total 262.33 last 83.33) - open/close 24200 (time 1761315366.17 total 272.33 last 89.68) - open/close 25039 (time 1761315376.17 total 282.33 last 83.85) - open/close 25964 (time 1761315386.17 total 292.34 last 92.50) - open/close 26804 (time 1761315396.18 total 302.34 last 83.96) - open/close 27636 (time 1761315406.18 total 312.35 last 83.13) - open/close 28396 (time 1761315416.19 total 322.35 last 75.95) - open/close 29128 (time 1761315426.20 total 332.36 last 73.12) - open/close 29941 (time 1761315436.21 total 342.37 last 81.22) - open/close 30868 (time 1761315446.22 total 352.38 last 92.66) - open/close 31830 (time 1761315456.22 total 362.39 last 96.14) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 370.49 seconds: 87.97 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1417 (time 1761315477.46 total 10.00 last 141.67) - /unlink 2702 (time 1761315487.46 total 20.00 last 128.49) - /unlink 4070 (time 1761315497.47 total 30.01 last 136.75) - /unlink 5731 (time 1761315507.47 total 40.01 last 166.06) - /unlink 7583 (time 1761315517.47 total 50.01 last 185.16) - /unlink 9035 (time 1761315527.47 total 60.01 last 145.17) - /unlink 10000 (time 1761315534.60 total 67.14 last 135.39) - /unlink 11538 (time 1761315544.60 total 77.14 last 153.78) - /unlink 13147 (time 1761315554.61 total 87.15 last 160.79) - /unlink 14560 (time 1761315564.61 total 97.15 last 141.22) - /unlink 15922 (time 1761315574.62 total 107.16 last 136.09) - /unlink 17506 (time 1761315584.62 total 117.16 last 158.37) - /unlink 19183 (time 1761315594.63 total 127.17 last 167.63) - /unlink 20000 (time 1761315599.89 total 132.44 last 155.13) - /unlink 21515 (time 1761315609.90 total 142.44 last 151.45) - /unlink 22928 (time 1761315619.90 total 152.45 last 141.20) - /unlink 24308 (time 1761315629.91 total 162.45 last 137.90) - /unlink 25801 (time 1761315639.92 total 172.46 last 149.15) - /unlink 27286 (time 1761315649.92 total 182.47 last 148.45) - /unlink 28765 (time 1761315659.93 total 192.47 last 147.83) - /unlink 30000 (time 1761315668.39 total 200.93 last 145.94) - /unlink 31526 (time 1761315678.40 total 210.94 last 152.51) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 218.45 seconds: 149.21 ops/second Waiting for MDT destroys to complete - open/close 837 (time 1761315712.54 total 10.00 last 83.67) - open/close 1687 (time 1761315722.55 total 20.01 last 84.92) - open/close 2483 (time 1761315732.55 total 30.02 last 79.57) - open/close 3280 (time 1761315742.56 total 40.02 last 79.66) - open/close 4151 (time 1761315752.57 total 50.03 last 87.00) - open/close 5042 (time 1761315762.57 total 60.03 last 89.10) - open/close 5886 (time 1761315772.57 total 70.03 last 84.38) - open/close 6621 (time 1761315782.58 total 80.04 last 73.43) - open/close 7432 (time 1761315792.59 total 90.05 last 81.04) - open/close 8172 (time 1761315802.60 total 100.06 last 73.94) - open/close 8974 (time 1761315812.60 total 110.07 last 80.15) - open/close 9787 (time 1761315822.61 total 120.07 last 81.28) - open/close 10000 (time 1761315825.39 total 122.85 last 76.57) - open/close 10821 (time 1761315835.40 total 132.86 last 82.04) - open/close 11602 (time 1761315845.41 total 142.87 last 78.02) - open/close 12415 (time 1761315855.41 total 152.88 last 81.24) - open/close 13178 (time 1761315865.42 total 162.88 last 76.29) - open/close 13972 (time 1761315875.42 total 172.89 last 79.33) - open/close 14716 (time 1761315885.43 total 182.89 last 74.35) - open/close 15585 (time 1761315895.45 total 192.91 last 86.76) - open/close 16382 (time 1761315905.45 total 202.91 last 79.69) - open/close 17198 (time 1761315915.45 total 212.91 last 81.60) - open/close 18056 (time 1761315925.46 total 222.92 last 85.74) - open/close 18896 (time 1761315935.46 total 232.92 last 83.94) - open/close 19701 (time 1761315945.47 total 242.93 last 80.42) - open/close 20000 (time 1761315949.08 total 246.54 last 82.85) - open/close 20787 (time 1761315959.08 total 256.54 last 78.69) - open/close 21582 (time 1761315969.09 total 266.55 last 79.46) - open/close 22351 (time 1761315979.09 total 276.56 last 76.86) - open/close 23148 (time 1761315989.10 total 286.57 last 79.62) - open/close 23894 (time 1761315999.12 total 296.58 last 74.48) - open/close 24703 (time 1761316009.13 total 306.59 last 80.81) - open/close 25500 (time 1761316019.13 total 316.59 last 79.68) - open/close 26254 (time 1761316029.15 total 326.61 last 75.29) - open/close 27060 (time 1761316039.15 total 336.61 last 80.58) - open/close 27838 (time 1761316049.15 total 346.62 last 77.78) - open/close 28635 (time 1761316059.16 total 356.62 last 79.68) - open/close 29472 (time 1761316069.16 total 366.63 last 83.63) - open/close 30000 (time 1761316075.36 total 372.82 last 85.27) - open/close 30747 (time 1761316085.36 total 382.82 last 74.67) - open/close 31537 (time 1761316095.37 total 392.83 last 78.92) - open/close 32337 (time 1761316105.37 total 402.83 last 79.99) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 405.66 seconds: 80.35 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1575 (time 1761316121.21 total 10.00 last 157.50) - /unlink 3057 (time 1761316131.22 total 20.00 last 148.13) - /unlink 4494 (time 1761316141.22 total 30.01 last 143.68) - /unlink 6199 (time 1761316151.23 total 40.02 last 170.24) - /unlink 8012 (time 1761316161.23 total 50.02 last 181.27) - /unlink 9855 (time 1761316171.24 total 60.03 last 184.24) - /unlink 11307 (time 1761316181.24 total 70.03 last 145.16) - /unlink 12844 (time 1761316191.24 total 80.03 last 153.69) - /unlink 14430 (time 1761316201.24 total 90.03 last 158.58) - /unlink 16265 (time 1761316211.25 total 100.03 last 183.45) - /unlink 17926 (time 1761316221.25 total 110.04 last 166.01) - /unlink 19469 (time 1761316231.26 total 120.04 last 154.22) - /unlink 20000 (time 1761316234.36 total 123.15 last 170.78) - /unlink 21581 (time 1761316244.37 total 133.16 last 157.98) - /unlink 23116 (time 1761316254.38 total 143.17 last 153.41) - /unlink 24467 (time 1761316264.38 total 153.17 last 135.03) - /unlink 25919 (time 1761316274.39 total 163.18 last 145.14) - /unlink 27870 (time 1761316284.39 total 173.18 last 195.02) - /unlink 29665 (time 1761316294.40 total 183.19 last 179.43) - /unlink 31702 (time 1761316304.40 total 193.19 last 203.65) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 199.02 seconds: 163.77 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg356-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg356-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing unload_modules_local oleg356-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg356-server: LNET ready to unload oleg356-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2