== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:58:24 (1776189504) Stopping clients: oleg301-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg301-client.virtnet /mnt/lustre2 (opts:-f) oleg301-server: oleg301-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing load_modules_local oleg301-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg301-server: Loading modules from /home/green/git/lustre-release/lustre oleg301-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg301-server: detected 4 online CPUs by sysfs oleg301-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg301-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg301-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg301-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg301-client.virtnet: -o user_xattr,flock oleg301-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 663 (time 1776189637.52 total 10.00 last 66.29) - open/close 1309 (time 1776189647.53 total 20.01 last 64.53) - open/close 1928 (time 1776189657.54 total 30.02 last 61.86) - open/close 2528 (time 1776189667.54 total 40.02 last 59.99) - open/close 3234 (time 1776189677.54 total 50.02 last 70.58) - open/close 3899 (time 1776189687.55 total 60.03 last 66.45) - open/close 4566 (time 1776189697.57 total 70.05 last 66.60) - open/close 5245 (time 1776189707.58 total 80.06 last 67.82) - open/close 5942 (time 1776189717.59 total 90.07 last 69.61) - open/close 6596 (time 1776189727.62 total 100.10 last 65.22) - open/close 7165 (time 1776189737.63 total 110.11 last 56.84) - open/close 7737 (time 1776189747.64 total 120.12 last 57.11) - open/close 8303 (time 1776189757.65 total 130.13 last 56.59) - open/close 8969 (time 1776189767.65 total 140.13 last 66.55) - open/close 9576 (time 1776189777.66 total 150.15 last 60.63) - open/close 10000 (time 1776189783.92 total 156.40 last 67.74) - open/close 10661 (time 1776189793.93 total 166.41 last 66.06) - open/close 11281 (time 1776189803.95 total 176.43 last 61.88) - open/close 11938 (time 1776189813.97 total 186.45 last 65.58) - open/close 12598 (time 1776189823.97 total 196.45 last 65.96) - open/close 13243 (time 1776189833.99 total 206.47 last 64.37) - open/close 13924 (time 1776189844.00 total 216.48 last 68.06) - open/close 14561 (time 1776189854.01 total 226.49 last 63.61) - open/close 15227 (time 1776189864.02 total 236.50 last 66.58) - open/close 15884 (time 1776189874.03 total 246.51 last 65.61) - open/close 16538 (time 1776189884.05 total 256.53 last 65.29) - open/close 17176 (time 1776189894.06 total 266.54 last 63.74) - open/close 17843 (time 1776189904.06 total 276.54 last 66.69) - open/close 18526 (time 1776189914.08 total 286.56 last 68.14) - open/close 19149 (time 1776189924.08 total 296.56 last 62.29) - open/close 19718 (time 1776189934.10 total 306.58 last 56.79) - open/close 20000 (time 1776189938.33 total 310.81 last 66.78) - open/close 20634 (time 1776189948.34 total 320.83 last 63.28) - open/close 21281 (time 1776189958.35 total 330.83 last 64.66) - open/close 21926 (time 1776189968.35 total 340.83 last 64.48) - open/close 22600 (time 1776189978.36 total 350.84 last 67.35) - open/close 23267 (time 1776189988.37 total 360.85 last 66.65) - open/close 23940 (time 1776189998.37 total 370.86 last 67.26) - open/close 24610 (time 1776190008.38 total 380.86 last 66.94) - open/close 25242 (time 1776190018.40 total 390.88 last 63.11) - open/close 25871 (time 1776190028.41 total 400.89 last 62.80) - open/close 26582 (time 1776190038.42 total 410.91 last 71.02) - open/close 27284 (time 1776190048.43 total 420.91 last 70.19) - open/close 27966 (time 1776190058.43 total 430.91 last 68.19) - open/close 28673 (time 1776190068.44 total 440.92 last 70.60) - open/close 29412 (time 1776190078.44 total 450.92 last 73.88) - open/close 30000 (time 1776190086.64 total 459.12 last 71.73) - open/close 30684 (time 1776190096.65 total 469.13 last 68.31) - open/close 31426 (time 1776190106.66 total 479.14 last 74.14) - open/close 32097 (time 1776190116.66 total 489.14 last 67.09) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 496.86 seconds: 65.60 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 277 79715 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 32851 47141 42% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33128 33128 0 100% /mnt/lustre - /unlink 1246 (time 1776190138.22 total 10.00 last 124.59) - /unlink 2529 (time 1776190148.22 total 20.00 last 128.28) - /unlink 3860 (time 1776190158.22 total 30.00 last 133.09) - /unlink 5270 (time 1776190168.23 total 40.01 last 140.97) - /unlink 6488 (time 1776190178.23 total 50.01 last 121.72) - /unlink 7762 (time 1776190188.23 total 60.01 last 127.38) - /unlink 9160 (time 1776190198.24 total 70.02 last 139.75) - /unlink 10000 (time 1776190204.83 total 76.60 last 127.51) - /unlink 11428 (time 1776190214.83 total 86.61 last 142.76) - /unlink 12931 (time 1776190224.83 total 96.61 last 150.25) - /unlink 14481 (time 1776190234.84 total 106.62 last 154.85) - /unlink 16098 (time 1776190244.84 total 116.62 last 161.70) - /unlink 17533 (time 1776190254.84 total 126.62 last 143.45) - /unlink 19155 (time 1776190264.85 total 136.63 last 162.11) - /unlink 20000 (time 1776190271.11 total 142.89 last 134.90) - /unlink 21083 (time 1776190281.11 total 152.89 last 108.29) - /unlink 22266 (time 1776190291.12 total 162.90 last 118.26) - /unlink 23314 (time 1776190301.12 total 172.90 last 104.79) - /unlink 24650 (time 1776190311.13 total 182.90 last 133.50) - /unlink 25892 (time 1776190321.13 total 192.91 last 124.11) - /unlink 27033 (time 1776190331.15 total 202.93 last 113.93) - /unlink 28395 (time 1776190341.15 total 212.93 last 136.19) - /unlink 29754 (time 1776190351.15 total 222.93 last 135.82) - /unlink 31091 (time 1776190361.16 total 232.94 last 133.65) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 242.74 seconds: 134.27 ops/second Waiting for MDT destroys to complete - open/close 878 (time 1776190397.92 total 10.01 last 87.74) - open/close 1674 (time 1776190407.92 total 20.01 last 79.54) - open/close 2394 (time 1776190417.94 total 30.03 last 71.88) - open/close 3036 (time 1776190427.95 total 40.04 last 64.15) - open/close 3621 (time 1776190437.95 total 50.04 last 58.47) - open/close 4241 (time 1776190447.95 total 60.05 last 61.98) - open/close 4906 (time 1776190457.96 total 70.05 last 66.44) - open/close 5554 (time 1776190467.98 total 80.07 last 64.72) - open/close 6249 (time 1776190477.98 total 90.07 last 69.46) - open/close 6899 (time 1776190487.99 total 100.08 last 64.93) - open/close 7602 (time 1776190498.00 total 110.09 last 70.25) - open/close 8319 (time 1776190508.01 total 120.11 last 71.59) - open/close 9022 (time 1776190518.02 total 130.11 last 70.24) - open/close 9725 (time 1776190528.05 total 140.14 last 70.09) - open/close 10000 (time 1776190531.87 total 143.97 last 71.96) - open/close 10665 (time 1776190541.88 total 153.97 last 66.46) - open/close 11295 (time 1776190551.89 total 163.98 last 62.92) - open/close 12062 (time 1776190561.90 total 173.99 last 76.67) - open/close 12846 (time 1776190571.91 total 184.00 last 78.31) - open/close 13573 (time 1776190581.91 total 194.00 last 72.69) - open/close 14211 (time 1776190591.91 total 204.00 last 63.80) - open/close 14912 (time 1776190601.91 total 214.00 last 70.09) - open/close 15691 (time 1776190611.92 total 224.01 last 77.85) - open/close 16466 (time 1776190621.93 total 234.02 last 77.42) - open/close 17180 (time 1776190631.94 total 244.03 last 71.32) - open/close 17898 (time 1776190641.95 total 254.04 last 71.76) - open/close 18622 (time 1776190651.96 total 264.06 last 72.26) - open/close 19299 (time 1776190661.97 total 274.06 last 67.66) - open/close 20000 (time 1776190671.88 total 283.97 last 70.75) - open/close 20741 (time 1776190681.88 total 293.98 last 74.05) - open/close 21464 (time 1776190691.89 total 303.98 last 72.27) - open/close 22221 (time 1776190701.90 total 313.99 last 75.64) - open/close 22908 (time 1776190711.91 total 324.00 last 68.59) - open/close 23613 (time 1776190721.91 total 334.01 last 70.50) - open/close 24287 (time 1776190731.93 total 344.02 last 67.31) - open/close 24927 (time 1776190741.93 total 354.02 last 63.97) - open/close 25624 (time 1776190751.95 total 364.04 last 69.56) - open/close 26247 (time 1776190761.96 total 374.05 last 62.25) - open/close 26875 (time 1776190771.97 total 384.06 last 62.73) - open/close 27561 (time 1776190781.97 total 394.06 last 68.59) - open/close 28256 (time 1776190791.98 total 404.07 last 69.45) - open/close 28952 (time 1776190801.99 total 414.08 last 69.53) - open/close 29696 (time 1776190811.99 total 424.09 last 74.37) - open/close 30000 (time 1776190816.42 total 428.51 last 68.76) - open/close 30690 (time 1776190826.42 total 438.51 last 68.97) - open/close 31402 (time 1776190836.43 total 448.52 last 71.16) - open/close 32187 (time 1776190846.44 total 458.53 last 78.42) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 463.66 seconds: 70.30 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 277 79715 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 32851 47141 42% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33128 33128 0 100% /mnt/lustre - /unlink 1209 (time 1776190865.39 total 10.05 last 120.32) - /unlink 2330 (time 1776190875.44 total 20.09 last 111.58) - /unlink 3444 (time 1776190885.45 total 30.11 last 111.21) - /unlink 4614 (time 1776190895.48 total 40.14 last 116.65) - /unlink 5814 (time 1776190905.51 total 50.17 last 119.68) - /unlink 7031 (time 1776190915.54 total 60.20 last 121.37) - /unlink 8195 (time 1776190925.55 total 70.21 last 116.25) - /unlink 9368 (time 1776190935.57 total 80.23 last 117.09) - /unlink 10000 (time 1776190941.21 total 85.87 last 111.94) - /unlink 11221 (time 1776190951.21 total 95.87 last 122.10) - /unlink 12551 (time 1776190961.22 total 105.88 last 132.94) - /unlink 13889 (time 1776190971.23 total 115.88 last 133.73) - /unlink 15142 (time 1776190981.23 total 125.89 last 125.28) - /unlink 16243 (time 1776190991.23 total 135.89 last 110.01) - /unlink 17394 (time 1776191001.24 total 145.90 last 115.04) - /unlink 18627 (time 1776191011.26 total 155.92 last 123.09) - /unlink 19927 (time 1776191021.26 total 165.92 last 129.90) - /unlink 21242 (time 1776191031.27 total 175.93 last 131.48) - /unlink 22583 (time 1776191041.27 total 185.93 last 134.10) - /unlink 23873 (time 1776191051.27 total 195.93 last 128.93) - /unlink 25462 (time 1776191061.27 total 205.93 last 158.90) - /unlink 27067 (time 1776191071.27 total 215.93 last 160.48) - /unlink 28554 (time 1776191081.28 total 225.94 last 148.67) - /unlink 29865 (time 1776191091.28 total 235.94 last 131.05) - /unlink 31209 (time 1776191101.28 total 245.94 last 134.37) - /unlink 32459 (time 1776191111.28 total 255.94 last 124.99) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 257.19 seconds: 126.73 ops/second Waiting for MDT destroys to complete - open/close 804 (time 1776191141.98 total 10.00 last 80.37) - open/close 1648 (time 1776191151.98 total 20.00 last 84.39) - open/close 2370 (time 1776191161.98 total 30.01 last 72.17) - open/close 3156 (time 1776191171.98 total 40.01 last 78.59) - open/close 3985 (time 1776191181.99 total 50.01 last 82.86) - open/close 4790 (time 1776191192.00 total 60.02 last 80.44) - open/close 5356 (time 1776191202.01 total 70.03 last 56.54) - open/close 5943 (time 1776191212.01 total 80.04 last 58.67) - open/close 6603 (time 1776191222.02 total 90.04 last 65.97) - open/close 7298 (time 1776191232.02 total 100.05 last 69.46) - open/close 7961 (time 1776191242.04 total 110.07 last 66.19) - open/close 8612 (time 1776191252.05 total 120.08 last 65.03) - open/close 9290 (time 1776191262.06 total 130.09 last 67.72) - open/close 9851 (time 1776191272.07 total 140.09 last 56.07) - open/close 10000 (time 1776191274.35 total 142.38 last 65.23) - open/close 10677 (time 1776191284.35 total 152.38 last 67.68) - open/close 11362 (time 1776191294.38 total 162.40 last 68.35) - open/close 12081 (time 1776191304.38 total 172.41 last 71.85) - open/close 12796 (time 1776191314.39 total 182.42 last 71.47) - open/close 13495 (time 1776191324.39 total 192.42 last 69.86) - open/close 14233 (time 1776191334.40 total 202.43 last 73.74) - open/close 14990 (time 1776191344.41 total 212.44 last 75.65) - open/close 15699 (time 1776191354.43 total 222.45 last 70.77) - open/close 16442 (time 1776191364.44 total 232.46 last 74.22) - open/close 17169 (time 1776191374.44 total 242.47 last 72.65) - open/close 17865 (time 1776191384.46 total 252.49 last 69.48) - open/close 18553 (time 1776191394.47 total 262.50 last 68.73) - open/close 19246 (time 1776191404.48 total 272.51 last 69.23) - open/close 20000 (time 1776191414.08 total 282.11 last 78.54) - open/close 20702 (time 1776191424.09 total 292.11 last 70.16) - open/close 21400 (time 1776191434.10 total 302.13 last 69.70) - open/close 22101 (time 1776191444.10 total 312.13 last 70.10) - open/close 22781 (time 1776191454.12 total 322.15 last 67.88) - open/close 23444 (time 1776191464.13 total 332.15 last 66.26) - open/close 24126 (time 1776191474.14 total 342.17 last 68.11) - open/close 24856 (time 1776191484.14 total 352.17 last 72.98) - open/close 25560 (time 1776191494.15 total 362.17 last 70.36) - open/close 26223 (time 1776191504.15 total 372.18 last 66.26) - open/close 26891 (time 1776191514.17 total 382.20 last 66.70) - open/close 27540 (time 1776191524.18 total 392.20 last 64.85) - open/close 28332 (time 1776191534.18 total 402.21 last 79.14) - open/close 29008 (time 1776191544.19 total 412.21 last 67.59) - open/close 29663 (time 1776191554.19 total 422.21 last 65.50) - open/close 30000 (time 1776191559.49 total 427.52 last 63.53) - open/close 30614 (time 1776191569.51 total 437.53 last 61.31) - open/close 31241 (time 1776191579.53 total 447.56 last 62.52) - open/close 31952 (time 1776191589.54 total 457.57 last 71.04) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 466.63 seconds: 69.85 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 277 79715 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 32851 47141 42% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33128 33128 0 100% /mnt/lustre - /unlink 1100 (time 1776191611.90 total 10.00 last 109.99) - /unlink 2213 (time 1776191621.91 total 20.00 last 111.26) - /unlink 3405 (time 1776191631.91 total 30.01 last 119.15) - /unlink 4578 (time 1776191641.92 total 40.01 last 117.24) - /unlink 5686 (time 1776191651.92 total 50.02 last 110.79) - /unlink 6779 (time 1776191661.92 total 60.02 last 109.28) - /unlink 7941 (time 1776191671.94 total 70.03 last 116.02) - /unlink 9109 (time 1776191681.94 total 80.03 last 116.79) - /unlink 10000 (time 1776191689.90 total 88.00 last 111.84) - /unlink 11112 (time 1776191699.91 total 98.00 last 111.17) - /unlink 12368 (time 1776191709.91 total 108.01 last 125.51) - /unlink 13626 (time 1776191719.91 total 118.01 last 125.79) - /unlink 14911 (time 1776191729.92 total 128.02 last 128.42) - /unlink 16466 (time 1776191739.93 total 138.02 last 155.40) - /unlink 17874 (time 1776191749.93 total 148.03 last 140.76) - /unlink 19151 (time 1776191759.93 total 158.03 last 127.68) - /unlink 20000 (time 1776191767.94 total 166.04 last 105.96) - /unlink 21081 (time 1776191777.96 total 176.05 last 107.96) - /unlink 22252 (time 1776191787.96 total 186.06 last 117.07) - /unlink 23428 (time 1776191797.96 total 196.06 last 117.56) - /unlink 24691 (time 1776191807.97 total 206.06 last 126.25) - /unlink 25868 (time 1776191817.97 total 216.07 last 117.69) - /unlink 27077 (time 1776191827.97 total 226.07 last 120.87) - /unlink 28324 (time 1776191837.97 total 236.07 last 124.69) - /unlink 29579 (time 1776191847.98 total 246.07 last 125.44) - /unlink 30000 (time 1776191851.24 total 249.34 last 129.07) - /unlink 31160 (time 1776191861.24 total 259.34 last 115.92) - /unlink 32412 (time 1776191871.25 total 269.35 last 125.09) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 270.76 seconds: 120.38 ops/second Waiting for MDT destroys to complete - open/close 750 (time 1776191899.36 total 10.01 last 74.95) - open/close 1448 (time 1776191909.36 total 20.02 last 69.74) - open/close 2090 (time 1776191919.38 total 30.03 last 64.11) - open/close 2796 (time 1776191929.40 total 40.06 last 70.41) - open/close 3519 (time 1776191939.41 total 50.06 last 72.23) - open/close 4285 (time 1776191949.43 total 60.09 last 76.44) - open/close 4925 (time 1776191959.44 total 70.09 last 63.97) - open/close 5663 (time 1776191969.45 total 80.10 last 73.76) - open/close 6376 (time 1776191979.47 total 90.12 last 71.13) - open/close 7074 (time 1776191989.48 total 100.14 last 69.69) - open/close 7769 (time 1776191999.49 total 110.14 last 69.49) - open/close 8358 (time 1776192009.49 total 120.14 last 58.86) - open/close 8945 (time 1776192019.50 total 130.15 last 58.69) - open/close 9614 (time 1776192029.50 total 140.15 last 66.86) - open/close 10000 (time 1776192035.88 total 146.53 last 60.50) - open/close 10564 (time 1776192045.89 total 156.54 last 56.37) - open/close 11151 (time 1776192055.89 total 166.54 last 58.67) - open/close 11850 (time 1776192065.91 total 176.56 last 69.81) - open/close 12570 (time 1776192075.91 total 186.57 last 71.94) - open/close 13275 (time 1776192085.93 total 196.58 last 70.40) - open/close 13939 (time 1776192095.93 total 206.59 last 66.36) - open/close 14666 (time 1776192105.95 total 216.60 last 72.61) - open/close 15313 (time 1776192115.96 total 226.61 last 64.60) - open/close 15893 (time 1776192125.98 total 236.63 last 57.92) - open/close 16583 (time 1776192135.98 total 246.63 last 68.97) - open/close 17267 (time 1776192145.99 total 256.64 last 68.37) - open/close 17937 (time 1776192156.00 total 266.65 last 66.93) - open/close 18596 (time 1776192166.00 total 276.65 last 65.87) - open/close 19293 (time 1776192176.01 total 286.66 last 69.67) - open/close 19933 (time 1776192186.02 total 296.67 last 63.89) - open/close 20523 (time 1776192196.04 total 306.69 last 58.91) - open/close 21248 (time 1776192206.05 total 316.70 last 72.41) - open/close 21929 (time 1776192216.06 total 326.71 last 68.01) - open/close 22511 (time 1776192226.06 total 336.71 last 58.19) - open/close 23101 (time 1776192236.07 total 346.72 last 58.96) - open/close 23770 (time 1776192246.09 total 356.74 last 66.76) - open/close 24456 (time 1776192256.11 total 366.76 last 68.49) - open/close 25176 (time 1776192266.12 total 376.77 last 71.92) - open/close 25830 (time 1776192276.12 total 386.77 last 65.39) - open/close 26508 (time 1776192286.13 total 396.78 last 67.77) - open/close 27175 (time 1776192296.14 total 406.79 last 66.63) - open/close 27852 (time 1776192306.14 total 416.79 last 67.67) - open/close 28473 (time 1776192316.15 total 426.81 last 62.01) - open/close 29156 (time 1776192326.17 total 436.82 last 68.22) - open/close 29731 (time 1776192336.18 total 446.83 last 57.40) - open/close 30000 (time 1776192340.57 total 451.22 last 61.37) - open/close 30606 (time 1776192350.58 total 461.23 last 60.54) - open/close 31245 (time 1776192360.59 total 471.24 last 63.82) - open/close 31954 (time 1776192370.60 total 481.25 last 70.81) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 491.24 seconds: 66.35 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 277 79715 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 32851 47141 42% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33128 33128 0 100% /mnt/lustre - /unlink 1222 (time 1776192394.32 total 10.02 last 121.99) - /unlink 2155 (time 1776192404.32 total 20.02 last 93.29) - /unlink 3004 (time 1776192414.33 total 30.02 last 84.88) - /unlink 3928 (time 1776192424.33 total 40.02 last 92.37) - /unlink 4983 (time 1776192434.33 total 50.03 last 105.47) - /unlink 6037 (time 1776192444.33 total 60.03 last 105.38) - /unlink 7306 (time 1776192454.34 total 70.03 last 126.87) - /unlink 8487 (time 1776192464.34 total 80.03 last 118.07) - /unlink 9626 (time 1776192474.34 total 90.04 last 113.83) - /unlink 10000 (time 1776192477.77 total 93.46 last 109.24) - /unlink 11027 (time 1776192487.77 total 103.46 last 102.67) - /unlink 12114 (time 1776192497.78 total 113.47 last 108.61) - /unlink 13329 (time 1776192507.78 total 123.47 last 121.49) - /unlink 14427 (time 1776192517.79 total 133.48 last 109.72) - /unlink 15690 (time 1776192527.80 total 143.49 last 126.16) - /unlink 16796 (time 1776192537.82 total 153.51 last 110.39) - /unlink 17852 (time 1776192547.82 total 163.52 last 105.54) - /unlink 18837 (time 1776192557.83 total 173.52 last 98.47) - /unlink 19804 (time 1776192567.83 total 183.52 last 96.70) - /unlink 20832 (time 1776192577.83 total 193.52 last 102.79) - /unlink 21958 (time 1776192587.83 total 203.53 last 112.53) - /unlink 23050 (time 1776192597.85 total 213.54 last 109.05) - /unlink 24089 (time 1776192607.86 total 223.55 last 103.76) - /unlink 24801 (time 1776192617.86 total 233.55 last 71.19) - /unlink 25730 (time 1776192627.87 total 243.56 last 92.84) - /unlink 26895 (time 1776192637.87 total 253.56 last 116.48) - /unlink 28091 (time 1776192647.87 total 263.57 last 119.55) - /unlink 29254 (time 1776192657.88 total 273.57 last 116.21) - /unlink 30000 (time 1776192664.56 total 280.26 last 111.62) - /unlink 31120 (time 1776192674.57 total 290.26 last 111.98) - /unlink 32385 (time 1776192684.57 total 300.27 last 126.43) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 302.28 seconds: 107.83 ops/second Waiting for MDT destroys to complete - open/close 660 (time 1776192714.44 total 10.01 last 65.91) - open/close 1290 (time 1776192724.44 total 20.02 last 62.99) - open/close 1845 (time 1776192734.44 total 30.02 last 55.48) - open/close 2354 (time 1776192744.46 total 40.03 last 50.82) - open/close 3033 (time 1776192754.48 total 50.05 last 67.76) - open/close 3722 (time 1776192764.48 total 60.06 last 68.88) - open/close 4386 (time 1776192774.49 total 70.06 last 66.35) - open/close 5059 (time 1776192784.49 total 80.07 last 67.27) - open/close 5748 (time 1776192794.49 total 90.07 last 68.90) - open/close 6453 (time 1776192804.51 total 100.09 last 70.38) - open/close 7172 (time 1776192814.52 total 110.09 last 71.87) - open/close 7721 (time 1776192824.53 total 120.10 last 54.82) - open/close 8228 (time 1776192834.54 total 130.12 last 50.63) - open/close 8792 (time 1776192844.55 total 140.13 last 56.36) - open/close 9333 (time 1776192854.55 total 150.13 last 54.09) - open/close 9949 (time 1776192864.56 total 160.13 last 61.57) - open/close 10641 (time 1776192874.56 total 170.14 last 69.15) - open/close 11292 (time 1776192884.58 total 180.15 last 65.02) - open/close 11918 (time 1776192894.58 total 190.16 last 62.57) - open/close 12584 (time 1776192904.60 total 200.17 last 66.52) - open/close 13194 (time 1776192914.60 total 210.18 last 60.94) - open/close 13790 (time 1776192924.61 total 220.18 last 59.57) - open/close 14417 (time 1776192934.61 total 230.18 last 62.70) - open/close 15057 (time 1776192944.62 total 240.19 last 63.94) - open/close 15683 (time 1776192954.63 total 250.20 last 62.56) - open/close 16285 (time 1776192964.63 total 260.20 last 60.18) - open/close 16921 (time 1776192974.64 total 270.22 last 63.51) - open/close 17572 (time 1776192984.65 total 280.22 last 65.05) - open/close 18162 (time 1776192994.65 total 290.23 last 58.98) - open/close 18715 (time 1776193004.66 total 300.23 last 55.29) - open/close 19163 (time 1776193014.67 total 310.24 last 44.74) - open/close 19700 (time 1776193024.67 total 320.24 last 53.69) - open/close 20000 (time 1776193030.47 total 326.04 last 51.71) - open/close 20490 (time 1776193040.50 total 336.08 last 48.84) - open/close 21076 (time 1776193050.51 total 346.08 last 58.57) - open/close 21742 (time 1776193060.51 total 356.09 last 66.58) - open/close 22383 (time 1776193070.52 total 366.09 last 64.04) - open/close 23069 (time 1776193080.53 total 376.11 last 68.52) - open/close 23670 (time 1776193090.54 total 386.11 last 60.05) - open/close 24276 (time 1776193100.56 total 396.14 last 60.45) - open/close 24899 (time 1776193110.58 total 406.16 last 62.18) - open/close 25423 (time 1776193120.60 total 416.18 last 52.31) - open/close 25980 (time 1776193130.61 total 426.19 last 55.63) - open/close 26614 (time 1776193140.63 total 436.20 last 63.31) - open/close 27172 (time 1776193150.64 total 446.21 last 55.76) - open/close 27767 (time 1776193160.64 total 456.22 last 59.46) - open/close 28382 (time 1776193170.65 total 466.23 last 61.43) - open/close 28991 (time 1776193180.67 total 476.24 last 60.80) - open/close 29584 (time 1776193190.68 total 486.25 last 59.24) - open/close 30000 (time 1776193196.92 total 492.49 last 66.69) - open/close 30672 (time 1776193206.93 total 502.51 last 67.11) - open/close 31334 (time 1776193216.95 total 512.53 last 66.05) - open/close 31980 (time 1776193226.97 total 522.54 last 64.51) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 532.38 seconds: 61.22 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 277 79715 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 32851 47141 42% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33128 33128 0 100% /mnt/lustre - /unlink 1184 (time 1776193251.35 total 10.00 last 118.38) - /unlink 2351 (time 1776193261.35 total 20.01 last 116.65) - /unlink 3504 (time 1776193271.36 total 30.02 last 115.18) - /unlink 4395 (time 1776193281.37 total 40.02 last 89.06) - /unlink 5540 (time 1776193291.37 total 50.03 last 114.42) - /unlink 6728 (time 1776193301.38 total 60.03 last 118.73) - /unlink 7911 (time 1776193311.38 total 70.03 last 118.28) - /unlink 9006 (time 1776193321.39 total 80.05 last 109.36) - /unlink 10000 (time 1776193330.15 total 88.81 last 113.46) - /unlink 11043 (time 1776193340.15 total 98.81 last 104.29) - /unlink 12061 (time 1776193350.16 total 108.82 last 101.72) - /unlink 13258 (time 1776193360.17 total 118.82 last 119.66) - /unlink 14386 (time 1776193370.18 total 128.83 last 112.63) - /unlink 15438 (time 1776193380.18 total 138.84 last 105.17) - /unlink 16632 (time 1776193390.19 total 148.84 last 119.34) - /unlink 17796 (time 1776193400.19 total 158.84 last 116.40) - /unlink 18989 (time 1776193410.19 total 168.84 last 119.27) - /unlink 20000 (time 1776193418.60 total 177.25 last 120.24) - /unlink 21102 (time 1776193428.60 total 187.26 last 110.14) - /unlink 22141 (time 1776193438.60 total 197.26 last 103.90) - /unlink 23281 (time 1776193448.61 total 207.27 last 113.93) - /unlink 24183 (time 1776193458.63 total 217.29 last 90.01) - /unlink 25290 (time 1776193468.63 total 227.29 last 110.70) - /unlink 26449 (time 1776193478.63 total 237.29 last 115.89) - /unlink 27673 (time 1776193488.64 total 247.30 last 122.29) - /unlink 28847 (time 1776193498.64 total 257.30 last 117.38) - /unlink 30000 (time 1776193508.49 total 267.14 last 117.11) - /unlink 31214 (time 1776193518.49 total 277.15 last 121.38) - /unlink 32344 (time 1776193528.49 total 287.15 last 112.99) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 289.18 seconds: 112.71 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg301-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server stop mds service on oleg301-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg301-server stop mds service on oleg301-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg301-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing unload_modules_local oleg301-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg301-server: LNET ready to unload oleg301-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2