== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:06:55 (1761322015) Stopping clients: oleg219-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg219-client.virtnet /mnt/lustre2 (opts:-f) oleg219-server: oleg219-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing load_modules_local oleg219-server: Loading modules from /home/green/git/lustre-release/lustre oleg219-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg219-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg219-server: detected 4 online CPUs by sysfs oleg219-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg219-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg219-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg219-client.virtnet: -o user_xattr,flock oleg219-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 268 32628 1% /mnt/lustre[OST:0] filesystem_summary: 32900 272 32628 1% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1181 (time 1761322075.37 total 10.01 last 118.01) - open/close 2415 (time 1761322085.37 total 20.01 last 123.37) - open/close 3863 (time 1761322095.38 total 30.01 last 144.73) - open/close 5465 (time 1761322105.38 total 40.02 last 160.10) - open/close 7211 (time 1761322115.39 total 50.02 last 174.53) - open/close 8961 (time 1761322125.39 total 60.03 last 174.93) - open/close 10000 (time 1761322133.03 total 67.67 last 135.99) - open/close 11250 (time 1761322143.04 total 77.67 last 124.93) - open/close 12547 (time 1761322153.04 total 87.68 last 129.67) - open/close 14152 (time 1761322163.04 total 97.68 last 160.49) - open/close 15652 (time 1761322173.04 total 107.68 last 149.93) - open/close 16989 (time 1761322183.05 total 117.69 last 133.64) - open/close 18437 (time 1761322193.05 total 127.69 last 144.79) - open/close 20000 (time 1761322202.98 total 137.62 last 157.43) - open/close 21822 (time 1761322212.98 total 147.62 last 182.17) - open/close 23332 (time 1761322222.99 total 157.62 last 150.88) - open/close 24814 (time 1761322232.99 total 167.63 last 148.06) - open/close 26342 (time 1761322243.00 total 177.64 last 152.78) - open/close 28444 (time 1761322253.00 total 187.64 last 210.18) - open/close 30000 (time 1761322259.17 total 193.81 last 251.88) - open/close 32217 (time 1761322269.18 total 203.82 last 221.62) open(/mnt/lustre/d106.conf-sanity/f-32660) error: No space left on device total: 32660 open/close in 205.75 seconds: 158.73 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32933 47059 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32933 32933 0 100% /mnt/lustre - /unlink 5860 (time 1761322282.87 total 10.00 last 585.98) - /unlink 10000 (time 1761322289.63 total 16.76 last 612.07) - /unlink 15049 (time 1761322299.63 total 26.77 last 504.76) - /unlink 19523 (time 1761322309.63 total 36.77 last 447.35) - /unlink 24819 (time 1761322319.64 total 46.77 last 529.47) - /unlink 29731 (time 1761322329.64 total 56.77 last 491.18) unlink(/mnt/lustre/d106.conf-sanity/f-32660) error: No such file or directory total: 32660 /unlink in 62.59 seconds: 521.81 ops/second Waiting for MDT destroys to complete - open/close 2733 (time 1761322356.16 total 10.00 last 273.26) - open/close 6040 (time 1761322366.16 total 20.00 last 330.65) - open/close 9679 (time 1761322376.16 total 30.00 last 363.89) - open/close 12982 (time 1761322386.16 total 40.01 last 330.22) - open/close 16337 (time 1761322396.16 total 50.01 last 335.39) - open/close 19947 (time 1761322406.17 total 60.01 last 360.94) - open/close 23093 (time 1761322416.17 total 70.01 last 314.57) - open/close 26194 (time 1761322426.17 total 80.01 last 310.04) - open/close 29759 (time 1761322436.17 total 90.02 last 356.44) open(/mnt/lustre/d106.conf-sanity/f-32660) error: No space left on device total: 32660 open/close in 98.53 seconds: 331.49 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32933 47059 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32933 32933 0 100% /mnt/lustre - /unlink 6244 (time 1761322456.21 total 10.00 last 624.35) - /unlink 10000 (time 1761322462.57 total 16.36 last 590.61) - /unlink 16897 (time 1761322472.57 total 26.36 last 689.65) - /unlink 20000 (time 1761322476.95 total 30.74 last 708.03) - /unlink 26498 (time 1761322486.95 total 40.74 last 649.72) - /unlink 30000 (time 1761322491.95 total 45.74 last 701.03) unlink(/mnt/lustre/d106.conf-sanity/f-32660) error: No such file or directory total: 32660 /unlink in 49.04 seconds: 665.99 ops/second Waiting for MDT destroys to complete - open/close 3055 (time 1761322534.76 total 10.00 last 305.45) - open/close 6460 (time 1761322544.76 total 20.00 last 340.46) - open/close 9397 (time 1761322554.76 total 30.01 last 293.63) - open/close 12159 (time 1761322564.76 total 40.01 last 276.17) - open/close 15465 (time 1761322574.77 total 50.01 last 330.54) - open/close 19121 (time 1761322584.77 total 60.01 last 365.51) - open/close 20000 (time 1761322587.34 total 62.58 last 342.00) - open/close 23490 (time 1761322597.34 total 72.58 last 348.94) - open/close 27203 (time 1761322607.34 total 82.58 last 371.26) - open/close 30000 (time 1761322615.89 total 91.13 last 327.23) open(/mnt/lustre/d106.conf-sanity/f-32660) error: No space left on device total: 32660 open/close in 98.57 seconds: 331.33 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32933 47059 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32933 32933 0 100% /mnt/lustre - /unlink 6245 (time 1761322634.86 total 10.00 last 624.46) - /unlink 10000 (time 1761322641.42 total 16.56 last 572.79) - /unlink 15851 (time 1761322651.42 total 26.56 last 585.02) - /unlink 20000 (time 1761322657.30 total 32.44 last 705.19) - /unlink 26424 (time 1761322667.30 total 42.44 last 642.37) - /unlink 30000 (time 1761322672.22 total 47.36 last 727.62) unlink(/mnt/lustre/d106.conf-sanity/f-32660) error: No such file or directory total: 32660 /unlink in 50.46 seconds: 647.25 ops/second Waiting for MDT destroys to complete - open/close 3610 (time 1761322697.94 total 10.00 last 360.99) - open/close 7298 (time 1761322707.95 total 20.00 last 368.73) - open/close 10000 (time 1761322716.01 total 28.07 last 335.07) - open/close 13790 (time 1761322726.01 total 38.07 last 378.90) - open/close 17456 (time 1761322736.01 total 48.07 last 366.54) - open/close 20000 (time 1761322741.99 total 54.05 last 425.38) - open/close 24001 (time 1761322752.00 total 64.05 last 400.06) - open/close 28161 (time 1761322762.00 total 74.05 last 416.00) - open/close 30000 (time 1761322766.18 total 78.24 last 439.53) open(/mnt/lustre/d106.conf-sanity/f-32660) error: No space left on device total: 32660 open/close in 84.35 seconds: 387.19 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32933 47059 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32933 32933 0 100% /mnt/lustre - /unlink 8041 (time 1761322783.77 total 10.00 last 804.08) - /unlink 10000 (time 1761322786.14 total 12.37 last 828.25) - /unlink 18687 (time 1761322796.14 total 22.37 last 868.69) - /unlink 25055 (time 1761322806.14 total 32.37 last 636.64) - /unlink 27098 (time 1761322816.14 total 42.37 last 204.26) - /unlink 29669 (time 1761322826.14 total 52.37 last 257.09) - /unlink 31908 (time 1761322836.15 total 62.37 last 223.84) unlink(/mnt/lustre/d106.conf-sanity/f-32660) error: No such file or directory total: 32660 /unlink in 65.97 seconds: 495.07 ops/second Waiting for MDT destroys to complete - open/close 902 (time 1761322865.50 total 10.00 last 90.19) - open/close 1858 (time 1761322875.51 total 20.01 last 95.54) - open/close 3051 (time 1761322885.51 total 30.01 last 119.22) - open/close 4372 (time 1761322895.51 total 40.02 last 132.08) - open/close 5521 (time 1761322905.52 total 50.02 last 114.87) - open/close 6757 (time 1761322915.53 total 60.03 last 123.46) - open/close 7888 (time 1761322925.53 total 70.03 last 113.08) - open/close 9138 (time 1761322935.54 total 80.04 last 124.88) - open/close 10000 (time 1761322942.18 total 86.68 last 129.76) - open/close 11573 (time 1761322952.18 total 96.69 last 157.28) - open/close 13251 (time 1761322962.18 total 106.69 last 167.80) - open/close 15062 (time 1761322972.18 total 116.69 last 181.09) - open/close 16614 (time 1761322982.19 total 126.69 last 155.16) - open/close 18798 (time 1761322992.19 total 136.69 last 218.40) - open/close 19795 (time 1761323002.19 total 146.69 last 99.70) - open/close 21074 (time 1761323012.19 total 156.69 last 127.89) - open/close 22525 (time 1761323022.19 total 166.69 last 145.06) - open/close 23858 (time 1761323032.19 total 176.70 last 133.25) - open/close 24923 (time 1761323042.20 total 186.70 last 106.49) - open/close 25960 (time 1761323052.20 total 196.70 last 103.68) - open/close 27010 (time 1761323062.21 total 206.71 last 104.92) - open/close 27998 (time 1761323072.21 total 216.71 last 98.79) - open/close 28950 (time 1761323082.21 total 226.71 last 95.17) - open/close 29915 (time 1761323092.22 total 236.72 last 96.39) - open/close 30927 (time 1761323102.23 total 246.73 last 101.15) - open/close 31842 (time 1761323112.23 total 256.73 last 91.47) open(/mnt/lustre/d106.conf-sanity/f-32660) error: No space left on device total: 32660 open/close in 264.22 seconds: 123.61 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32933 47059 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32933 32933 0 100% /mnt/lustre - /unlink 1868 (time 1761323132.24 total 10.00 last 186.79) - /unlink 3546 (time 1761323142.24 total 20.00 last 167.77) - /unlink 5349 (time 1761323152.25 total 30.01 last 180.17) - /unlink 7327 (time 1761323162.25 total 40.01 last 197.70) - /unlink 9159 (time 1761323172.26 total 50.02 last 183.12) - /unlink 10000 (time 1761323177.16 total 54.92 last 171.62) - /unlink 11587 (time 1761323187.16 total 64.92 last 158.68) - /unlink 13337 (time 1761323197.16 total 74.92 last 174.93) - /unlink 15179 (time 1761323207.17 total 84.93 last 184.03) - /unlink 16784 (time 1761323217.18 total 94.94 last 160.41) - /unlink 18259 (time 1761323227.19 total 104.95 last 147.34) - /unlink 19793 (time 1761323237.19 total 114.95 last 153.40) - /unlink 21502 (time 1761323247.20 total 124.96 last 170.77) - /unlink 23292 (time 1761323257.20 total 134.96 last 178.97) - /unlink 25055 (time 1761323267.20 total 144.96 last 176.22) - /unlink 27137 (time 1761323277.20 total 154.96 last 208.19) - /unlink 28874 (time 1761323287.21 total 164.97 last 173.69) - /unlink 30000 (time 1761323294.69 total 172.45 last 150.50) - /unlink 31755 (time 1761323304.69 total 182.45 last 175.42) unlink(/mnt/lustre/d106.conf-sanity/f-32660) error: No such file or directory total: 32660 /unlink in 186.60 seconds: 175.02 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg219-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg219-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server stop mds service on oleg219-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing unload_modules_local oleg219-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg219-server: LNET ready to unload oleg219-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg219-client: oleg219-client: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-client: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2