== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:27:18 (1760452038) Stopping clients: oleg333-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg333-client.virtnet /mnt/lustre2 (opts:-f) oleg333-server: oleg333-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing load_modules_local oleg333-server: Loading modules from /home/green/git/lustre-release/lustre oleg333-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg333-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg333-server: detected 4 online CPUs by sysfs oleg333-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg333-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg333-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg333-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg333-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg333-server: oleg333-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg333-client.virtnet: -o user_xattr,flock oleg333-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1455 (time 1760452106.30 total 10.00 last 145.43) - open/close 2872 (time 1760452116.30 total 20.01 last 141.65) - open/close 4252 (time 1760452126.31 total 30.01 last 137.97) - open/close 5755 (time 1760452136.31 total 40.01 last 150.23) - open/close 7643 (time 1760452146.31 total 50.02 last 188.77) - open/close 9132 (time 1760452156.32 total 60.02 last 148.85) - open/close 10000 (time 1760452163.15 total 66.85 last 127.04) - open/close 11160 (time 1760452173.15 total 76.86 last 115.93) - open/close 12378 (time 1760452183.16 total 86.86 last 121.72) - open/close 13560 (time 1760452193.16 total 96.87 last 118.15) - open/close 14888 (time 1760452203.17 total 106.88 last 132.69) - open/close 16649 (time 1760452213.17 total 116.88 last 176.05) - open/close 18510 (time 1760452223.18 total 126.88 last 186.06) - open/close 20000 (time 1760452230.09 total 133.80 last 215.50) - open/close 22078 (time 1760452240.09 total 143.80 last 207.78) - open/close 24626 (time 1760452250.09 total 153.80 last 254.77) - open/close 27214 (time 1760452260.09 total 163.80 last 258.80) - open/close 29853 (time 1760452270.10 total 173.80 last 263.82) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 182.82 seconds: 178.29 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 5149 (time 1760452290.64 total 10.00 last 514.88) - /unlink 10000 (time 1760452300.08 total 19.44 last 513.77) - /unlink 15486 (time 1760452310.08 total 29.44 last 548.53) - /unlink 20000 (time 1760452316.63 total 35.99 last 689.79) - /unlink 25049 (time 1760452326.63 total 45.99 last 504.81) - /unlink 30000 (time 1760452335.61 total 54.97 last 551.08) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 58.24 seconds: 559.69 ops/second Waiting for MDT destroys to complete - open/close 3014 (time 1760452363.96 total 10.00 last 301.34) - open/close 5875 (time 1760452373.96 total 20.00 last 286.07) - open/close 9532 (time 1760452383.96 total 30.00 last 365.66) - open/close 12856 (time 1760452393.96 total 40.00 last 332.38) - open/close 15687 (time 1760452403.96 total 50.01 last 283.01) - open/close 18062 (time 1760452413.96 total 60.01 last 237.43) - open/close 20000 (time 1760452419.88 total 65.93 last 327.68) - open/close 22970 (time 1760452429.88 total 75.93 last 296.94) - open/close 27018 (time 1760452439.88 total 85.93 last 404.72) - open/close 30000 (time 1760452446.51 total 92.56 last 449.69) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 99.06 seconds: 329.04 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 7731 (time 1760452464.45 total 10.00 last 773.03) - /unlink 10000 (time 1760452467.20 total 12.76 last 823.69) - /unlink 17635 (time 1760452477.20 total 22.76 last 763.50) - /unlink 20000 (time 1760452480.41 total 25.97 last 736.04) - /unlink 27594 (time 1760452490.42 total 35.97 last 759.37) - /unlink 30000 (time 1760452493.76 total 39.31 last 719.63) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 42.38 seconds: 769.01 ops/second Waiting for MDT destroys to complete - open/close 4778 (time 1760452546.85 total 10.00 last 477.79) - open/close 9220 (time 1760452556.85 total 20.00 last 444.19) - open/close 13931 (time 1760452566.85 total 30.00 last 471.02) - open/close 18413 (time 1760452576.85 total 40.00 last 448.17) - open/close 20000 (time 1760452580.03 total 43.18 last 499.12) - open/close 24782 (time 1760452590.03 total 53.18 last 478.20) - open/close 29459 (time 1760452600.03 total 63.18 last 467.58) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 69.62 seconds: 468.15 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 8589 (time 1760452617.85 total 10.00 last 858.83) - /unlink 17966 (time 1760452627.85 total 20.00 last 937.59) - /unlink 20000 (time 1760452630.22 total 22.37 last 859.82) - /unlink 29695 (time 1760452640.22 total 32.37 last 969.42) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 35.16 seconds: 927.13 ops/second Waiting for MDT destroys to complete - open/close 5113 (time 1760452680.30 total 10.00 last 511.26) - open/close 10000 (time 1760452689.02 total 18.72 last 560.75) - open/close 15411 (time 1760452699.02 total 28.72 last 541.07) - open/close 20000 (time 1760452708.31 total 38.01 last 494.01) - open/close 24847 (time 1760452718.31 total 48.01 last 484.64) - open/close 29430 (time 1760452728.31 total 58.01 last 458.13) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 65.99 seconds: 493.95 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 9120 (time 1760452747.67 total 10.00 last 911.94) - /unlink 17902 (time 1760452757.67 total 20.00 last 878.18) - /unlink 20000 (time 1760452760.04 total 22.37 last 885.88) - /unlink 28712 (time 1760452770.04 total 32.37 last 871.13) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 37.52 seconds: 868.72 ops/second Waiting for MDT destroys to complete - open/close 5187 (time 1760452794.10 total 10.00 last 518.60) - open/close 10000 (time 1760452803.85 total 19.75 last 493.57) - open/close 15280 (time 1760452813.85 total 29.75 last 527.97) - open/close 19999 (time 1760452823.85 total 39.76 last 471.81) - open/close 24758 (time 1760452833.85 total 49.76 last 475.86) - open/close 30000 (time 1760452843.17 total 59.08 last 562.33) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 63.80 seconds: 510.86 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 9682 (time 1760452859.21 total 10.00 last 968.13) - /unlink 19262 (time 1760452869.21 total 20.00 last 957.99) - /unlink 28402 (time 1760452879.21 total 30.00 last 913.98) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 34.59 seconds: 942.32 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg333-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg333-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg333-server stop mds service on oleg333-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg333-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg333-server' oleg333-server: oleg333-server.virtnet: executing unload_modules_local oleg333-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg333-server: LNET ready to unload oleg333-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-client: ssh exited with exit code 2 pdsh@oleg333-client: oleg333-server: ssh exited with exit code 2