== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:53:21 (1747828401) Stopping clients: oleg149-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg149-client.virtnet /mnt/lustre2 (opts:-f) oleg149-server: oleg149-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg149-server' oleg149-server: oleg149-server.virtnet: executing load_modules_local oleg149-server: Loading modules from /home/green/git/lustre-release/lustre oleg149-server: detected 4 online CPUs by sysfs oleg149-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg149-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg149-server: oleg149-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg149-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg149-server: oleg149-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg149-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg149-server: oleg149-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg149-client: oleg149-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg149-client.virtnet: -o user_xattr,flock oleg149-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 3899 (time 1747828446.58 total 10.00 last 389.82) - open/close 9221 (time 1747828456.58 total 20.00 last 532.09) - open/close 14602 (time 1747828466.58 total 30.00 last 538.05) - open/close 20000 (time 1747828475.94 total 39.36 last 576.87) - open/close 25460 (time 1747828485.94 total 49.36 last 545.99) - open/close 30000 (time 1747828493.95 total 57.38 last 566.53) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 62.14 seconds: 524.54 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1747828508.74 total 9.15 last 1092.43) - /unlink 19828 (time 1747828518.74 total 19.15 last 982.74) - /unlink 28715 (time 1747828528.74 total 29.16 last 888.62) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 33.01 seconds: 987.51 ops/second Waiting for MDT destroys to complete - open/close 5147 (time 1747828570.33 total 10.00 last 514.68) - open/close 10000 (time 1747828579.59 total 19.26 last 524.03) - open/close 15442 (time 1747828589.59 total 29.26 last 544.17) - open/close 20000 (time 1747828597.95 total 37.62 last 545.18) - open/close 25621 (time 1747828607.95 total 47.62 last 562.05) - open/close 30000 (time 1747828615.06 total 54.74 last 615.73) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 58.66 seconds: 555.60 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1747828629.24 total 9.45 last 1057.93) - /unlink 20000 (time 1747828637.58 total 17.79 last 1200.06) - /unlink 30000 (time 1747828646.68 total 26.88 last 1099.04) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 29.47 seconds: 1105.99 ops/second Waiting for MDT destroys to complete - open/close 5734 (time 1747828710.99 total 10.00 last 573.39) - open/close 10000 (time 1747828718.21 total 17.22 last 591.03) - open/close 15830 (time 1747828728.21 total 27.22 last 582.93) - open/close 20000 (time 1747828735.42 total 34.44 last 577.81) - open/close 26152 (time 1747828745.42 total 44.44 last 615.19) - open/close 30000 (time 1747828751.72 total 50.73 last 611.01) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 54.93 seconds: 593.32 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1747828766.09 total 9.31 last 1074.64) - /unlink 20000 (time 1747828774.86 total 18.08 last 1140.12) - /unlink 30000 (time 1747828782.77 total 25.99 last 1264.25) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 28.09 seconds: 1160.22 ops/second Waiting for MDT destroys to complete - open/close 5679 (time 1747828825.89 total 10.00 last 567.89) - open/close 10000 (time 1747828834.06 total 18.17 last 529.13) - open/close 15698 (time 1747828844.06 total 28.17 last 569.77) - open/close 20000 (time 1747828851.58 total 35.69 last 571.87) - open/close 25805 (time 1747828861.58 total 45.69 last 580.50) - open/close 30000 (time 1747828869.03 total 53.14 last 563.30) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 57.79 seconds: 563.99 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1747828883.89 total 9.34 last 1070.97) - /unlink 20000 (time 1747828893.20 total 18.64 last 1074.60) - /unlink 30000 (time 1747828902.84 total 28.28 last 1037.20) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 30.71 seconds: 1061.35 ops/second Waiting for MDT destroys to complete - open/close 5802 (time 1747828950.07 total 10.00 last 580.16) - open/close 10000 (time 1747828957.60 total 17.53 last 557.90) - open/close 15802 (time 1747828967.60 total 27.53 last 580.12) - open/close 20000 (time 1747828974.78 total 34.71 last 584.78) - open/close 25732 (time 1747828984.78 total 44.71 last 573.20) - open/close 30000 (time 1747828992.24 total 52.17 last 571.55) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 56.77 seconds: 574.15 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1747829006.97 total 9.23 last 1082.94) - /unlink 20000 (time 1747829016.02 total 18.29 last 1104.58) - /unlink 30000 (time 1747829024.92 total 27.19 last 1123.64) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 29.46 seconds: 1106.54 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg149-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg149-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg149-server stop mds service on oleg149-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg149-server stop mds service on oleg149-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg149-server LNET ready to unload unloading modules on: 'oleg149-server' oleg149-server: oleg149-server.virtnet: executing unload_modules_local oleg149-server: LNET ready to unload modules unloaded. pdsh@oleg149-client: oleg149-client: ssh exited with exit code 2 pdsh@oleg149-client: oleg149-server: ssh exited with exit code 2 pdsh@oleg149-client: oleg149-client: ssh exited with exit code 2 pdsh@oleg149-client: oleg149-server: ssh exited with exit code 2