== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:32:46 (1751319166) Stopping clients: oleg256-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg256-client.virtnet /mnt/lustre2 (opts:-f) oleg256-server: oleg256-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing load_modules_local oleg256-server: Loading modules from /home/green/git/lustre-release/lustre oleg256-server: detected 4 online CPUs by sysfs oleg256-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg256-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg256-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg256-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg256-client.virtnet: -o user_xattr,flock oleg256-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 6358 (time 1751319205.86 total 10.00 last 635.70) - open/close 10000 (time 1751319212.05 total 16.19 last 588.46) - open/close 16391 (time 1751319222.05 total 26.19 last 638.98) - open/close 20000 (time 1751319227.72 total 31.86 last 636.96) - open/close 26526 (time 1751319237.72 total 41.86 last 652.53) - open/close 30000 (time 1751319243.26 total 47.40 last 627.44) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 51.67 seconds: 630.78 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1751319257.11 total 8.77 last 1140.36) - /unlink 20000 (time 1751319265.73 total 17.39 last 1159.69) - /unlink 30000 (time 1751319273.88 total 25.54 last 1227.76) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 28.07 seconds: 1161.25 ops/second Waiting for MDT destroys to complete - open/close 7232 (time 1751319340.80 total 10.00 last 723.14) - open/close 10000 (time 1751319345.62 total 14.82 last 573.85) - open/close 15610 (time 1751319355.62 total 24.83 last 560.96) - open/close 20000 (time 1751319363.94 total 33.14 last 527.69) - open/close 25695 (time 1751319373.94 total 43.14 last 569.49) - open/close 30000 (time 1751319381.48 total 50.69 last 570.75) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 55.55 seconds: 586.77 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1751319397.13 total 9.85 last 1015.47) - /unlink 20000 (time 1751319406.78 total 19.50 last 1035.94) - /unlink 30000 (time 1751319416.24 total 28.96 last 1057.44) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 31.42 seconds: 1037.39 ops/second Waiting for MDT destroys to complete - open/close 5862 (time 1751319461.09 total 10.00 last 586.19) - open/close 10000 (time 1751319468.35 total 17.26 last 570.36) - open/close 15592 (time 1751319478.35 total 27.26 last 559.14) - open/close 20000 (time 1751319486.17 total 35.08 last 563.67) - open/close 25527 (time 1751319496.17 total 45.08 last 552.59) - open/close 30000 (time 1751319504.47 total 53.38 last 538.67) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 58.07 seconds: 561.24 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1751319519.72 total 9.66 last 1035.26) - /unlink 20000 (time 1751319529.33 total 19.27 last 1040.90) - /unlink 30000 (time 1751319538.49 total 28.43 last 1091.41) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 30.84 seconds: 1056.99 ops/second Waiting for MDT destroys to complete - open/close 6050 (time 1751319586.66 total 10.00 last 604.99) - open/close 10000 (time 1751319593.39 total 16.73 last 587.19) - open/close 16059 (time 1751319603.39 total 26.73 last 605.89) - open/close 20000 (time 1751319609.95 total 33.29 last 600.74) - open/close 25873 (time 1751319619.95 total 43.29 last 587.23) - open/close 30000 (time 1751319626.85 total 50.19 last 598.38) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 54.48 seconds: 598.25 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1751319641.27 total 9.28 last 1077.13) - /unlink 20000 (time 1751319649.83 total 17.84 last 1168.19) - /unlink 30000 (time 1751319658.73 total 26.74 last 1123.92) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 28.90 seconds: 1127.64 ops/second Waiting for MDT destroys to complete - open/close 6146 (time 1751319703.89 total 10.00 last 614.59) - open/close 10000 (time 1751319710.10 total 16.21 last 620.45) - open/close 16220 (time 1751319720.10 total 26.21 last 621.93) - open/close 20000 (time 1751319726.41 total 32.53 last 598.81) - open/close 25846 (time 1751319736.42 total 42.53 last 584.50) - open/close 30000 (time 1751319743.33 total 49.44 last 600.57) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 53.98 seconds: 603.87 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1751319757.83 total 9.07 last 1102.48) - /unlink 20000 (time 1751319766.87 total 18.12 last 1105.45) - /unlink 30000 (time 1751319775.58 total 26.83 last 1147.85) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 29.07 seconds: 1121.08 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg256-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg256-server LNET ready to unload unloading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing unload_modules_local oleg256-server: LNET ready to unload modules unloaded. pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2