== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 04:01:28 (1715587288) Stopping clients: oleg420-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg420-client.virtnet /mnt/lustre2 (opts:-f) oleg420-server: oleg420-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: detected 4 online CPUs by sysfs oleg420-server: Force libcfs to create 2 CPU partitions oleg420-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg420-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg420-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg420-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock oleg420-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5158 (time 1715587326.98 total 10.00 last 515.69) - open/close 10000 (time 1715587335.94 total 18.96 last 540.43) - open/close 15579 (time 1715587345.95 total 28.96 last 557.81) - open/close 20000 (time 1715587354.72 total 37.73 last 504.12) - open/close 25182 (time 1715587364.72 total 47.73 last 518.18) - open/close 30000 (time 1715587373.97 total 56.98 last 520.82) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 62.02 seconds: 525.03 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9499 (time 1715587389.94 total 10.00 last 949.84) - /unlink 18483 (time 1715587399.94 total 20.00 last 898.36) - /unlink 28276 (time 1715587409.94 total 30.00 last 979.24) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 34.48 seconds: 944.30 ops/second Waiting for MDT destroys to complete - open/close 4689 (time 1715587471.86 total 10.00 last 468.81) - open/close 9970 (time 1715587481.86 total 20.00 last 528.08) - open/close 14820 (time 1715587491.86 total 30.00 last 484.92) - open/close 19664 (time 1715587501.86 total 40.01 last 484.32) - open/close 24153 (time 1715587511.86 total 50.01 last 448.80) - open/close 29255 (time 1715587521.86 total 60.01 last 510.12) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 65.91 seconds: 494.02 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7593 (time 1715587538.58 total 10.00 last 759.28) - /unlink 10000 (time 1715587541.25 total 12.67 last 902.05) - /unlink 19366 (time 1715587551.25 total 22.67 last 936.55) - /unlink 30000 (time 1715587560.73 total 32.15 last 1122.21) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 34.69 seconds: 938.60 ops/second Waiting for MDT destroys to complete - open/close 5391 (time 1715587605.65 total 10.00 last 539.00) - open/close 10000 (time 1715587614.52 total 18.87 last 519.57) - open/close 15447 (time 1715587624.53 total 28.87 last 544.64) - open/close 20000 (time 1715587633.85 total 38.20 last 488.04) - open/close 24784 (time 1715587643.86 total 48.20 last 478.33) - open/close 30000 (time 1715587652.63 total 56.98 last 594.53) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 61.45 seconds: 529.90 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9683 (time 1715587667.94 total 10.00 last 968.19) - /unlink 19025 (time 1715587677.94 total 20.00 last 934.12) - /unlink 29925 (time 1715587687.94 total 30.00 last 1089.96) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 32.64 seconds: 997.50 ops/second Waiting for MDT destroys to complete - open/close 4696 (time 1715587733.15 total 10.00 last 469.52) - open/close 9925 (time 1715587743.16 total 20.00 last 522.86) - open/close 15767 (time 1715587753.16 total 30.00 last 584.18) - open/close 20000 (time 1715587760.30 total 37.15 last 592.42) - open/close 25803 (time 1715587770.30 total 47.15 last 580.28) - open/close 30000 (time 1715587778.24 total 55.08 last 528.95) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 59.85 seconds: 544.01 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32746 47246 41% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33001 33001 0 100% /mnt/lustre - /unlink 9624 (time 1715587793.89 total 10.00 last 962.33) - /unlink 19693 (time 1715587803.89 total 20.00 last 1006.88) - /unlink 29104 (time 1715587813.89 total 30.00 last 941.10) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 33.61 seconds: 968.87 ops/second Waiting for MDT destroys to complete - open/close 5574 (time 1715587856.63 total 10.00 last 557.37) - open/close 10000 (time 1715587864.64 total 18.01 last 552.63) - open/close 15432 (time 1715587874.64 total 28.01 last 543.17) - open/close 20000 (time 1715587883.50 total 36.87 last 515.67) - open/close 25263 (time 1715587893.50 total 46.87 last 526.25) - open/close 30000 (time 1715587902.33 total 55.70 last 536.62) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 60.53 seconds: 537.88 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9433 (time 1715587918.09 total 10.00 last 943.24) - /unlink 19368 (time 1715587928.09 total 20.00 last 993.46) - /unlink 29408 (time 1715587938.09 total 30.00 last 1003.80) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 33.21 seconds: 980.31 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg420-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg420-server unloading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg420-client: oleg420-client: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-client: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2