== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:42:30 (1734259350) Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 oleg408-server: oleg408-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg408-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg408-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg408-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg408-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4272 (time 1734259402.83 total 10.00 last 427.09) - open/close 8621 (time 1734259412.83 total 20.00 last 434.87) - open/close 10000 (time 1734259416.23 total 23.40 last 405.58) - open/close 14563 (time 1734259426.23 total 33.40 last 456.26) - open/close 19356 (time 1734259436.23 total 43.41 last 479.26) - open/close 24053 (time 1734259446.23 total 53.41 last 469.68) - open/close 29037 (time 1734259456.23 total 63.41 last 498.35) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 71.26 seconds: 456.91 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 8483 (time 1734259475.02 total 10.00 last 848.29) - /unlink 17525 (time 1734259485.02 total 20.00 last 904.20) - /unlink 20000 (time 1734259487.16 total 22.14 last 1154.23) - /unlink 30000 (time 1734259496.06 total 31.04 last 1123.64) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 33.27 seconds: 978.55 ops/second Waiting for MDT destroys to complete oleg408-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg408-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 - open/close 6188 (time 1734259558.08 total 10.00 last 618.73) - open/close 10000 (time 1734259564.01 total 15.94 last 642.07) - open/close 16395 (time 1734259574.01 total 25.94 last 639.46) - open/close 20000 (time 1734259580.00 total 31.93 last 601.71) - open/close 25658 (time 1734259590.01 total 41.93 last 565.76) - open/close 30000 (time 1734259597.17 total 49.09 last 606.10) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 53.14 seconds: 612.73 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1734259611.73 total 9.69 last 1032.36) - /unlink 19452 (time 1734259621.73 total 19.69 last 945.13) - /unlink 29117 (time 1734259631.73 total 29.69 last 966.45) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 33.30 seconds: 977.82 ops/second Waiting for MDT destroys to complete oleg408-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg408-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 - open/close 4775 (time 1734259692.39 total 10.00 last 477.42) - open/close 10000 (time 1734259702.30 total 19.92 last 527.05) - open/close 15795 (time 1734259712.30 total 29.92 last 579.46) - open/close 20000 (time 1734259718.77 total 36.39 last 649.84) - open/close 25892 (time 1734259728.77 total 46.39 last 589.14) - open/close 30000 (time 1734259736.63 total 54.25 last 522.62) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 58.52 seconds: 556.38 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1734259750.58 total 8.86 last 1128.35) - /unlink 20000 (time 1734259759.84 total 18.12 last 1080.51) - /unlink 29699 (time 1734259769.84 total 28.12 last 969.89) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 30.90 seconds: 1053.67 ops/second Waiting for MDT destroys to complete oleg408-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 oleg408-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 - open/close 5724 (time 1734259811.21 total 10.00 last 572.37) - open/close 10000 (time 1734259819.07 total 17.86 last 543.87) - open/close 15512 (time 1734259829.07 total 27.86 last 551.19) - open/close 20000 (time 1734259837.39 total 36.18 last 539.33) - open/close 25865 (time 1734259847.39 total 46.19 last 586.42) - open/close 30000 (time 1734259854.69 total 53.48 last 566.95) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 58.06 seconds: 560.79 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9798 (time 1734259870.14 total 10.00 last 979.73) - /unlink 19811 (time 1734259880.14 total 20.00 last 1001.21) - /unlink 30000 (time 1734259890.11 total 29.97 last 1022.17) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 32.47 seconds: 1002.92 ops/second Waiting for MDT destroys to complete oleg408-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg408-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 - open/close 5502 (time 1734259930.12 total 10.00 last 550.11) - open/close 10000 (time 1734259938.24 total 18.12 last 554.38) - open/close 15755 (time 1734259948.24 total 28.12 last 575.43) - open/close 20000 (time 1734259955.60 total 35.48 last 576.54) - open/close 25632 (time 1734259965.60 total 45.48 last 563.11) - open/close 30000 (time 1734259973.26 total 53.13 last 570.86) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 57.65 seconds: 564.81 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1734259988.61 total 10.00 last 1000.28) - /unlink 20000 (time 1734259998.42 total 19.80 last 1019.63) - /unlink 29978 (time 1734260008.42 total 29.81 last 997.72) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 32.40 seconds: 1004.83 ops/second Waiting for MDT destroys to complete oleg408-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 oleg408-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2