== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:30:46 (1731346246) Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f) oleg160-server: oleg160-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing load_modules_local oleg160-server: Loading modules from /home/green/git/lustre-release/lustre oleg160-server: detected 4 online CPUs by sysfs oleg160-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg160-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg160-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg160-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg160-client.virtnet: -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5108 (time 1731346285.65 total 10.00 last 510.71) - open/close 8559 (time 1731346295.65 total 20.00 last 345.06) - open/close 10000 (time 1731346299.73 total 24.08 last 353.70) - open/close 13100 (time 1731346309.73 total 34.08 last 309.95) - open/close 16309 (time 1731346319.73 total 44.08 last 320.87) - open/close 18736 (time 1731346329.74 total 54.08 last 242.60) - open/close 20000 (time 1731346334.50 total 58.84 last 265.51) - open/close 22762 (time 1731346344.50 total 68.84 last 276.19) - open/close 25745 (time 1731346354.50 total 78.85 last 298.24) - open/close 28500 (time 1731346364.50 total 88.85 last 275.45) - open/close 30000 (time 1731346369.88 total 94.22 last 278.99) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 103.34 seconds: 315.07 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7897 (time 1731346390.25 total 10.00 last 789.65) - /unlink 10000 (time 1731346392.48 total 12.23 last 944.83) - /unlink 19665 (time 1731346402.48 total 22.23 last 966.46) - /unlink 29433 (time 1731346412.48 total 32.23 last 976.73) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 36.23 seconds: 898.78 ops/second Waiting for MDT destroys to complete oleg160-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg160-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 - open/close 5537 (time 1731346450.83 total 10.00 last 553.67) - open/close 10000 (time 1731346459.05 total 18.21 last 543.41) - open/close 13575 (time 1731346469.05 total 28.22 last 357.28) - open/close 15754 (time 1731346479.06 total 38.22 last 217.86) - open/close 18596 (time 1731346489.06 total 48.22 last 284.15) - open/close 20000 (time 1731346492.88 total 52.05 last 367.23) - open/close 24257 (time 1731346502.88 total 62.05 last 425.64) - open/close 27919 (time 1731346512.89 total 72.05 last 365.97) - open/close 30000 (time 1731346521.14 total 80.30 last 252.23) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 87.59 seconds: 371.74 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 8679 (time 1731346539.36 total 10.00 last 867.83) - /unlink 18279 (time 1731346549.36 total 20.00 last 959.96) - /unlink 20000 (time 1731346551.71 total 22.36 last 729.63) - /unlink 29742 (time 1731346561.71 total 32.36 last 974.16) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 34.95 seconds: 931.51 ops/second Waiting for MDT destroys to complete oleg160-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg160-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 - open/close 4920 (time 1731346619.91 total 10.00 last 491.98) - open/close 9982 (time 1731346629.91 total 20.00 last 506.11) - open/close 15231 (time 1731346639.91 total 30.00 last 524.82) - open/close 20000 (time 1731346648.29 total 38.38 last 569.19) - open/close 25569 (time 1731346658.29 total 48.38 last 556.89) - open/close 30000 (time 1731346666.74 total 56.83 last 524.35) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 62.25 seconds: 523.07 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1731346682.91 total 9.81 last 1019.87) - /unlink 19635 (time 1731346692.91 total 19.81 last 963.43) - /unlink 29342 (time 1731346702.91 total 29.81 last 970.61) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 33.12 seconds: 983.12 ops/second Waiting for MDT destroys to complete oleg160-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg160-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 - open/close 5303 (time 1731346749.09 total 10.00 last 530.28) - open/close 10000 (time 1731346757.68 total 18.59 last 546.55) - open/close 15491 (time 1731346767.69 total 28.60 last 549.00) - open/close 20000 (time 1731346775.89 total 36.80 last 549.80) - open/close 25366 (time 1731346785.89 total 46.80 last 536.54) - open/close 30000 (time 1731346794.52 total 55.43 last 537.11) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 60.09 seconds: 541.90 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9692 (time 1731346810.07 total 10.00 last 969.18) - /unlink 19283 (time 1731346820.07 total 20.00 last 958.99) - /unlink 28907 (time 1731346830.07 total 30.00 last 962.32) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 33.79 seconds: 963.68 ops/second Waiting for MDT destroys to complete oleg160-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg160-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 - open/close 5359 (time 1731346874.52 total 10.00 last 535.86) - open/close 10000 (time 1731346882.85 total 18.33 last 557.11) - open/close 15613 (time 1731346892.85 total 28.33 last 561.21) - open/close 20000 (time 1731346900.70 total 36.18 last 558.86) - open/close 25546 (time 1731346910.70 total 46.18 last 554.59) - open/close 30000 (time 1731346918.66 total 54.14 last 559.59) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 58.86 seconds: 553.17 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 8646 (time 1731346934.22 total 10.00 last 864.58) - /unlink 17904 (time 1731346944.22 total 20.00 last 925.76) - /unlink 20000 (time 1731346946.56 total 22.34 last 895.40) - /unlink 29953 (time 1731346956.56 total 32.34 last 995.23) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 34.98 seconds: 930.82 ops/second Waiting for MDT destroys to complete oleg160-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg160-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg160-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg160-server unloading modules via unload_modules_local on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2