== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 02:16:57 (1733642217) Stopping clients: oleg123-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg123-client.virtnet /mnt/lustre2 (opts:-f) oleg123-server: oleg123-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing load_modules_local oleg123-server: Loading modules from /home/green/git/lustre-release/lustre oleg123-server: detected 4 online CPUs by sysfs oleg123-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg123-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg123-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg123-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5687 (time 1733642255.13 total 10.00 last 568.61) - open/close 10000 (time 1733642262.63 total 17.50 last 574.98) - open/close 15154 (time 1733642272.63 total 27.50 last 515.35) - open/close 20000 (time 1733642281.55 total 36.42 last 543.31) - open/close 25441 (time 1733642291.55 total 46.42 last 544.06) - open/close 30000 (time 1733642299.78 total 54.65 last 554.48) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 59.31 seconds: 548.97 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1733642315.15 total 9.82 last 1018.17) - /unlink 20000 (time 1733642324.82 total 19.49 last 1034.76) - /unlink 30000 (time 1733642334.69 total 29.36 last 1013.22) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 31.86 seconds: 1022.07 ops/second Waiting for MDT destroys to complete oleg123-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg123-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 - open/close 5744 (time 1733642381.09 total 10.00 last 574.32) - open/close 10000 (time 1733642388.63 total 17.54 last 564.26) - open/close 15454 (time 1733642398.63 total 27.54 last 545.36) - open/close 20000 (time 1733642406.41 total 35.32 last 584.33) - open/close 26290 (time 1733642416.41 total 45.33 last 628.92) - open/close 30000 (time 1733642421.86 total 50.77 last 680.94) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 54.90 seconds: 593.13 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1733642435.74 total 8.97 last 1114.63) - /unlink 20000 (time 1733642444.75 total 17.98 last 1110.04) - /unlink 30000 (time 1733642453.47 total 26.71 last 1145.84) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 28.85 seconds: 1128.63 ops/second Waiting for MDT destroys to complete oleg123-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg123-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 - open/close 5556 (time 1733642497.25 total 10.00 last 555.54) - open/close 10000 (time 1733642505.23 total 17.98 last 556.82) - open/close 15667 (time 1733642515.23 total 27.98 last 566.63) - open/close 20000 (time 1733642522.92 total 35.68 last 563.19) - open/close 25454 (time 1733642532.92 total 45.68 last 545.34) - open/close 30000 (time 1733642541.60 total 54.35 last 524.16) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 59.00 seconds: 551.84 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9973 (time 1733642557.16 total 10.00 last 997.28) - /unlink 20000 (time 1733642567.02 total 19.86 last 1016.76) - /unlink 29990 (time 1733642577.02 total 29.86 last 998.92) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 32.45 seconds: 1003.49 ops/second Waiting for MDT destroys to complete oleg123-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg123-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 - open/close 5941 (time 1733642639.24 total 10.00 last 594.05) - open/close 10000 (time 1733642645.98 total 16.74 last 602.24) - open/close 15913 (time 1733642655.98 total 26.74 last 591.25) - open/close 20000 (time 1733642663.10 total 33.87 last 573.73) - open/close 25894 (time 1733642673.10 total 43.87 last 589.36) - open/close 30000 (time 1733642680.05 total 50.81 last 591.02) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 55.22 seconds: 589.68 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1733642695.20 total 9.90 last 1010.26) - /unlink 20000 (time 1733642704.85 total 19.55 last 1035.97) - /unlink 30000 (time 1733642714.29 total 28.99 last 1059.31) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 31.39 seconds: 1037.16 ops/second Waiting for MDT destroys to complete oleg123-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg123-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 - open/close 5768 (time 1733642747.46 total 10.00 last 576.71) - open/close 10000 (time 1733642754.83 total 17.37 last 574.51) - open/close 15749 (time 1733642764.83 total 27.37 last 574.81) - open/close 20000 (time 1733642772.30 total 34.84 last 569.10) - open/close 25728 (time 1733642782.30 total 44.84 last 572.75) - open/close 30000 (time 1733642789.90 total 52.45 last 561.55) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 56.85 seconds: 572.70 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1733642805.01 total 9.82 last 1017.87) - /unlink 20000 (time 1733642814.73 total 19.54 last 1028.96) - /unlink 30000 (time 1733642824.25 total 29.06 last 1050.49) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 31.45 seconds: 1035.22 ops/second Waiting for MDT destroys to complete oleg123-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg123-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg123-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg123-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server stop mds service on oleg123-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg123-server stop mds service on oleg123-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg123-server unloading modules via unload_modules_local on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg123-client: oleg123-client: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-client: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 2