== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:06:07 (1718316367) Stopping clients: oleg127-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg127-client.virtnet /mnt/lustre2 (opts:-f) oleg127-server: oleg127-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg127-server' oleg127-server: oleg127-server.virtnet: executing load_modules_local oleg127-server: Loading modules from /home/green/git/lustre-release/lustre oleg127-server: detected 4 online CPUs by sysfs oleg127-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg127-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg127-server: oleg127-server.virtnet: executing set_default_debug -1 all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg127-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg127-server: oleg127-server.virtnet: executing set_default_debug -1 all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg127-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug -1 all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg127-client.virtnet: -o user_xattr,flock oleg127-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2274 (time 1718316411.25 total 10.00 last 227.33) - open/close 4830 (time 1718316421.26 total 20.01 last 255.49) - open/close 7792 (time 1718316431.26 total 30.01 last 296.14) - open/close 10000 (time 1718316437.61 total 36.36 last 347.68) - open/close 13696 (time 1718316447.61 total 46.36 last 369.57) - open/close 17393 (time 1718316457.61 total 56.36 last 369.70) - open/close 20000 (time 1718316465.55 total 64.30 last 328.53) - open/close 23093 (time 1718316475.55 total 74.30 last 309.27) - open/close 25783 (time 1718316485.55 total 84.30 last 268.99) - open/close 28889 (time 1718316495.55 total 94.30 last 310.52) - open/close 30000 (time 1718316498.34 total 97.09 last 398.07) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 103.44 seconds: 314.76 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7043 (time 1718316515.66 total 10.00 last 704.28) - /unlink 10000 (time 1718316519.50 total 13.84 last 770.13) - /unlink 19480 (time 1718316529.50 total 23.84 last 947.93) - /unlink 28812 (time 1718316539.50 total 33.84 last 933.08) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 38.09 seconds: 854.82 ops/second Waiting for MDT destroys to complete - open/close 5228 (time 1718316583.43 total 10.00 last 522.72) - open/close 10000 (time 1718316591.95 total 18.52 last 560.16) - open/close 15451 (time 1718316601.95 total 28.52 last 545.04) - open/close 20000 (time 1718316609.83 total 36.40 last 577.64) - open/close 25798 (time 1718316619.83 total 46.40 last 579.71) - open/close 30000 (time 1718316629.54 total 56.11 last 432.85) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 61.33 seconds: 530.89 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1718316645.26 total 9.62 last 1039.01) - /unlink 20000 (time 1718316654.41 total 18.78 last 1092.30) - /unlink 30000 (time 1718316663.68 total 28.04 last 1079.35) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 30.48 seconds: 1068.22 ops/second Waiting for MDT destroys to complete - open/close 5409 (time 1718316710.45 total 10.00 last 540.85) - open/close 10000 (time 1718316717.93 total 17.49 last 613.28) - open/close 16422 (time 1718316727.94 total 27.49 last 642.12) - open/close 20000 (time 1718316733.59 total 33.14 last 633.03) - open/close 26081 (time 1718316743.59 total 43.14 last 608.04) - open/close 30000 (time 1718316749.79 total 49.34 last 632.26) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 53.39 seconds: 609.89 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1718316764.31 total 9.67 last 1034.40) - /unlink 20000 (time 1718316774.08 total 19.44 last 1022.83) - /unlink 30000 (time 1718316783.99 total 29.36 last 1008.97) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 31.85 seconds: 1022.40 ops/second Waiting for MDT destroys to complete - open/close 5843 (time 1718316844.60 total 10.00 last 584.26) - open/close 10000 (time 1718316851.47 total 16.87 last 605.51) - open/close 16027 (time 1718316861.47 total 26.87 last 602.63) - open/close 20000 (time 1718316867.82 total 33.22 last 625.40) - open/close 26380 (time 1718316877.82 total 43.22 last 637.97) - open/close 30000 (time 1718316884.50 total 49.90 last 541.96) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 54.93 seconds: 592.74 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32816 47176 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33071 33071 0 100% /mnt/lustre - /unlink 10000 (time 1718316900.32 total 9.81 last 1019.39) - /unlink 20000 (time 1718316909.20 total 18.69 last 1125.88) - /unlink 30000 (time 1718316918.55 total 28.04 last 1070.26) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 30.72 seconds: 1059.89 ops/second Waiting for MDT destroys to complete - open/close 5576 (time 1718316966.81 total 10.00 last 557.53) - open/close 10000 (time 1718316974.36 total 17.55 last 586.06) - open/close 16361 (time 1718316984.36 total 27.55 last 636.07) - open/close 20000 (time 1718316990.12 total 33.30 last 632.53) - open/close 26169 (time 1718317000.12 total 43.30 last 616.82) - open/close 30000 (time 1718317006.93 total 50.11 last 562.56) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 54.41 seconds: 598.39 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1718317020.95 total 8.90 last 1123.99) - /unlink 20000 (time 1718317030.08 total 18.03 last 1095.32) - /unlink 30000 (time 1718317039.17 total 27.12 last 1099.73) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 29.25 seconds: 1113.13 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg127-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg127-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg127-server stop mds service on oleg127-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg127-server stop mds service on oleg127-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg127-server unloading modules on: 'oleg127-server' oleg127-server: oleg127-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg127-client: oleg127-client: ssh exited with exit code 2 pdsh@oleg127-client: oleg127-server: ssh exited with exit code 2 pdsh@oleg127-client: oleg127-client: ssh exited with exit code 2 pdsh@oleg127-client: oleg127-server: ssh exited with exit code 2