== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:17:52 (1748989072) Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f) oleg160-server: oleg160-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing load_modules_local oleg160-server: Loading modules from /home/green/git/lustre-release/lustre oleg160-server: detected 4 online CPUs by sysfs oleg160-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg160-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg160-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg160-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg160-client.virtnet: -o user_xattr,flock oleg160-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 5546 (time 1748989113.89 total 10.00 last 554.52) - open/close 10000 (time 1748989122.04 total 18.15 last 546.28) - open/close 15521 (time 1748989132.04 total 28.16 last 552.09) - open/close 20000 (time 1748989140.08 total 36.19 last 557.12) - open/close 25498 (time 1748989150.08 total 46.19 last 549.79) - open/close 30000 (time 1748989158.09 total 54.20 last 562.49) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 59.04 seconds: 552.04 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1748989173.53 total 9.70 last 1030.82) - /unlink 20000 (time 1748989183.41 total 19.58 last 1011.82) - /unlink 30000 (time 1748989193.29 total 29.46 last 1012.12) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 32.01 seconds: 1018.17 ops/second Waiting for MDT destroys to complete - open/close 5622 (time 1748989232.79 total 10.00 last 562.16) - open/close 10000 (time 1748989240.58 total 17.80 last 561.59) - open/close 15356 (time 1748989250.58 total 27.80 last 535.52) - open/close 20000 (time 1748989259.76 total 36.97 last 506.11) - open/close 25083 (time 1748989269.76 total 46.97 last 508.29) - open/close 30000 (time 1748989278.81 total 56.03 last 543.23) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 60.68 seconds: 537.19 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1748989293.88 total 9.55 last 1047.23) - /unlink 20000 (time 1748989303.38 total 19.05 last 1052.00) - /unlink 30000 (time 1748989312.69 total 28.37 last 1073.94) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 30.82 seconds: 1057.58 ops/second Waiting for MDT destroys to complete - open/close 5779 (time 1748989360.37 total 10.00 last 577.89) - open/close 10000 (time 1748989367.64 total 17.27 last 580.49) - open/close 15736 (time 1748989377.64 total 27.27 last 573.53) - open/close 20000 (time 1748989384.99 total 34.63 last 579.77) - open/close 25738 (time 1748989394.99 total 44.63 last 573.72) - open/close 30000 (time 1748989402.45 total 52.08 last 571.72) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 56.59 seconds: 575.98 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 10000 (time 1748989417.30 total 9.49 last 1053.31) - /unlink 20000 (time 1748989426.80 total 18.99 last 1052.74) - /unlink 30000 (time 1748989436.49 total 28.69 last 1031.62) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 30.96 seconds: 1052.77 ops/second Waiting for MDT destroys to complete - open/close 4259 (time 1748989487.85 total 10.00 last 425.76) - open/close 7940 (time 1748989497.85 total 20.00 last 368.05) - open/close 10000 (time 1748989503.30 total 25.45 last 378.14) - open/close 13201 (time 1748989513.30 total 35.45 last 320.01) - open/close 15980 (time 1748989523.30 total 45.46 last 277.90) - open/close 19992 (time 1748989533.30 total 55.46 last 401.14) - open/close 23832 (time 1748989543.30 total 65.46 last 383.99) - open/close 26986 (time 1748989553.31 total 75.46 last 315.29) - open/close 29784 (time 1748989563.31 total 85.46 last 279.76) - open/close 32391 (time 1748989573.31 total 95.46 last 260.66) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 96.23 seconds: 338.70 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 5926 (time 1748989585.36 total 10.00 last 592.56) - /unlink 10000 (time 1748989591.99 total 16.63 last 614.79) - /unlink 16406 (time 1748989601.99 total 26.63 last 640.58) - /unlink 20000 (time 1748989608.17 total 32.81 last 581.48) - /unlink 25946 (time 1748989618.17 total 42.81 last 594.60) - /unlink 30000 (time 1748989623.97 total 48.61 last 698.73) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 52.00 seconds: 626.76 ops/second Waiting for MDT destroys to complete - open/close 3031 (time 1748989658.46 total 10.00 last 303.10) - open/close 6413 (time 1748989668.46 total 20.00 last 338.20) - open/close 9123 (time 1748989678.47 total 30.00 last 270.90) - open/close 10000 (time 1748989681.17 total 32.70 last 324.79) - open/close 12758 (time 1748989691.17 total 42.71 last 275.70) - open/close 15397 (time 1748989701.18 total 52.71 last 263.80) - open/close 18071 (time 1748989711.18 total 62.71 last 267.35) - open/close 20000 (time 1748989718.27 total 69.80 last 272.05) - open/close 23094 (time 1748989728.27 total 79.81 last 309.36) - open/close 26162 (time 1748989738.27 total 89.81 last 306.72) - open/close 28941 (time 1748989748.27 total 99.81 last 277.90) - open/close 30000 (time 1748989752.22 total 103.75 last 268.32) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 113.73 seconds: 286.59 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 5355 (time 1748989773.52 total 10.00 last 535.46) - /unlink 10000 (time 1748989782.08 total 18.57 last 542.22) - /unlink 16044 (time 1748989792.11 total 28.59 last 603.00) - /unlink 20000 (time 1748989798.00 total 34.49 last 670.83) - /unlink 26115 (time 1748989808.01 total 44.49 last 611.45) - /unlink 30000 (time 1748989814.59 total 51.07 last 590.09) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 54.20 seconds: 601.34 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg160-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg160-server LNET ready to unload unloading modules on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing unload_modules_local oleg160-server: LNET ready to unload modules unloaded. pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2