== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:26:00 (1734258360) Stopping clients: oleg401-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg401-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg401-client: oleg401-server: ssh exited with exit code 2 oleg401-server: oleg401-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg401-server' oleg401-server: oleg401-server.virtnet: executing load_modules_local oleg401-server: Loading modules from /home/green/git/lustre-release/lustre oleg401-server: detected 4 online CPUs by sysfs oleg401-server: Force libcfs to create 2 CPU partitions oleg401-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg401-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg401-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg401-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 302 32594 1% /mnt/lustre[OST:0] filesystem_summary: 32866 272 32594 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3706 (time 1734258407.89 total 10.00 last 370.55) - open/close 7221 (time 1734258417.89 total 20.00 last 351.49) - open/close 10000 (time 1734258427.03 total 29.14 last 304.15) - open/close 13346 (time 1734258437.03 total 39.14 last 334.51) - open/close 15778 (time 1734258447.03 total 49.14 last 243.14) - open/close 17932 (time 1734258457.03 total 59.15 last 215.33) - open/close 20000 (time 1734258465.26 total 67.38 last 251.25) - open/close 23039 (time 1734258475.27 total 77.38 last 303.83) - open/close 25924 (time 1734258485.27 total 87.38 last 288.45) - open/close 28781 (time 1734258495.27 total 97.39 last 285.54) - open/close 30000 (time 1734258500.06 total 102.18 last 254.52) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 111.33 seconds: 293.06 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 5171 (time 1734258520.37 total 10.00 last 516.99) - /unlink 10000 (time 1734258530.07 total 19.71 last 497.43) - /unlink 16141 (time 1734258540.07 total 29.71 last 614.09) - /unlink 20000 (time 1734258547.65 total 37.28 last 509.48) - /unlink 25166 (time 1734258557.65 total 47.29 last 516.53) - /unlink 30000 (time 1734258566.96 total 56.60 last 518.98) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 59.70 seconds: 546.49 ops/second Waiting for MDT destroys to complete oleg401-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg401-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 - open/close 3256 (time 1734258590.64 total 10.00 last 325.48) - open/close 6047 (time 1734258600.64 total 20.00 last 279.10) - open/close 8653 (time 1734258610.64 total 30.01 last 260.53) - open/close 10000 (time 1734258615.82 total 35.19 last 259.92) - open/close 12508 (time 1734258625.82 total 45.19 last 250.77) - open/close 15953 (time 1734258635.82 total 55.19 last 344.48) - open/close 19037 (time 1734258645.82 total 65.19 last 308.36) - open/close 20000 (time 1734258649.44 total 68.81 last 266.47) - open/close 23204 (time 1734258659.44 total 78.81 last 320.39) - open/close 26574 (time 1734258669.44 total 88.81 last 336.96) - open/close 30000 (time 1734258678.60 total 97.96 last 374.12) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 106.22 seconds: 307.15 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 5871 (time 1734258698.01 total 10.00 last 587.07) - /unlink 10000 (time 1734258706.32 total 18.32 last 496.53) - /unlink 14621 (time 1734258716.32 total 28.32 last 462.07) - /unlink 20000 (time 1734258725.73 total 37.72 last 572.17) - /unlink 26222 (time 1734258735.73 total 47.72 last 622.16) - /unlink 30000 (time 1734258740.68 total 52.67 last 763.01) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 56.58 seconds: 576.60 ops/second Waiting for MDT destroys to complete oleg401-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg401-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 - open/close 4026 (time 1734258767.65 total 10.00 last 402.57) - open/close 7425 (time 1734258777.65 total 20.00 last 339.86) - open/close 10000 (time 1734258784.39 total 26.74 last 382.37) - open/close 13288 (time 1734258794.39 total 36.74 last 328.77) - open/close 16248 (time 1734258804.39 total 46.74 last 295.99) - open/close 19263 (time 1734258814.39 total 56.74 last 301.45) - open/close 20000 (time 1734258817.27 total 59.61 last 256.34) - open/close 23279 (time 1734258827.27 total 69.62 last 327.83) - open/close 26734 (time 1734258837.27 total 79.62 last 345.28) - open/close 29063 (time 1734258847.28 total 89.63 last 232.78) - open/close 30000 (time 1734258850.73 total 93.08 last 271.63) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 102.15 seconds: 319.40 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 6238 (time 1734258870.88 total 10.00 last 623.76) - /unlink 10000 (time 1734258876.04 total 15.17 last 728.45) - /unlink 17276 (time 1734258886.04 total 25.17 last 727.57) - /unlink 20000 (time 1734258889.89 total 29.01 last 708.00) - /unlink 26634 (time 1734258899.89 total 39.01 last 663.29) - /unlink 30000 (time 1734258904.73 total 43.86 last 695.38) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 47.31 seconds: 689.66 ops/second Waiting for MDT destroys to complete oleg401-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg401-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 - open/close 4270 (time 1734258947.89 total 10.00 last 426.99) - open/close 8337 (time 1734258957.89 total 20.00 last 406.68) - open/close 10000 (time 1734258961.95 total 24.06 last 409.81) - open/close 13919 (time 1734258971.95 total 34.06 last 391.80) - open/close 17695 (time 1734258981.95 total 44.06 last 377.56) - open/close 20000 (time 1734258988.60 total 50.72 last 346.31) - open/close 23400 (time 1734258998.61 total 60.72 last 339.96) - open/close 27000 (time 1734259008.61 total 70.72 last 359.92) - open/close 30000 (time 1734259016.31 total 78.42 last 389.54) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 86.42 seconds: 377.53 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 7158 (time 1734259035.25 total 10.00 last 715.73) - /unlink 10000 (time 1734259039.47 total 14.21 last 674.51) - /unlink 16506 (time 1734259049.47 total 24.21 last 650.57) - /unlink 20000 (time 1734259054.30 total 29.05 last 722.43) - /unlink 26522 (time 1734259064.31 total 39.05 last 652.14) - /unlink 30000 (time 1734259068.62 total 43.36 last 806.70) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 46.99 seconds: 694.26 ops/second Waiting for MDT destroys to complete oleg401-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg401-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 - open/close 3344 (time 1734259101.54 total 10.00 last 334.37) - open/close 7698 (time 1734259111.54 total 20.00 last 435.39) - open/close 10000 (time 1734259116.97 total 25.44 last 423.22) - open/close 14460 (time 1734259126.98 total 35.44 last 445.95) - open/close 19387 (time 1734259136.98 total 45.44 last 492.69) - open/close 23801 (time 1734259146.98 total 55.44 last 441.35) - open/close 28791 (time 1734259156.98 total 65.44 last 498.91) - open/close 30000 (time 1734259159.61 total 68.07 last 460.04) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 73.69 seconds: 442.75 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 10000 (time 1734259176.06 total 9.97 last 1002.64) - /unlink 19164 (time 1734259186.06 total 19.97 last 916.32) - /unlink 29852 (time 1734259196.06 total 29.97 last 1068.76) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 32.58 seconds: 1001.38 ops/second Waiting for MDT destroys to complete oleg401-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg401-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg401-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg401-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg401-server stop mds service on oleg401-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg401-server unloading modules via unload_modules_local on: 'oleg401-server' oleg401-server: oleg401-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg401-client: oleg401-client: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-server: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-client: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-server: ssh exited with exit code 2