== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:48:37 (1734259717) Stopping clients: oleg327-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg327-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2 oleg327-server: oleg327-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing load_modules_local oleg327-server: Loading modules from /home/green/git/lustre-release/lustre oleg327-server: detected 4 online CPUs by sysfs oleg327-server: Force libcfs to create 2 CPU partitions oleg327-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg327-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg327-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg327-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg327-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg327-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg327-client.virtnet: -o user_xattr,flock 192.168.203.127@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2402 (time 1734259775.08 total 10.00 last 240.14) - open/close 4923 (time 1734259785.08 total 20.00 last 252.08) - open/close 7464 (time 1734259795.08 total 30.01 last 254.02) - open/close 9620 (time 1734259805.08 total 40.01 last 215.56) - open/close 11826 (time 1734259815.08 total 50.01 last 220.59) - open/close 14233 (time 1734259825.09 total 60.01 last 240.62) - open/close 16927 (time 1734259835.09 total 70.01 last 269.38) - open/close 19615 (time 1734259845.09 total 80.02 last 268.73) - open/close 22342 (time 1734259855.09 total 90.02 last 272.61) - open/close 24909 (time 1734259865.09 total 100.02 last 256.69) - open/close 27157 (time 1734259875.09 total 110.02 last 224.78) - open/close 29330 (time 1734259885.10 total 120.02 last 217.27) - open/close 30000 (time 1734259888.32 total 123.25 last 207.84) - open/close 31987 (time 1734259898.32 total 133.25 last 198.68) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 136.00 seconds: 239.42 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 3929 (time 1734259912.35 total 10.00 last 392.86) - /unlink 8385 (time 1734259922.35 total 20.00 last 445.54) - /unlink 10000 (time 1734259925.85 total 23.50 last 462.06) - /unlink 14301 (time 1734259935.85 total 33.50 last 430.03) - /unlink 18384 (time 1734259945.85 total 43.50 last 408.29) - /unlink 20000 (time 1734259949.96 total 47.61 last 393.07) - /unlink 24239 (time 1734259959.96 total 57.61 last 423.83) - /unlink 28319 (time 1734259969.96 total 67.61 last 407.99) - /unlink 30000 (time 1734259973.70 total 71.35 last 450.15) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 77.64 seconds: 419.34 ops/second Waiting for MDT destroys to complete oleg327-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg327-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 - open/close 2614 (time 1734259995.83 total 10.00 last 261.39) - open/close 5654 (time 1734260005.83 total 20.00 last 303.96) - open/close 8653 (time 1734260015.83 total 30.00 last 299.88) - open/close 10000 (time 1734260020.57 total 34.74 last 284.27) - open/close 12847 (time 1734260030.57 total 44.74 last 284.63) - open/close 15748 (time 1734260040.57 total 54.74 last 290.09) - open/close 18501 (time 1734260050.58 total 64.74 last 275.27) - open/close 20000 (time 1734260055.55 total 69.72 last 301.07) - open/close 23784 (time 1734260065.56 total 79.72 last 378.34) - open/close 26827 (time 1734260075.56 total 89.73 last 304.27) - open/close 29805 (time 1734260085.56 total 99.73 last 297.70) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 109.32 seconds: 297.83 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5543 (time 1734260106.17 total 10.00 last 554.25) - /unlink 10000 (time 1734260114.02 total 17.85 last 568.02) - /unlink 15196 (time 1734260124.02 total 27.85 last 519.52) - /unlink 20000 (time 1734260133.52 total 37.35 last 505.65) - /unlink 25208 (time 1734260143.52 total 47.35 last 520.78) - /unlink 30000 (time 1734260151.96 total 55.79 last 568.05) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 60.07 seconds: 542.06 ops/second Waiting for MDT destroys to complete oleg327-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg327-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 - open/close 3338 (time 1734260176.86 total 10.00 last 333.71) - open/close 6611 (time 1734260186.87 total 20.00 last 327.24) - open/close 9967 (time 1734260196.87 total 30.01 last 335.51) - open/close 13010 (time 1734260206.87 total 40.01 last 304.28) - open/close 16122 (time 1734260216.87 total 50.01 last 311.14) - open/close 19121 (time 1734260226.87 total 60.01 last 299.83) - open/close 20000 (time 1734260230.04 total 63.18 last 277.71) - open/close 22838 (time 1734260240.04 total 73.18 last 283.75) - open/close 25781 (time 1734260250.04 total 83.18 last 294.28) - open/close 29140 (time 1734260260.04 total 93.18 last 335.80) - open/close 30000 (time 1734260262.43 total 95.57 last 360.77) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 104.19 seconds: 312.50 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4791 (time 1734260282.10 total 10.00 last 479.09) - /unlink 9917 (time 1734260292.10 total 20.00 last 512.51) - /unlink 14325 (time 1734260302.10 total 30.00 last 440.77) - /unlink 19143 (time 1734260312.11 total 40.00 last 481.74) - /unlink 23895 (time 1734260322.11 total 50.01 last 475.12) - /unlink 29394 (time 1734260332.11 total 60.01 last 549.82) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 66.08 seconds: 492.76 ops/second Waiting for MDT destroys to complete oleg327-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg327-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 - open/close 3113 (time 1734260356.28 total 10.00 last 311.18) - open/close 5865 (time 1734260366.28 total 20.00 last 275.20) - open/close 8597 (time 1734260376.28 total 30.00 last 273.19) - open/close 10000 (time 1734260381.57 total 35.30 last 264.88) - open/close 12572 (time 1734260391.58 total 45.31 last 257.10) - open/close 14968 (time 1734260401.58 total 55.31 last 239.58) - open/close 17213 (time 1734260411.58 total 65.31 last 224.45) - open/close 19779 (time 1734260421.58 total 75.31 last 256.58) - open/close 22309 (time 1734260431.58 total 85.31 last 252.92) - open/close 24738 (time 1734260441.59 total 95.31 last 242.84) - open/close 27182 (time 1734260451.59 total 105.31 last 244.40) - open/close 29755 (time 1734260461.59 total 115.32 last 257.22) - open/close 32550 (time 1734260471.59 total 125.32 last 279.46) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 125.36 seconds: 259.72 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5906 (time 1734260482.64 total 10.00 last 590.54) - /unlink 10000 (time 1734260489.77 total 17.13 last 573.89) - /unlink 15750 (time 1734260499.77 total 27.14 last 574.94) - /unlink 20000 (time 1734260506.45 total 33.82 last 636.22) - /unlink 26079 (time 1734260516.45 total 43.82 last 607.83) - /unlink 30000 (time 1734260522.60 total 49.96 last 638.02) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 53.78 seconds: 605.40 ops/second Waiting for MDT destroys to complete oleg327-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg327-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 - open/close 2904 (time 1734260547.13 total 10.00 last 290.30) - open/close 5772 (time 1734260557.13 total 20.00 last 286.78) - open/close 8670 (time 1734260567.13 total 30.01 last 289.71) - open/close 10000 (time 1734260571.05 total 33.92 last 339.55) - open/close 13132 (time 1734260581.05 total 43.93 last 313.14) - open/close 16543 (time 1734260591.06 total 53.93 last 340.99) - open/close 19688 (time 1734260601.06 total 63.93 last 314.48) - open/close 22607 (time 1734260611.06 total 73.93 last 291.85) - open/close 25798 (time 1734260621.06 total 83.93 last 319.03) - open/close 28864 (time 1734260631.06 total 93.94 last 306.56) - open/close 30000 (time 1734260634.99 total 97.86 last 289.22) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 106.55 seconds: 305.58 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5816 (time 1734260654.71 total 10.00 last 581.53) - /unlink 10000 (time 1734260660.89 total 16.19 last 676.51) - /unlink 16144 (time 1734260670.89 total 26.19 last 614.35) - /unlink 20000 (time 1734260677.70 total 33.00 last 566.05) - /unlink 25947 (time 1734260687.71 total 43.00 last 594.63) - /unlink 30000 (time 1734260694.05 total 49.34 last 639.16) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 53.41 seconds: 609.67 ops/second Waiting for MDT destroys to complete oleg327-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 oleg327-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg327-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg327-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg327-server stop mds service on oleg327-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg327-server stop mds service on oleg327-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg327-server unloading modules via unload_modules_local on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2