== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:49:34 (1734259774) Stopping clients: oleg246-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg246-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2 oleg246-server: oleg246-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing load_modules_local oleg246-server: Loading modules from /home/green/git/lustre-release/lustre oleg246-server: detected 4 online CPUs by sysfs oleg246-server: Force libcfs to create 2 CPU partitions oleg246-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg246-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg246-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg246-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg246-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg246-client.virtnet: -o user_xattr,flock 192.168.202.146@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2382 (time 1734259834.63 total 10.00 last 238.18) - open/close 4703 (time 1734259844.64 total 20.00 last 232.05) - open/close 6805 (time 1734259854.64 total 30.00 last 210.18) - open/close 8880 (time 1734259864.64 total 40.01 last 207.42) - open/close 10000 (time 1734259869.58 total 44.94 last 226.91) - open/close 12029 (time 1734259879.58 total 54.95 last 202.84) - open/close 14252 (time 1734259889.58 total 64.95 last 222.30) - open/close 16538 (time 1734259899.58 total 74.95 last 228.59) - open/close 18906 (time 1734259909.58 total 84.95 last 236.77) - open/close 20000 (time 1734259914.64 total 90.01 last 216.15) - open/close 22500 (time 1734259924.64 total 100.01 last 249.93) - open/close 25214 (time 1734259934.64 total 110.01 last 271.38) - open/close 27909 (time 1734259944.65 total 120.01 last 269.49) - open/close 30000 (time 1734259951.71 total 127.08 last 295.81) - open/close 32529 (time 1734259961.72 total 137.08 last 252.87) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 137.21 seconds: 237.30 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5270 (time 1734259972.88 total 10.00 last 526.99) - /unlink 10000 (time 1734259981.33 total 18.46 last 559.20) - /unlink 15841 (time 1734259991.33 total 28.46 last 584.09) - /unlink 20000 (time 1734260000.13 total 37.25 last 472.84) - /unlink 25502 (time 1734260010.13 total 47.26 last 550.13) - /unlink 30000 (time 1734260019.25 total 56.38 last 493.21) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 61.25 seconds: 531.57 ops/second Waiting for MDT destroys to complete oleg246-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg246-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 - open/close 2725 (time 1734260039.76 total 10.00 last 272.43) - open/close 5046 (time 1734260049.76 total 20.00 last 232.09) - open/close 7732 (time 1734260059.76 total 30.01 last 268.53) - open/close 10000 (time 1734260067.21 total 37.45 last 304.52) - open/close 12828 (time 1734260077.21 total 47.46 last 282.66) - open/close 15360 (time 1734260087.21 total 57.46 last 253.19) - open/close 17916 (time 1734260097.21 total 67.46 last 255.56) - open/close 20000 (time 1734260105.63 total 75.88 last 247.66) - open/close 22725 (time 1734260115.63 total 85.88 last 272.48) - open/close 25151 (time 1734260125.63 total 95.88 last 242.56) - open/close 27491 (time 1734260135.63 total 105.88 last 233.96) - open/close 30000 (time 1734260145.62 total 115.86 last 251.31) - open/close 32424 (time 1734260155.62 total 125.86 last 242.35) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 126.36 seconds: 257.68 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32797 47195 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33052 33052 0 100% /mnt/lustre - /unlink 4645 (time 1734260167.33 total 10.00 last 464.42) - /unlink 9047 (time 1734260177.33 total 20.00 last 440.19) - /unlink 10000 (time 1734260179.47 total 22.13 last 447.38) - /unlink 13796 (time 1734260189.47 total 32.13 last 379.56) - /unlink 18116 (time 1734260199.47 total 42.13 last 431.98) - /unlink 20000 (time 1734260203.26 total 45.93 last 496.27) - /unlink 24570 (time 1734260213.26 total 55.93 last 456.99) - /unlink 29030 (time 1734260223.26 total 65.93 last 445.95) - /unlink 30000 (time 1734260225.91 total 68.57 last 367.06) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 75.27 seconds: 432.57 ops/second Waiting for MDT destroys to complete oleg246-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg246-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 - open/close 2420 (time 1734260249.62 total 10.00 last 241.98) - open/close 4780 (time 1734260259.63 total 20.00 last 235.91) - open/close 7255 (time 1734260269.63 total 30.00 last 247.49) - open/close 10000 (time 1734260279.59 total 39.97 last 275.56) - open/close 12723 (time 1734260289.59 total 49.97 last 272.25) - open/close 15288 (time 1734260299.59 total 59.97 last 256.45) - open/close 17472 (time 1734260309.60 total 69.97 last 218.34) - open/close 19760 (time 1734260319.60 total 79.97 last 228.77) - open/close 22181 (time 1734260329.60 total 89.98 last 242.08) - open/close 24424 (time 1734260339.60 total 99.98 last 224.27) - open/close 26596 (time 1734260349.61 total 109.98 last 217.05) - open/close 28853 (time 1734260359.61 total 119.99 last 225.65) - open/close 30000 (time 1734260364.55 total 124.93 last 232.04) - open/close 32418 (time 1734260374.55 total 134.93 last 241.79) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 135.47 seconds: 240.36 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4039 (time 1734260386.17 total 10.00 last 403.88) - /unlink 8382 (time 1734260396.17 total 20.00 last 434.25) - /unlink 10000 (time 1734260399.15 total 22.98 last 542.89) - /unlink 14941 (time 1734260409.16 total 32.98 last 494.00) - /unlink 19560 (time 1734260419.16 total 42.99 last 461.83) - /unlink 24439 (time 1734260429.16 total 52.99 last 487.89) - /unlink 28613 (time 1734260439.16 total 62.99 last 417.34) - /unlink 30000 (time 1734260442.40 total 66.22 last 428.45) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 70.94 seconds: 459.00 ops/second Waiting for MDT destroys to complete oleg246-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg246-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 - open/close 3066 (time 1734260465.23 total 10.00 last 306.52) - open/close 5708 (time 1734260475.23 total 20.01 last 264.12) - open/close 8017 (time 1734260485.23 total 30.01 last 230.83) - open/close 10000 (time 1734260493.45 total 38.22 last 241.50) - open/close 12767 (time 1734260503.45 total 48.22 last 276.67) - open/close 15519 (time 1734260513.45 total 58.22 last 275.19) - open/close 17972 (time 1734260523.45 total 68.22 last 245.23) - open/close 20000 (time 1734260531.48 total 76.26 last 252.43) - open/close 22567 (time 1734260541.49 total 86.26 last 256.64) - open/close 24988 (time 1734260551.49 total 96.26 last 242.04) - open/close 27207 (time 1734260561.49 total 106.26 last 221.88) - open/close 29417 (time 1734260571.49 total 116.27 last 220.97) - open/close 30000 (time 1734260573.65 total 118.42 last 270.14) - open/close 32271 (time 1734260583.65 total 128.43 last 227.06) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 129.75 seconds: 250.94 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5523 (time 1734260596.11 total 10.00 last 552.24) - /unlink 10000 (time 1734260604.58 total 18.47 last 528.65) - /unlink 14047 (time 1734260614.58 total 28.47 last 404.65) - /unlink 19668 (time 1734260624.58 total 38.47 last 562.05) - /unlink 25850 (time 1734260634.58 total 48.47 last 618.15) - /unlink 30000 (time 1734260640.65 total 54.54 last 683.90) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 58.53 seconds: 556.26 ops/second Waiting for MDT destroys to complete oleg246-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 oleg246-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 - open/close 3592 (time 1734260661.94 total 10.00 last 359.16) - open/close 6777 (time 1734260671.94 total 20.00 last 318.49) - open/close 10000 (time 1734260681.29 total 29.35 last 344.90) - open/close 12773 (time 1734260691.29 total 39.35 last 277.26) - open/close 15564 (time 1734260701.29 total 49.35 last 279.09) - open/close 18560 (time 1734260711.29 total 59.35 last 299.53) - open/close 20000 (time 1734260715.53 total 63.59 last 339.79) - open/close 23129 (time 1734260725.53 total 73.59 last 312.86) - open/close 26273 (time 1734260735.54 total 83.59 last 314.28) - open/close 28962 (time 1734260745.54 total 93.60 last 268.76) - open/close 30000 (time 1734260749.44 total 97.50 last 266.38) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 106.56 seconds: 305.56 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5086 (time 1734260769.61 total 10.00 last 508.52) - /unlink 10000 (time 1734260777.66 total 18.05 last 610.71) - /unlink 15653 (time 1734260787.66 total 28.05 last 565.20) - /unlink 20000 (time 1734260795.03 total 35.42 last 589.56) - /unlink 27067 (time 1734260805.03 total 45.42 last 706.63) - /unlink 30000 (time 1734260809.83 total 50.22 last 611.55) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 54.39 seconds: 598.63 ops/second Waiting for MDT destroys to complete oleg246-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg246-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg246-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg246-server unloading modules via unload_modules_local on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg246-client: oleg246-client: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-client: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2