== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:05:51 (1735099551) Stopping clients: oleg305-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg305-client.virtnet /mnt/lustre2 (opts:-f) oleg305-server: oleg305-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg305-server' oleg305-server: oleg305-server.virtnet: executing load_modules_local oleg305-server: Loading modules from /home/green/git/lustre-release/lustre oleg305-server: detected 4 online CPUs by sysfs oleg305-server: MODOPTS_LIBCFS= oleg305-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg305-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg305-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg305-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3622 (time 1735099596.88 total 10.00 last 362.13) - open/close 6738 (time 1735099606.88 total 20.00 last 311.57) - open/close 10000 (time 1735099616.85 total 29.98 last 327.00) - open/close 12733 (time 1735099626.86 total 39.98 last 273.18) - open/close 15196 (time 1735099636.86 total 49.98 last 246.27) - open/close 17690 (time 1735099646.86 total 59.98 last 249.38) - open/close 20000 (time 1735099653.62 total 66.75 last 341.42) - open/close 22827 (time 1735099663.63 total 76.75 last 282.63) - open/close 25815 (time 1735099673.63 total 86.75 last 298.77) - open/close 28959 (time 1735099683.63 total 96.75 last 314.36) - open/close 30000 (time 1735099686.12 total 99.24 last 418.76) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 105.57 seconds: 308.43 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7555 (time 1735099703.41 total 10.00 last 755.42) - /unlink 10000 (time 1735099706.12 total 12.71 last 901.32) - /unlink 17520 (time 1735099716.13 total 22.71 last 751.95) - /unlink 20000 (time 1735099719.43 total 26.02 last 749.35) - /unlink 28610 (time 1735099729.44 total 36.02 last 860.98) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 41.19 seconds: 790.55 ops/second Waiting for MDT destroys to complete oleg305-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg305-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 - open/close 4683 (time 1735099767.80 total 10.00 last 468.22) - open/close 10000 (time 1735099777.01 total 19.21 last 577.71) - open/close 13978 (time 1735099787.01 total 29.21 last 397.78) - open/close 17932 (time 1735099797.01 total 39.21 last 395.30) - open/close 20000 (time 1735099803.33 total 45.53 last 327.15) - open/close 24182 (time 1735099813.34 total 55.53 last 418.05) - open/close 27588 (time 1735099823.34 total 65.53 last 340.58) - open/close 30000 (time 1735099830.46 total 72.66 last 338.56) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 81.96 seconds: 397.24 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5649 (time 1735099850.89 total 10.00 last 564.82) - /unlink 10000 (time 1735099858.50 total 17.62 last 571.46) - /unlink 16000 (time 1735099868.51 total 27.62 last 599.83) - /unlink 20000 (time 1735099872.79 total 31.90 last 933.61) - /unlink 27203 (time 1735099882.79 total 41.90 last 720.27) - /unlink 30000 (time 1735099888.15 total 47.26 last 521.97) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 51.88 seconds: 627.56 ops/second Waiting for MDT destroys to complete oleg305-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg305-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 - open/close 4167 (time 1735099917.34 total 10.00 last 416.64) - open/close 8972 (time 1735099927.34 total 20.00 last 480.40) - open/close 10000 (time 1735099929.69 total 22.35 last 438.68) - open/close 13643 (time 1735099939.69 total 32.35 last 364.30) - open/close 17804 (time 1735099949.69 total 42.35 last 416.05) - open/close 20000 (time 1735099955.47 total 48.13 last 379.67) - open/close 23754 (time 1735099965.47 total 58.13 last 375.39) - open/close 28146 (time 1735099975.47 total 68.13 last 439.16) - open/close 30000 (time 1735099980.34 total 73.00 last 381.22) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 78.80 seconds: 413.18 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4120 (time 1735099997.11 total 10.00 last 411.93) - /unlink 6215 (time 1735100007.12 total 20.01 last 209.43) - /unlink 8812 (time 1735100017.13 total 30.02 last 259.44) - /unlink 10000 (time 1735100021.42 total 34.31 last 276.85) - /unlink 14164 (time 1735100031.42 total 44.31 last 416.35) - /unlink 18241 (time 1735100041.42 total 54.31 last 407.67) - /unlink 20000 (time 1735100045.11 total 58.00 last 477.11) - /unlink 23944 (time 1735100055.11 total 68.00 last 394.32) - /unlink 27618 (time 1735100065.11 total 78.00 last 367.37) - /unlink 30000 (time 1735100071.16 total 84.05 last 393.50) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 90.50 seconds: 359.78 ops/second Waiting for MDT destroys to complete oleg305-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg305-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 - open/close 3641 (time 1735100095.80 total 10.00 last 364.09) - open/close 6600 (time 1735100105.80 total 20.00 last 295.89) - open/close 9895 (time 1735100115.80 total 30.00 last 329.45) - open/close 13913 (time 1735100125.81 total 40.00 last 401.71) - open/close 17411 (time 1735100135.81 total 50.01 last 349.75) - open/close 20000 (time 1735100143.18 total 57.38 last 351.03) - open/close 22563 (time 1735100153.18 total 67.38 last 256.27) - open/close 25159 (time 1735100163.19 total 77.38 last 259.55) - open/close 27890 (time 1735100173.19 total 87.38 last 273.08) - open/close 30000 (time 1735100182.77 total 96.97 last 220.10) - open/close 32171 (time 1735100192.77 total 106.97 last 217.08) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 108.62 seconds: 299.77 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5570 (time 1735100205.73 total 10.00 last 556.99) - /unlink 10000 (time 1735100212.96 total 17.23 last 613.14) - /unlink 15593 (time 1735100222.96 total 27.23 last 559.20) - /unlink 20000 (time 1735100232.39 total 36.66 last 467.15) - /unlink 27299 (time 1735100242.40 total 46.66 last 729.74) - /unlink 30000 (time 1735100248.63 total 52.90 last 433.20) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 58.83 seconds: 553.44 ops/second Waiting for MDT destroys to complete oleg305-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg305-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 - open/close 3258 (time 1735100272.83 total 10.00 last 325.75) - open/close 7634 (time 1735100282.83 total 20.00 last 437.52) - open/close 10000 (time 1735100290.22 total 27.39 last 320.50) - open/close 14674 (time 1735100300.22 total 37.39 last 467.31) - open/close 20000 (time 1735100309.96 total 47.13 last 546.65) - open/close 25736 (time 1735100319.96 total 57.13 last 573.59) - open/close 30000 (time 1735100327.86 total 65.03 last 540.11) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 70.03 seconds: 464.93 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9278 (time 1735100343.74 total 10.00 last 927.76) - /unlink 19103 (time 1735100353.75 total 20.01 last 981.33) - /unlink 29185 (time 1735100363.75 total 30.01 last 1008.12) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 33.29 seconds: 978.17 ops/second Waiting for MDT destroys to complete oleg305-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 oleg305-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg305-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg305-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg305-server stop mds service on oleg305-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg305-server stop mds service on oleg305-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg305-server unloading modules via unload_modules_local on: 'oleg305-server' oleg305-server: oleg305-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg305-client: oleg305-client: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-client: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2