== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:18:43 (1730801923) Stopping clients: oleg225-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg225-client.virtnet /mnt/lustre2 (opts:-f) oleg225-server: oleg225-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing load_modules_local oleg225-server: Loading modules from /home/green/git/lustre-release/lustre oleg225-server: detected 4 online CPUs by sysfs oleg225-server: Force libcfs to create 2 CPU partitions oleg225-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg225-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg225-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg225-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg225-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1459 (time 1730801979.42 total 10.00 last 145.87) - open/close 2818 (time 1730801989.42 total 20.01 last 135.83) - open/close 4316 (time 1730801999.43 total 30.01 last 149.77) - open/close 5798 (time 1730802009.43 total 40.02 last 148.10) - open/close 7262 (time 1730802019.43 total 50.02 last 146.38) - open/close 9263 (time 1730802029.43 total 60.02 last 200.07) - open/close 10000 (time 1730802033.29 total 63.87 last 191.16) - open/close 11456 (time 1730802043.29 total 73.88 last 145.56) - open/close 13175 (time 1730802053.29 total 83.88 last 171.89) - open/close 14671 (time 1730802063.30 total 93.88 last 149.51) - open/close 16371 (time 1730802073.30 total 103.89 last 169.95) - open/close 17931 (time 1730802083.31 total 113.89 last 155.93) - open/close 19528 (time 1730802093.31 total 123.89 last 159.69) - open/close 20000 (time 1730802095.87 total 126.45 last 184.10) - open/close 21599 (time 1730802105.87 total 136.46 last 159.87) - open/close 23125 (time 1730802115.88 total 146.46 last 152.52) - open/close 24719 (time 1730802125.88 total 156.47 last 159.34) - open/close 26424 (time 1730802135.89 total 166.47 last 170.41) - open/close 28111 (time 1730802145.89 total 176.47 last 168.67) - open/close 29864 (time 1730802155.89 total 186.48 last 175.26) - open/close 31768 (time 1730802165.90 total 196.48 last 190.33) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 200.50 seconds: 162.40 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 3383 (time 1730802181.30 total 10.00 last 338.22) - /unlink 7064 (time 1730802191.30 total 20.00 last 368.08) - /unlink 10000 (time 1730802199.43 total 28.13 last 361.19) - /unlink 13085 (time 1730802209.43 total 38.13 last 308.43) - /unlink 16744 (time 1730802219.43 total 48.13 last 365.87) - /unlink 20000 (time 1730802227.95 total 56.65 last 382.25) - /unlink 23202 (time 1730802237.95 total 66.65 last 320.15) - /unlink 26448 (time 1730802247.95 total 76.66 last 324.56) - /unlink 30000 (time 1730802257.83 total 86.53 last 359.71) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 94.35 seconds: 345.09 ops/second Waiting for MDT destroys to complete oleg225-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 oleg225-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 - open/close 1969 (time 1730802285.44 total 10.00 last 196.84) - open/close 3770 (time 1730802295.45 total 20.01 last 180.02) - open/close 5619 (time 1730802305.45 total 30.01 last 184.87) - open/close 7421 (time 1730802315.45 total 40.01 last 180.19) - open/close 9046 (time 1730802325.45 total 50.01 last 162.50) - open/close 10000 (time 1730802330.76 total 55.32 last 179.56) - open/close 11681 (time 1730802340.77 total 65.33 last 168.04) - open/close 13540 (time 1730802350.77 total 75.33 last 185.79) - open/close 16055 (time 1730802360.77 total 85.33 last 251.47) - open/close 19230 (time 1730802370.77 total 95.33 last 317.44) - open/close 20000 (time 1730802372.85 total 97.41 last 371.77) - open/close 22283 (time 1730802382.85 total 107.41 last 228.28) - open/close 24263 (time 1730802392.85 total 117.41 last 197.97) - open/close 26469 (time 1730802402.85 total 127.41 last 220.57) - open/close 28176 (time 1730802412.85 total 137.41 last 170.70) - open/close 30000 (time 1730802421.83 total 146.39 last 203.20) - open/close 31861 (time 1730802431.83 total 156.39 last 186.09) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 160.54 seconds: 202.82 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 3746 (time 1730802447.32 total 10.00 last 374.52) - /unlink 7258 (time 1730802457.32 total 20.00 last 351.17) - /unlink 10000 (time 1730802463.43 total 26.11 last 448.84) - /unlink 15245 (time 1730802473.43 total 36.11 last 524.46) - /unlink 20000 (time 1730802480.57 total 43.25 last 665.79) - /unlink 27023 (time 1730802490.57 total 53.26 last 702.24) - /unlink 30000 (time 1730802495.65 total 58.33 last 586.33) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 62.27 seconds: 522.87 ops/second Waiting for MDT destroys to complete oleg225-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg225-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 - open/close 3328 (time 1730802516.39 total 10.00 last 332.71) - open/close 6246 (time 1730802526.39 total 20.01 last 291.66) - open/close 9950 (time 1730802536.39 total 30.01 last 370.38) - open/close 12621 (time 1730802546.40 total 40.01 last 266.95) - open/close 15321 (time 1730802556.40 total 50.02 last 269.95) - open/close 19608 (time 1730802566.40 total 60.02 last 428.62) - open/close 23266 (time 1730802576.40 total 70.02 last 365.69) - open/close 26045 (time 1730802586.41 total 80.02 last 277.87) - open/close 28795 (time 1730802596.41 total 90.02 last 274.99) - open/close 30000 (time 1730802601.40 total 95.01 last 241.47) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 104.57 seconds: 311.36 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 6112 (time 1730802621.99 total 10.00 last 611.10) - /unlink 10000 (time 1730802628.39 total 16.39 last 608.37) - /unlink 17996 (time 1730802638.39 total 26.39 last 799.59) - /unlink 20000 (time 1730802640.87 total 28.88 last 806.17) - /unlink 25617 (time 1730802650.87 total 38.88 last 561.63) - /unlink 30000 (time 1730802656.87 total 44.88 last 730.94) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 48.55 seconds: 670.71 ops/second Waiting for MDT destroys to complete oleg225-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg225-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 - open/close 5816 (time 1730802701.32 total 10.00 last 581.49) - open/close 10000 (time 1730802709.68 total 18.36 last 500.72) - open/close 14215 (time 1730802719.68 total 28.36 last 421.38) - open/close 18037 (time 1730802729.68 total 38.36 last 382.17) - open/close 20000 (time 1730802733.79 total 42.47 last 477.90) - open/close 25234 (time 1730802743.79 total 52.47 last 523.39) - open/close 30000 (time 1730802752.15 total 60.83 last 569.85) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 65.80 seconds: 494.82 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7865 (time 1730802768.08 total 10.00 last 786.48) - /unlink 10000 (time 1730802770.65 total 12.58 last 828.84) - /unlink 16605 (time 1730802780.65 total 22.58 last 660.47) - /unlink 20000 (time 1730802784.21 total 26.13 last 954.30) - /unlink 29538 (time 1730802794.21 total 36.13 last 953.80) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 40.83 seconds: 797.40 ops/second Waiting for MDT destroys to complete oleg225-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 oleg225-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 - open/close 4746 (time 1730802830.84 total 10.00 last 474.51) - open/close 8326 (time 1730802840.84 total 20.00 last 357.99) - open/close 10000 (time 1730802845.14 total 24.30 last 389.75) - open/close 14843 (time 1730802855.14 total 34.30 last 484.25) - open/close 19361 (time 1730802865.14 total 44.30 last 451.73) - open/close 23710 (time 1730802875.14 total 54.30 last 434.89) - open/close 28452 (time 1730802885.14 total 64.30 last 474.19) - open/close 30000 (time 1730802887.89 total 67.05 last 563.90) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 72.23 seconds: 450.77 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 8998 (time 1730802903.92 total 10.00 last 899.70) - /unlink 18538 (time 1730802913.92 total 20.00 last 954.00) - /unlink 28819 (time 1730802923.92 total 30.00 last 1028.06) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 33.52 seconds: 971.40 ops/second Waiting for MDT destroys to complete oleg225-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg225-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg225-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg225-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server stop mds service on oleg225-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server stop mds service on oleg225-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg225-server unloading modules via unload_modules_local on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2