== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:50:26 (1736985026) Stopping clients: oleg455-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg455-client.virtnet /mnt/lustre2 (opts:-f) oleg455-server: oleg455-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing load_modules_local oleg455-server: Loading modules from /home/green/git/lustre-release/lustre oleg455-server: detected 4 online CPUs by sysfs oleg455-server: Force libcfs to create 2 CPU partitions oleg455-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg455-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg455-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg455-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg455-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg455-server: oleg455-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg455-client.virtnet: -o user_xattr,flock oleg455-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1757 (time 1736985084.43 total 10.00 last 175.69) - open/close 3733 (time 1736985094.44 total 20.00 last 197.55) - open/close 5553 (time 1736985104.44 total 30.01 last 181.92) - open/close 7495 (time 1736985114.44 total 40.01 last 194.16) - open/close 9441 (time 1736985124.45 total 50.02 last 194.50) - open/close 10000 (time 1736985127.63 total 53.19 last 175.95) - open/close 11595 (time 1736985137.63 total 63.19 last 159.49) - open/close 13086 (time 1736985147.63 total 73.19 last 149.07) - open/close 14571 (time 1736985157.63 total 83.20 last 148.47) - open/close 16033 (time 1736985167.64 total 93.20 last 146.12) - open/close 17546 (time 1736985177.64 total 103.21 last 151.25) - open/close 19234 (time 1736985187.64 total 113.21 last 168.78) - open/close 20000 (time 1736985191.27 total 116.84 last 210.84) - open/close 21989 (time 1736985201.27 total 126.84 last 198.89) - open/close 23714 (time 1736985211.28 total 136.84 last 172.45) - open/close 25512 (time 1736985221.28 total 146.85 last 179.67) - open/close 27150 (time 1736985231.29 total 156.86 last 163.66) - open/close 29119 (time 1736985241.29 total 166.86 last 196.90) - open/close 30000 (time 1736985245.81 total 171.37 last 195.16) - open/close 31835 (time 1736985255.81 total 181.37 last 183.46) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 185.03 seconds: 176.15 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 4016 (time 1736985270.76 total 10.00 last 401.49) - /unlink 7749 (time 1736985280.76 total 20.00 last 373.25) - /unlink 10000 (time 1736985287.21 total 26.46 last 348.80) - /unlink 13388 (time 1736985297.21 total 36.46 last 338.76) - /unlink 16853 (time 1736985307.21 total 46.46 last 346.42) - /unlink 20000 (time 1736985316.41 total 55.65 last 342.34) - /unlink 23261 (time 1736985326.41 total 65.66 last 326.02) - /unlink 26261 (time 1736985336.41 total 75.66 last 299.95) - /unlink 29920 (time 1736985346.41 total 85.66 last 365.80) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 92.62 seconds: 351.89 ops/second Waiting for MDT destroys to complete - open/close 1895 (time 1736985371.49 total 10.00 last 189.44) - open/close 3789 (time 1736985381.49 total 20.01 last 189.36) - open/close 5540 (time 1736985391.49 total 30.01 last 175.06) - open/close 7554 (time 1736985401.49 total 40.01 last 201.40) - open/close 9774 (time 1736985411.49 total 50.01 last 221.90) - open/close 11881 (time 1736985421.49 total 60.01 last 210.68) - open/close 15030 (time 1736985431.50 total 70.01 last 314.88) - open/close 18047 (time 1736985441.50 total 80.01 last 301.69) - open/close 20000 (time 1736985448.59 total 87.10 last 275.47) - open/close 22738 (time 1736985458.59 total 97.11 last 273.73) - open/close 25256 (time 1736985468.59 total 107.11 last 251.80) - open/close 28319 (time 1736985478.59 total 117.11 last 306.20) - open/close 30000 (time 1736985482.61 total 121.13 last 418.11) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 128.35 seconds: 253.95 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 6388 (time 1736985500.82 total 10.00 last 638.78) - /unlink 10000 (time 1736985506.63 total 15.81 last 621.83) - /unlink 15596 (time 1736985516.63 total 25.81 last 559.56) - /unlink 20000 (time 1736985523.86 total 33.04 last 609.04) - /unlink 27575 (time 1736985533.87 total 43.04 last 757.46) - /unlink 30000 (time 1736985537.91 total 47.08 last 599.96) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 51.62 seconds: 631.44 ops/second Waiting for MDT destroys to complete - open/close 3100 (time 1736985562.48 total 10.00 last 309.98) - open/close 6933 (time 1736985572.49 total 20.00 last 383.22) - open/close 9860 (time 1736985582.49 total 30.00 last 292.64) - open/close 13848 (time 1736985592.49 total 40.01 last 398.72) - open/close 17194 (time 1736985602.49 total 50.01 last 334.51) - open/close 20000 (time 1736985610.53 total 58.05 last 349.15) - open/close 23223 (time 1736985620.53 total 68.05 last 322.24) - open/close 25876 (time 1736985630.54 total 78.05 last 265.17) - open/close 28451 (time 1736985640.54 total 88.06 last 257.44) - open/close 30000 (time 1736985646.06 total 93.58 last 280.40) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 102.27 seconds: 318.71 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 6079 (time 1736985665.82 total 10.00 last 607.84) - /unlink 10000 (time 1736985670.80 total 14.98 last 787.61) - /unlink 16630 (time 1736985680.80 total 24.98 last 663.00) - /unlink 20000 (time 1736985686.46 total 30.64 last 595.56) - /unlink 27233 (time 1736985696.46 total 40.64 last 723.27) - /unlink 30000 (time 1736985700.86 total 45.04 last 628.95) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 49.28 seconds: 661.35 ops/second Waiting for MDT destroys to complete - open/close 3435 (time 1736985733.64 total 10.00 last 343.43) - open/close 7161 (time 1736985743.64 total 20.00 last 372.55) - open/close 10000 (time 1736985750.68 total 27.04 last 403.46) - open/close 14302 (time 1736985760.68 total 37.04 last 430.12) - open/close 17192 (time 1736985770.68 total 47.04 last 288.98) - open/close 19986 (time 1736985780.68 total 57.04 last 279.38) - open/close 23097 (time 1736985790.69 total 67.04 last 311.07) - open/close 26176 (time 1736985800.69 total 77.04 last 307.90) - open/close 28862 (time 1736985810.69 total 87.05 last 268.54) - open/close 30000 (time 1736985814.18 total 90.54 last 326.19) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 98.70 seconds: 330.22 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 4302 (time 1736985833.38 total 10.00 last 430.16) - /unlink 10000 (time 1736985842.88 total 19.50 last 599.70) - /unlink 15607 (time 1736985852.88 total 29.51 last 560.54) - /unlink 20000 (time 1736985862.14 total 38.77 last 474.32) - /unlink 26187 (time 1736985872.15 total 48.77 last 618.64) - /unlink 30000 (time 1736985877.38 total 54.00 last 728.32) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 57.81 seconds: 563.81 ops/second Waiting for MDT destroys to complete - open/close 3421 (time 1736985901.32 total 10.00 last 341.99) - open/close 6797 (time 1736985911.32 total 20.00 last 337.56) - open/close 10000 (time 1736985920.37 total 29.05 last 354.02) - open/close 13898 (time 1736985930.37 total 39.05 last 389.80) - open/close 18330 (time 1736985940.37 total 49.05 last 443.09) - open/close 20000 (time 1736985944.98 total 53.67 last 362.02) - open/close 23560 (time 1736985954.98 total 63.67 last 355.99) - open/close 26927 (time 1736985964.98 total 73.67 last 336.65) - open/close 30000 (time 1736985973.78 total 82.47 last 349.12) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 90.75 seconds: 359.16 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 7341 (time 1736985993.12 total 10.00 last 734.05) - /unlink 10000 (time 1736985997.16 total 14.04 last 658.00) - /unlink 17946 (time 1736986007.16 total 24.04 last 794.57) - /unlink 20000 (time 1736986009.43 total 26.32 last 902.24) - /unlink 28583 (time 1736986019.43 total 36.32 last 858.27) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 40.92 seconds: 796.57 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg455-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg455-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg455-server stop mds service on oleg455-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg455-server stop mds service on oleg455-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg455-server LNET ready to unload unloading modules on: 'oleg455-server' oleg455-server: oleg455-server.virtnet: executing unload_modules_local oleg455-server: LNET ready to unload modules unloaded. pdsh@oleg455-client: oleg455-client: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-client: ssh exited with exit code 2 pdsh@oleg455-client: oleg455-server: ssh exited with exit code 2