== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:24:46 (1736263486) Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) oleg408-server: oleg408-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg408-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg408-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg408-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock oleg408-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1816 (time 1736263549.69 total 10.00 last 181.52) - open/close 3642 (time 1736263559.69 total 20.00 last 182.60) - open/close 5495 (time 1736263569.69 total 30.01 last 185.26) - open/close 7318 (time 1736263579.69 total 40.01 last 182.29) - open/close 9098 (time 1736263589.70 total 50.01 last 177.96) - open/close 10000 (time 1736263595.40 total 55.71 last 158.18) - open/close 11527 (time 1736263605.40 total 65.72 last 152.61) - open/close 13102 (time 1736263615.41 total 75.72 last 157.49) - open/close 14571 (time 1736263625.41 total 85.72 last 146.84) - open/close 16195 (time 1736263635.41 total 95.72 last 162.40) - open/close 17835 (time 1736263645.41 total 105.72 last 163.97) - open/close 19389 (time 1736263655.41 total 115.73 last 155.37) - open/close 20000 (time 1736263658.76 total 119.07 last 182.80) - open/close 22014 (time 1736263668.76 total 129.07 last 201.32) - open/close 24455 (time 1736263678.76 total 139.07 last 244.09) - open/close 26928 (time 1736263688.76 total 149.08 last 247.23) - open/close 29281 (time 1736263698.77 total 159.08 last 235.20) - open/close 30000 (time 1736263702.09 total 162.40 last 216.64) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 172.12 seconds: 189.37 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32785 47207 41% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33040 33040 0 100% /mnt/lustre - /unlink 4062 (time 1736263722.88 total 10.00 last 406.09) - /unlink 7942 (time 1736263732.88 total 20.00 last 387.91) - /unlink 10000 (time 1736263738.40 total 25.53 last 372.73) - /unlink 13924 (time 1736263748.40 total 35.53 last 392.40) - /unlink 17682 (time 1736263758.40 total 45.53 last 375.79) - /unlink 20000 (time 1736263764.16 total 51.28 last 402.63) - /unlink 23996 (time 1736263774.16 total 61.29 last 399.54) - /unlink 27582 (time 1736263784.16 total 71.29 last 358.56) - /unlink 30000 (time 1736263790.76 total 77.89 last 366.29) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 84.09 seconds: 387.63 ops/second Waiting for MDT destroys to complete - open/close 1972 (time 1736263818.44 total 10.00 last 197.17) - open/close 4507 (time 1736263828.44 total 20.00 last 253.47) - open/close 6807 (time 1736263838.44 total 30.00 last 230.00) - open/close 9186 (time 1736263848.44 total 40.00 last 237.87) - open/close 10000 (time 1736263851.38 total 42.94 last 277.15) - open/close 12749 (time 1736263861.38 total 52.94 last 274.80) - open/close 15363 (time 1736263871.38 total 62.95 last 261.30) - open/close 17623 (time 1736263881.39 total 72.95 last 225.92) - open/close 20000 (time 1736263890.13 total 81.70 last 271.75) - open/close 22924 (time 1736263900.13 total 91.70 last 292.38) - open/close 25842 (time 1736263910.14 total 101.70 last 291.75) - open/close 29064 (time 1736263920.14 total 111.70 last 322.19) - open/close 30000 (time 1736263922.60 total 114.17 last 379.75) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 122.06 seconds: 267.04 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 6013 (time 1736263941.58 total 10.00 last 601.23) - /unlink 10000 (time 1736263950.89 total 19.31 last 428.40) - /unlink 14769 (time 1736263960.89 total 29.31 last 476.86) - /unlink 19716 (time 1736263970.89 total 39.31 last 494.58) - /unlink 24312 (time 1736263980.89 total 49.31 last 459.56) - /unlink 28683 (time 1736263990.89 total 59.31 last 437.08) - /unlink 30000 (time 1736263993.67 total 62.09 last 473.33) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 66.28 seconds: 491.76 ops/second Waiting for MDT destroys to complete - open/close 2895 (time 1736264016.69 total 10.00 last 289.41) - open/close 5516 (time 1736264026.69 total 20.01 last 262.04) - open/close 8236 (time 1736264036.69 total 30.01 last 271.96) - open/close 10000 (time 1736264042.86 total 36.17 last 286.12) - open/close 12709 (time 1736264052.86 total 46.17 last 270.85) - open/close 15292 (time 1736264062.86 total 56.18 last 258.22) - open/close 17634 (time 1736264072.87 total 66.18 last 234.14) - open/close 20000 (time 1736264081.37 total 74.68 last 278.27) - open/close 22562 (time 1736264091.37 total 84.68 last 256.18) - open/close 25122 (time 1736264101.37 total 94.68 last 255.97) - open/close 27469 (time 1736264111.37 total 104.69 last 234.66) - open/close 29874 (time 1736264121.37 total 114.69 last 240.45) - open/close 32567 (time 1736264131.37 total 124.69 last 269.27) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 124.78 seconds: 261.21 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 4623 (time 1736264142.64 total 10.00 last 462.24) - /unlink 9421 (time 1736264152.64 total 20.00 last 479.73) - /unlink 14324 (time 1736264162.65 total 30.00 last 490.27) - /unlink 19168 (time 1736264172.65 total 40.00 last 484.34) - /unlink 20000 (time 1736264174.79 total 42.15 last 387.29) - /unlink 25152 (time 1736264184.80 total 52.15 last 515.12) - /unlink 30000 (time 1736264192.87 total 60.22 last 600.78) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 65.25 seconds: 499.54 ops/second Waiting for MDT destroys to complete - open/close 3348 (time 1736264217.81 total 10.00 last 334.71) - open/close 6951 (time 1736264227.81 total 20.00 last 360.27) - open/close 10000 (time 1736264234.96 total 27.15 last 426.66) - open/close 13587 (time 1736264244.96 total 37.15 last 358.70) - open/close 17490 (time 1736264254.96 total 47.15 last 390.28) - open/close 20000 (time 1736264260.36 total 52.55 last 464.75) - open/close 24170 (time 1736264270.36 total 62.55 last 416.92) - open/close 27385 (time 1736264280.36 total 72.55 last 321.48) - open/close 30000 (time 1736264287.33 total 79.52 last 375.32) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 85.26 seconds: 382.27 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 8926 (time 1736264303.96 total 10.00 last 892.60) - /unlink 19180 (time 1736264313.96 total 20.00 last 1025.22) - /unlink 29175 (time 1736264323.96 total 30.00 last 999.47) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 33.51 seconds: 972.72 ops/second Waiting for MDT destroys to complete - open/close 5629 (time 1736264369.69 total 10.00 last 562.87) - open/close 10000 (time 1736264377.61 total 17.92 last 552.25) - open/close 15241 (time 1736264387.61 total 27.92 last 524.01) - open/close 20000 (time 1736264396.97 total 37.28 last 508.24) - open/close 25970 (time 1736264406.97 total 47.28 last 596.94) - open/close 30000 (time 1736264414.81 total 55.12 last 514.42) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 59.97 seconds: 543.55 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 9465 (time 1736264430.58 total 10.00 last 946.45) - /unlink 18921 (time 1736264440.58 total 20.00 last 945.59) - /unlink 29006 (time 1736264450.58 total 30.00 last 1008.43) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 33.77 seconds: 965.11 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg408-server LNET ready to unload unloading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local oleg408-server: LNET ready to unload modules unloaded. pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2