== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 08:46:06 (1747399566) Stopping clients: oleg453-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg453-client.virtnet /mnt/lustre2 (opts:-f) oleg453-server: oleg453-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing load_modules_local oleg453-server: Loading modules from /home/green/git/lustre-release/lustre oleg453-server: detected 4 online CPUs by sysfs oleg453-server: Force libcfs to create 2 CPU partitions oleg453-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg453-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg453-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg453-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg453-client.virtnet: -o user_xattr,flock oleg453-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 963 (time 1747399660.88 total 10.01 last 96.24) - open/close 1930 (time 1747399670.88 total 20.01 last 96.67) - open/close 2897 (time 1747399680.89 total 30.01 last 96.67) - open/close 3902 (time 1747399690.89 total 40.02 last 100.44) - open/close 4844 (time 1747399700.90 total 50.03 last 94.12) - open/close 5805 (time 1747399710.90 total 60.03 last 96.07) - open/close 6866 (time 1747399720.90 total 70.03 last 106.08) - open/close 8110 (time 1747399730.91 total 80.04 last 124.34) - open/close 9381 (time 1747399740.92 total 90.04 last 127.01) - open/close 10000 (time 1747399746.16 total 95.29 last 118.04) - open/close 11212 (time 1747399756.16 total 105.29 last 121.18) - open/close 12403 (time 1747399766.17 total 115.29 last 119.02) - open/close 13608 (time 1747399776.17 total 125.30 last 120.47) - open/close 14818 (time 1747399786.17 total 135.30 last 120.99) - open/close 16000 (time 1747399796.18 total 145.30 last 118.12) - open/close 17217 (time 1747399806.18 total 155.31 last 121.62) - open/close 18491 (time 1747399816.19 total 165.32 last 127.33) - open/close 19741 (time 1747399826.19 total 175.32 last 124.94) - open/close 20000 (time 1747399828.32 total 177.45 last 121.51) - open/close 21177 (time 1747399838.33 total 187.46 last 117.64) - open/close 22414 (time 1747399848.33 total 197.46 last 123.68) - open/close 23631 (time 1747399858.33 total 207.46 last 121.69) - open/close 25097 (time 1747399868.34 total 217.46 last 146.54) - open/close 27008 (time 1747399878.34 total 227.47 last 191.03) - open/close 28861 (time 1747399888.34 total 237.47 last 185.27) - open/close 30000 (time 1747399894.69 total 243.82 last 179.28) - open/close 31821 (time 1747399904.70 total 253.82 last 182.09) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 258.23 seconds: 126.22 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 3868 (time 1747399920.39 total 10.00 last 386.78) - /unlink 6901 (time 1747399930.39 total 20.00 last 303.29) - /unlink 10000 (time 1747399939.67 total 29.27 last 334.16) - /unlink 13432 (time 1747399949.67 total 39.28 last 343.16) - /unlink 16916 (time 1747399959.67 total 49.28 last 348.37) - /unlink 20000 (time 1747399967.14 total 56.75 last 412.84) - /unlink 23397 (time 1747399977.14 total 66.75 last 339.69) - /unlink 27441 (time 1747399987.14 total 76.75 last 404.37) - /unlink 30000 (time 1747399996.34 total 85.95 last 278.11) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 93.92 seconds: 347.05 ops/second Waiting for MDT destroys to complete - open/close 1710 (time 1747400027.25 total 10.00 last 170.98) - open/close 3077 (time 1747400037.25 total 20.01 last 136.64) - open/close 4530 (time 1747400047.25 total 30.01 last 145.29) - open/close 6238 (time 1747400057.25 total 40.01 last 170.76) - open/close 8081 (time 1747400067.26 total 50.01 last 184.28) - open/close 9882 (time 1747400077.26 total 60.01 last 180.02) - open/close 11839 (time 1747400087.26 total 70.02 last 195.64) - open/close 14177 (time 1747400097.26 total 80.02 last 233.76) - open/close 16527 (time 1747400107.27 total 90.02 last 234.94) - open/close 18625 (time 1747400117.27 total 100.02 last 209.79) - open/close 20000 (time 1747400124.18 total 106.94 last 198.80) - open/close 22113 (time 1747400134.19 total 116.94 last 211.26) - open/close 24376 (time 1747400144.19 total 126.95 last 226.13) - open/close 26873 (time 1747400154.19 total 136.95 last 249.69) - open/close 29187 (time 1747400164.20 total 146.95 last 231.37) - open/close 30000 (time 1747400167.66 total 150.41 last 234.78) - open/close 32543 (time 1747400177.66 total 160.41 last 254.25) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 160.67 seconds: 202.86 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 4857 (time 1747400189.07 total 10.00 last 485.63) - /unlink 9831 (time 1747400199.07 total 20.00 last 497.29) - /unlink 14159 (time 1747400209.07 total 30.00 last 432.80) - /unlink 18925 (time 1747400219.07 total 40.00 last 476.56) - /unlink 20000 (time 1747400221.40 total 42.33 last 462.78) - /unlink 24612 (time 1747400231.40 total 52.33 last 461.15) - /unlink 28900 (time 1747400241.40 total 62.33 last 428.78) - /unlink 30000 (time 1747400244.02 total 64.95 last 419.94) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 71.04 seconds: 458.81 ops/second Waiting for MDT destroys to complete - open/close 2578 (time 1747400272.52 total 10.00 last 257.73) - open/close 5564 (time 1747400282.52 total 20.01 last 298.49) - open/close 8411 (time 1747400292.52 total 30.01 last 284.69) - open/close 10000 (time 1747400298.16 total 35.65 last 281.72) - open/close 13017 (time 1747400308.16 total 45.65 last 301.66) - open/close 15863 (time 1747400318.17 total 55.65 last 284.52) - open/close 19371 (time 1747400328.17 total 65.65 last 350.70) - open/close 23144 (time 1747400338.17 total 75.66 last 377.25) - open/close 26768 (time 1747400348.17 total 85.66 last 362.30) - open/close 29530 (time 1747400358.17 total 95.66 last 276.19) - open/close 32335 (time 1747400368.18 total 105.66 last 280.44) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 106.50 seconds: 306.05 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 6035 (time 1747400380.02 total 10.00 last 603.47) - /unlink 10000 (time 1747400387.62 total 17.60 last 521.80) - /unlink 15678 (time 1747400397.62 total 27.60 last 567.76) - /unlink 20000 (time 1747400405.52 total 35.51 last 546.64) - /unlink 25055 (time 1747400415.52 total 45.51 last 505.46) - /unlink 30000 (time 1747400423.91 total 53.90 last 589.45) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 58.57 seconds: 556.47 ops/second Waiting for MDT destroys to complete - open/close 3497 (time 1747400453.06 total 10.00 last 349.67) - open/close 7518 (time 1747400463.07 total 20.00 last 401.99) - open/close 10000 (time 1747400469.84 total 26.78 last 366.14) - open/close 14002 (time 1747400479.84 total 36.78 last 400.19) - open/close 18539 (time 1747400489.85 total 46.78 last 453.62) - open/close 20000 (time 1747400493.11 total 50.05 last 447.95) - open/close 24171 (time 1747400503.11 total 60.05 last 417.09) - open/close 28532 (time 1747400513.11 total 70.05 last 436.04) - open/close 30000 (time 1747400517.11 total 74.05 last 367.21) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 81.55 seconds: 399.69 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 7016 (time 1747400535.59 total 10.00 last 701.59) - /unlink 10000 (time 1747400540.07 total 14.49 last 665.21) - /unlink 18127 (time 1747400550.07 total 24.49 last 812.68) - /unlink 20000 (time 1747400552.17 total 26.58 last 892.68) - /unlink 28608 (time 1747400562.17 total 36.58 last 860.77) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 41.15 seconds: 792.01 ops/second Waiting for MDT destroys to complete - open/close 2208 (time 1747400589.71 total 10.00 last 220.80) - open/close 3702 (time 1747400599.72 total 20.01 last 149.30) - open/close 5787 (time 1747400609.72 total 30.01 last 208.47) - open/close 8148 (time 1747400619.72 total 40.01 last 236.04) - open/close 10000 (time 1747400627.81 total 48.11 last 228.79) - open/close 12024 (time 1747400637.82 total 58.11 last 202.33) - open/close 13938 (time 1747400647.82 total 68.11 last 191.35) - open/close 16001 (time 1747400657.82 total 78.11 last 206.26) - open/close 18903 (time 1747400667.82 total 88.11 last 290.17) - open/close 20000 (time 1747400670.77 total 91.06 last 372.87) - open/close 23492 (time 1747400680.77 total 101.06 last 349.17) - open/close 26691 (time 1747400690.77 total 111.06 last 319.90) - open/close 28786 (time 1747400700.77 total 121.06 last 209.43) - open/close 30000 (time 1747400706.37 total 126.66 last 216.96) - open/close 31917 (time 1747400716.37 total 136.66 last 191.60) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 140.85 seconds: 231.40 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 4289 (time 1747400732.06 total 10.00 last 428.84) - /unlink 9605 (time 1747400742.06 total 20.00 last 531.59) - /unlink 15299 (time 1747400752.06 total 30.00 last 569.37) - /unlink 20000 (time 1747400760.55 total 38.50 last 553.49) - /unlink 25494 (time 1747400770.56 total 48.50 last 549.32) - /unlink 30000 (time 1747400776.96 total 54.90 last 703.71) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 57.71 seconds: 564.80 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg453-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg453-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg453-server stop mds service on oleg453-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg453-server stop mds service on oleg453-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg453-server LNET ready to unload unloading modules on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing unload_modules_local oleg453-server: LNET ready to unload modules unloaded. pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2