== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 08:47:32 (1747399652) Stopping clients: oleg208-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg208-client.virtnet /mnt/lustre2 (opts:-f) oleg208-server: oleg208-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing load_modules_local oleg208-server: Loading modules from /home/green/git/lustre-release/lustre oleg208-server: detected 4 online CPUs by sysfs oleg208-server: Force libcfs to create 2 CPU partitions oleg208-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg208-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg208-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg208-client.virtnet: -o user_xattr,flock oleg208-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1411 (time 1747399703.45 total 10.00 last 141.08) - open/close 2918 (time 1747399713.46 total 20.00 last 150.68) - open/close 4504 (time 1747399723.46 total 30.00 last 158.58) - open/close 6045 (time 1747399733.46 total 40.01 last 154.08) - open/close 7789 (time 1747399743.46 total 50.01 last 174.33) - open/close 9569 (time 1747399753.46 total 60.01 last 177.99) - open/close 10000 (time 1747399755.58 total 62.13 last 203.18) - open/close 11909 (time 1747399765.59 total 72.13 last 190.86) - open/close 13460 (time 1747399775.59 total 82.14 last 155.05) - open/close 15145 (time 1747399785.59 total 92.14 last 168.45) - open/close 16685 (time 1747399795.59 total 102.14 last 153.97) - open/close 18103 (time 1747399805.59 total 112.14 last 141.80) - open/close 19638 (time 1747399815.59 total 122.14 last 153.47) - open/close 20000 (time 1747399817.82 total 124.37 last 162.41) - open/close 21625 (time 1747399827.83 total 134.37 last 162.47) - open/close 23235 (time 1747399837.83 total 144.37 last 160.99) - open/close 24838 (time 1747399847.83 total 154.38 last 160.28) - open/close 26281 (time 1747399857.83 total 164.38 last 144.28) - open/close 27652 (time 1747399867.83 total 174.38 last 137.03) - open/close 28989 (time 1747399877.84 total 184.38 last 133.68) - open/close 30000 (time 1747399885.61 total 192.16 last 130.00) - open/close 31324 (time 1747399895.62 total 202.17 last 132.30) - open/close 32625 (time 1747399905.62 total 212.17 last 130.06) - open/close 34102 (time 1747399915.63 total 222.18 last 147.62) - open/close 35455 (time 1747399925.63 total 232.18 last 135.23) - open/close 36816 (time 1747399935.64 total 242.19 last 136.03) - open/close 38253 (time 1747399945.64 total 252.19 last 143.68) - open/close 39738 (time 1747399955.64 total 262.19 last 148.45) - open/close 41159 (time 1747399965.64 total 272.19 last 142.09) - open/close 42431 (time 1747399975.64 total 282.19 last 127.20) - open/close 43980 (time 1747399985.65 total 292.19 last 154.86) - open/close 45501 (time 1747399995.65 total 302.20 last 152.01) - open/close 46801 (time 1747400005.66 total 312.21 last 129.90) - open/close 48219 (time 1747400015.66 total 322.21 last 141.79) - open/close 49663 (time 1747400025.66 total 332.21 last 144.36) - open/close 50000 (time 1747400027.94 total 334.49 last 147.75) - open/close 51453 (time 1747400037.95 total 344.49 last 145.27) - open/close 52858 (time 1747400047.95 total 354.50 last 140.46) - open/close 54436 (time 1747400057.95 total 364.50 last 157.79) - open/close 55888 (time 1747400067.96 total 374.51 last 145.09) - open/close 57305 (time 1747400077.97 total 384.51 last 141.59) - open/close 58727 (time 1747400087.97 total 394.51 last 142.19) - open/close 60000 (time 1747400095.65 total 402.20 last 165.62) - open/close 61660 (time 1747400105.66 total 412.20 last 165.94) - open/close 63460 (time 1747400115.66 total 422.21 last 179.89) total: 64768 open/close in 429.15 seconds: 150.92 ops/second - /unlink 2901 (time 1747400133.89 total 10.00 last 290.02) - /unlink 5621 (time 1747400143.89 total 20.00 last 271.94) - /unlink 8274 (time 1747400153.89 total 30.01 last 265.25) - /unlink 10000 (time 1747400160.66 total 36.77 last 255.03) - /unlink 12763 (time 1747400170.66 total 46.78 last 276.29) - /unlink 16018 (time 1747400180.66 total 56.78 last 325.49) - /unlink 19381 (time 1747400190.66 total 66.78 last 336.26) - /unlink 22929 (time 1747400200.66 total 76.78 last 354.73) - /unlink 26156 (time 1747400210.67 total 86.78 last 322.67) - /unlink 29132 (time 1747400220.67 total 96.78 last 297.55) - /unlink 30000 (time 1747400223.68 total 99.80 last 287.63) - /unlink 32530 (time 1747400233.69 total 109.80 last 252.91) - /unlink 35209 (time 1747400243.69 total 119.81 last 267.77) - /unlink 38310 (time 1747400253.69 total 129.81 last 310.06) - /unlink 40000 (time 1747400259.81 total 135.92 last 276.35) - /unlink 43431 (time 1747400269.81 total 145.92 last 343.10) - /unlink 47216 (time 1747400279.81 total 155.92 last 378.50) - /unlink 50000 (time 1747400288.66 total 164.77 last 314.73) - /unlink 53347 (time 1747400298.66 total 174.77 last 334.61) - /unlink 57056 (time 1747400308.66 total 184.77 last 370.87) - /unlink 60000 (time 1747400316.86 total 192.97 last 359.16) - /unlink 63350 (time 1747400326.86 total 202.97 last 334.91) total: 64768 /unlink in 207.40 seconds: 312.29 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2568 (time 1747400362.28 total 10.00 last 256.79) - open/close 5025 (time 1747400372.29 total 20.00 last 245.63) - open/close 7049 (time 1747400382.29 total 30.01 last 202.35) - open/close 9106 (time 1747400392.29 total 40.01 last 205.69) - open/close 10000 (time 1747400397.47 total 45.19 last 172.45) - open/close 11826 (time 1747400407.48 total 55.19 last 182.55) - open/close 13654 (time 1747400417.48 total 65.20 last 182.74) - open/close 15442 (time 1747400427.49 total 75.20 last 178.64) - open/close 17151 (time 1747400437.49 total 85.21 last 170.82) - open/close 19062 (time 1747400447.49 total 95.21 last 191.08) - open/close 20000 (time 1747400452.38 total 100.10 last 191.97) - open/close 22158 (time 1747400462.38 total 110.10 last 215.76) - open/close 24468 (time 1747400472.38 total 120.10 last 230.95) - open/close 26608 (time 1747400482.39 total 130.10 last 213.94) - open/close 28716 (time 1747400492.39 total 140.10 last 210.78) - open/close 30000 (time 1747400496.24 total 143.96 last 333.14) - open/close 32952 (time 1747400506.24 total 153.96 last 295.17) - open/close 35832 (time 1747400516.25 total 163.96 last 287.89) - open/close 38532 (time 1747400526.25 total 173.96 last 269.98) - open/close 40000 (time 1747400531.94 total 179.66 last 257.93) - open/close 42434 (time 1747400541.94 total 189.66 last 243.37) - open/close 45128 (time 1747400551.94 total 199.66 last 269.35) - open/close 48136 (time 1747400561.94 total 209.66 last 300.78) - open/close 50000 (time 1747400568.40 total 216.11 last 288.81) - open/close 53049 (time 1747400578.40 total 226.12 last 304.83) - open/close 55108 (time 1747400588.40 total 236.12 last 205.83) - open/close 56401 (time 1747400598.41 total 246.13 last 129.22) - open/close 58137 (time 1747400608.41 total 256.13 last 173.55) - open/close 60000 (time 1747400618.05 total 265.77 last 193.25) - open/close 61944 (time 1747400628.05 total 275.77 last 194.35) - open/close 63566 (time 1747400638.06 total 285.77 last 162.18) total: 64768 open/close in 292.29 seconds: 221.59 ops/second - /unlink 2995 (time 1747400656.03 total 10.00 last 299.42) - /unlink 5797 (time 1747400666.04 total 20.01 last 280.00) - /unlink 8242 (time 1747400676.04 total 30.01 last 244.49) - /unlink 10000 (time 1747400682.43 total 36.40 last 274.96) - /unlink 12742 (time 1747400692.43 total 46.41 last 274.16) - /unlink 15185 (time 1747400702.43 total 56.41 last 244.27) - /unlink 17559 (time 1747400712.43 total 66.41 last 237.36) - /unlink 19838 (time 1747400722.43 total 76.41 last 227.89) - /unlink 22089 (time 1747400732.43 total 86.41 last 225.09) - /unlink 25106 (time 1747400742.44 total 96.41 last 301.65) - /unlink 28115 (time 1747400752.44 total 106.41 last 300.85) - /unlink 30000 (time 1747400758.09 total 112.06 last 333.47) - /unlink 34242 (time 1747400768.09 total 122.07 last 424.16) - /unlink 37678 (time 1747400778.09 total 132.07 last 343.60) - /unlink 40000 (time 1747400784.19 total 138.17 last 380.51) - /unlink 45059 (time 1747400794.20 total 148.17 last 505.81) - /unlink 50000 (time 1747400801.89 total 155.87 last 641.77) - /unlink 55250 (time 1747400811.89 total 165.87 last 524.99) - /unlink 59545 (time 1747400821.90 total 175.87 last 429.36) - /unlink 63371 (time 1747400831.90 total 185.87 last 382.53) total: 64768 /unlink in 188.56 seconds: 343.49 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2375 (time 1747400868.04 total 10.00 last 237.46) - open/close 4755 (time 1747400878.04 total 20.01 last 237.91) - open/close 7025 (time 1747400888.04 total 30.01 last 226.95) - open/close 9903 (time 1747400898.05 total 40.01 last 287.76) - open/close 13546 (time 1747400908.05 total 50.01 last 364.27) - open/close 17382 (time 1747400918.05 total 60.01 last 383.60) - open/close 20000 (time 1747400927.79 total 69.75 last 268.76) - open/close 23148 (time 1747400937.79 total 79.75 last 314.73) - open/close 25856 (time 1747400947.79 total 89.76 last 270.74) - open/close 29169 (time 1747400957.79 total 99.76 last 331.28) - open/close 30000 (time 1747400960.05 total 102.02 last 367.58) - open/close 33389 (time 1747400970.06 total 112.02 last 338.86) - open/close 37790 (time 1747400980.06 total 122.02 last 440.06) - open/close 40000 (time 1747400985.22 total 127.19 last 427.71) - open/close 45040 (time 1747400995.22 total 137.19 last 503.92) - open/close 49223 (time 1747401005.23 total 147.19 last 418.14) - open/close 50000 (time 1747401008.03 total 149.99 last 277.71) - open/close 53652 (time 1747401018.03 total 159.99 last 365.19) - open/close 58478 (time 1747401028.03 total 169.99 last 482.57) - open/close 60000 (time 1747401030.84 total 172.80 last 541.08) - open/close 64686 (time 1747401040.84 total 182.80 last 468.56) total: 64768 open/close in 182.97 seconds: 353.98 ops/second - /unlink 6726 (time 1747401051.95 total 10.00 last 672.52) - /unlink 10000 (time 1747401057.32 total 15.37 last 610.28) - /unlink 17932 (time 1747401067.32 total 25.37 last 793.10) - /unlink 20000 (time 1747401069.53 total 27.58 last 933.91) - /unlink 30000 (time 1747401079.45 total 37.49 last 1008.92) - /unlink 39644 (time 1747401089.45 total 47.49 last 964.36) - /unlink 48118 (time 1747401099.45 total 57.49 last 847.32) - /unlink 50000 (time 1747401101.97 total 60.02 last 745.26) - /unlink 57897 (time 1747401111.97 total 70.02 last 789.63) - /unlink 60000 (time 1747401114.40 total 72.44 last 867.86) total: 64768 /unlink in 77.86 seconds: 831.82 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5535 (time 1747401157.49 total 10.00 last 553.49) - open/close 10000 (time 1747401167.06 total 19.57 last 466.63) - open/close 14623 (time 1747401177.06 total 29.57 last 462.24) - open/close 19597 (time 1747401187.07 total 39.57 last 497.31) - open/close 23789 (time 1747401197.07 total 49.57 last 419.12) - open/close 28264 (time 1747401207.07 total 59.57 last 447.48) - open/close 30000 (time 1747401211.81 total 64.32 last 365.88) - open/close 34565 (time 1747401221.81 total 74.32 last 456.42) - open/close 38456 (time 1747401231.82 total 84.32 last 389.04) - open/close 40000 (time 1747401235.32 total 87.82 last 441.16) - open/close 45419 (time 1747401245.32 total 97.82 last 541.81) - open/close 50000 (time 1747401254.07 total 106.58 last 523.44) - open/close 55209 (time 1747401264.07 total 116.58 last 520.85) - open/close 59692 (time 1747401274.07 total 126.58 last 448.25) total: 64768 open/close in 136.38 seconds: 474.91 ops/second - /unlink 8176 (time 1747401294.71 total 10.00 last 817.58) - /unlink 10000 (time 1747401297.19 total 12.48 last 736.26) - /unlink 16951 (time 1747401307.19 total 22.48 last 695.05) - /unlink 20000 (time 1747401312.11 total 27.39 last 620.29) - /unlink 26318 (time 1747401322.11 total 37.39 last 631.75) - /unlink 30000 (time 1747401327.53 total 42.81 last 679.43) - /unlink 37363 (time 1747401337.53 total 52.81 last 736.24) - /unlink 40000 (time 1747401340.66 total 55.94 last 843.19) - /unlink 48332 (time 1747401350.66 total 65.94 last 833.19) - /unlink 50000 (time 1747401353.11 total 68.40 last 678.56) - /unlink 55361 (time 1747401363.11 total 78.40 last 536.09) - /unlink 60000 (time 1747401371.77 total 87.06 last 535.80) total: 64768 /unlink in 95.56 seconds: 677.74 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2805 (time 1747401413.13 total 10.00 last 280.45) - open/close 5792 (time 1747401423.13 total 20.00 last 298.66) - open/close 8638 (time 1747401433.13 total 30.00 last 284.59) - open/close 10000 (time 1747401437.85 total 34.72 last 288.94) - open/close 12834 (time 1747401447.85 total 44.72 last 283.39) - open/close 15246 (time 1747401457.85 total 54.72 last 241.10) - open/close 18042 (time 1747401467.85 total 64.72 last 279.60) - open/close 20000 (time 1747401474.64 total 71.51 last 288.28) - open/close 22677 (time 1747401484.64 total 81.51 last 267.70) - open/close 24860 (time 1747401494.65 total 91.51 last 218.30) - open/close 27302 (time 1747401504.65 total 101.52 last 244.14) - open/close 29992 (time 1747401514.65 total 111.52 last 268.92) - open/close 33294 (time 1747401524.65 total 121.52 last 330.08) - open/close 36156 (time 1747401534.66 total 131.53 last 286.11) - open/close 39731 (time 1747401544.66 total 141.53 last 357.38) - open/close 42878 (time 1747401554.66 total 151.53 last 314.66) - open/close 46185 (time 1747401564.66 total 161.53 last 330.66) - open/close 50000 (time 1747401573.09 total 169.96 last 452.93) - open/close 55061 (time 1747401583.09 total 179.96 last 506.07) - open/close 60000 (time 1747401592.08 total 188.95 last 549.19) - open/close 64566 (time 1747401602.08 total 198.95 last 456.49) total: 64768 open/close in 199.34 seconds: 324.91 ops/second - /unlink 9830 (time 1747401613.29 total 10.00 last 982.93) - /unlink 19361 (time 1747401623.30 total 20.00 last 953.01) - /unlink 27252 (time 1747401633.30 total 30.00 last 788.96) - /unlink 30000 (time 1747401638.29 total 34.99 last 550.67) - /unlink 37905 (time 1747401648.29 total 44.99 last 790.50) - /unlink 40000 (time 1747401650.67 total 47.38 last 879.25) - /unlink 47545 (time 1747401660.67 total 57.38 last 754.49) - /unlink 50000 (time 1747401663.56 total 60.27 last 849.38) - /unlink 58188 (time 1747401673.56 total 70.27 last 818.71) - /unlink 60000 (time 1747401675.84 total 72.54 last 796.07) total: 64768 /unlink in 79.93 seconds: 810.33 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg208-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg208-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg208-server stop mds service on oleg208-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg208-server LNET ready to unload unloading modules on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing unload_modules_local oleg208-server: LNET ready to unload modules unloaded. pdsh@oleg208-client: oleg208-client: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-client: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2