== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:53:24 (1718398404) Stopping clients: oleg250-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg250-client.virtnet /mnt/lustre2 (opts:-f) oleg250-server: oleg250-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing load_modules_local oleg250-server: Loading modules from /home/green/git/lustre-release/lustre oleg250-server: detected 4 online CPUs by sysfs oleg250-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg250-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg250-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg250-server: oleg250-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg250-client.virtnet: -o user_xattr,flock oleg250-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1713 (time 1718398445.89 total 10.00 last 171.27) - open/close 3629 (time 1718398455.89 total 20.00 last 191.59) - open/close 5842 (time 1718398465.89 total 30.00 last 221.25) - open/close 8203 (time 1718398475.89 total 40.01 last 236.05) - open/close 10000 (time 1718398483.06 total 47.17 last 250.71) - open/close 12407 (time 1718398493.06 total 57.17 last 240.69) - open/close 15221 (time 1718398503.06 total 67.18 last 281.35) - open/close 17819 (time 1718398513.06 total 77.18 last 259.78) - open/close 20000 (time 1718398522.34 total 86.45 last 235.14) - open/close 22151 (time 1718398532.34 total 96.46 last 215.03) - open/close 24237 (time 1718398542.34 total 106.46 last 208.53) - open/close 26540 (time 1718398552.34 total 116.46 last 230.27) - open/close 28841 (time 1718398562.35 total 126.46 last 230.05) - open/close 30000 (time 1718398568.97 total 133.09 last 174.95) - open/close 32080 (time 1718398578.97 total 143.09 last 207.96) - open/close 34168 (time 1718398588.97 total 153.09 last 208.76) - open/close 36259 (time 1718398598.98 total 163.09 last 209.04) - open/close 38267 (time 1718398608.98 total 173.10 last 200.65) - open/close 40000 (time 1718398618.73 total 182.85 last 177.85) - open/close 42407 (time 1718398628.73 total 192.85 last 240.62) - open/close 44619 (time 1718398638.74 total 202.85 last 221.10) - open/close 47006 (time 1718398648.74 total 212.86 last 238.61) - open/close 49560 (time 1718398658.74 total 222.86 last 255.30) - open/close 50000 (time 1718398660.93 total 225.05 last 201.11) - open/close 52616 (time 1718398670.93 total 235.05 last 261.59) - open/close 54960 (time 1718398680.94 total 245.05 last 234.33) - open/close 57180 (time 1718398690.94 total 255.06 last 221.92) - open/close 59302 (time 1718398700.94 total 265.06 last 212.15) - open/close 60000 (time 1718398704.27 total 268.39 last 209.60) - open/close 62216 (time 1718398714.28 total 278.39 last 221.50) - open/close 64467 (time 1718398724.28 total 288.40 last 225.02) total: 64768 open/close in 289.81 seconds: 223.48 ops/second - /unlink 4691 (time 1718398736.77 total 10.00 last 469.02) - /unlink 9279 (time 1718398746.78 total 20.00 last 458.72) - /unlink 10000 (time 1718398748.82 total 22.05 last 352.87) - /unlink 13799 (time 1718398758.82 total 32.05 last 379.90) - /unlink 17685 (time 1718398768.82 total 42.05 last 388.53) - /unlink 20000 (time 1718398774.30 total 47.52 last 422.81) - /unlink 24850 (time 1718398784.30 total 57.53 last 484.92) - /unlink 28986 (time 1718398794.30 total 67.53 last 413.54) - /unlink 30000 (time 1718398796.74 total 69.96 last 416.30) - /unlink 35024 (time 1718398806.74 total 79.96 last 502.40) - /unlink 40000 (time 1718398816.64 total 89.86 last 502.54) - /unlink 44665 (time 1718398826.64 total 99.87 last 466.42) - /unlink 49198 (time 1718398836.64 total 109.87 last 453.29) - /unlink 53807 (time 1718398846.64 total 119.87 last 460.78) - /unlink 58170 (time 1718398856.64 total 129.87 last 436.23) - /unlink 60000 (time 1718398860.56 total 133.78 last 467.71) - /unlink 64668 (time 1718398870.56 total 143.78 last 466.72) total: 64768 /unlink in 144.00 seconds: 449.77 ops/second - open/close 2831 (time 1718398881.99 total 10.00 last 283.02) - open/close 5714 (time 1718398891.99 total 20.00 last 288.27) - open/close 8693 (time 1718398901.99 total 30.00 last 297.87) - open/close 10000 (time 1718398906.57 total 34.59 last 285.26) - open/close 12810 (time 1718398916.58 total 44.59 last 280.84) - open/close 15727 (time 1718398926.58 total 54.59 last 291.69) - open/close 18251 (time 1718398936.58 total 64.59 last 252.37) - open/close 20000 (time 1718398943.21 total 71.23 last 263.70) - open/close 22687 (time 1718398953.22 total 81.23 last 268.64) - open/close 25800 (time 1718398963.22 total 91.23 last 311.22) - open/close 28562 (time 1718398973.22 total 101.23 last 276.13) - open/close 30000 (time 1718398978.25 total 106.26 last 285.88) - open/close 32026 (time 1718398988.25 total 116.27 last 202.56) - open/close 33014 (time 1718398998.26 total 126.27 last 98.78) - open/close 33988 (time 1718399008.26 total 136.27 last 97.38) - open/close 34899 (time 1718399018.27 total 146.28 last 91.02) - open/close 36021 (time 1718399028.27 total 156.28 last 112.16) - open/close 37315 (time 1718399038.28 total 166.29 last 129.28) - open/close 38734 (time 1718399048.28 total 176.29 last 141.88) - open/close 40000 (time 1718399058.25 total 186.26 last 127.04) - open/close 41081 (time 1718399068.25 total 196.26 last 108.08) - open/close 42122 (time 1718399078.25 total 206.27 last 104.05) - open/close 43241 (time 1718399088.26 total 216.27 last 111.87) - open/close 44459 (time 1718399098.26 total 226.28 last 121.71) - open/close 46021 (time 1718399108.27 total 236.28 last 156.18) - open/close 47452 (time 1718399118.27 total 246.28 last 143.06) - open/close 48623 (time 1718399128.27 total 256.28 last 117.09) - open/close 49924 (time 1718399138.28 total 266.29 last 130.00) - open/close 51228 (time 1718399148.28 total 276.29 last 130.39) - open/close 52409 (time 1718399158.28 total 286.29 last 118.10) - open/close 53739 (time 1718399168.28 total 296.30 last 132.94) - open/close 55086 (time 1718399178.29 total 306.30 last 134.64) - open/close 56670 (time 1718399188.29 total 316.30 last 158.39) - open/close 58154 (time 1718399198.29 total 326.30 last 148.34) - open/close 59246 (time 1718399208.29 total 336.31 last 109.18) - open/close 60000 (time 1718399215.26 total 343.27 last 108.22) - open/close 61015 (time 1718399225.26 total 353.28 last 101.46) - open/close 62058 (time 1718399235.27 total 363.28 last 104.29) - open/close 63058 (time 1718399245.27 total 373.28 last 99.95) - open/close 64145 (time 1718399255.28 total 383.29 last 108.64) total: 64768 open/close in 388.50 seconds: 166.71 ops/second - /unlink 1769 (time 1718399272.43 total 10.00 last 176.84) - /unlink 3315 (time 1718399282.43 total 20.01 last 154.55) - /unlink 5083 (time 1718399292.43 total 30.01 last 176.72) - /unlink 6801 (time 1718399302.43 total 40.01 last 171.78) - /unlink 8418 (time 1718399312.44 total 50.02 last 161.62) - /unlink 10000 (time 1718399321.42 total 59.00 last 176.07) - /unlink 11846 (time 1718399331.43 total 69.01 last 184.49) - /unlink 14074 (time 1718399341.43 total 79.01 last 222.78) - /unlink 16149 (time 1718399351.43 total 89.01 last 207.48) - /unlink 19256 (time 1718399361.43 total 99.01 last 310.66) - /unlink 20000 (time 1718399363.78 total 101.36 last 317.30) - /unlink 22832 (time 1718399373.78 total 111.36 last 283.14) - /unlink 25331 (time 1718399383.78 total 121.36 last 249.89) - /unlink 27624 (time 1718399393.78 total 131.36 last 229.25) - /unlink 29898 (time 1718399403.79 total 141.37 last 227.31) - /unlink 31813 (time 1718399413.79 total 151.37 last 191.47) - /unlink 33798 (time 1718399423.79 total 161.37 last 198.47) - /unlink 35615 (time 1718399433.80 total 171.38 last 181.50) - /unlink 37481 (time 1718399443.80 total 181.38 last 186.58) - /unlink 39302 (time 1718399453.81 total 191.39 last 181.99) - /unlink 40000 (time 1718399457.31 total 194.88 last 199.61) - /unlink 41938 (time 1718399467.31 total 204.88 last 193.79) - /unlink 43890 (time 1718399477.31 total 214.89 last 195.15) - /unlink 45822 (time 1718399487.31 total 224.89 last 193.19) - /unlink 47949 (time 1718399497.31 total 234.89 last 212.62) - /unlink 50000 (time 1718399506.01 total 243.59 last 235.70) - /unlink 52310 (time 1718399516.02 total 253.60 last 230.93) - /unlink 55308 (time 1718399526.02 total 263.60 last 299.76) - /unlink 58704 (time 1718399536.02 total 273.60 last 339.57) - /unlink 60000 (time 1718399540.47 total 278.05 last 290.90) - /unlink 63030 (time 1718399550.48 total 288.05 last 302.98) total: 64768 /unlink in 294.68 seconds: 219.79 ops/second - open/close 1516 (time 1718399568.84 total 10.00 last 151.58) - open/close 3083 (time 1718399578.85 total 20.01 last 156.64) - open/close 4712 (time 1718399588.85 total 30.01 last 162.84) - open/close 6224 (time 1718399598.85 total 40.01 last 151.16) - open/close 7649 (time 1718399608.85 total 50.01 last 142.48) - open/close 9100 (time 1718399618.86 total 60.02 last 145.04) - open/close 10000 (time 1718399624.57 total 65.73 last 157.64) - open/close 11684 (time 1718399634.57 total 75.73 last 168.37) - open/close 13673 (time 1718399644.57 total 85.73 last 198.85) - open/close 15965 (time 1718399654.58 total 95.74 last 229.10) - open/close 19282 (time 1718399664.58 total 105.74 last 331.58) - open/close 20000 (time 1718399666.62 total 107.78 last 351.93) - open/close 23457 (time 1718399676.62 total 117.78 last 345.66) - open/close 26714 (time 1718399686.62 total 127.78 last 325.65) - open/close 30000 (time 1718399694.70 total 135.86 last 406.90) - open/close 33588 (time 1718399704.70 total 145.86 last 358.74) - open/close 37288 (time 1718399714.70 total 155.86 last 369.89) - open/close 40000 (time 1718399722.01 total 163.17 last 371.19) - open/close 43818 (time 1718399732.01 total 173.17 last 381.75) - open/close 47639 (time 1718399742.01 total 183.17 last 382.06) - open/close 50000 (time 1718399747.57 total 188.73 last 424.63) - open/close 54113 (time 1718399757.57 total 198.73 last 411.27) - open/close 58181 (time 1718399767.57 total 208.73 last 406.78) - open/close 60000 (time 1718399772.74 total 213.90 last 352.06) - open/close 63663 (time 1718399782.74 total 223.90 last 366.28) total: 64768 open/close in 227.33 seconds: 284.91 ops/second - /unlink 7232 (time 1718399797.11 total 10.00 last 723.14) - /unlink 10000 (time 1718399800.59 total 13.47 last 797.14) - /unlink 16969 (time 1718399810.59 total 23.47 last 696.90) - /unlink 20000 (time 1718399814.19 total 27.07 last 841.69) - /unlink 28224 (time 1718399824.19 total 37.08 last 822.35) - /unlink 30000 (time 1718399826.48 total 39.37 last 774.85) - /unlink 38269 (time 1718399836.48 total 49.37 last 826.88) - /unlink 40000 (time 1718399838.78 total 51.66 last 753.77) - /unlink 48574 (time 1718399848.78 total 61.66 last 857.33) - /unlink 57669 (time 1718399858.78 total 71.67 last 909.45) - /unlink 60000 (time 1718399861.14 total 74.03 last 984.52) total: 64768 /unlink in 79.12 seconds: 818.60 ops/second - open/close 5186 (time 1718399877.05 total 10.00 last 518.60) - open/close 10000 (time 1718399886.05 total 19.00 last 534.83) - open/close 14271 (time 1718399896.05 total 29.00 last 427.09) - open/close 19259 (time 1718399906.05 total 39.00 last 498.77) - open/close 25003 (time 1718399916.05 total 49.00 last 574.40) - open/close 30000 (time 1718399925.15 total 58.11 last 548.71) - open/close 35124 (time 1718399935.15 total 68.11 last 512.39) - open/close 40000 (time 1718399944.91 total 77.87 last 499.69) - open/close 45153 (time 1718399954.91 total 87.87 last 515.23) - open/close 50000 (time 1718399964.29 total 97.24 last 516.93) - open/close 54823 (time 1718399974.29 total 107.25 last 482.27) - open/close 59652 (time 1718399984.29 total 117.25 last 482.83) - open/close 64430 (time 1718399994.30 total 127.25 last 477.69) total: 64768 open/close in 128.19 seconds: 505.25 ops/second - /unlink 8700 (time 1718400006.12 total 10.00 last 869.97) - /unlink 17683 (time 1718400016.12 total 20.00 last 898.28) - /unlink 20000 (time 1718400018.71 total 22.59 last 894.19) - /unlink 28759 (time 1718400028.71 total 32.59 last 875.90) - /unlink 37905 (time 1718400038.71 total 42.59 last 914.51) - /unlink 40000 (time 1718400041.11 total 44.99 last 873.60) - /unlink 49028 (time 1718400051.11 total 54.99 last 902.77) - /unlink 57996 (time 1718400061.11 total 64.99 last 896.78) - /unlink 60000 (time 1718400063.28 total 67.16 last 922.48) total: 64768 /unlink in 72.24 seconds: 896.61 ops/second - open/close 5312 (time 1718400079.55 total 10.00 last 531.12) - open/close 10000 (time 1718400088.27 total 18.73 last 537.33) - open/close 14958 (time 1718400098.27 total 28.73 last 495.71) - open/close 20000 (time 1718400108.02 total 38.48 last 517.22) - open/close 24746 (time 1718400118.02 total 48.48 last 474.60) - open/close 28566 (time 1718400128.02 total 58.48 last 381.95) - open/close 30000 (time 1718400130.41 total 60.87 last 600.30) - open/close 35712 (time 1718400140.41 total 70.87 last 571.18) - open/close 40000 (time 1718400147.70 total 78.15 last 588.59) - open/close 44794 (time 1718400157.70 total 88.15 last 479.35) - open/close 50000 (time 1718400167.58 total 98.04 last 526.66) - open/close 55025 (time 1718400177.59 total 108.04 last 502.42) - open/close 59926 (time 1718400187.59 total 118.04 last 490.05) total: 64768 open/close in 127.85 seconds: 506.58 ops/second - /unlink 8710 (time 1718400208.25 total 10.00 last 870.99) - /unlink 17262 (time 1718400218.25 total 20.00 last 855.14) - /unlink 20000 (time 1718400221.86 total 23.61 last 758.30) - /unlink 27752 (time 1718400231.86 total 33.61 last 775.11) - /unlink 30000 (time 1718400235.10 total 36.85 last 694.50) - /unlink 37451 (time 1718400245.10 total 46.85 last 745.06) - /unlink 40000 (time 1718400248.29 total 50.04 last 798.44) - /unlink 48874 (time 1718400258.29 total 60.04 last 887.35) - /unlink 58159 (time 1718400268.29 total 70.04 last 928.47) - /unlink 60000 (time 1718400270.54 total 72.29 last 820.24) total: 64768 /unlink in 77.60 seconds: 834.66 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg250-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg250-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg250-server stop mds service on oleg250-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg250-server LNET ready to unload unloading modules on: 'oleg250-server' oleg250-server: oleg250-server.virtnet: executing unload_modules_local oleg250-server: LNET ready to unload modules unloaded. pdsh@oleg250-client: oleg250-client: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-server: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-client: ssh exited with exit code 2 pdsh@oleg250-client: oleg250-server: ssh exited with exit code 2