== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:35:59 (1734258959) Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 oleg229-server: oleg229-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: libkmod: kmod_module_get_size: failed to read coresize from /sys/module/acpi_cpufreq oleg229-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg229-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg229-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg229-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg229-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg229-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16266 458 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16451 643 15808 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1636 (time 1734259026.34 total 10.00 last 163.53) - open/close 3297 (time 1734259036.34 total 20.01 last 166.04) - open/close 5018 (time 1734259046.34 total 30.01 last 172.09) - open/close 6795 (time 1734259056.34 total 40.01 last 177.67) - open/close 8602 (time 1734259066.35 total 50.01 last 180.61) - open/close 10000 (time 1734259073.88 total 57.54 last 185.67) - open/close 11756 (time 1734259083.88 total 67.55 last 175.57) - open/close 13346 (time 1734259093.88 total 77.55 last 158.97) - open/close 15127 (time 1734259103.88 total 87.55 last 178.09) - open/close 16846 (time 1734259113.88 total 97.55 last 171.89) - open/close 18903 (time 1734259123.89 total 107.56 last 205.58) - open/close 20000 (time 1734259130.40 total 114.07 last 168.33) - open/close 21628 (time 1734259140.40 total 124.07 last 162.78) - open/close 23478 (time 1734259150.41 total 134.08 last 184.93) - open/close 25156 (time 1734259160.41 total 144.08 last 167.76) - open/close 27049 (time 1734259170.42 total 154.08 last 189.22) - open/close 28719 (time 1734259180.42 total 164.09 last 166.91) - open/close 30000 (time 1734259188.12 total 171.79 last 166.43) - open/close 31926 (time 1734259198.12 total 181.79 last 192.56) - open/close 33479 (time 1734259208.12 total 191.79 last 155.25) - open/close 35543 (time 1734259218.13 total 201.80 last 206.32) - open/close 37057 (time 1734259228.13 total 211.80 last 151.35) - open/close 38627 (time 1734259238.13 total 221.80 last 156.92) - open/close 40000 (time 1734259245.40 total 229.07 last 188.96) - open/close 41745 (time 1734259255.40 total 239.07 last 174.48) - open/close 43592 (time 1734259265.40 total 249.07 last 184.66) - open/close 45431 (time 1734259275.41 total 259.08 last 183.81) - open/close 47703 (time 1734259285.41 total 269.08 last 227.13) - open/close 49825 (time 1734259295.42 total 279.09 last 212.11) - open/close 51497 (time 1734259305.42 total 289.09 last 167.16) - open/close 53187 (time 1734259315.42 total 299.09 last 168.99) - open/close 54803 (time 1734259325.42 total 309.09 last 161.59) - open/close 56600 (time 1734259335.42 total 319.09 last 179.62) - open/close 58471 (time 1734259345.43 total 329.10 last 187.04) - open/close 60000 (time 1734259353.52 total 337.19 last 189.02) - open/close 62104 (time 1734259363.52 total 347.19 last 210.38) - open/close 63761 (time 1734259373.52 total 357.19 last 165.65) total: 64768 open/close in 362.44 seconds: 178.70 ops/second - /unlink 3287 (time 1734259389.97 total 10.00 last 328.60) - /unlink 6334 (time 1734259399.97 total 20.00 last 304.70) - /unlink 9816 (time 1734259409.97 total 30.01 last 348.08) - /unlink 12729 (time 1734259419.97 total 40.01 last 291.21) - /unlink 15734 (time 1734259429.97 total 50.01 last 300.45) - /unlink 18608 (time 1734259439.97 total 60.01 last 287.40) - /unlink 20000 (time 1734259445.36 total 65.40 last 258.37) - /unlink 22720 (time 1734259455.37 total 75.40 last 271.86) - /unlink 25594 (time 1734259465.37 total 85.41 last 287.27) - /unlink 28880 (time 1734259475.37 total 95.41 last 328.49) - /unlink 30000 (time 1734259478.59 total 98.63 last 348.29) - /unlink 32761 (time 1734259488.59 total 108.63 last 276.08) - /unlink 35872 (time 1734259498.59 total 118.63 last 311.09) - /unlink 38742 (time 1734259508.60 total 128.63 last 286.90) - /unlink 40000 (time 1734259512.45 total 132.49 last 326.20) - /unlink 43223 (time 1734259522.45 total 142.49 last 322.28) - /unlink 46142 (time 1734259532.45 total 152.49 last 291.88) - /unlink 49099 (time 1734259542.46 total 162.49 last 295.62) - /unlink 50000 (time 1734259545.64 total 165.68 last 282.73) - /unlink 53033 (time 1734259555.65 total 175.68 last 303.21) - /unlink 56248 (time 1734259565.65 total 185.68 last 321.46) - /unlink 59613 (time 1734259575.65 total 195.68 last 336.47) - /unlink 62691 (time 1734259585.65 total 205.69 last 307.72) total: 64768 /unlink in 212.00 seconds: 305.51 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2116 (time 1734259617.72 total 10.00 last 211.58) - open/close 4097 (time 1734259627.72 total 20.00 last 198.09) - open/close 6428 (time 1734259637.72 total 30.00 last 233.10) - open/close 8698 (time 1734259647.72 total 40.00 last 226.96) - open/close 10000 (time 1734259653.43 total 45.71 last 228.29) - open/close 12120 (time 1734259663.43 total 55.71 last 211.93) - open/close 13933 (time 1734259673.43 total 65.71 last 181.27) - open/close 15852 (time 1734259683.43 total 75.71 last 191.89) - open/close 17763 (time 1734259693.43 total 85.71 last 191.08) - open/close 20000 (time 1734259703.38 total 95.66 last 224.86) - open/close 22114 (time 1734259713.38 total 105.66 last 211.38) - open/close 23904 (time 1734259723.39 total 115.67 last 178.93) - open/close 25955 (time 1734259733.39 total 125.67 last 205.04) - open/close 27628 (time 1734259743.39 total 135.67 last 167.27) - open/close 29640 (time 1734259753.39 total 145.67 last 201.16) - open/close 31417 (time 1734259763.39 total 155.68 last 177.66) - open/close 33063 (time 1734259773.40 total 165.68 last 164.53) - open/close 34774 (time 1734259783.40 total 175.68 last 171.07) - open/close 36587 (time 1734259793.40 total 185.68 last 181.23) - open/close 38678 (time 1734259803.41 total 195.69 last 209.03) - open/close 40000 (time 1734259809.25 total 201.54 last 226.11) - open/close 42124 (time 1734259819.26 total 211.54 last 212.39) - open/close 44341 (time 1734259829.26 total 221.54 last 221.69) - open/close 46370 (time 1734259839.26 total 231.54 last 202.81) - open/close 48582 (time 1734259849.26 total 241.54 last 221.19) - open/close 50000 (time 1734259858.02 total 250.30 last 161.85) - open/close 51977 (time 1734259868.02 total 260.30 last 197.70) - open/close 53876 (time 1734259878.03 total 270.31 last 189.76) - open/close 55790 (time 1734259888.03 total 280.31 last 191.40) - open/close 58017 (time 1734259898.03 total 290.31 last 222.65) - open/close 59966 (time 1734259908.03 total 300.31 last 194.86) - open/close 62120 (time 1734259918.03 total 310.32 last 215.37) - open/close 64689 (time 1734259928.04 total 320.32 last 256.83) total: 64768 open/close in 320.63 seconds: 202.00 ops/second - /unlink 4631 (time 1734259939.38 total 10.00 last 463.08) - /unlink 9384 (time 1734259949.38 total 20.00 last 475.24) - /unlink 14243 (time 1734259959.38 total 30.00 last 485.85) - /unlink 19177 (time 1734259969.38 total 40.00 last 493.40) - /unlink 24167 (time 1734259979.38 total 50.00 last 498.98) - /unlink 29283 (time 1734259989.39 total 60.00 last 511.56) - /unlink 33957 (time 1734259999.39 total 70.01 last 467.35) - /unlink 39135 (time 1734260009.39 total 80.01 last 517.73) - /unlink 43844 (time 1734260019.39 total 90.01 last 470.90) - /unlink 48994 (time 1734260029.39 total 100.01 last 515.00) - /unlink 53990 (time 1734260039.39 total 110.01 last 499.52) - /unlink 58279 (time 1734260049.39 total 120.01 last 428.85) - /unlink 60000 (time 1734260053.30 total 123.92 last 440.43) total: 64768 /unlink in 133.13 seconds: 486.51 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2738 (time 1734260087.78 total 10.00 last 273.78) - open/close 5595 (time 1734260097.78 total 20.00 last 285.62) - open/close 8302 (time 1734260107.78 total 30.00 last 270.69) - open/close 10000 (time 1734260113.41 total 35.64 last 301.43) - open/close 12719 (time 1734260123.42 total 45.64 last 271.82) - open/close 15451 (time 1734260133.42 total 55.64 last 273.15) - open/close 18145 (time 1734260143.42 total 65.64 last 269.33) - open/close 20000 (time 1734260149.76 total 71.98 last 292.60) - open/close 22923 (time 1734260159.76 total 81.99 last 292.23) - open/close 25942 (time 1734260169.77 total 91.99 last 301.85) - open/close 28786 (time 1734260179.77 total 101.99 last 284.37) - open/close 30000 (time 1734260184.77 total 107.00 last 242.45) - open/close 32520 (time 1734260194.77 total 117.00 last 251.97) - open/close 35453 (time 1734260204.78 total 127.00 last 293.24) - open/close 38189 (time 1734260214.78 total 137.00 last 273.59) - open/close 40000 (time 1734260221.73 total 143.95 last 260.65) - open/close 42293 (time 1734260231.73 total 153.95 last 229.26) - open/close 45030 (time 1734260241.73 total 163.95 last 273.70) - open/close 47772 (time 1734260251.73 total 173.95 last 274.16) - open/close 50000 (time 1734260260.22 total 182.45 last 262.30) - open/close 52798 (time 1734260270.23 total 192.45 last 279.71) - open/close 55743 (time 1734260280.23 total 202.45 last 294.47) - open/close 58522 (time 1734260290.23 total 212.45 last 277.81) - open/close 60000 (time 1734260295.54 total 217.76 last 278.48) - open/close 62463 (time 1734260305.54 total 227.76 last 246.20) - open/close 64700 (time 1734260315.54 total 237.77 last 223.66) total: 64768 open/close in 238.05 seconds: 272.07 ops/second - /unlink 4119 (time 1734260326.90 total 10.00 last 411.81) - /unlink 8227 (time 1734260336.90 total 20.00 last 410.75) - /unlink 10000 (time 1734260341.57 total 24.67 last 380.18) - /unlink 13868 (time 1734260351.57 total 34.67 last 386.80) - /unlink 17842 (time 1734260361.57 total 44.67 last 397.34) - /unlink 20000 (time 1734260366.79 total 49.89 last 413.12) - /unlink 24298 (time 1734260376.80 total 59.90 last 429.67) - /unlink 28377 (time 1734260386.80 total 69.90 last 407.89) - /unlink 30000 (time 1734260390.82 total 73.92 last 403.09) - /unlink 34882 (time 1734260400.82 total 83.92 last 488.19) - /unlink 39549 (time 1734260410.82 total 93.92 last 466.62) - /unlink 43870 (time 1734260420.83 total 103.92 last 432.08) - /unlink 48467 (time 1734260430.83 total 113.93 last 459.62) - /unlink 50000 (time 1734260434.42 total 117.52 last 426.88) - /unlink 54173 (time 1734260444.42 total 127.52 last 417.24) - /unlink 59985 (time 1734260454.42 total 137.52 last 581.16) total: 64768 /unlink in 146.08 seconds: 443.37 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2477 (time 1734260488.27 total 10.00 last 247.65) - open/close 5053 (time 1734260498.27 total 20.00 last 257.60) - open/close 8125 (time 1734260508.27 total 30.00 last 307.18) - open/close 10000 (time 1734260515.29 total 37.03 last 266.83) - open/close 12493 (time 1734260525.29 total 47.03 last 249.29) - open/close 15085 (time 1734260535.30 total 57.03 last 259.15) - open/close 17627 (time 1734260545.30 total 67.03 last 254.14) - open/close 20000 (time 1734260554.51 total 76.25 last 257.53) - open/close 22619 (time 1734260564.51 total 86.25 last 261.87) - open/close 25371 (time 1734260574.52 total 96.25 last 275.17) - open/close 28116 (time 1734260584.52 total 106.25 last 274.50) - open/close 30000 (time 1734260590.43 total 112.16 last 318.76) - open/close 33554 (time 1734260600.43 total 122.16 last 355.38) - open/close 36652 (time 1734260610.43 total 132.16 last 309.79) - open/close 39433 (time 1734260620.43 total 142.16 last 278.05) - open/close 42744 (time 1734260630.43 total 152.17 last 331.05) - open/close 46192 (time 1734260640.43 total 162.17 last 344.73) - open/close 49232 (time 1734260650.43 total 172.17 last 303.93) - open/close 50000 (time 1734260652.82 total 174.56 last 321.36) - open/close 53535 (time 1734260662.83 total 184.56 last 353.40) - open/close 56690 (time 1734260672.83 total 194.57 last 315.42) - open/close 60000 (time 1734260682.71 total 204.44 last 335.08) - open/close 62834 (time 1734260692.71 total 214.45 last 283.32) total: 64768 open/close in 221.34 seconds: 292.62 ops/second - /unlink 4815 (time 1734260710.73 total 10.00 last 481.43) - /unlink 10000 (time 1734260720.21 total 19.48 last 547.24) - /unlink 15373 (time 1734260730.21 total 29.48 last 537.24) - /unlink 20000 (time 1734260739.49 total 38.75 last 498.78) - /unlink 24775 (time 1734260749.49 total 48.75 last 477.47) - /unlink 29916 (time 1734260759.49 total 58.76 last 514.05) - /unlink 35023 (time 1734260769.49 total 68.76 last 510.65) - /unlink 40000 (time 1734260777.95 total 77.21 last 588.48) - /unlink 45653 (time 1734260787.95 total 87.22 last 565.21) - /unlink 50000 (time 1734260795.82 total 95.09 last 551.82) - /unlink 56320 (time 1734260805.83 total 105.09 last 631.92) - /unlink 60000 (time 1734260812.00 total 111.27 last 595.57) total: 64768 /unlink in 118.40 seconds: 547.02 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3963 (time 1734260848.56 total 10.00 last 396.22) - open/close 7875 (time 1734260858.56 total 20.00 last 391.18) - open/close 10000 (time 1734260863.78 total 25.22 last 407.46) - open/close 13439 (time 1734260873.78 total 35.22 last 343.80) - open/close 17805 (time 1734260883.78 total 45.22 last 436.58) - open/close 20000 (time 1734260888.31 total 49.75 last 484.76) - open/close 24772 (time 1734260898.31 total 59.75 last 477.13) - open/close 29065 (time 1734260908.31 total 69.75 last 429.26) - open/close 30000 (time 1734260910.51 total 71.95 last 426.00) - open/close 34254 (time 1734260920.51 total 81.95 last 425.31) - open/close 38003 (time 1734260930.51 total 91.95 last 374.90) - open/close 40000 (time 1734260935.34 total 96.78 last 413.67) - open/close 44261 (time 1734260945.34 total 106.78 last 426.03) - open/close 48244 (time 1734260955.34 total 116.78 last 398.27) - open/close 50000 (time 1734260959.44 total 120.88 last 428.08) - open/close 53937 (time 1734260969.44 total 130.88 last 393.69) - open/close 58389 (time 1734260979.44 total 140.88 last 445.17) - open/close 60000 (time 1734260982.81 total 144.25 last 478.51) - open/close 64664 (time 1734260992.81 total 154.25 last 466.39) total: 64768 open/close in 154.47 seconds: 419.30 ops/second - /unlink 6272 (time 1734261003.97 total 10.00 last 627.18) - /unlink 10000 (time 1734261009.24 total 15.27 last 707.04) - /unlink 18200 (time 1734261019.24 total 25.27 last 819.95) - /unlink 20000 (time 1734261021.40 total 27.43 last 833.96) - /unlink 27574 (time 1734261031.40 total 37.43 last 757.36) - /unlink 30000 (time 1734261034.59 total 40.63 last 759.21) - /unlink 38077 (time 1734261044.59 total 50.63 last 807.62) - /unlink 40000 (time 1734261046.87 total 52.91 last 843.96) - /unlink 48730 (time 1734261056.87 total 62.91 last 872.90) - /unlink 58150 (time 1734261066.87 total 72.91 last 941.97) total: 64768 /unlink in 79.67 seconds: 812.93 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server unloading modules via unload_modules_local on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2