== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:36:24 (1734258984) Stopping clients: oleg252-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg252-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg252-client: oleg252-server: ssh exited with exit code 2 oleg252-server: oleg252-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing load_modules_local oleg252-server: Loading modules from /home/green/git/lustre-release/lustre oleg252-server: detected 4 online CPUs by sysfs oleg252-server: Force libcfs to create 2 CPU partitions oleg252-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg252-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg252-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg252-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg252-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg252-client.virtnet: -o user_xattr,flock 192.168.202.152@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16266 458 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16451 643 15808 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1896 (time 1734259052.51 total 10.00 last 189.53) - open/close 3554 (time 1734259062.51 total 20.01 last 165.74) - open/close 5428 (time 1734259072.51 total 30.01 last 187.40) - open/close 7269 (time 1734259082.51 total 40.01 last 184.09) - open/close 8861 (time 1734259092.51 total 50.01 last 159.16) - open/close 10000 (time 1734259098.78 total 56.28 last 181.78) - open/close 11625 (time 1734259108.79 total 66.28 last 162.42) - open/close 13607 (time 1734259118.79 total 76.28 last 198.13) - open/close 15491 (time 1734259128.79 total 86.29 last 188.35) - open/close 17177 (time 1734259138.79 total 96.29 last 168.55) - open/close 19016 (time 1734259148.80 total 106.29 last 183.89) - open/close 20000 (time 1734259154.54 total 112.04 last 171.28) - open/close 21783 (time 1734259164.54 total 122.04 last 178.26) - open/close 23682 (time 1734259174.55 total 132.04 last 189.79) - open/close 25256 (time 1734259184.55 total 142.05 last 157.37) - open/close 27256 (time 1734259194.56 total 152.05 last 199.90) - open/close 28831 (time 1734259204.56 total 162.05 last 157.50) - open/close 30000 (time 1734259211.20 total 168.69 last 175.96) - open/close 32014 (time 1734259221.20 total 178.70 last 201.39) - open/close 33467 (time 1734259231.21 total 188.70 last 145.19) - open/close 35102 (time 1734259241.21 total 198.71 last 163.43) - open/close 36910 (time 1734259251.21 total 208.71 last 180.79) - open/close 38704 (time 1734259261.22 total 218.72 last 179.25) - open/close 40000 (time 1734259268.03 total 225.53 last 190.22) - open/close 41835 (time 1734259278.04 total 235.53 last 183.46) - open/close 44073 (time 1734259288.04 total 245.53 last 223.80) - open/close 46018 (time 1734259298.04 total 255.53 last 194.49) - open/close 47708 (time 1734259308.04 total 265.53 last 168.97) - open/close 49298 (time 1734259318.04 total 275.54 last 158.94) - open/close 50000 (time 1734259322.56 total 280.05 last 155.48) - open/close 51730 (time 1734259332.56 total 290.06 last 172.92) - open/close 53576 (time 1734259342.57 total 300.06 last 184.53) - open/close 55414 (time 1734259352.57 total 310.06 last 183.74) - open/close 57569 (time 1734259362.57 total 320.07 last 215.41) - open/close 59176 (time 1734259372.58 total 330.07 last 160.62) - open/close 60000 (time 1734259377.24 total 334.73 last 176.74) - open/close 61951 (time 1734259387.24 total 344.74 last 195.07) - open/close 63721 (time 1734259397.25 total 354.74 last 176.92) total: 64768 open/close in 360.72 seconds: 179.55 ops/second - /unlink 3255 (time 1734259414.43 total 10.00 last 325.48) - /unlink 6177 (time 1734259424.44 total 20.00 last 292.15) - /unlink 9281 (time 1734259434.44 total 30.01 last 310.19) - /unlink 10000 (time 1734259437.07 total 32.63 last 273.85) - /unlink 12717 (time 1734259447.07 total 42.64 last 271.65) - /unlink 15456 (time 1734259457.07 total 52.64 last 273.79) - /unlink 18433 (time 1734259467.08 total 62.64 last 297.65) - /unlink 20000 (time 1734259472.08 total 67.64 last 313.27) - /unlink 23295 (time 1734259482.08 total 77.65 last 329.46) - /unlink 26189 (time 1734259492.08 total 87.65 last 289.34) - /unlink 29416 (time 1734259502.09 total 97.65 last 322.58) - /unlink 30000 (time 1734259504.36 total 99.93 last 256.43) - /unlink 32971 (time 1734259514.36 total 109.93 last 297.06) - /unlink 36204 (time 1734259524.37 total 119.93 last 323.29) - /unlink 39008 (time 1734259534.37 total 129.93 last 280.36) - /unlink 40000 (time 1734259537.83 total 133.39 last 286.68) - /unlink 42994 (time 1734259547.83 total 143.39 last 299.35) - /unlink 45993 (time 1734259557.83 total 153.40 last 299.86) - /unlink 49197 (time 1734259567.83 total 163.40 last 320.37) - /unlink 50000 (time 1734259569.94 total 165.51 last 380.57) - /unlink 52915 (time 1734259579.94 total 175.51 last 291.48) - /unlink 56122 (time 1734259589.94 total 185.51 last 320.65) - /unlink 59708 (time 1734259599.94 total 195.51 last 358.56) - /unlink 63178 (time 1734259609.94 total 205.51 last 346.97) total: 64768 /unlink in 210.00 seconds: 308.42 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2471 (time 1734259642.83 total 10.00 last 247.03) - open/close 4603 (time 1734259652.84 total 20.01 last 213.12) - open/close 6728 (time 1734259662.84 total 30.01 last 212.46) - open/close 8515 (time 1734259672.84 total 40.01 last 178.65) - open/close 10000 (time 1734259680.43 total 47.60 last 195.60) - open/close 12108 (time 1734259690.44 total 57.61 last 210.75) - open/close 14469 (time 1734259700.44 total 67.61 last 236.05) - open/close 16834 (time 1734259710.44 total 77.61 last 236.47) - open/close 18690 (time 1734259720.44 total 87.61 last 185.56) - open/close 20000 (time 1734259726.47 total 93.64 last 217.26) - open/close 22045 (time 1734259736.47 total 103.64 last 204.45) - open/close 23704 (time 1734259746.48 total 113.65 last 165.85) - open/close 25696 (time 1734259756.48 total 123.65 last 199.20) - open/close 27393 (time 1734259766.48 total 133.65 last 169.68) - open/close 29102 (time 1734259776.48 total 143.65 last 170.80) - open/close 30000 (time 1734259781.66 total 148.82 last 173.64) - open/close 31729 (time 1734259791.66 total 158.83 last 172.87) - open/close 33761 (time 1734259801.66 total 168.83 last 203.16) - open/close 35991 (time 1734259811.66 total 178.83 last 223.00) - open/close 38156 (time 1734259821.66 total 188.83 last 216.46) - open/close 40000 (time 1734259830.11 total 197.28 last 218.35) - open/close 42047 (time 1734259840.11 total 207.28 last 204.68) - open/close 44212 (time 1734259850.11 total 217.28 last 216.43) - open/close 45883 (time 1734259860.11 total 227.28 last 167.06) - open/close 47920 (time 1734259870.12 total 237.29 last 203.61) - open/close 49760 (time 1734259880.12 total 247.29 last 183.88) - open/close 51715 (time 1734259890.13 total 257.30 last 195.43) - open/close 53874 (time 1734259900.13 total 267.30 last 215.89) - open/close 55909 (time 1734259910.13 total 277.30 last 203.48) - open/close 58020 (time 1734259920.13 total 287.30 last 211.07) - open/close 60000 (time 1734259927.56 total 294.73 last 266.36) - open/close 62670 (time 1734259937.57 total 304.74 last 266.87) total: 64768 open/close in 312.62 seconds: 207.18 ops/second - /unlink 4831 (time 1734259956.45 total 10.00 last 483.05) - /unlink 9413 (time 1734259966.46 total 20.00 last 458.18) - /unlink 14228 (time 1734259976.46 total 30.00 last 481.46) - /unlink 19265 (time 1734259986.46 total 40.00 last 503.70) - /unlink 23851 (time 1734259996.46 total 50.00 last 458.59) - /unlink 28792 (time 1734260006.46 total 60.00 last 494.03) - /unlink 30000 (time 1734260008.96 total 62.51 last 482.38) - /unlink 34551 (time 1734260018.96 total 72.51 last 455.09) - /unlink 39479 (time 1734260028.96 total 82.51 last 492.73) - /unlink 44323 (time 1734260038.96 total 92.51 last 484.33) - /unlink 48579 (time 1734260048.97 total 102.51 last 425.54) - /unlink 50000 (time 1734260052.62 total 106.17 last 388.69) - /unlink 54872 (time 1734260062.62 total 116.17 last 487.19) - /unlink 60000 (time 1734260071.89 total 125.44 last 553.35) total: 64768 /unlink in 135.16 seconds: 479.19 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2618 (time 1734260108.24 total 10.00 last 261.78) - open/close 5487 (time 1734260118.24 total 20.00 last 286.89) - open/close 8110 (time 1734260128.25 total 30.00 last 262.22) - open/close 10000 (time 1734260135.98 total 37.74 last 244.25) - open/close 12744 (time 1734260145.99 total 47.74 last 274.38) - open/close 15404 (time 1734260155.99 total 57.74 last 265.98) - open/close 18332 (time 1734260165.99 total 67.75 last 292.73) - open/close 20000 (time 1734260171.87 total 73.62 last 283.77) - open/close 22687 (time 1734260181.87 total 83.62 last 268.70) - open/close 25057 (time 1734260191.87 total 93.63 last 236.95) - open/close 27774 (time 1734260201.87 total 103.63 last 271.66) - open/close 30000 (time 1734260209.82 total 111.57 last 280.16) - open/close 32574 (time 1734260219.82 total 121.58 last 257.31) - open/close 34930 (time 1734260229.82 total 131.58 last 235.55) - open/close 37538 (time 1734260239.82 total 141.58 last 260.75) - open/close 40000 (time 1734260249.11 total 150.87 last 265.09) - open/close 42680 (time 1734260259.11 total 160.87 last 267.95) - open/close 45390 (time 1734260269.12 total 170.87 last 270.93) - open/close 48285 (time 1734260279.12 total 180.88 last 289.41) - open/close 50000 (time 1734260285.77 total 187.53 last 257.66) - open/close 52846 (time 1734260295.78 total 197.53 last 284.52) - open/close 55255 (time 1734260305.78 total 207.54 last 240.87) - open/close 57468 (time 1734260315.78 total 217.54 last 221.22) - open/close 59846 (time 1734260325.79 total 227.54 last 237.73) - open/close 62271 (time 1734260335.79 total 237.54 last 242.46) - open/close 64428 (time 1734260345.79 total 247.55 last 215.67) total: 64768 open/close in 248.87 seconds: 260.25 ops/second - /unlink 3780 (time 1734260358.16 total 10.00 last 377.94) - /unlink 7886 (time 1734260368.16 total 20.00 last 410.53) - /unlink 10000 (time 1734260373.10 total 24.95 last 427.35) - /unlink 14159 (time 1734260383.11 total 34.95 last 415.89) - /unlink 18169 (time 1734260393.11 total 44.95 last 400.97) - /unlink 20000 (time 1734260397.18 total 49.02 last 449.93) - /unlink 24661 (time 1734260407.18 total 59.02 last 466.06) - /unlink 29083 (time 1734260417.18 total 69.02 last 442.12) - /unlink 30000 (time 1734260419.23 total 71.08 last 446.17) - /unlink 34515 (time 1734260429.23 total 81.08 last 451.44) - /unlink 38403 (time 1734260439.24 total 91.08 last 388.74) - /unlink 40000 (time 1734260443.08 total 94.92 last 415.78) - /unlink 45465 (time 1734260453.08 total 104.92 last 546.50) - /unlink 50000 (time 1734260461.32 total 113.16 last 550.36) - /unlink 54917 (time 1734260471.32 total 123.16 last 491.64) - /unlink 59444 (time 1734260481.32 total 133.16 last 452.69) - /unlink 63594 (time 1734260491.32 total 143.16 last 414.99) total: 64768 /unlink in 145.90 seconds: 443.92 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2534 (time 1734260524.84 total 10.00 last 253.33) - open/close 5134 (time 1734260534.84 total 20.00 last 259.95) - open/close 7595 (time 1734260544.84 total 30.01 last 246.07) - open/close 10000 (time 1734260554.61 total 39.78 last 246.10) - open/close 12510 (time 1734260564.61 total 49.78 last 250.93) - open/close 15231 (time 1734260574.62 total 59.78 last 272.05) - open/close 17842 (time 1734260584.62 total 69.78 last 261.06) - open/close 20000 (time 1734260591.61 total 76.77 last 308.75) - open/close 23469 (time 1734260601.61 total 86.78 last 346.86) - open/close 26369 (time 1734260611.61 total 96.78 last 289.94) - open/close 29156 (time 1734260621.61 total 106.78 last 278.70) - open/close 30000 (time 1734260624.25 total 109.42 last 319.97) - open/close 33333 (time 1734260634.25 total 119.42 last 333.24) - open/close 36844 (time 1734260644.25 total 129.42 last 351.08) - open/close 39590 (time 1734260654.25 total 139.42 last 274.55) - open/close 43082 (time 1734260664.25 total 149.42 last 349.15) - open/close 46238 (time 1734260674.26 total 159.42 last 315.53) - open/close 49595 (time 1734260684.26 total 169.42 last 335.69) - open/close 52248 (time 1734260694.26 total 179.43 last 265.24) - open/close 55000 (time 1734260704.26 total 189.43 last 275.14) - open/close 57951 (time 1734260714.26 total 199.43 last 295.05) - open/close 60000 (time 1734260720.95 total 206.11 last 306.59) - open/close 63031 (time 1734260730.95 total 216.11 last 303.05) total: 64768 open/close in 222.20 seconds: 291.48 ops/second - /unlink 4740 (time 1734260748.06 total 10.00 last 473.97) - /unlink 9740 (time 1734260758.06 total 20.00 last 499.95) - /unlink 14683 (time 1734260768.06 total 30.00 last 494.22) - /unlink 20000 (time 1734260777.15 total 39.09 last 585.15) - /unlink 25549 (time 1734260787.15 total 49.09 last 554.84) - /unlink 30000 (time 1734260795.12 total 57.06 last 558.21) - /unlink 36185 (time 1734260805.13 total 67.07 last 618.41) - /unlink 40000 (time 1734260811.34 total 73.28 last 613.78) - /unlink 46514 (time 1734260821.34 total 83.28 last 651.36) - /unlink 50000 (time 1734260826.39 total 88.33 last 690.41) - /unlink 56341 (time 1734260836.39 total 98.33 last 634.03) - /unlink 60000 (time 1734260842.00 total 103.94 last 652.67) total: 64768 /unlink in 110.80 seconds: 584.53 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3922 (time 1734260880.12 total 10.00 last 392.17) - open/close 8664 (time 1734260890.12 total 20.00 last 474.08) - open/close 10000 (time 1734260893.11 total 23.00 last 446.32) - open/close 14620 (time 1734260903.11 total 33.00 last 461.99) - open/close 18818 (time 1734260913.11 total 43.00 last 419.79) - open/close 20000 (time 1734260915.74 total 45.63 last 449.28) - open/close 23716 (time 1734260925.75 total 55.63 last 371.54) - open/close 27672 (time 1734260935.75 total 65.63 last 395.58) - open/close 30000 (time 1734260941.40 total 71.28 last 412.08) - open/close 33958 (time 1734260951.40 total 81.28 last 395.75) - open/close 38129 (time 1734260961.40 total 91.28 last 417.02) - open/close 40000 (time 1734260965.89 total 95.78 last 416.29) - open/close 44125 (time 1734260975.89 total 105.78 last 412.48) - open/close 48654 (time 1734260985.90 total 115.78 last 452.83) - open/close 50000 (time 1734260988.59 total 118.48 last 499.08) - open/close 53828 (time 1734260998.59 total 128.48 last 382.71) - open/close 57682 (time 1734261008.60 total 138.48 last 385.34) - open/close 60000 (time 1734261013.54 total 143.42 last 469.20) - open/close 64555 (time 1734261023.54 total 153.42 last 455.49) total: 64768 open/close in 153.92 seconds: 420.80 ops/second - /unlink 7211 (time 1734261034.92 total 10.00 last 721.05) - /unlink 10000 (time 1734261038.42 total 13.50 last 796.77) - /unlink 18052 (time 1734261048.42 total 23.50 last 805.12) - /unlink 20000 (time 1734261050.70 total 25.79 last 851.20) - /unlink 28944 (time 1734261060.70 total 35.79 last 894.40) - /unlink 38354 (time 1734261070.71 total 45.79 last 940.98) - /unlink 48162 (time 1734261080.71 total 55.79 last 980.74) - /unlink 57552 (time 1734261090.71 total 65.79 last 938.96) - /unlink 60000 (time 1734261093.06 total 68.15 last 1038.39) total: 64768 /unlink in 72.85 seconds: 889.06 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg252-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg252-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg252-server stop mds service on oleg252-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg252-server stop mds service on oleg252-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg252-server unloading modules via unload_modules_local on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg252-client: oleg252-client: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-server: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-client: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-server: ssh exited with exit code 2