== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:36:48 (1734259008) Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 oleg257-server: oleg257-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: detected 4 online CPUs by sysfs oleg257-server: Force libcfs to create 2 CPU partitions oleg257-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg257-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg257-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg257-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg257-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg257-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock 192.168.202.157@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16266 458 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16451 643 15808 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1967 (time 1734259077.66 total 10.01 last 196.57) - open/close 3562 (time 1734259087.67 total 20.01 last 159.42) - open/close 5261 (time 1734259097.67 total 30.01 last 169.87) - open/close 6934 (time 1734259107.67 total 40.01 last 167.29) - open/close 8830 (time 1734259117.67 total 50.02 last 189.57) - open/close 10000 (time 1734259123.34 total 55.68 last 206.38) - open/close 11745 (time 1734259133.35 total 65.69 last 174.41) - open/close 13396 (time 1734259143.35 total 75.69 last 165.03) - open/close 15232 (time 1734259153.35 total 85.70 last 183.55) - open/close 16950 (time 1734259163.35 total 95.70 last 171.78) - open/close 18875 (time 1734259173.35 total 105.70 last 192.49) - open/close 20000 (time 1734259180.31 total 112.65 last 161.73) - open/close 21784 (time 1734259190.31 total 122.66 last 178.34) - open/close 23568 (time 1734259200.32 total 132.67 last 178.26) - open/close 25267 (time 1734259210.33 total 142.67 last 169.83) - open/close 27322 (time 1734259220.33 total 152.67 last 205.44) - open/close 28733 (time 1734259230.33 total 162.67 last 141.09) - open/close 30000 (time 1734259238.44 total 170.78 last 156.27) - open/close 31812 (time 1734259248.44 total 180.78 last 181.16) - open/close 33654 (time 1734259258.44 total 190.79 last 184.13) - open/close 35548 (time 1734259268.45 total 200.79 last 189.31) - open/close 37464 (time 1734259278.45 total 210.79 last 191.56) - open/close 39800 (time 1734259288.45 total 220.80 last 233.55) - open/close 41770 (time 1734259298.45 total 230.80 last 196.95) - open/close 43518 (time 1734259308.46 total 240.80 last 174.76) - open/close 45149 (time 1734259318.46 total 250.80 last 163.04) - open/close 46793 (time 1734259328.46 total 260.81 last 164.37) - open/close 48615 (time 1734259338.46 total 270.81 last 182.20) - open/close 50000 (time 1734259346.05 total 278.39 last 182.52) - open/close 51890 (time 1734259356.05 total 288.40 last 188.98) - open/close 53933 (time 1734259366.06 total 298.40 last 204.20) - open/close 55559 (time 1734259376.06 total 308.41 last 162.47) - open/close 57566 (time 1734259386.07 total 318.41 last 200.67) - open/close 59308 (time 1734259396.07 total 328.41 last 174.14) - open/close 60000 (time 1734259399.79 total 332.14 last 185.86) - open/close 61981 (time 1734259409.79 total 342.14 last 198.07) - open/close 63675 (time 1734259419.80 total 352.14 last 169.37) total: 64768 open/close in 358.75 seconds: 180.54 ops/second - /unlink 3107 (time 1734259437.68 total 10.00 last 310.63) - /unlink 5881 (time 1734259447.68 total 20.00 last 277.39) - /unlink 8613 (time 1734259457.68 total 30.00 last 273.16) - /unlink 10000 (time 1734259462.40 total 34.73 last 293.61) - /unlink 13097 (time 1734259472.41 total 44.73 last 309.60) - /unlink 16413 (time 1734259482.41 total 54.74 last 331.46) - /unlink 19202 (time 1734259492.41 total 64.74 last 278.86) - /unlink 20000 (time 1734259495.05 total 67.37 last 302.70) - /unlink 22945 (time 1734259505.05 total 77.38 last 294.45) - /unlink 25915 (time 1734259515.05 total 87.38 last 296.92) - /unlink 29238 (time 1734259525.05 total 97.38 last 332.28) - /unlink 30000 (time 1734259527.56 total 99.88 last 304.44) - /unlink 32644 (time 1734259537.56 total 109.88 last 264.35) - /unlink 35609 (time 1734259547.56 total 119.88 last 296.45) - /unlink 38592 (time 1734259557.56 total 129.89 last 298.20) - /unlink 40000 (time 1734259562.26 total 134.59 last 299.65) - /unlink 43318 (time 1734259572.26 total 144.59 last 331.76) - /unlink 46069 (time 1734259582.26 total 154.59 last 275.05) - /unlink 49259 (time 1734259592.27 total 164.59 last 318.98) - /unlink 52827 (time 1734259602.27 total 174.59 last 356.79) - /unlink 56170 (time 1734259612.27 total 184.59 last 334.26) - /unlink 59677 (time 1734259622.27 total 194.59 last 350.63) - /unlink 63491 (time 1734259632.27 total 204.60 last 381.35) total: 64768 /unlink in 208.10 seconds: 311.24 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2109 (time 1734259663.71 total 10.00 last 210.86) - open/close 3908 (time 1734259673.71 total 20.00 last 179.88) - open/close 6017 (time 1734259683.72 total 30.01 last 210.84) - open/close 8097 (time 1734259693.72 total 40.01 last 207.93) - open/close 10000 (time 1734259701.25 total 47.54 last 252.77) - open/close 12387 (time 1734259711.25 total 57.54 last 238.65) - open/close 14314 (time 1734259721.25 total 67.54 last 192.69) - open/close 16675 (time 1734259731.25 total 77.54 last 236.05) - open/close 18418 (time 1734259741.26 total 87.55 last 174.20) - open/close 20000 (time 1734259749.82 total 96.11 last 184.82) - open/close 21970 (time 1734259759.82 total 106.11 last 196.92) - open/close 23655 (time 1734259769.82 total 116.11 last 168.47) - open/close 25518 (time 1734259779.83 total 126.12 last 186.20) - open/close 27202 (time 1734259789.83 total 136.12 last 168.39) - open/close 29182 (time 1734259799.83 total 146.12 last 197.99) - open/close 30000 (time 1734259803.27 total 149.56 last 237.87) - open/close 32231 (time 1734259813.27 total 159.56 last 223.05) - open/close 34514 (time 1734259823.27 total 169.56 last 228.23) - open/close 36680 (time 1734259833.28 total 179.57 last 216.56) - open/close 38769 (time 1734259843.28 total 189.57 last 208.89) - open/close 40000 (time 1734259848.35 total 194.64 last 242.79) - open/close 41635 (time 1734259858.35 total 204.64 last 163.41) - open/close 43617 (time 1734259868.35 total 214.64 last 198.18) - open/close 45474 (time 1734259878.35 total 224.64 last 185.69) - open/close 47339 (time 1734259888.35 total 234.64 last 186.50) - open/close 49551 (time 1734259898.35 total 244.64 last 221.17) - open/close 50000 (time 1734259900.90 total 247.19 last 176.29) - open/close 52049 (time 1734259910.90 total 257.19 last 204.87) - open/close 54155 (time 1734259920.90 total 267.19 last 210.59) - open/close 56786 (time 1734259930.90 total 277.19 last 263.08) - open/close 59533 (time 1734259940.91 total 287.20 last 274.64) - open/close 62302 (time 1734259950.91 total 297.20 last 276.85) total: 64768 open/close in 306.43 seconds: 211.37 ops/second - /unlink 4866 (time 1734259971.15 total 10.00 last 486.50) - /unlink 9892 (time 1734259981.15 total 20.00 last 502.55) - /unlink 14924 (time 1734259991.15 total 30.00 last 503.14) - /unlink 19538 (time 1734260001.15 total 40.00 last 461.37) - /unlink 24710 (time 1734260011.15 total 50.01 last 517.14) - /unlink 29398 (time 1734260021.15 total 60.01 last 468.68) - /unlink 34597 (time 1734260031.16 total 70.01 last 519.81) - /unlink 39596 (time 1734260041.16 total 80.01 last 499.88) - /unlink 43693 (time 1734260051.16 total 90.01 last 409.60) - /unlink 48527 (time 1734260061.16 total 100.01 last 483.38) - /unlink 50000 (time 1734260063.79 total 102.65 last 559.20) - /unlink 55729 (time 1734260073.79 total 112.65 last 572.86) - /unlink 60000 (time 1734260082.59 total 121.45 last 485.42) total: 64768 /unlink in 131.17 seconds: 493.78 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2905 (time 1734260118.96 total 10.00 last 290.43) - open/close 5718 (time 1734260128.97 total 20.01 last 281.21) - open/close 8325 (time 1734260138.97 total 30.01 last 260.64) - open/close 10000 (time 1734260144.90 total 35.93 last 282.70) - open/close 12804 (time 1734260154.90 total 45.94 last 280.31) - open/close 15931 (time 1734260164.90 total 55.94 last 312.61) - open/close 18797 (time 1734260174.90 total 65.94 last 286.59) - open/close 20000 (time 1734260179.06 total 70.10 last 289.07) - open/close 22562 (time 1734260189.07 total 80.10 last 256.15) - open/close 25188 (time 1734260199.07 total 90.10 last 262.57) - open/close 28101 (time 1734260209.07 total 100.11 last 291.22) - open/close 30000 (time 1734260216.21 total 107.25 last 265.91) - open/close 32639 (time 1734260226.21 total 117.25 last 263.83) - open/close 35022 (time 1734260236.22 total 127.25 last 238.20) - open/close 37822 (time 1734260246.22 total 137.26 last 279.99) - open/close 40000 (time 1734260254.25 total 145.29 last 271.19) - open/close 42733 (time 1734260264.25 total 155.29 last 273.29) - open/close 45641 (time 1734260274.25 total 165.29 last 290.73) - open/close 48497 (time 1734260284.25 total 175.29 last 285.55) - open/close 50000 (time 1734260289.59 total 180.63 last 281.71) - open/close 52722 (time 1734260299.59 total 190.63 last 272.10) - open/close 55110 (time 1734260309.59 total 200.63 last 238.80) - open/close 57420 (time 1734260319.60 total 210.63 last 230.91) - open/close 59877 (time 1734260329.60 total 220.64 last 245.66) - open/close 62271 (time 1734260339.60 total 230.64 last 239.39) - open/close 64544 (time 1734260349.60 total 240.64 last 227.23) total: 64768 open/close in 241.63 seconds: 268.05 ops/second - /unlink 3911 (time 1734260361.71 total 10.00 last 390.99) - /unlink 8189 (time 1734260371.71 total 20.00 last 427.78) - /unlink 10000 (time 1734260376.10 total 24.39 last 413.24) - /unlink 14079 (time 1734260386.10 total 34.39 last 407.84) - /unlink 18447 (time 1734260396.10 total 44.39 last 436.73) - /unlink 20000 (time 1734260399.28 total 47.57 last 488.36) - /unlink 24730 (time 1734260409.28 total 57.57 last 472.97) - /unlink 29152 (time 1734260419.28 total 67.57 last 442.13) - /unlink 33712 (time 1734260429.28 total 77.57 last 455.97) - /unlink 37820 (time 1734260439.28 total 87.57 last 410.71) - /unlink 40000 (time 1734260444.31 total 92.60 last 433.72) - /unlink 45711 (time 1734260454.31 total 102.60 last 571.08) - /unlink 50000 (time 1734260461.99 total 110.28 last 558.23) - /unlink 55033 (time 1734260471.99 total 120.28 last 503.29) - /unlink 59671 (time 1734260481.99 total 130.28 last 463.79) - /unlink 63932 (time 1734260492.00 total 140.29 last 426.03) total: 64768 /unlink in 142.15 seconds: 455.63 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2684 (time 1734260519.33 total 10.07 last 266.60) - open/close 5256 (time 1734260529.34 total 20.07 last 257.13) - open/close 7817 (time 1734260539.34 total 30.07 last 256.02) - open/close 10000 (time 1734260547.84 total 38.58 last 256.75) - open/close 12591 (time 1734260557.84 total 48.58 last 259.07) - open/close 15213 (time 1734260567.85 total 58.58 last 262.10) - open/close 17984 (time 1734260577.85 total 68.58 last 277.02) - open/close 20000 (time 1734260585.04 total 75.77 last 280.50) - open/close 23353 (time 1734260595.04 total 85.77 last 335.24) - open/close 26802 (time 1734260605.04 total 95.77 last 344.89) - open/close 29595 (time 1734260615.04 total 105.78 last 279.20) - open/close 32692 (time 1734260625.05 total 115.78 last 309.64) - open/close 36084 (time 1734260635.05 total 125.78 last 339.14) - open/close 39676 (time 1734260645.05 total 135.78 last 359.18) - open/close 42488 (time 1734260655.05 total 145.78 last 281.11) - open/close 46035 (time 1734260665.05 total 155.79 last 354.62) - open/close 49229 (time 1734260675.05 total 165.79 last 319.37) - open/close 50000 (time 1734260677.40 total 168.14 last 328.14) - open/close 53210 (time 1734260687.41 total 178.14 last 320.94) - open/close 55928 (time 1734260697.41 total 188.14 last 271.76) - open/close 58636 (time 1734260707.41 total 198.14 last 270.74) - open/close 60000 (time 1734260711.94 total 202.67 last 301.23) - open/close 63165 (time 1734260721.94 total 212.67 last 316.50) total: 64768 open/close in 218.11 seconds: 296.95 ops/second - /unlink 5061 (time 1734260738.32 total 10.00 last 506.05) - /unlink 9957 (time 1734260748.33 total 20.00 last 489.54) - /unlink 15066 (time 1734260758.33 total 30.00 last 510.79) - /unlink 20000 (time 1734260768.32 total 39.99 last 493.87) - /unlink 25970 (time 1734260778.32 total 50.00 last 596.94) - /unlink 30000 (time 1734260785.28 total 56.96 last 578.96) - /unlink 35657 (time 1734260795.28 total 66.96 last 565.59) - /unlink 40000 (time 1734260802.14 total 73.81 last 633.49) - /unlink 46282 (time 1734260812.14 total 83.82 last 628.14) - /unlink 50000 (time 1734260817.83 total 89.51 last 652.81) - /unlink 56809 (time 1734260827.83 total 99.51 last 680.88) - /unlink 60000 (time 1734260832.61 total 104.29 last 667.65) total: 64768 /unlink in 111.33 seconds: 581.75 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3789 (time 1734260870.27 total 10.00 last 378.82) - open/close 7792 (time 1734260880.28 total 20.00 last 400.26) - open/close 10000 (time 1734260884.92 total 24.65 last 475.33) - open/close 14736 (time 1734260894.92 total 34.65 last 473.54) - open/close 19251 (time 1734260904.92 total 44.65 last 451.47) - open/close 23623 (time 1734260914.93 total 54.65 last 437.10) - open/close 27460 (time 1734260924.93 total 64.65 last 383.67) - open/close 30000 (time 1734260931.81 total 71.53 last 369.14) - open/close 33960 (time 1734260941.81 total 81.53 last 395.98) - open/close 37705 (time 1734260951.81 total 91.54 last 374.45) - open/close 40000 (time 1734260957.84 total 97.56 last 380.82) - open/close 43928 (time 1734260967.84 total 107.56 last 392.77) - open/close 48077 (time 1734260977.84 total 117.56 last 414.89) - open/close 50000 (time 1734260982.25 total 121.97 last 436.12) - open/close 54375 (time 1734260992.25 total 131.98 last 437.41) - open/close 58119 (time 1734261002.25 total 141.98 last 374.36) - open/close 60000 (time 1734261007.10 total 146.82 last 388.01) - open/close 64559 (time 1734261017.10 total 156.82 last 455.86) total: 64768 open/close in 157.24 seconds: 411.91 ops/second - /unlink 7863 (time 1734261028.38 total 10.00 last 786.22) - /unlink 10000 (time 1734261031.39 total 13.02 last 708.11) - /unlink 17975 (time 1734261041.39 total 23.02 last 797.48) - /unlink 20000 (time 1734261044.09 total 25.71 last 752.36) - /unlink 28123 (time 1734261054.09 total 35.71 last 812.27) - /unlink 30000 (time 1734261056.13 total 37.76 last 918.28) - /unlink 39247 (time 1734261066.13 total 47.76 last 924.70) - /unlink 48965 (time 1734261076.13 total 57.76 last 971.73) - /unlink 58757 (time 1734261086.13 total 67.76 last 979.14) total: 64768 /unlink in 73.99 seconds: 875.38 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg257-server unloading modules via unload_modules_local on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2