== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:35:45 (1734258945) Stopping clients: oleg334-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg334-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2 oleg334-server: oleg334-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing load_modules_local oleg334-server: Loading modules from /home/green/git/lustre-release/lustre oleg334-server: detected 4 online CPUs by sysfs oleg334-server: Force libcfs to create 2 CPU partitions oleg334-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg334-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg334-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg334-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg334-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16266 458 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16451 643 15808 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1670 (time 1734259020.82 total 10.00 last 166.99) - open/close 3219 (time 1734259030.82 total 20.00 last 154.86) - open/close 4773 (time 1734259040.83 total 30.00 last 155.39) - open/close 6290 (time 1734259050.83 total 40.01 last 151.68) - open/close 7938 (time 1734259060.83 total 50.01 last 164.72) - open/close 9429 (time 1734259070.84 total 60.01 last 149.03) - open/close 10000 (time 1734259074.99 total 64.17 last 137.31) - open/close 11381 (time 1734259084.99 total 74.17 last 138.10) - open/close 12818 (time 1734259095.00 total 84.18 last 143.60) - open/close 14135 (time 1734259105.01 total 94.19 last 131.63) - open/close 15599 (time 1734259115.01 total 104.19 last 146.35) - open/close 17330 (time 1734259125.01 total 114.19 last 173.04) - open/close 18796 (time 1734259135.02 total 124.20 last 146.52) - open/close 20000 (time 1734259143.52 total 132.70 last 141.62) - open/close 21485 (time 1734259153.52 total 142.70 last 148.50) - open/close 23042 (time 1734259163.52 total 152.70 last 155.66) - open/close 24674 (time 1734259173.53 total 162.71 last 163.11) - open/close 26279 (time 1734259183.54 total 172.71 last 160.40) - open/close 27736 (time 1734259193.54 total 182.72 last 145.68) - open/close 29288 (time 1734259203.54 total 192.72 last 155.18) - open/close 30000 (time 1734259208.45 total 197.63 last 145.04) - open/close 31384 (time 1734259218.45 total 207.63 last 138.40) - open/close 32915 (time 1734259228.45 total 217.63 last 153.07) - open/close 34457 (time 1734259238.46 total 227.63 last 154.09) - open/close 36321 (time 1734259248.46 total 237.64 last 186.37) - open/close 38309 (time 1734259258.46 total 247.64 last 198.79) - open/close 40000 (time 1734259267.89 total 257.07 last 179.33) - open/close 41500 (time 1734259277.89 total 267.07 last 149.97) - open/close 43231 (time 1734259287.89 total 277.07 last 173.09) - open/close 44786 (time 1734259297.90 total 287.07 last 155.42) - open/close 46382 (time 1734259307.90 total 297.07 last 159.59) - open/close 47921 (time 1734259317.90 total 307.08 last 153.85) - open/close 49695 (time 1734259327.90 total 317.08 last 177.35) - open/close 51455 (time 1734259337.90 total 327.08 last 175.94) - open/close 53195 (time 1734259347.91 total 337.09 last 173.92) - open/close 54744 (time 1734259357.91 total 347.09 last 154.86) - open/close 56283 (time 1734259367.91 total 357.09 last 153.86) - open/close 57854 (time 1734259377.92 total 367.10 last 157.02) - open/close 59388 (time 1734259387.92 total 377.10 last 153.32) - open/close 60000 (time 1734259391.69 total 380.87 last 162.67) - open/close 61619 (time 1734259401.69 total 390.87 last 161.84) - open/close 63121 (time 1734259411.69 total 400.87 last 150.17) - open/close 64633 (time 1734259421.69 total 410.87 last 151.19) total: 64768 open/close in 411.71 seconds: 157.31 ops/second - /unlink 2555 (time 1734259434.06 total 10.00 last 255.42) - /unlink 5196 (time 1734259444.06 total 20.00 last 264.07) - /unlink 7837 (time 1734259454.06 total 30.01 last 264.02) - /unlink 10000 (time 1734259462.69 total 38.63 last 250.70) - /unlink 12572 (time 1734259472.69 total 48.64 last 257.13) - /unlink 15368 (time 1734259482.70 total 58.64 last 279.53) - /unlink 18285 (time 1734259492.70 total 68.64 last 291.67) - /unlink 20000 (time 1734259498.65 total 74.60 last 287.87) - /unlink 22474 (time 1734259508.66 total 84.60 last 247.34) - /unlink 25344 (time 1734259518.66 total 94.60 last 287.00) - /unlink 28406 (time 1734259528.66 total 104.60 last 306.15) - /unlink 30000 (time 1734259533.05 total 109.00 last 362.87) - /unlink 32919 (time 1734259543.05 total 119.00 last 291.87) - /unlink 35672 (time 1734259553.06 total 129.00 last 275.19) - /unlink 38581 (time 1734259563.06 total 139.00 last 290.85) - /unlink 40000 (time 1734259568.17 total 144.12 last 277.39) - /unlink 42656 (time 1734259578.18 total 154.12 last 265.51) - /unlink 45846 (time 1734259588.18 total 164.12 last 318.99) - /unlink 49045 (time 1734259598.18 total 174.12 last 319.88) - /unlink 50000 (time 1734259601.05 total 177.00 last 332.15) - /unlink 53649 (time 1734259611.05 total 187.00 last 364.86) - /unlink 57264 (time 1734259621.05 total 197.00 last 361.48) - /unlink 60000 (time 1734259628.71 total 204.65 last 357.57) - /unlink 63592 (time 1734259638.71 total 214.65 last 359.14) total: 64768 /unlink in 217.89 seconds: 297.25 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2001 (time 1734259670.90 total 10.01 last 199.90) - open/close 3960 (time 1734259680.90 total 20.01 last 195.85) - open/close 5969 (time 1734259690.91 total 30.02 last 200.85) - open/close 8123 (time 1734259700.91 total 40.02 last 215.37) - open/close 10000 (time 1734259709.41 total 48.52 last 220.68) - open/close 11972 (time 1734259719.42 total 58.53 last 197.09) - open/close 13784 (time 1734259729.42 total 68.53 last 181.17) - open/close 15499 (time 1734259739.42 total 78.53 last 171.49) - open/close 17291 (time 1734259749.42 total 88.53 last 179.14) - open/close 18936 (time 1734259759.42 total 98.53 last 164.49) - open/close 20000 (time 1734259765.22 total 104.33 last 183.52) - open/close 21977 (time 1734259775.22 total 114.33 last 197.63) - open/close 24029 (time 1734259785.23 total 124.34 last 205.15) - open/close 26099 (time 1734259795.23 total 134.34 last 206.97) - open/close 27861 (time 1734259805.23 total 144.34 last 176.18) - open/close 29726 (time 1734259815.23 total 154.34 last 186.49) - open/close 31771 (time 1734259825.23 total 164.34 last 204.45) - open/close 34058 (time 1734259835.24 total 174.35 last 228.62) - open/close 36261 (time 1734259845.24 total 184.35 last 220.26) - open/close 38528 (time 1734259855.24 total 194.35 last 226.68) - open/close 40000 (time 1734259861.94 total 201.05 last 219.64) - open/close 41956 (time 1734259871.95 total 211.06 last 195.51) - open/close 43882 (time 1734259881.95 total 221.06 last 192.51) - open/close 45668 (time 1734259891.95 total 231.06 last 178.52) - open/close 47382 (time 1734259901.96 total 241.07 last 171.36) - open/close 49282 (time 1734259911.96 total 251.07 last 189.95) - open/close 50000 (time 1734259915.48 total 254.59 last 204.18) - open/close 52100 (time 1734259925.48 total 264.59 last 209.98) - open/close 54160 (time 1734259935.48 total 274.59 last 205.98) - open/close 56089 (time 1734259945.48 total 284.59 last 192.85) - open/close 58072 (time 1734259955.48 total 294.59 last 198.27) - open/close 59995 (time 1734259965.48 total 304.59 last 192.25) - open/close 62051 (time 1734259975.49 total 314.60 last 205.55) - open/close 64098 (time 1734259985.49 total 324.60 last 204.69) total: 64768 open/close in 327.81 seconds: 197.58 ops/second - /unlink 3996 (time 1734259999.94 total 10.00 last 399.54) - /unlink 8547 (time 1734260009.94 total 20.00 last 455.04) - /unlink 10000 (time 1734260013.28 total 23.34 last 435.33) - /unlink 13949 (time 1734260023.28 total 33.34 last 394.86) - /unlink 18060 (time 1734260033.28 total 43.34 last 411.09) - /unlink 20000 (time 1734260038.17 total 48.23 last 396.58) - /unlink 23874 (time 1734260048.17 total 58.24 last 387.34) - /unlink 28653 (time 1734260058.17 total 68.24 last 477.87) - /unlink 30000 (time 1734260060.55 total 70.62 last 565.86) - /unlink 35026 (time 1734260070.56 total 80.62 last 502.51) - /unlink 39501 (time 1734260080.56 total 90.62 last 447.45) - /unlink 44081 (time 1734260090.56 total 100.62 last 457.94) - /unlink 48401 (time 1734260100.56 total 110.62 last 431.83) - /unlink 50000 (time 1734260104.13 total 114.19 last 448.77) - /unlink 54716 (time 1734260114.13 total 124.19 last 471.59) - /unlink 59072 (time 1734260124.13 total 134.19 last 435.54) - /unlink 60000 (time 1734260126.28 total 136.34 last 431.65) - /unlink 64183 (time 1734260136.28 total 146.34 last 418.22) total: 64768 /unlink in 147.75 seconds: 438.36 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2950 (time 1734260167.17 total 10.00 last 294.98) - open/close 5690 (time 1734260177.17 total 20.00 last 273.92) - open/close 8417 (time 1734260187.17 total 30.00 last 272.66) - open/close 10000 (time 1734260192.67 total 35.50 last 287.84) - open/close 12498 (time 1734260202.68 total 45.51 last 249.72) - open/close 15091 (time 1734260212.68 total 55.51 last 259.24) - open/close 17720 (time 1734260222.68 total 65.51 last 262.89) - open/close 20000 (time 1734260231.65 total 74.48 last 254.18) - open/close 22547 (time 1734260241.65 total 84.48 last 254.63) - open/close 25149 (time 1734260251.66 total 94.49 last 260.13) - open/close 28095 (time 1734260261.66 total 104.49 last 294.59) - open/close 30000 (time 1734260268.59 total 111.42 last 274.75) - open/close 32350 (time 1734260278.59 total 121.42 last 234.94) - open/close 34745 (time 1734260288.59 total 131.42 last 239.50) - open/close 36954 (time 1734260298.59 total 141.42 last 220.88) - open/close 39111 (time 1734260308.59 total 151.42 last 215.68) - open/close 40000 (time 1734260312.26 total 155.09 last 242.55) - open/close 42247 (time 1734260322.26 total 165.09 last 224.69) - open/close 44665 (time 1734260332.26 total 175.09 last 241.79) - open/close 46997 (time 1734260342.26 total 185.09 last 233.17) - open/close 49451 (time 1734260352.26 total 195.09 last 245.31) - open/close 50000 (time 1734260354.50 total 197.33 last 245.68) - open/close 52261 (time 1734260364.50 total 207.33 last 226.06) - open/close 54543 (time 1734260374.50 total 217.34 last 228.10) - open/close 56659 (time 1734260384.51 total 227.34 last 211.46) - open/close 58733 (time 1734260394.51 total 237.34 last 207.36) - open/close 60000 (time 1734260400.83 total 243.66 last 200.70) - open/close 61917 (time 1734260410.83 total 253.66 last 191.60) - open/close 64146 (time 1734260420.83 total 263.66 last 222.87) total: 64768 open/close in 266.55 seconds: 242.99 ops/second - /unlink 3680 (time 1734260434.92 total 10.00 last 367.98) - /unlink 7205 (time 1734260444.93 total 20.00 last 352.43) - /unlink 10000 (time 1734260452.23 total 27.31 last 382.67) - /unlink 13869 (time 1734260462.23 total 37.31 last 386.87) - /unlink 17924 (time 1734260472.23 total 47.31 last 405.45) - /unlink 20000 (time 1734260477.02 total 52.09 last 433.80) - /unlink 24606 (time 1734260487.02 total 62.10 last 460.49) - /unlink 28842 (time 1734260497.02 total 72.10 last 423.52) - /unlink 30000 (time 1734260499.71 total 74.79 last 430.25) - /unlink 34566 (time 1734260509.72 total 84.79 last 456.53) - /unlink 38955 (time 1734260519.72 total 94.79 last 438.89) - /unlink 40000 (time 1734260521.94 total 97.02 last 469.35) - /unlink 44912 (time 1734260531.94 total 107.02 last 491.17) - /unlink 49619 (time 1734260541.95 total 117.02 last 470.58) - /unlink 53497 (time 1734260551.95 total 127.02 last 387.79) - /unlink 57734 (time 1734260561.95 total 137.02 last 423.57) - /unlink 60000 (time 1734260567.33 total 142.40 last 421.26) total: 64768 /unlink in 152.37 seconds: 425.08 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2503 (time 1734260607.94 total 10.00 last 250.22) - open/close 5167 (time 1734260617.95 total 20.00 last 266.36) - open/close 7797 (time 1734260627.95 total 30.01 last 262.94) - open/close 10000 (time 1734260637.02 total 39.08 last 242.90) - open/close 12655 (time 1734260647.02 total 49.08 last 265.43) - open/close 15486 (time 1734260657.02 total 59.08 last 283.04) - open/close 18493 (time 1734260667.02 total 69.08 last 300.66) - open/close 20000 (time 1734260672.87 total 74.93 last 257.72) - open/close 22641 (time 1734260682.87 total 84.93 last 264.04) - open/close 25377 (time 1734260692.88 total 94.93 last 273.54) - open/close 28029 (time 1734260702.88 total 104.94 last 265.17) - open/close 30000 (time 1734260710.06 total 112.12 last 274.53) - open/close 33002 (time 1734260720.06 total 122.12 last 300.15) - open/close 36233 (time 1734260730.06 total 132.12 last 323.07) - open/close 39495 (time 1734260740.06 total 142.12 last 326.17) - open/close 43074 (time 1734260750.06 total 152.12 last 357.78) - open/close 46263 (time 1734260760.06 total 162.12 last 318.82) - open/close 50000 (time 1734260769.97 total 172.03 last 377.46) - open/close 54713 (time 1734260779.97 total 182.03 last 471.27) - open/close 59064 (time 1734260789.97 total 192.03 last 435.07) - open/close 64410 (time 1734260799.97 total 202.03 last 534.59) total: 64768 open/close in 202.80 seconds: 319.37 ops/second - /unlink 7765 (time 1734260811.62 total 10.00 last 776.44) - /unlink 10000 (time 1734260814.49 total 12.87 last 778.82) - /unlink 17828 (time 1734260824.49 total 22.87 last 782.79) - /unlink 20000 (time 1734260827.37 total 25.74 last 755.77) - /unlink 28048 (time 1734260837.37 total 35.74 last 804.79) - /unlink 30000 (time 1734260839.85 total 38.23 last 785.78) - /unlink 37610 (time 1734260849.85 total 48.23 last 760.92) - /unlink 40000 (time 1734260853.12 total 51.50 last 731.30) - /unlink 47914 (time 1734260863.12 total 61.50 last 791.40) - /unlink 50000 (time 1734260865.47 total 63.85 last 887.84) - /unlink 59580 (time 1734260875.47 total 73.85 last 957.94) total: 64768 /unlink in 79.11 seconds: 818.66 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4938 (time 1734260913.59 total 10.00 last 493.67) - open/close 9288 (time 1734260923.59 total 20.00 last 434.93) - open/close 14312 (time 1734260933.59 total 30.00 last 502.36) - open/close 19963 (time 1734260943.59 total 40.01 last 565.08) - open/close 25482 (time 1734260953.59 total 50.01 last 551.86) - open/close 30000 (time 1734260963.18 total 59.60 last 471.11) - open/close 34669 (time 1734260973.18 total 69.60 last 466.88) - open/close 39232 (time 1734260983.18 total 79.60 last 456.20) - open/close 43842 (time 1734260993.18 total 89.60 last 460.97) - open/close 48605 (time 1734261003.19 total 99.60 last 476.24) - open/close 50000 (time 1734261006.39 total 102.81 last 434.71) - open/close 54652 (time 1734261016.40 total 112.81 last 465.16) - open/close 59239 (time 1734261026.40 total 122.81 last 458.65) - open/close 63862 (time 1734261036.40 total 132.81 last 462.21) total: 64768 open/close in 134.71 seconds: 480.80 ops/second - /unlink 8248 (time 1734261049.17 total 10.00 last 824.73) - /unlink 10000 (time 1734261051.22 total 12.05 last 853.19) - /unlink 18252 (time 1734261061.22 total 22.05 last 825.19) - /unlink 20000 (time 1734261063.39 total 24.22 last 805.46) - /unlink 28479 (time 1734261073.39 total 34.22 last 847.90) - /unlink 36983 (time 1734261083.39 total 44.23 last 850.30) - /unlink 40000 (time 1734261086.94 total 47.77 last 850.86) - /unlink 48338 (time 1734261096.94 total 57.77 last 833.79) - /unlink 56812 (time 1734261106.94 total 67.77 last 847.31) - /unlink 60000 (time 1734261110.58 total 71.41 last 875.69) total: 64768 /unlink in 76.76 seconds: 843.78 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg334-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg334-server unloading modules via unload_modules_local on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2