== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:34:39 (1734258879) Stopping clients: oleg325-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg325-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg325-client: oleg325-server: ssh exited with exit code 2 oleg325-server: oleg325-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg325-server' oleg325-server: oleg325-server.virtnet: executing load_modules_local oleg325-server: Loading modules from /home/green/git/lustre-release/lustre oleg325-server: detected 4 online CPUs by sysfs oleg325-server: Force libcfs to create 2 CPU partitions oleg325-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg325-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg325-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg325-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg325-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg325-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg325-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg325-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg325-server: oleg325-server.virtnet: executing set_default_debug -1 all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16266 458 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16451 643 15808 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1423 (time 1734258960.71 total 10.00 last 142.27) - open/close 2928 (time 1734258970.72 total 20.01 last 150.39) - open/close 4423 (time 1734258980.72 total 30.01 last 149.49) - open/close 6037 (time 1734258990.72 total 40.01 last 161.37) - open/close 7690 (time 1734259000.73 total 50.02 last 165.20) - open/close 9505 (time 1734259010.73 total 60.02 last 181.42) - open/close 10000 (time 1734259013.60 total 62.89 last 172.90) - open/close 11582 (time 1734259023.60 total 72.89 last 158.15) - open/close 13056 (time 1734259033.60 total 82.89 last 147.37) - open/close 14493 (time 1734259043.60 total 92.89 last 143.65) - open/close 15983 (time 1734259053.61 total 102.90 last 148.89) - open/close 17479 (time 1734259063.63 total 112.92 last 149.34) - open/close 18904 (time 1734259073.64 total 122.93 last 142.36) - open/close 20000 (time 1734259081.65 total 130.94 last 136.88) - open/close 21471 (time 1734259091.65 total 140.94 last 147.07) - open/close 22865 (time 1734259101.65 total 150.94 last 139.38) - open/close 24259 (time 1734259111.65 total 160.94 last 139.34) - open/close 26002 (time 1734259121.66 total 170.95 last 174.26) - open/close 27502 (time 1734259131.67 total 180.96 last 149.86) - open/close 28967 (time 1734259141.67 total 190.96 last 146.43) - open/close 30000 (time 1734259148.82 total 198.11 last 144.57) - open/close 31479 (time 1734259158.82 total 208.11 last 147.88) - open/close 33094 (time 1734259168.82 total 218.11 last 161.48) - open/close 34746 (time 1734259178.82 total 228.11 last 165.11) - open/close 36250 (time 1734259188.83 total 238.11 last 150.38) - open/close 37765 (time 1734259198.83 total 248.12 last 151.49) - open/close 39282 (time 1734259208.83 total 258.12 last 151.65) - open/close 40000 (time 1734259213.96 total 263.25 last 139.99) - open/close 41453 (time 1734259223.96 total 273.25 last 145.26) - open/close 42995 (time 1734259233.96 total 283.25 last 154.17) - open/close 44645 (time 1734259243.96 total 293.25 last 164.96) - open/close 46688 (time 1734259253.97 total 303.26 last 204.17) - open/close 48554 (time 1734259263.97 total 313.26 last 186.59) - open/close 50000 (time 1734259273.35 total 322.64 last 154.23) - open/close 51619 (time 1734259283.35 total 332.64 last 161.88) - open/close 53374 (time 1734259293.35 total 342.64 last 175.47) - open/close 54908 (time 1734259303.35 total 352.64 last 153.33) - open/close 56471 (time 1734259313.36 total 362.65 last 156.27) - open/close 58018 (time 1734259323.36 total 372.65 last 154.66) - open/close 59926 (time 1734259333.36 total 382.65 last 190.80) - open/close 61749 (time 1734259343.36 total 392.65 last 182.24) - open/close 63274 (time 1734259353.36 total 402.65 last 152.49) total: 64768 open/close in 412.49 seconds: 157.02 ops/second - /unlink 2647 (time 1734259374.62 total 10.00 last 264.70) - /unlink 5269 (time 1734259384.63 total 20.00 last 262.12) - /unlink 7997 (time 1734259394.63 total 30.01 last 272.70) - /unlink 10000 (time 1734259401.89 total 37.26 last 276.03) - /unlink 12649 (time 1734259411.89 total 47.27 last 264.80) - /unlink 15270 (time 1734259421.89 total 57.27 last 262.02) - /unlink 17881 (time 1734259431.90 total 67.27 last 261.02) - /unlink 20000 (time 1734259440.03 total 75.40 last 260.64) - /unlink 22628 (time 1734259450.03 total 85.41 last 262.74) - /unlink 25208 (time 1734259460.03 total 95.41 last 257.98) - /unlink 27931 (time 1734259470.03 total 105.41 last 272.22) - /unlink 30000 (time 1734259477.64 total 113.02 last 271.77) - /unlink 32984 (time 1734259487.65 total 123.02 last 298.36) - /unlink 35775 (time 1734259497.65 total 133.02 last 279.08) - /unlink 38278 (time 1734259507.65 total 143.03 last 250.22) - /unlink 40000 (time 1734259513.72 total 149.09 last 283.87) - /unlink 42991 (time 1734259523.72 total 159.10 last 298.93) - /unlink 46424 (time 1734259533.73 total 169.10 last 343.14) - /unlink 49345 (time 1734259543.73 total 179.11 last 291.93) - /unlink 50000 (time 1734259546.02 total 181.40 last 285.77) - /unlink 52789 (time 1734259556.03 total 191.40 last 278.84) - /unlink 55826 (time 1734259566.03 total 201.41 last 303.61) - /unlink 58459 (time 1734259576.03 total 211.41 last 263.25) - /unlink 60000 (time 1734259581.08 total 216.46 last 305.24) - /unlink 63321 (time 1734259591.08 total 226.46 last 332.02) total: 64768 /unlink in 230.81 seconds: 280.61 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2152 (time 1734259621.33 total 10.00 last 215.13) - open/close 4235 (time 1734259631.33 total 20.00 last 208.29) - open/close 6305 (time 1734259641.33 total 30.00 last 206.99) - open/close 8290 (time 1734259651.33 total 40.00 last 198.50) - open/close 10000 (time 1734259659.92 total 48.59 last 199.24) - open/close 11924 (time 1734259669.92 total 58.59 last 192.36) - open/close 13847 (time 1734259679.92 total 68.59 last 192.30) - open/close 15837 (time 1734259689.92 total 78.59 last 198.95) - open/close 17898 (time 1734259699.92 total 88.59 last 206.06) - open/close 20000 (time 1734259709.50 total 98.18 last 219.36) - open/close 22018 (time 1734259719.51 total 108.18 last 201.68) - open/close 23882 (time 1734259729.52 total 118.19 last 186.30) - open/close 25678 (time 1734259739.52 total 128.19 last 179.59) - open/close 27590 (time 1734259749.52 total 138.19 last 191.18) - open/close 29416 (time 1734259759.52 total 148.19 last 182.54) - open/close 30000 (time 1734259762.49 total 151.16 last 196.93) - open/close 31991 (time 1734259772.49 total 161.16 last 199.05) - open/close 34023 (time 1734259782.49 total 171.16 last 203.19) - open/close 36104 (time 1734259792.49 total 181.16 last 208.08) - open/close 37965 (time 1734259802.49 total 191.16 last 186.10) - open/close 39655 (time 1734259812.50 total 201.17 last 168.91) - open/close 41575 (time 1734259822.50 total 211.17 last 191.95) - open/close 43828 (time 1734259832.50 total 221.17 last 225.29) - open/close 46052 (time 1734259842.50 total 231.17 last 222.37) - open/close 48410 (time 1734259852.50 total 241.18 last 235.74) - open/close 50000 (time 1734259860.04 total 248.71 last 210.88) - open/close 52100 (time 1734259870.04 total 258.72 last 209.99) - open/close 54065 (time 1734259880.05 total 268.72 last 196.45) - open/close 55923 (time 1734259890.05 total 278.72 last 185.69) - open/close 57788 (time 1734259900.06 total 288.73 last 186.42) - open/close 59745 (time 1734259910.06 total 298.73 last 195.65) - open/close 61979 (time 1734259920.06 total 308.73 last 223.39) - open/close 64130 (time 1734259930.06 total 318.74 last 214.97) total: 64768 open/close in 321.54 seconds: 201.43 ops/second - /unlink 3502 (time 1734259944.05 total 10.00 last 350.08) - /unlink 7025 (time 1734259954.05 total 20.01 last 352.24) - /unlink 10000 (time 1734259962.58 total 28.53 last 349.03) - /unlink 13511 (time 1734259972.58 total 38.53 last 351.07) - /unlink 16947 (time 1734259982.58 total 48.53 last 343.57) - /unlink 20000 (time 1734259990.76 total 56.71 last 373.18) - /unlink 24063 (time 1734260000.76 total 66.71 last 406.24) - /unlink 28700 (time 1734260010.76 total 76.71 last 463.68) - /unlink 30000 (time 1734260013.74 total 79.70 last 435.72) - /unlink 34092 (time 1734260023.75 total 89.70 last 409.16) - /unlink 38272 (time 1734260033.75 total 99.70 last 417.99) - /unlink 40000 (time 1734260037.95 total 103.90 last 411.24) - /unlink 43855 (time 1734260047.95 total 113.90 last 385.43) - /unlink 48641 (time 1734260057.95 total 123.90 last 478.57) - /unlink 50000 (time 1734260060.30 total 126.25 last 579.35) - /unlink 54997 (time 1734260070.30 total 136.25 last 499.60) - /unlink 59463 (time 1734260080.30 total 146.25 last 446.49) - /unlink 63967 (time 1734260090.30 total 156.25 last 450.37) total: 64768 /unlink in 158.25 seconds: 409.27 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2529 (time 1734260121.69 total 10.00 last 252.84) - open/close 4986 (time 1734260131.69 total 20.00 last 245.67) - open/close 7214 (time 1734260141.69 total 30.00 last 222.78) - open/close 9807 (time 1734260151.70 total 40.01 last 259.28) - open/close 12654 (time 1734260161.70 total 50.01 last 284.62) - open/close 15379 (time 1734260171.70 total 60.01 last 272.44) - open/close 18023 (time 1734260181.70 total 70.01 last 264.31) - open/close 20000 (time 1734260188.90 total 77.21 last 274.75) - open/close 22586 (time 1734260198.90 total 87.21 last 258.59) - open/close 25079 (time 1734260208.90 total 97.21 last 249.26) - open/close 27599 (time 1734260218.91 total 107.22 last 251.86) - open/close 30000 (time 1734260228.87 total 117.18 last 240.96) - open/close 32346 (time 1734260238.88 total 127.19 last 234.47) - open/close 34696 (time 1734260248.88 total 137.19 last 234.99) - open/close 37668 (time 1734260258.88 total 147.19 last 297.13) - open/close 40000 (time 1734260266.78 total 155.09 last 295.08) - open/close 42577 (time 1734260276.78 total 165.09 last 257.66) - open/close 45012 (time 1734260286.79 total 175.10 last 243.43) - open/close 47480 (time 1734260296.79 total 185.10 last 246.80) - open/close 49699 (time 1734260306.79 total 195.10 last 221.86) - open/close 52170 (time 1734260316.79 total 205.10 last 247.05) - open/close 54679 (time 1734260326.84 total 215.15 last 249.57) - open/close 57295 (time 1734260336.85 total 225.16 last 261.54) - open/close 59850 (time 1734260346.85 total 235.16 last 255.46) - open/close 62609 (time 1734260356.85 total 245.16 last 275.82) total: 64768 open/close in 253.83 seconds: 255.16 ops/second - /unlink 4281 (time 1734260376.55 total 10.00 last 428.09) - /unlink 8256 (time 1734260386.55 total 20.00 last 397.43) - /unlink 10000 (time 1734260390.65 total 24.11 last 424.94) - /unlink 13953 (time 1734260400.66 total 34.11 last 395.29) - /unlink 17604 (time 1734260410.66 total 44.11 last 365.08) - /unlink 20000 (time 1734260416.38 total 49.83 last 418.60) - /unlink 24120 (time 1734260426.38 total 59.83 last 411.93) - /unlink 28252 (time 1734260436.38 total 69.83 last 413.10) - /unlink 30000 (time 1734260440.78 total 74.23 last 397.72) - /unlink 33983 (time 1734260450.78 total 84.23 last 398.23) - /unlink 38119 (time 1734260460.78 total 94.23 last 413.56) - /unlink 40000 (time 1734260464.90 total 98.35 last 456.33) - /unlink 44604 (time 1734260474.90 total 108.36 last 460.39) - /unlink 49810 (time 1734260484.90 total 118.36 last 520.59) - /unlink 54614 (time 1734260494.90 total 128.36 last 480.39) - /unlink 59843 (time 1734260504.91 total 138.36 last 522.81) total: 64768 /unlink in 147.71 seconds: 438.49 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3131 (time 1734260543.48 total 10.00 last 313.02) - open/close 5660 (time 1734260553.48 total 20.00 last 252.90) - open/close 8414 (time 1734260563.48 total 30.00 last 275.37) - open/close 10000 (time 1734260568.85 total 35.38 last 295.24) - open/close 12983 (time 1734260578.85 total 45.38 last 298.26) - open/close 16145 (time 1734260588.86 total 55.38 last 316.16) - open/close 19183 (time 1734260598.86 total 65.38 last 303.75) - open/close 20000 (time 1734260601.74 total 68.26 last 283.56) - open/close 22813 (time 1734260611.74 total 78.26 last 281.24) - open/close 25794 (time 1734260621.74 total 88.27 last 298.03) - open/close 28683 (time 1734260631.74 total 98.27 last 288.86) - open/close 30000 (time 1734260636.66 total 103.19 last 267.69) - open/close 32857 (time 1734260646.67 total 113.19 last 285.63) - open/close 35850 (time 1734260656.67 total 123.19 last 299.22) - open/close 39202 (time 1734260666.67 total 133.19 last 335.12) - open/close 40000 (time 1734260669.37 total 135.90 last 295.23) - open/close 42836 (time 1734260679.38 total 145.90 last 283.50) - open/close 45788 (time 1734260689.38 total 155.90 last 295.11) - open/close 48911 (time 1734260699.38 total 165.90 last 312.27) - open/close 50000 (time 1734260703.04 total 169.57 last 297.36) - open/close 53214 (time 1734260713.05 total 179.57 last 321.33) - open/close 56826 (time 1734260723.05 total 189.57 last 361.15) - open/close 60000 (time 1734260731.60 total 198.12 last 371.25) - open/close 63733 (time 1734260741.60 total 208.12 last 373.29) total: 64768 open/close in 210.68 seconds: 307.42 ops/second - /unlink 6534 (time 1734260755.03 total 10.00 last 653.37) - /unlink 10000 (time 1734260760.20 total 15.17 last 670.92) - /unlink 17690 (time 1734260770.20 total 25.17 last 768.96) - /unlink 20000 (time 1734260772.96 total 27.93 last 836.05) - /unlink 28891 (time 1734260782.96 total 37.93 last 889.00) - /unlink 37650 (time 1734260792.96 total 47.93 last 875.89) - /unlink 40000 (time 1734260795.40 total 50.37 last 964.47) - /unlink 49724 (time 1734260805.40 total 60.37 last 972.38) - /unlink 57553 (time 1734260815.40 total 70.37 last 782.89) - /unlink 60000 (time 1734260818.57 total 73.54 last 772.35) total: 64768 /unlink in 79.12 seconds: 818.56 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4650 (time 1734260859.15 total 10.00 last 464.99) - open/close 10000 (time 1734260868.98 total 19.83 last 544.38) - open/close 16050 (time 1734260878.98 total 29.83 last 604.96) - open/close 20000 (time 1734260885.94 total 36.78 last 567.85) - open/close 24900 (time 1734260895.94 total 46.79 last 489.97) - open/close 29874 (time 1734260905.94 total 56.79 last 497.33) - open/close 34885 (time 1734260915.99 total 66.83 last 498.80) - open/close 39587 (time 1734260925.99 total 76.83 last 470.18) - open/close 45053 (time 1734260935.99 total 86.83 last 546.58) - open/close 50000 (time 1734260944.59 total 95.44 last 574.70) - open/close 55601 (time 1734260954.60 total 105.44 last 560.02) - open/close 60000 (time 1734260964.00 total 114.85 last 467.71) - open/close 64749 (time 1734260974.00 total 124.85 last 474.89) total: 64768 open/close in 124.88 seconds: 518.62 ops/second - /unlink 8255 (time 1734260984.91 total 10.00 last 825.42) - /unlink 16391 (time 1734260994.91 total 20.00 last 813.59) - /unlink 20000 (time 1734260999.18 total 24.27 last 845.94) - /unlink 28259 (time 1734261009.18 total 34.27 last 825.87) - /unlink 30000 (time 1734261011.21 total 36.30 last 858.23) - /unlink 38334 (time 1734261021.21 total 46.30 last 833.32) - /unlink 46327 (time 1734261031.21 total 56.30 last 799.23) - /unlink 50000 (time 1734261035.53 total 60.61 last 850.88) - /unlink 58730 (time 1734261045.53 total 70.62 last 872.92) total: 64768 /unlink in 77.31 seconds: 837.75 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg325-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg325-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg325-server stop mds service on oleg325-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg325-server stop mds service on oleg325-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg325-server unloading modules via unload_modules_local on: 'oleg325-server' oleg325-server: oleg325-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg325-client: oleg325-client: ssh exited with exit code 2 pdsh@oleg325-client: oleg325-server: ssh exited with exit code 2 pdsh@oleg325-client: oleg325-client: ssh exited with exit code 2 pdsh@oleg325-client: oleg325-server: ssh exited with exit code 2