== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:36:08 (1734258968) Stopping clients: oleg214-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg214-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: oleg214-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing load_modules_local oleg214-server: Loading modules from /home/green/git/lustre-release/lustre oleg214-server: detected 4 online CPUs by sysfs oleg214-server: Force libcfs to create 2 CPU partitions oleg214-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg214-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg214-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg214-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg214-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock 192.168.202.114@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16131 291 15840 2% /mnt/lustre[OST:0] filesystem_summary: 16483 643 15840 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1591 (time 1734259033.17 total 10.00 last 159.05) - open/close 3219 (time 1734259043.18 total 20.01 last 162.74) - open/close 5027 (time 1734259053.18 total 30.01 last 180.74) - open/close 6655 (time 1734259063.18 total 40.01 last 162.74) - open/close 8498 (time 1734259073.19 total 50.02 last 184.22) - open/close 10000 (time 1734259081.52 total 58.35 last 180.27) - open/close 11608 (time 1734259091.52 total 68.35 last 160.80) - open/close 13330 (time 1734259101.52 total 78.35 last 172.14) - open/close 14907 (time 1734259111.53 total 88.36 last 157.60) - open/close 17003 (time 1734259121.53 total 98.36 last 209.54) - open/close 18765 (time 1734259131.54 total 108.37 last 176.00) - open/close 20000 (time 1734259139.35 total 116.18 last 158.13) - open/close 21770 (time 1734259149.36 total 126.19 last 176.94) - open/close 23439 (time 1734259159.36 total 136.19 last 166.90) - open/close 25269 (time 1734259169.36 total 146.19 last 182.99) - open/close 26983 (time 1734259179.36 total 156.20 last 171.30) - open/close 28627 (time 1734259189.36 total 166.20 last 164.38) - open/close 30000 (time 1734259196.41 total 173.24 last 194.95) - open/close 31570 (time 1734259206.41 total 183.24 last 156.95) - open/close 33422 (time 1734259216.41 total 193.24 last 185.19) - open/close 35044 (time 1734259226.42 total 203.25 last 162.08) - open/close 36544 (time 1734259236.42 total 213.25 last 149.97) - open/close 38256 (time 1734259246.43 total 223.26 last 171.10) - open/close 39986 (time 1734259256.43 total 233.26 last 172.93) - open/close 41664 (time 1734259266.44 total 243.27 last 167.73) - open/close 43435 (time 1734259276.44 total 253.27 last 177.04) - open/close 45653 (time 1734259286.44 total 263.27 last 221.76) - open/close 47580 (time 1734259296.44 total 273.27 last 192.67) - open/close 49238 (time 1734259306.44 total 283.28 last 165.78) - open/close 50000 (time 1734259310.98 total 287.82 last 167.81) - open/close 51439 (time 1734259320.98 total 297.82 last 143.89) - open/close 53103 (time 1734259330.99 total 307.82 last 166.33) - open/close 54822 (time 1734259340.99 total 317.82 last 171.85) - open/close 56710 (time 1734259350.99 total 327.82 last 188.80) - open/close 58623 (time 1734259360.99 total 337.82 last 191.29) - open/close 60000 (time 1734259368.24 total 345.07 last 189.93) - open/close 61705 (time 1734259378.25 total 355.08 last 170.41) - open/close 63585 (time 1734259388.25 total 365.08 last 187.98) total: 64768 open/close in 371.59 seconds: 174.30 ops/second - /unlink 3090 (time 1734259406.01 total 10.00 last 308.97) - /unlink 6256 (time 1734259416.01 total 20.00 last 316.55) - /unlink 9247 (time 1734259426.01 total 30.00 last 299.05) - /unlink 10000 (time 1734259428.64 total 32.63 last 286.39) - /unlink 12836 (time 1734259438.64 total 42.63 last 283.56) - /unlink 15353 (time 1734259448.65 total 52.64 last 251.64) - /unlink 17951 (time 1734259458.65 total 62.64 last 259.80) - /unlink 20000 (time 1734259465.86 total 69.85 last 284.26) - /unlink 23226 (time 1734259475.86 total 79.85 last 322.49) - /unlink 26109 (time 1734259485.86 total 89.85 last 288.24) - /unlink 29029 (time 1734259495.86 total 99.85 last 291.99) - /unlink 30000 (time 1734259499.15 total 103.14 last 295.08) - /unlink 32667 (time 1734259509.15 total 113.14 last 266.64) - /unlink 35779 (time 1734259519.16 total 123.15 last 311.09) - /unlink 38696 (time 1734259529.16 total 133.15 last 291.66) - /unlink 40000 (time 1734259534.24 total 138.23 last 256.64) - /unlink 42803 (time 1734259544.24 total 148.23 last 280.23) - /unlink 45552 (time 1734259554.24 total 158.23 last 274.90) - /unlink 48521 (time 1734259564.24 total 168.23 last 296.85) - /unlink 50000 (time 1734259569.06 total 173.05 last 307.37) - /unlink 52950 (time 1734259579.06 total 183.05 last 294.94) - /unlink 56085 (time 1734259589.06 total 193.05 last 313.42) - /unlink 59400 (time 1734259599.06 total 203.05 last 331.45) - /unlink 62727 (time 1734259609.06 total 213.05 last 332.68) total: 64768 /unlink in 219.38 seconds: 295.23 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2265 (time 1734259644.03 total 10.01 last 226.38) - open/close 4359 (time 1734259654.03 total 20.01 last 209.33) - open/close 6296 (time 1734259664.03 total 30.01 last 193.65) - open/close 8078 (time 1734259674.04 total 40.02 last 178.13) - open/close 10000 (time 1734259683.64 total 49.62 last 200.20) - open/close 12008 (time 1734259693.64 total 59.62 last 200.74) - open/close 14317 (time 1734259703.64 total 69.62 last 230.87) - open/close 16440 (time 1734259713.64 total 79.62 last 212.29) - open/close 18276 (time 1734259723.64 total 89.62 last 183.58) - open/close 20000 (time 1734259732.03 total 98.01 last 205.52) - open/close 21654 (time 1734259742.03 total 108.01 last 165.38) - open/close 23450 (time 1734259752.03 total 118.01 last 179.58) - open/close 25264 (time 1734259762.04 total 128.01 last 181.37) - open/close 26874 (time 1734259772.04 total 138.02 last 160.97) - open/close 28529 (time 1734259782.04 total 148.02 last 165.47) - open/close 30000 (time 1734259790.82 total 156.80 last 167.56) - open/close 31896 (time 1734259800.82 total 166.80 last 189.60) - open/close 34134 (time 1734259810.82 total 176.80 last 223.74) - open/close 36221 (time 1734259820.83 total 186.80 last 208.61) - open/close 38320 (time 1734259830.83 total 196.81 last 209.86) - open/close 40000 (time 1734259839.35 total 205.33 last 197.17) - open/close 42094 (time 1734259849.35 total 215.33 last 209.32) - open/close 43710 (time 1734259859.35 total 225.33 last 161.57) - open/close 45584 (time 1734259869.36 total 235.34 last 187.31) - open/close 47366 (time 1734259879.37 total 245.34 last 178.09) - open/close 49198 (time 1734259889.37 total 255.35 last 183.14) - open/close 50000 (time 1734259893.04 total 259.02 last 218.44) - open/close 51892 (time 1734259903.04 total 269.02 last 189.18) - open/close 53877 (time 1734259913.04 total 279.02 last 198.45) - open/close 55902 (time 1734259923.05 total 289.03 last 202.40) - open/close 58291 (time 1734259933.05 total 299.03 last 238.89) - open/close 60000 (time 1734259940.23 total 306.20 last 238.08) - open/close 62385 (time 1734259950.23 total 316.21 last 238.44) - open/close 64742 (time 1734259960.23 total 326.21 last 235.62) total: 64768 open/close in 326.31 seconds: 198.49 ops/second - /unlink 4154 (time 1734259971.38 total 10.00 last 415.39) - /unlink 8614 (time 1734259981.38 total 20.00 last 446.00) - /unlink 10000 (time 1734259984.42 total 23.04 last 455.67) - /unlink 14160 (time 1734259994.43 total 33.04 last 415.94) - /unlink 18209 (time 1734260004.43 total 43.04 last 404.86) - /unlink 20000 (time 1734260008.40 total 47.02 last 450.78) - /unlink 24013 (time 1734260018.40 total 57.02 last 401.30) - /unlink 28157 (time 1734260028.40 total 67.02 last 414.39) - /unlink 30000 (time 1734260032.71 total 71.33 last 427.14) - /unlink 34120 (time 1734260042.72 total 81.33 last 411.96) - /unlink 37624 (time 1734260052.72 total 91.34 last 350.33) - /unlink 40000 (time 1734260058.56 total 97.18 last 406.34) - /unlink 44447 (time 1734260068.57 total 107.18 last 444.62) - /unlink 48637 (time 1734260078.57 total 117.19 last 418.91) - /unlink 50000 (time 1734260082.36 total 120.98 last 359.61) - /unlink 53979 (time 1734260092.36 total 130.98 last 397.88) - /unlink 57945 (time 1734260102.36 total 140.98 last 396.47) - /unlink 60000 (time 1734260107.66 total 146.28 last 387.98) - /unlink 64466 (time 1734260117.66 total 156.28 last 446.56) total: 64768 /unlink in 157.15 seconds: 412.14 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2428 (time 1734260145.32 total 10.00 last 242.78) - open/close 4890 (time 1734260155.32 total 20.00 last 246.16) - open/close 7587 (time 1734260165.32 total 30.00 last 269.70) - open/close 10000 (time 1734260175.22 total 39.91 last 243.65) - open/close 12283 (time 1734260185.22 total 49.91 last 228.23) - open/close 14422 (time 1734260195.23 total 59.91 last 213.82) - open/close 16923 (time 1734260205.23 total 69.91 last 250.08) - open/close 19239 (time 1734260215.23 total 79.91 last 231.59) - open/close 20000 (time 1734260218.47 total 83.15 last 234.98) - open/close 22123 (time 1734260228.47 total 93.16 last 212.22) - open/close 24315 (time 1734260238.48 total 103.16 last 219.12) - open/close 26663 (time 1734260248.48 total 113.16 last 234.77) - open/close 29006 (time 1734260258.48 total 123.16 last 234.27) - open/close 30000 (time 1734260262.83 total 127.51 last 228.47) - open/close 32526 (time 1734260272.83 total 137.52 last 252.55) - open/close 35098 (time 1734260282.83 total 147.52 last 257.10) - open/close 37634 (time 1734260292.84 total 157.52 last 253.55) - open/close 39957 (time 1734260302.84 total 167.52 last 232.23) - open/close 42002 (time 1734260312.84 total 177.53 last 204.42) - open/close 44147 (time 1734260322.85 total 187.53 last 214.41) - open/close 46342 (time 1734260332.85 total 197.54 last 219.42) - open/close 48335 (time 1734260342.86 total 207.54 last 199.18) - open/close 50000 (time 1734260351.07 total 215.76 last 202.72) - open/close 52038 (time 1734260361.07 total 225.76 last 203.74) - open/close 54225 (time 1734260371.08 total 235.76 last 218.63) - open/close 56345 (time 1734260381.08 total 245.76 last 211.96) - open/close 58367 (time 1734260391.08 total 255.77 last 202.11) - open/close 60000 (time 1734260398.03 total 262.71 last 235.10) - open/close 62395 (time 1734260408.03 total 272.72 last 239.43) - open/close 64646 (time 1734260418.03 total 282.72 last 225.02) total: 64768 open/close in 283.25 seconds: 228.66 ops/second - /unlink 3792 (time 1734260429.66 total 10.00 last 379.18) - /unlink 7362 (time 1734260439.66 total 20.00 last 356.98) - /unlink 10000 (time 1734260446.50 total 26.84 last 385.90) - /unlink 14811 (time 1734260456.50 total 36.84 last 481.02) - /unlink 19206 (time 1734260466.50 total 46.84 last 439.43) - /unlink 20000 (time 1734260468.69 total 49.03 last 363.34) - /unlink 23906 (time 1734260478.69 total 59.03 last 390.60) - /unlink 27357 (time 1734260488.69 total 69.03 last 345.05) - /unlink 30000 (time 1734260496.00 total 76.34 last 361.44) - /unlink 34173 (time 1734260506.00 total 86.34 last 417.25) - /unlink 38084 (time 1734260516.00 total 96.34 last 391.08) - /unlink 40000 (time 1734260521.14 total 101.48 last 373.18) - /unlink 43695 (time 1734260531.14 total 111.48 last 369.40) - /unlink 47586 (time 1734260541.14 total 121.48 last 389.06) - /unlink 50000 (time 1734260547.55 total 127.88 last 376.86) - /unlink 53781 (time 1734260557.55 total 137.89 last 378.03) - /unlink 57239 (time 1734260567.55 total 147.89 last 345.74) - /unlink 60000 (time 1734260574.59 total 154.92 last 392.46) - /unlink 63897 (time 1734260584.59 total 164.92 last 389.68) total: 64768 /unlink in 167.06 seconds: 387.69 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2274 (time 1734260616.30 total 10.00 last 227.36) - open/close 5080 (time 1734260626.30 total 20.00 last 280.53) - open/close 7888 (time 1734260636.30 total 30.00 last 280.78) - open/close 10000 (time 1734260642.90 total 36.61 last 319.85) - open/close 12449 (time 1734260652.90 total 46.61 last 244.87) - open/close 15559 (time 1734260662.91 total 56.61 last 310.86) - open/close 18313 (time 1734260672.91 total 66.62 last 275.32) - open/close 20000 (time 1734260678.61 total 72.32 last 295.98) - open/close 22744 (time 1734260688.61 total 82.32 last 274.29) - open/close 25200 (time 1734260698.62 total 92.32 last 245.50) - open/close 27559 (time 1734260708.62 total 102.33 last 235.86) - open/close 30000 (time 1734260718.14 total 111.85 last 256.39) - open/close 32403 (time 1734260728.14 total 121.85 last 240.24) - open/close 34687 (time 1734260738.15 total 131.85 last 228.33) - open/close 36844 (time 1734260748.15 total 141.85 last 215.69) - open/close 39181 (time 1734260758.15 total 151.85 last 233.68) - open/close 40000 (time 1734260761.69 total 155.40 last 231.22) - open/close 42332 (time 1734260771.69 total 165.40 last 233.11) - open/close 45119 (time 1734260781.70 total 175.40 last 278.57) - open/close 47643 (time 1734260791.70 total 185.41 last 252.36) - open/close 50000 (time 1734260799.78 total 193.48 last 291.83) - open/close 52742 (time 1734260809.78 total 203.49 last 274.10) - open/close 55755 (time 1734260819.80 total 213.51 last 300.62) - open/close 58493 (time 1734260829.80 total 223.51 last 273.76) - open/close 60000 (time 1734260835.12 total 228.83 last 283.46) - open/close 62812 (time 1734260845.12 total 238.83 last 281.12) total: 64768 open/close in 245.85 seconds: 263.44 ops/second - /unlink 4926 (time 1734260863.15 total 10.00 last 492.60) - /unlink 9331 (time 1734260873.15 total 20.00 last 440.43) - /unlink 14694 (time 1734260883.15 total 30.00 last 536.29) - /unlink 20000 (time 1734260891.40 total 38.26 last 642.81) - /unlink 26050 (time 1734260901.40 total 48.26 last 604.98) - /unlink 30000 (time 1734260908.40 total 55.25 last 565.00) - /unlink 35286 (time 1734260918.40 total 65.25 last 528.56) - /unlink 40000 (time 1734260928.02 total 74.88 last 489.66) - /unlink 44791 (time 1734260938.02 total 84.88 last 479.06) - /unlink 50000 (time 1734260947.88 total 94.73 last 528.73) - /unlink 55003 (time 1734260957.88 total 104.73 last 500.25) - /unlink 60000 (time 1734260967.17 total 114.02 last 537.81) total: 64768 /unlink in 122.80 seconds: 527.43 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2799 (time 1734261006.62 total 10.00 last 279.88) - open/close 6553 (time 1734261016.62 total 20.00 last 375.38) - open/close 10000 (time 1734261026.05 total 29.43 last 365.46) - open/close 13355 (time 1734261036.05 total 39.44 last 335.29) - open/close 17210 (time 1734261046.06 total 49.44 last 385.40) - open/close 20000 (time 1734261052.95 total 56.33 last 405.01) - open/close 24816 (time 1734261062.95 total 66.33 last 481.51) - open/close 30000 (time 1734261072.85 total 76.24 last 523.49) - open/close 35554 (time 1734261082.85 total 86.24 last 555.30) - open/close 40000 (time 1734261091.77 total 95.16 last 498.52) - open/close 45288 (time 1734261101.77 total 105.16 last 528.73) - open/close 50000 (time 1734261110.94 total 114.33 last 513.91) - open/close 54633 (time 1734261120.98 total 124.36 last 461.53) - open/close 58668 (time 1734261130.98 total 134.37 last 403.45) - open/close 60000 (time 1734261134.68 total 138.07 last 359.76) - open/close 64259 (time 1734261144.68 total 148.07 last 425.88) total: 64768 open/close in 149.26 seconds: 433.92 ops/second - /unlink 7596 (time 1734261156.80 total 10.00 last 759.55) - /unlink 10000 (time 1734261159.74 total 12.94 last 816.73) - /unlink 17585 (time 1734261169.74 total 22.94 last 758.45) - /unlink 20000 (time 1734261173.00 total 26.20 last 740.89) - /unlink 27596 (time 1734261183.00 total 36.20 last 759.58) - /unlink 30000 (time 1734261186.13 total 39.33 last 768.68) - /unlink 37194 (time 1734261196.13 total 49.33 last 719.38) - /unlink 40000 (time 1734261199.72 total 52.92 last 782.31) - /unlink 47876 (time 1734261209.72 total 62.92 last 787.55) - /unlink 50000 (time 1734261212.41 total 65.61 last 788.52) - /unlink 58214 (time 1734261222.41 total 75.61 last 821.37) - /unlink 60000 (time 1734261224.66 total 77.87 last 792.86) total: 64768 /unlink in 83.94 seconds: 771.58 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg214-server unloading modules via unload_modules_local on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-client: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-client: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2