== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:18:18 (1734257898) Stopping clients: oleg211-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg211-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2 oleg211-server: oleg211-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg211-server' oleg211-server: oleg211-server.virtnet: executing load_modules_local oleg211-server: Loading modules from /home/green/git/lustre-release/lustre oleg211-server: detected 4 online CPUs by sysfs oleg211-server: Force libcfs to create 2 CPU partitions oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg211-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg211-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg211-client.virtnet: -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1437 (time 1734257963.32 total 10.00 last 143.67) - open/close 2877 (time 1734257973.32 total 20.00 last 143.99) - open/close 4535 (time 1734257983.32 total 30.00 last 165.77) - open/close 5965 (time 1734257993.32 total 40.01 last 142.95) - open/close 7371 (time 1734258003.33 total 50.01 last 140.56) - open/close 8448 (time 1734258013.33 total 60.01 last 107.69) - open/close 9488 (time 1734258023.33 total 70.01 last 103.97) - open/close 10000 (time 1734258027.73 total 74.42 last 116.34) - open/close 11116 (time 1734258037.74 total 84.42 last 111.54) - open/close 12209 (time 1734258047.74 total 94.42 last 109.28) - open/close 13294 (time 1734258057.74 total 104.43 last 108.46) - open/close 14479 (time 1734258067.75 total 114.43 last 118.43) - open/close 15575 (time 1734258077.75 total 124.43 last 109.57) - open/close 16727 (time 1734258087.76 total 134.44 last 115.16) - open/close 17840 (time 1734258097.76 total 144.45 last 111.21) - open/close 18886 (time 1734258107.77 total 154.45 last 104.55) - open/close 20000 (time 1734258117.60 total 164.28 last 113.35) - open/close 21106 (time 1734258127.60 total 174.28 last 110.58) - open/close 22200 (time 1734258137.60 total 184.29 last 109.34) - open/close 23425 (time 1734258147.61 total 194.29 last 122.38) - open/close 24688 (time 1734258157.62 total 204.30 last 126.21) - open/close 25859 (time 1734258167.62 total 214.30 last 117.08) - open/close 26993 (time 1734258177.63 total 224.31 last 113.33) - open/close 28354 (time 1734258187.63 total 234.32 last 136.00) - open/close 29743 (time 1734258197.64 total 244.32 last 138.88) - open/close 30000 (time 1734258199.85 total 246.53 last 115.98) - open/close 31329 (time 1734258209.86 total 256.54 last 132.81) - open/close 32492 (time 1734258219.86 total 266.55 last 116.24) - open/close 33834 (time 1734258229.87 total 276.55 last 134.16) - open/close 35329 (time 1734258239.87 total 286.55 last 149.50) - open/close 36652 (time 1734258249.87 total 296.55 last 132.23) - open/close 37970 (time 1734258259.88 total 306.56 last 131.70) - open/close 39291 (time 1734258269.88 total 316.57 last 132.05) - open/close 40000 (time 1734258275.25 total 321.93 last 132.11) - open/close 41280 (time 1734258285.25 total 331.94 last 127.95) - open/close 42634 (time 1734258295.26 total 341.94 last 135.33) - open/close 44063 (time 1734258305.26 total 351.95 last 142.84) - open/close 45435 (time 1734258315.27 total 361.95 last 137.14) - open/close 46762 (time 1734258325.27 total 371.95 last 132.65) - open/close 47959 (time 1734258335.27 total 381.96 last 119.65) - open/close 49230 (time 1734258345.28 total 391.96 last 127.08) - open/close 50000 (time 1734258351.18 total 397.87 last 130.37) - open/close 51369 (time 1734258361.19 total 407.87 last 136.86) - open/close 52668 (time 1734258371.19 total 417.87 last 129.86) - open/close 53958 (time 1734258381.19 total 427.87 last 128.98) - open/close 55401 (time 1734258391.19 total 437.87 last 144.29) - open/close 57120 (time 1734258401.19 total 447.88 last 171.84) - open/close 58739 (time 1734258411.20 total 457.88 last 161.80) - open/close 60000 (time 1734258418.33 total 465.01 last 176.97) - open/close 61698 (time 1734258428.33 total 475.02 last 169.68) - open/close 63253 (time 1734258438.34 total 485.02 last 155.45) total: 64768 open/close in 494.83 seconds: 130.89 ops/second - /unlink 2814 (time 1734258459.91 total 10.00 last 281.31) - /unlink 5582 (time 1734258469.92 total 20.01 last 276.68) - /unlink 7726 (time 1734258479.92 total 30.01 last 214.31) - /unlink 9833 (time 1734258489.92 total 40.01 last 210.65) - /unlink 12426 (time 1734258499.92 total 50.02 last 259.26) - /unlink 14534 (time 1734258509.92 total 60.02 last 210.79) - /unlink 16692 (time 1734258519.93 total 70.02 last 215.75) - /unlink 18987 (time 1734258529.93 total 80.02 last 229.44) - /unlink 20000 (time 1734258535.72 total 85.81 last 174.97) - /unlink 22052 (time 1734258545.73 total 95.82 last 205.08) - /unlink 24511 (time 1734258555.73 total 105.82 last 245.74) - /unlink 26683 (time 1734258565.74 total 115.83 last 217.09) - /unlink 28784 (time 1734258575.74 total 125.83 last 210.04) - /unlink 30000 (time 1734258580.71 total 130.80 last 244.56) - /unlink 32396 (time 1734258590.71 total 140.81 last 239.55) - /unlink 34873 (time 1734258600.71 total 150.81 last 247.70) - /unlink 37282 (time 1734258610.71 total 160.81 last 240.89) - /unlink 39654 (time 1734258620.72 total 170.81 last 237.05) - /unlink 41873 (time 1734258630.72 total 180.81 last 221.84) - /unlink 43758 (time 1734258640.72 total 190.82 last 188.47) - /unlink 45857 (time 1734258650.73 total 200.82 last 209.89) - /unlink 48154 (time 1734258660.73 total 210.82 last 229.66) - /unlink 50000 (time 1734258669.38 total 219.47 last 213.38) - /unlink 52294 (time 1734258679.38 total 229.47 last 229.36) - /unlink 55114 (time 1734258689.38 total 239.48 last 281.89) - /unlink 57682 (time 1734258699.39 total 249.48 last 256.76) - /unlink 60000 (time 1734258706.71 total 256.80 last 316.60) - /unlink 63116 (time 1734258716.71 total 266.80 last 311.55) total: 64768 /unlink in 272.71 seconds: 237.50 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1513 (time 1734258748.20 total 10.00 last 151.26) - open/close 3215 (time 1734258758.20 total 20.00 last 170.19) - open/close 5299 (time 1734258768.21 total 30.01 last 208.36) - open/close 7101 (time 1734258778.21 total 40.01 last 180.20) - open/close 8966 (time 1734258788.21 total 50.01 last 186.43) - open/close 10000 (time 1734258793.29 total 55.09 last 203.43) - open/close 11922 (time 1734258803.29 total 65.09 last 192.18) - open/close 13364 (time 1734258813.30 total 75.10 last 144.11) - open/close 14919 (time 1734258823.30 total 85.10 last 155.48) - open/close 16701 (time 1734258833.30 total 95.10 last 178.15) - open/close 18221 (time 1734258843.30 total 105.10 last 152.00) - open/close 20000 (time 1734258852.32 total 114.12 last 197.38) - open/close 22214 (time 1734258862.32 total 124.12 last 221.31) - open/close 24823 (time 1734258872.32 total 134.12 last 260.89) - open/close 27181 (time 1734258882.33 total 144.13 last 235.68) - open/close 29231 (time 1734258892.33 total 154.13 last 204.97) - open/close 30000 (time 1734258895.26 total 157.05 last 262.75) - open/close 31903 (time 1734258905.26 total 167.06 last 190.24) - open/close 33706 (time 1734258915.27 total 177.06 last 180.18) - open/close 35455 (time 1734258925.27 total 187.07 last 174.84) - open/close 37579 (time 1734258935.27 total 197.07 last 212.39) - open/close 39777 (time 1734258945.27 total 207.07 last 219.78) - open/close 42176 (time 1734258955.27 total 217.07 last 239.88) - open/close 44658 (time 1734258965.27 total 227.07 last 248.18) - open/close 46900 (time 1734258975.28 total 237.07 last 224.13) - open/close 49048 (time 1734258985.28 total 247.08 last 214.73) - open/close 50000 (time 1734258989.77 total 251.57 last 212.07) - open/close 52426 (time 1734258999.77 total 261.57 last 242.52) - open/close 55003 (time 1734259009.77 total 271.57 last 257.60) - open/close 57351 (time 1734259019.78 total 281.58 last 234.74) - open/close 59269 (time 1734259029.78 total 291.58 last 191.77) - open/close 60000 (time 1734259033.43 total 295.23 last 200.20) - open/close 62068 (time 1734259043.43 total 305.23 last 206.75) - open/close 64163 (time 1734259053.44 total 315.24 last 209.41) total: 64768 open/close in 318.37 seconds: 203.43 ops/second - /unlink 4218 (time 1734259067.80 total 10.00 last 421.70) - /unlink 8339 (time 1734259077.80 total 20.00 last 412.10) - /unlink 10000 (time 1734259082.05 total 24.25 last 390.78) - /unlink 13717 (time 1734259092.06 total 34.26 last 371.62) - /unlink 17574 (time 1734259102.06 total 44.26 last 385.62) - /unlink 20000 (time 1734259109.09 total 51.29 last 344.87) - /unlink 24582 (time 1734259119.09 total 61.29 last 458.13) - /unlink 28744 (time 1734259129.10 total 71.30 last 416.08) - /unlink 30000 (time 1734259132.33 total 74.53 last 388.31) - /unlink 33837 (time 1734259142.33 total 84.53 last 383.64) - /unlink 37697 (time 1734259152.33 total 94.53 last 385.96) - /unlink 40000 (time 1734259158.02 total 100.22 last 404.83) - /unlink 44205 (time 1734259168.02 total 110.22 last 420.47) - /unlink 48163 (time 1734259178.02 total 120.22 last 395.79) - /unlink 50000 (time 1734259183.35 total 125.55 last 344.78) - /unlink 54447 (time 1734259193.35 total 135.55 last 444.57) - /unlink 58298 (time 1734259203.36 total 145.55 last 385.07) - /unlink 60000 (time 1734259207.79 total 149.99 last 383.67) - /unlink 64668 (time 1734259217.79 total 159.99 last 466.79) total: 64768 /unlink in 160.21 seconds: 404.27 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2106 (time 1734259243.95 total 10.00 last 210.54) - open/close 4416 (time 1734259253.96 total 20.01 last 230.95) - open/close 6569 (time 1734259263.96 total 30.01 last 215.22) - open/close 8878 (time 1734259273.96 total 40.01 last 230.85) - open/close 10000 (time 1734259278.03 total 44.08 last 275.59) - open/close 12759 (time 1734259288.03 total 54.08 last 275.90) - open/close 15267 (time 1734259298.04 total 64.09 last 250.73) - open/close 17478 (time 1734259308.04 total 74.09 last 221.02) - open/close 19393 (time 1734259318.04 total 84.09 last 191.42) - open/close 20000 (time 1734259321.21 total 87.26 last 191.58) - open/close 22176 (time 1734259331.22 total 97.27 last 217.43) - open/close 24453 (time 1734259341.22 total 107.27 last 227.64) - open/close 27027 (time 1734259351.23 total 117.28 last 257.28) - open/close 29502 (time 1734259361.23 total 127.28 last 247.42) - open/close 31817 (time 1734259371.23 total 137.28 last 231.44) - open/close 33874 (time 1734259381.23 total 147.28 last 205.70) - open/close 36131 (time 1734259391.23 total 157.28 last 225.66) - open/close 38405 (time 1734259401.24 total 167.29 last 227.29) - open/close 40000 (time 1734259408.09 total 174.14 last 232.78) - open/close 42322 (time 1734259418.09 total 184.14 last 232.13) - open/close 44347 (time 1734259428.10 total 194.15 last 202.46) - open/close 46492 (time 1734259438.10 total 204.15 last 214.48) - open/close 48420 (time 1734259448.10 total 214.15 last 192.80) - open/close 50000 (time 1734259455.90 total 221.95 last 202.52) - open/close 52140 (time 1734259465.90 total 231.95 last 213.94) - open/close 54630 (time 1734259475.90 total 241.95 last 248.99) - open/close 56800 (time 1734259485.91 total 251.95 last 216.94) - open/close 59093 (time 1734259495.91 total 261.96 last 229.21) - open/close 60000 (time 1734259500.09 total 266.14 last 216.79) - open/close 62075 (time 1734259510.09 total 276.14 last 207.49) - open/close 64365 (time 1734259520.10 total 286.14 last 228.95) total: 64768 open/close in 288.32 seconds: 224.64 ops/second - /unlink 3632 (time 1734259533.40 total 10.00 last 363.16) - /unlink 7433 (time 1734259543.40 total 20.00 last 380.06) - /unlink 10000 (time 1734259550.58 total 27.18 last 357.86) - /unlink 13901 (time 1734259560.58 total 37.18 last 390.09) - /unlink 18082 (time 1734259570.58 total 47.18 last 418.09) - /unlink 20000 (time 1734259575.29 total 51.89 last 407.22) - /unlink 24207 (time 1734259585.29 total 61.89 last 420.68) - /unlink 28679 (time 1734259595.29 total 71.89 last 447.14) - /unlink 30000 (time 1734259598.04 total 74.64 last 479.42) - /unlink 34618 (time 1734259608.05 total 84.64 last 461.77) - /unlink 38934 (time 1734259618.05 total 94.65 last 431.54) - /unlink 40000 (time 1734259620.64 total 97.24 last 410.75) - /unlink 45390 (time 1734259630.64 total 107.24 last 538.98) - /unlink 50000 (time 1734259639.24 total 115.84 last 536.13) - /unlink 55275 (time 1734259649.24 total 125.84 last 527.43) - /unlink 60000 (time 1734259657.86 total 134.46 last 548.46) - /unlink 64423 (time 1734259667.86 total 144.46 last 442.27) total: 64768 /unlink in 145.31 seconds: 445.73 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2803 (time 1734259694.16 total 10.00 last 280.29) - open/close 5877 (time 1734259704.16 total 20.00 last 307.38) - open/close 8622 (time 1734259714.16 total 30.00 last 274.42) - open/close 10000 (time 1734259720.40 total 36.24 last 220.90) - open/close 12694 (time 1734259730.40 total 46.24 last 269.35) - open/close 15081 (time 1734259740.40 total 56.25 last 238.64) - open/close 17465 (time 1734259750.41 total 66.25 last 238.30) - open/close 19973 (time 1734259760.41 total 76.25 last 250.70) - open/close 22136 (time 1734259770.41 total 86.25 last 216.29) - open/close 24314 (time 1734259780.41 total 96.26 last 217.73) - open/close 26641 (time 1734259790.41 total 106.26 last 232.70) - open/close 29038 (time 1734259800.41 total 116.26 last 239.69) - open/close 30000 (time 1734259803.68 total 119.52 last 294.77) - open/close 33045 (time 1734259813.68 total 129.53 last 304.35) - open/close 35829 (time 1734259823.68 total 139.53 last 278.37) - open/close 38696 (time 1734259833.69 total 149.53 last 286.64) - open/close 40000 (time 1734259838.67 total 154.51 last 261.82) - open/close 42794 (time 1734259848.67 total 164.51 last 279.33) - open/close 45085 (time 1734259858.67 total 174.52 last 229.01) - open/close 47642 (time 1734259868.67 total 184.52 last 255.68) - open/close 50000 (time 1734259878.60 total 194.45 last 237.43) - open/close 52626 (time 1734259888.61 total 204.45 last 262.52) - open/close 55283 (time 1734259898.61 total 214.45 last 265.64) - open/close 57885 (time 1734259908.61 total 224.46 last 260.17) - open/close 60000 (time 1734259916.65 total 232.49 last 263.12) - open/close 62855 (time 1734259926.65 total 242.50 last 285.41) total: 64768 open/close in 249.05 seconds: 260.06 ops/second - /unlink 5456 (time 1734259944.19 total 10.00 last 545.48) - /unlink 10000 (time 1734259951.93 total 17.74 last 587.28) - /unlink 15230 (time 1734259961.93 total 27.74 last 522.97) - /unlink 20000 (time 1734259970.34 total 36.16 last 566.72) - /unlink 26102 (time 1734259980.34 total 46.16 last 610.17) - /unlink 30000 (time 1734259986.32 total 52.14 last 651.77) - /unlink 35449 (time 1734259996.33 total 62.14 last 544.84) - /unlink 40000 (time 1734260004.65 total 70.46 last 546.99) - /unlink 45570 (time 1734260014.65 total 80.46 last 556.93) - /unlink 50000 (time 1734260022.83 total 88.64 last 541.32) - /unlink 55925 (time 1734260032.83 total 98.65 last 592.46) - /unlink 60000 (time 1734260039.88 total 105.69 last 578.35) total: 64768 /unlink in 115.38 seconds: 561.33 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3046 (time 1734260080.54 total 10.00 last 304.47) - open/close 5883 (time 1734260090.55 total 20.00 last 283.69) - open/close 8678 (time 1734260100.55 total 30.01 last 279.41) - open/close 10000 (time 1734260105.56 total 35.02 last 263.57) - open/close 13135 (time 1734260115.57 total 45.03 last 313.37) - open/close 15898 (time 1734260125.57 total 55.03 last 276.26) - open/close 18715 (time 1734260135.57 total 65.03 last 281.60) - open/close 20000 (time 1734260140.05 total 69.51 last 287.22) - open/close 23019 (time 1734260150.05 total 79.51 last 301.80) - open/close 26075 (time 1734260160.05 total 89.51 last 305.57) - open/close 29144 (time 1734260170.05 total 99.51 last 306.86) - open/close 30000 (time 1734260173.14 total 102.59 last 277.70) - open/close 32811 (time 1734260183.14 total 112.60 last 281.02) - open/close 35236 (time 1734260193.14 total 122.60 last 242.47) - open/close 38031 (time 1734260203.14 total 132.60 last 279.46) - open/close 40000 (time 1734260210.11 total 139.57 last 282.45) - open/close 42656 (time 1734260220.12 total 149.58 last 265.47) - open/close 45033 (time 1734260230.12 total 159.58 last 237.60) - open/close 47703 (time 1734260240.13 total 169.59 last 266.73) - open/close 50000 (time 1734260248.64 total 178.10 last 270.10) - open/close 52689 (time 1734260258.64 total 188.10 last 268.89) - open/close 55470 (time 1734260268.64 total 198.10 last 278.08) - open/close 58514 (time 1734260278.64 total 208.10 last 304.32) - open/close 60000 (time 1734260284.03 total 213.48 last 275.91) - open/close 63153 (time 1734260294.03 total 223.49 last 315.29) total: 64768 open/close in 229.33 seconds: 282.43 ops/second - /unlink 4315 (time 1734260310.92 total 10.00 last 431.41) - /unlink 8953 (time 1734260320.93 total 20.00 last 463.78) - /unlink 10000 (time 1734260323.04 total 22.12 last 494.70) - /unlink 14943 (time 1734260333.04 total 32.12 last 494.24) - /unlink 19281 (time 1734260343.04 total 42.12 last 433.73) - /unlink 23709 (time 1734260353.04 total 52.12 last 442.77) - /unlink 28363 (time 1734260363.05 total 62.12 last 465.35) - /unlink 30000 (time 1734260366.10 total 65.18 last 535.68) - /unlink 34874 (time 1734260376.10 total 75.18 last 487.38) - /unlink 39298 (time 1734260386.10 total 85.18 last 442.40) - /unlink 44309 (time 1734260396.10 total 95.18 last 501.01) - /unlink 49882 (time 1734260406.10 total 105.18 last 557.30) - /unlink 55105 (time 1734260416.10 total 115.18 last 522.30) - /unlink 60000 (time 1734260425.36 total 124.44 last 528.90) total: 64768 /unlink in 133.64 seconds: 484.66 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg211-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg211-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server stop mds service on oleg211-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server unloading modules via unload_modules_local on: 'oleg211-server' oleg211-server: oleg211-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2