== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:33:45 (1734258825) Stopping clients: oleg153-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg153-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg153-client: oleg153-server: ssh exited with exit code 2 oleg153-server: oleg153-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg153-server' oleg153-server: oleg153-server.virtnet: executing load_modules_local oleg153-server: Loading modules from /home/green/git/lustre-release/lustre oleg153-server: detected 4 online CPUs by sysfs oleg153-server: Force libcfs to create 2 CPU partitions oleg153-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg153-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg153-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg153-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg153-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg153-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg153-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg153-server: oleg153-server.virtnet: executing set_default_debug -1 all pdsh@oleg153-client: oleg153-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg153-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg153-client.virtnet: -o user_xattr,flock 192.168.201.153@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16266 458 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16451 643 15808 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1498 (time 1734258901.60 total 10.00 last 149.78) - open/close 2954 (time 1734258911.60 total 20.00 last 145.57) - open/close 4427 (time 1734258921.61 total 30.01 last 147.18) - open/close 6054 (time 1734258931.61 total 40.01 last 162.69) - open/close 7732 (time 1734258941.61 total 50.02 last 167.73) - open/close 9430 (time 1734258951.61 total 60.02 last 169.79) - open/close 10000 (time 1734258954.99 total 63.39 last 168.78) - open/close 11490 (time 1734258965.00 total 73.40 last 148.84) - open/close 12842 (time 1734258975.01 total 83.41 last 135.13) - open/close 14202 (time 1734258985.01 total 93.41 last 135.94) - open/close 15538 (time 1734258995.01 total 103.42 last 133.56) - open/close 16975 (time 1734259005.02 total 113.43 last 143.56) - open/close 18391 (time 1734259015.03 total 123.43 last 141.54) - open/close 19879 (time 1734259025.03 total 133.43 last 148.80) - open/close 21326 (time 1734259035.03 total 143.43 last 144.69) - open/close 22635 (time 1734259045.03 total 153.43 last 130.85) - open/close 23994 (time 1734259055.04 total 163.44 last 135.81) - open/close 25330 (time 1734259065.04 total 173.45 last 133.52) - open/close 26610 (time 1734259075.04 total 183.45 last 128.00) - open/close 28024 (time 1734259085.05 total 193.45 last 141.36) - open/close 29431 (time 1734259095.05 total 203.45 last 140.65) - open/close 30000 (time 1734259099.50 total 207.90 last 127.88) - open/close 31382 (time 1734259109.50 total 217.90 last 138.18) - open/close 32658 (time 1734259119.50 total 227.91 last 127.56) - open/close 33921 (time 1734259129.51 total 237.92 last 126.18) - open/close 35196 (time 1734259139.52 total 247.92 last 127.46) - open/close 36472 (time 1734259149.52 total 257.93 last 127.48) - open/close 37646 (time 1734259159.53 total 267.93 last 117.34) - open/close 38872 (time 1734259169.53 total 277.94 last 122.55) - open/close 40000 (time 1734259178.07 total 286.47 last 132.16) - open/close 41284 (time 1734259188.08 total 296.48 last 128.29) - open/close 42551 (time 1734259198.08 total 306.49 last 126.65) - open/close 43835 (time 1734259208.09 total 316.49 last 128.32) - open/close 45093 (time 1734259218.09 total 326.50 last 125.75) - open/close 46289 (time 1734259228.10 total 336.50 last 119.49) - open/close 47590 (time 1734259238.10 total 346.51 last 130.05) - open/close 48876 (time 1734259248.11 total 356.51 last 128.53) - open/close 50000 (time 1734259256.39 total 364.79 last 135.81) - open/close 51385 (time 1734259266.39 total 374.80 last 138.42) - open/close 52833 (time 1734259276.40 total 384.80 last 144.71) - open/close 54396 (time 1734259286.40 total 394.81 last 156.22) - open/close 55760 (time 1734259296.40 total 404.81 last 136.39) - open/close 57154 (time 1734259306.41 total 414.81 last 139.34) - open/close 58489 (time 1734259316.41 total 424.81 last 133.47) - open/close 59808 (time 1734259326.42 total 434.82 last 131.78) - open/close 61052 (time 1734259336.43 total 444.83 last 124.30) - open/close 62254 (time 1734259346.43 total 454.83 last 120.17) - open/close 63683 (time 1734259356.43 total 464.84 last 142.86) total: 64768 open/close in 473.18 seconds: 136.88 ops/second - /unlink 2187 (time 1734259376.60 total 10.00 last 218.68) - /unlink 4860 (time 1734259386.61 total 20.00 last 267.24) - /unlink 7415 (time 1734259396.61 total 30.01 last 255.43) - /unlink 10000 (time 1734259406.56 total 39.96 last 259.72) - /unlink 12633 (time 1734259416.56 total 49.96 last 263.28) - /unlink 15380 (time 1734259426.57 total 59.96 last 274.61) - /unlink 18034 (time 1734259436.57 total 69.97 last 265.29) - /unlink 20000 (time 1734259444.44 total 77.83 last 249.91) - /unlink 22478 (time 1734259454.44 total 87.83 last 247.79) - /unlink 25275 (time 1734259464.44 total 97.84 last 279.63) - /unlink 28265 (time 1734259474.44 total 107.84 last 298.93) - /unlink 30000 (time 1734259480.30 total 113.70 last 296.28) - /unlink 33041 (time 1734259490.30 total 123.70 last 304.09) - /unlink 36084 (time 1734259500.30 total 133.70 last 304.21) - /unlink 39091 (time 1734259510.30 total 143.70 last 300.65) - /unlink 40000 (time 1734259512.96 total 146.36 last 341.70) - /unlink 43020 (time 1734259522.96 total 156.36 last 301.96) - /unlink 46076 (time 1734259532.97 total 166.37 last 305.45) - /unlink 48763 (time 1734259542.97 total 176.37 last 268.58) - /unlink 50000 (time 1734259548.18 total 181.57 last 237.72) - /unlink 52286 (time 1734259558.18 total 191.58 last 228.57) - /unlink 54754 (time 1734259568.18 total 201.58 last 246.73) - /unlink 57460 (time 1734259578.18 total 211.58 last 270.54) - /unlink 59937 (time 1734259588.19 total 221.58 last 247.62) - /unlink 62494 (time 1734259598.19 total 231.59 last 255.55) total: 64768 /unlink in 240.05 seconds: 269.81 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1484 (time 1734259631.75 total 10.00 last 148.35) - open/close 3053 (time 1734259641.75 total 20.00 last 156.89) - open/close 4659 (time 1734259651.75 total 30.01 last 160.56) - open/close 6232 (time 1734259661.75 total 40.01 last 157.28) - open/close 7818 (time 1734259671.75 total 50.01 last 158.57) - open/close 9324 (time 1734259681.76 total 60.01 last 150.53) - open/close 10000 (time 1734259685.76 total 64.02 last 168.70) - open/close 11665 (time 1734259695.77 total 74.02 last 166.46) - open/close 13183 (time 1734259705.77 total 84.03 last 151.75) - open/close 14698 (time 1734259715.77 total 94.03 last 151.44) - open/close 16097 (time 1734259725.78 total 104.03 last 139.86) - open/close 17570 (time 1734259735.78 total 114.04 last 147.21) - open/close 19258 (time 1734259745.78 total 124.04 last 168.77) - open/close 20000 (time 1734259750.49 total 128.75 last 157.65) - open/close 21601 (time 1734259760.49 total 138.75 last 160.09) - open/close 23130 (time 1734259770.50 total 148.75 last 152.83) - open/close 24968 (time 1734259780.50 total 158.76 last 183.75) - open/close 26794 (time 1734259790.50 total 168.76 last 182.57) - open/close 28524 (time 1734259800.51 total 178.76 last 172.91) - open/close 30000 (time 1734259809.41 total 187.67 last 165.71) - open/close 31762 (time 1734259819.41 total 197.67 last 176.18) - open/close 33374 (time 1734259829.42 total 207.68 last 161.11) - open/close 34849 (time 1734259839.42 total 217.68 last 147.48) - open/close 36351 (time 1734259849.43 total 227.69 last 150.09) - open/close 38032 (time 1734259859.43 total 237.69 last 168.09) - open/close 39457 (time 1734259869.44 total 247.69 last 142.42) - open/close 40000 (time 1734259873.11 total 251.36 last 147.89) - open/close 41504 (time 1734259883.11 total 261.37 last 150.34) - open/close 43157 (time 1734259893.11 total 271.37 last 165.29) - open/close 44756 (time 1734259903.11 total 281.37 last 159.86) - open/close 46516 (time 1734259913.12 total 291.38 last 175.90) - open/close 48378 (time 1734259923.12 total 301.38 last 186.11) - open/close 50000 (time 1734259932.29 total 310.55 last 176.91) - open/close 51872 (time 1734259942.29 total 320.55 last 187.20) - open/close 53682 (time 1734259952.30 total 330.55 last 180.89) - open/close 55614 (time 1734259962.30 total 340.56 last 193.13) - open/close 57389 (time 1734259972.31 total 350.56 last 177.42) - open/close 59295 (time 1734259982.31 total 360.57 last 190.50) - open/close 60000 (time 1734259986.09 total 364.35 last 186.45) - open/close 61887 (time 1734259996.10 total 374.35 last 188.66) - open/close 63898 (time 1734260006.10 total 384.35 last 201.04) total: 64768 open/close in 388.92 seconds: 166.53 ops/second - /unlink 3378 (time 1734260021.82 total 10.01 last 337.62) - /unlink 6747 (time 1734260031.82 total 20.01 last 336.76) - /unlink 9957 (time 1734260041.82 total 30.01 last 320.94) - /unlink 13223 (time 1734260051.83 total 40.01 last 326.53) - /unlink 16271 (time 1734260061.83 total 50.01 last 304.80) - /unlink 18992 (time 1734260071.83 total 60.02 last 272.05) - /unlink 20000 (time 1734260075.34 total 63.53 last 287.05) - /unlink 22845 (time 1734260085.34 total 73.53 last 284.46) - /unlink 25994 (time 1734260095.34 total 83.53 last 314.82) - /unlink 28920 (time 1734260105.35 total 93.53 last 292.56) - /unlink 30000 (time 1734260108.92 total 97.11 last 301.82) - /unlink 33045 (time 1734260118.93 total 107.11 last 304.43) - /unlink 36304 (time 1734260128.93 total 117.11 last 325.86) - /unlink 39656 (time 1734260138.93 total 127.12 last 335.16) - /unlink 43010 (time 1734260148.93 total 137.12 last 335.35) - /unlink 46888 (time 1734260158.93 total 147.12 last 387.73) - /unlink 50000 (time 1734260167.20 total 155.39 last 376.44) - /unlink 53774 (time 1734260177.20 total 165.39 last 377.36) - /unlink 58194 (time 1734260187.20 total 175.39 last 441.91) - /unlink 60000 (time 1734260191.00 total 179.19 last 474.89) - /unlink 64405 (time 1734260201.01 total 189.19 last 440.36) total: 64768 /unlink in 190.01 seconds: 340.87 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2842 (time 1734260227.22 total 10.00 last 284.19) - open/close 5752 (time 1734260237.22 total 20.00 last 290.91) - open/close 8200 (time 1734260247.22 total 30.00 last 244.77) - open/close 10000 (time 1734260254.27 total 37.06 last 255.28) - open/close 12610 (time 1734260264.28 total 47.06 last 260.93) - open/close 15287 (time 1734260274.28 total 57.06 last 267.61) - open/close 18171 (time 1734260284.28 total 67.07 last 288.31) - open/close 20000 (time 1734260291.10 total 73.88 last 268.45) - open/close 22714 (time 1734260301.10 total 83.88 last 271.28) - open/close 25047 (time 1734260311.10 total 93.89 last 233.22) - open/close 27144 (time 1734260321.10 total 103.89 last 209.67) - open/close 29413 (time 1734260331.11 total 113.89 last 226.84) - open/close 30000 (time 1734260333.76 total 116.54 last 221.51) - open/close 32166 (time 1734260343.76 total 126.54 last 216.58) - open/close 34632 (time 1734260353.76 total 136.54 last 246.56) - open/close 37612 (time 1734260363.76 total 146.54 last 297.95) - open/close 40000 (time 1734260372.54 total 155.32 last 272.13) - open/close 42566 (time 1734260382.54 total 165.32 last 256.56) - open/close 45205 (time 1734260392.54 total 175.33 last 263.80) - open/close 47868 (time 1734260402.54 total 185.33 last 266.29) - open/close 50000 (time 1734260410.84 total 193.62 last 257.00) - open/close 52311 (time 1734260420.84 total 203.62 last 231.07) - open/close 54759 (time 1734260430.84 total 213.63 last 244.73) - open/close 56911 (time 1734260440.84 total 223.63 last 215.18) - open/close 58830 (time 1734260450.85 total 233.63 last 191.85) - open/close 60000 (time 1734260456.29 total 239.07 last 215.09) - open/close 62137 (time 1734260466.29 total 249.07 last 213.66) - open/close 64056 (time 1734260476.29 total 259.07 last 191.84) total: 64768 open/close in 262.52 seconds: 246.72 ops/second - /unlink 3561 (time 1734260490.84 total 10.00 last 355.97) - /unlink 7123 (time 1734260500.84 total 20.00 last 356.18) - /unlink 10000 (time 1734260508.03 total 27.20 last 400.07) - /unlink 14395 (time 1734260518.03 total 37.20 last 439.44) - /unlink 19163 (time 1734260528.03 total 47.20 last 476.70) - /unlink 23847 (time 1734260538.04 total 57.20 last 468.35) - /unlink 28150 (time 1734260548.04 total 67.20 last 430.30) - /unlink 30000 (time 1734260552.00 total 71.17 last 466.09) - /unlink 34269 (time 1734260562.01 total 81.17 last 426.81) - /unlink 39436 (time 1734260572.01 total 91.17 last 516.63) - /unlink 44735 (time 1734260582.01 total 101.17 last 529.84) - /unlink 49714 (time 1734260592.01 total 111.18 last 497.79) - /unlink 54280 (time 1734260602.01 total 121.18 last 456.54) - /unlink 60000 (time 1734260611.80 total 130.96 last 584.67) total: 64768 /unlink in 138.85 seconds: 466.46 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3113 (time 1734260647.87 total 10.00 last 311.21) - open/close 5854 (time 1734260657.87 total 20.00 last 274.06) - open/close 8403 (time 1734260667.87 total 30.01 last 254.84) - open/close 10000 (time 1734260673.37 total 35.50 last 290.48) - open/close 12939 (time 1734260683.37 total 45.51 last 293.88) - open/close 16024 (time 1734260693.37 total 55.51 last 308.48) - open/close 18823 (time 1734260703.37 total 65.51 last 279.84) - open/close 20000 (time 1734260707.06 total 69.19 last 319.37) - open/close 23021 (time 1734260717.06 total 79.19 last 302.09) - open/close 25936 (time 1734260727.06 total 89.19 last 291.48) - open/close 28939 (time 1734260737.06 total 99.20 last 300.22) - open/close 30000 (time 1734260740.48 total 102.61 last 310.79) - open/close 33167 (time 1734260750.48 total 112.61 last 316.64) - open/close 36736 (time 1734260760.48 total 122.61 last 356.86) - open/close 40000 (time 1734260768.70 total 130.83 last 397.29) - open/close 43892 (time 1734260778.70 total 140.83 last 389.15) - open/close 47730 (time 1734260788.70 total 150.83 last 383.71) - open/close 50000 (time 1734260794.56 total 156.69 last 387.53) - open/close 55133 (time 1734260804.56 total 166.69 last 513.22) - open/close 60000 (time 1734260813.60 total 175.73 last 538.54) total: 64768 open/close in 185.44 seconds: 349.26 ops/second - /unlink 8045 (time 1734260834.19 total 10.00 last 804.49) - /unlink 10000 (time 1734260836.38 total 12.19 last 894.41) - /unlink 18053 (time 1734260846.38 total 22.19 last 805.16) - /unlink 20000 (time 1734260849.08 total 24.89 last 721.01) - /unlink 27582 (time 1734260859.08 total 34.89 last 758.17) - /unlink 30000 (time 1734260862.86 total 38.66 last 640.46) - /unlink 36713 (time 1734260872.86 total 48.66 last 671.26) - /unlink 40000 (time 1734260878.13 total 53.94 last 622.92) - /unlink 47263 (time 1734260888.13 total 63.94 last 726.29) - /unlink 50000 (time 1734260891.72 total 67.53 last 762.69) - /unlink 58744 (time 1734260901.72 total 77.53 last 874.33) total: 64768 /unlink in 83.90 seconds: 771.99 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4981 (time 1734260938.93 total 10.00 last 498.03) - open/close 9628 (time 1734260948.93 total 20.00 last 464.64) - open/close 14788 (time 1734260958.93 total 30.00 last 515.96) - open/close 20000 (time 1734260968.58 total 39.65 last 540.28) - open/close 25472 (time 1734260978.58 total 49.65 last 547.12) - open/close 30000 (time 1734260987.44 total 58.51 last 511.05) - open/close 35088 (time 1734260997.44 total 68.51 last 508.76) - open/close 40000 (time 1734261007.00 total 78.07 last 514.13) - open/close 45049 (time 1734261017.00 total 88.07 last 504.88) - open/close 49839 (time 1734261027.00 total 98.07 last 478.97) - open/close 54473 (time 1734261037.00 total 108.07 last 463.38) - open/close 59272 (time 1734261047.00 total 118.07 last 479.82) - open/close 63957 (time 1734261057.00 total 128.07 last 468.39) total: 64768 open/close in 129.93 seconds: 498.47 ops/second - /unlink 7393 (time 1734261069.80 total 10.00 last 739.22) - /unlink 10000 (time 1734261073.00 total 13.21 last 813.05) - /unlink 17607 (time 1734261083.01 total 23.21 last 760.58) - /unlink 20000 (time 1734261085.48 total 25.69 last 965.72) - /unlink 28496 (time 1734261095.48 total 35.69 last 849.54) - /unlink 36919 (time 1734261105.49 total 45.69 last 842.23) - /unlink 40000 (time 1734261109.28 total 49.48 last 811.78) - /unlink 48171 (time 1734261119.28 total 59.48 last 817.05) - /unlink 50000 (time 1734261121.44 total 61.64 last 848.93) - /unlink 57870 (time 1734261131.44 total 71.64 last 786.97) - /unlink 60000 (time 1734261134.01 total 74.21 last 828.91) total: 64768 /unlink in 79.67 seconds: 812.94 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg153-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg153-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg153-server stop mds service on oleg153-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg153-server stop mds service on oleg153-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg153-server unloading modules via unload_modules_local on: 'oleg153-server' oleg153-server: oleg153-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg153-client: oleg153-client: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-server: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-client: ssh exited with exit code 2 pdsh@oleg153-client: oleg153-server: ssh exited with exit code 2