== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:55:57 (1735098957) Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) oleg408-server: oleg408-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg408-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg408-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2790 (time 1735099002.06 total 10.00 last 278.97) - open/close 5004 (time 1735099012.06 total 20.00 last 221.38) - open/close 7585 (time 1735099022.07 total 30.01 last 258.00) - open/close 10000 (time 1735099030.88 total 38.82 last 274.05) - open/close 13315 (time 1735099040.88 total 48.82 last 331.49) - open/close 16456 (time 1735099050.88 total 58.82 last 314.02) - open/close 19220 (time 1735099060.88 total 68.82 last 276.34) - open/close 20000 (time 1735099063.72 total 71.66 last 275.09) - open/close 24072 (time 1735099073.72 total 81.66 last 407.11) - open/close 28090 (time 1735099083.72 total 91.66 last 401.77) - open/close 30000 (time 1735099088.50 total 96.43 last 400.18) - open/close 33344 (time 1735099098.50 total 106.44 last 334.36) - open/close 36141 (time 1735099108.50 total 116.44 last 279.66) - open/close 39724 (time 1735099118.50 total 126.44 last 358.30) - open/close 43467 (time 1735099128.50 total 136.44 last 374.22) - open/close 47811 (time 1735099138.50 total 146.44 last 434.39) - open/close 50000 (time 1735099143.55 total 151.49 last 433.62) - open/close 54380 (time 1735099153.55 total 161.49 last 437.98) - open/close 58767 (time 1735099163.55 total 171.49 last 438.67) - open/close 60000 (time 1735099165.78 total 173.72 last 552.96) total: 64768 open/close in 182.89 seconds: 354.13 ops/second - /unlink 8659 (time 1735099185.83 total 10.00 last 865.85) - /unlink 14946 (time 1735099195.83 total 20.00 last 628.67) - /unlink 20000 (time 1735099204.12 total 28.30 last 609.28) - /unlink 26784 (time 1735099214.12 total 38.30 last 678.31) - /unlink 30000 (time 1735099219.48 total 43.65 last 600.31) - /unlink 36107 (time 1735099229.48 total 53.66 last 610.61) - /unlink 40000 (time 1735099236.03 total 60.20 last 594.76) - /unlink 45980 (time 1735099246.03 total 70.20 last 597.94) - /unlink 50000 (time 1735099253.43 total 77.60 last 543.49) - /unlink 56723 (time 1735099263.43 total 87.60 last 672.23) - /unlink 60000 (time 1735099268.12 total 92.30 last 697.70) total: 64768 /unlink in 99.96 seconds: 647.96 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3389 (time 1735099304.97 total 10.00 last 338.78) - open/close 6993 (time 1735099314.97 total 20.01 last 360.33) - open/close 9878 (time 1735099324.98 total 30.01 last 288.40) - open/close 12281 (time 1735099334.98 total 40.01 last 240.23) - open/close 13659 (time 1735099344.98 total 50.01 last 137.79) - open/close 14791 (time 1735099354.98 total 60.02 last 113.15) - open/close 16239 (time 1735099364.99 total 70.02 last 144.78) - open/close 17587 (time 1735099374.99 total 80.02 last 134.75) - open/close 19134 (time 1735099384.99 total 90.02 last 154.67) - open/close 20000 (time 1735099390.97 total 96.01 last 144.78) - open/close 21436 (time 1735099400.97 total 106.01 last 143.57) - open/close 23033 (time 1735099410.98 total 116.01 last 159.63) - open/close 25100 (time 1735099420.98 total 126.01 last 206.69) - open/close 28037 (time 1735099430.98 total 136.01 last 293.65) - open/close 30000 (time 1735099436.82 total 141.86 last 335.97) - open/close 32622 (time 1735099446.83 total 151.86 last 262.09) - open/close 35364 (time 1735099456.83 total 161.86 last 274.20) - open/close 37537 (time 1735099466.83 total 171.87 last 217.18) - open/close 40000 (time 1735099476.39 total 181.42 last 257.87) - open/close 42590 (time 1735099486.39 total 191.42 last 258.91) - open/close 45067 (time 1735099496.39 total 201.43 last 247.62) - open/close 47362 (time 1735099506.39 total 211.43 last 229.47) - open/close 49774 (time 1735099516.40 total 221.43 last 241.15) - open/close 52335 (time 1735099526.40 total 231.43 last 256.09) - open/close 54745 (time 1735099536.40 total 241.43 last 240.98) - open/close 57575 (time 1735099546.40 total 251.43 last 282.97) - open/close 60000 (time 1735099555.93 total 260.96 last 254.50) - open/close 62711 (time 1735099565.93 total 270.96 last 270.97) total: 64768 open/close in 280.34 seconds: 231.04 ops/second - /unlink 3662 (time 1735099586.57 total 10.00 last 366.10) - /unlink 7462 (time 1735099596.57 total 20.00 last 379.98) - /unlink 10000 (time 1735099603.94 total 27.37 last 344.48) - /unlink 13514 (time 1735099613.94 total 37.37 last 351.39) - /unlink 17227 (time 1735099623.94 total 47.37 last 371.21) - /unlink 20000 (time 1735099632.50 total 55.92 last 324.26) - /unlink 23282 (time 1735099642.50 total 65.93 last 328.16) - /unlink 26659 (time 1735099652.50 total 75.93 last 337.67) - /unlink 30000 (time 1735099660.44 total 83.87 last 420.68) - /unlink 33799 (time 1735099670.44 total 93.87 last 379.83) - /unlink 36957 (time 1735099680.44 total 103.87 last 315.78) - /unlink 40000 (time 1735099690.17 total 113.60 last 312.73) - /unlink 43410 (time 1735099700.17 total 123.60 last 340.95) - /unlink 46643 (time 1735099710.18 total 133.60 last 323.25) - /unlink 49878 (time 1735099720.18 total 143.61 last 323.45) - /unlink 53020 (time 1735099730.18 total 153.61 last 314.15) - /unlink 56782 (time 1735099740.18 total 163.61 last 376.18) - /unlink 60000 (time 1735099748.85 total 172.27 last 371.36) - /unlink 64240 (time 1735099758.85 total 182.28 last 423.94) total: 64768 /unlink in 183.32 seconds: 353.31 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2382 (time 1735099785.52 total 10.00 last 238.16) - open/close 4859 (time 1735099795.52 total 20.00 last 247.64) - open/close 7699 (time 1735099805.53 total 30.01 last 283.90) - open/close 10000 (time 1735099815.09 total 39.57 last 240.69) - open/close 12735 (time 1735099825.09 total 49.57 last 273.41) - open/close 15806 (time 1735099835.09 total 59.57 last 307.00) - open/close 18714 (time 1735099845.09 total 69.57 last 290.78) - open/close 20000 (time 1735099849.73 total 74.21 last 277.21) - open/close 23182 (time 1735099859.74 total 84.22 last 318.14) - open/close 26050 (time 1735099869.74 total 94.22 last 286.73) - open/close 29024 (time 1735099879.74 total 104.22 last 297.32) - open/close 30000 (time 1735099883.16 total 107.64 last 285.47) - open/close 32934 (time 1735099893.16 total 117.64 last 293.37) - open/close 35663 (time 1735099903.16 total 127.64 last 272.81) - open/close 38380 (time 1735099913.16 total 137.64 last 271.69) - open/close 40000 (time 1735099919.13 total 143.61 last 271.59) - open/close 42798 (time 1735099929.13 total 153.61 last 279.73) - open/close 45644 (time 1735099939.13 total 163.61 last 284.53) - open/close 47987 (time 1735099949.14 total 173.62 last 234.26) - open/close 49863 (time 1735099959.14 total 183.62 last 187.51) - open/close 51808 (time 1735099969.15 total 193.63 last 194.41) - open/close 54019 (time 1735099979.15 total 203.63 last 221.01) - open/close 56289 (time 1735099989.15 total 213.63 last 226.98) - open/close 57726 (time 1735099999.16 total 223.64 last 143.59) - open/close 58711 (time 1735100009.16 total 233.64 last 98.48) - open/close 59704 (time 1735100019.17 total 243.65 last 99.21) - open/close 60000 (time 1735100021.96 total 246.43 last 106.27) - open/close 61196 (time 1735100031.96 total 256.44 last 119.56) - open/close 62296 (time 1735100041.96 total 266.44 last 109.95) - open/close 63542 (time 1735100051.97 total 276.45 last 124.52) total: 64768 open/close in 286.04 seconds: 226.43 ops/second - /unlink 2392 (time 1735100073.23 total 10.01 last 239.04) - /unlink 4692 (time 1735100083.23 total 20.01 last 229.99) - /unlink 7120 (time 1735100093.23 total 30.01 last 242.76) - /unlink 9545 (time 1735100103.23 total 40.01 last 242.45) - /unlink 12801 (time 1735100113.23 total 50.01 last 325.55) - /unlink 16318 (time 1735100123.23 total 60.02 last 351.59) - /unlink 19773 (time 1735100133.23 total 70.02 last 345.48) - /unlink 23164 (time 1735100143.24 total 80.02 last 339.09) - /unlink 26815 (time 1735100153.24 total 90.02 last 365.07) - /unlink 30000 (time 1735100162.42 total 99.20 last 346.81) - /unlink 33264 (time 1735100172.42 total 109.20 last 326.39) - /unlink 36621 (time 1735100182.42 total 119.20 last 335.62) - /unlink 39400 (time 1735100192.43 total 129.21 last 277.81) - /unlink 40000 (time 1735100194.52 total 131.30 last 286.46) - /unlink 42889 (time 1735100204.52 total 141.30 last 288.88) - /unlink 45889 (time 1735100214.52 total 151.30 last 299.98) - /unlink 48763 (time 1735100224.52 total 161.31 last 287.31) - /unlink 50000 (time 1735100228.53 total 165.31 last 308.74) - /unlink 53126 (time 1735100238.53 total 175.31 last 312.58) - /unlink 56034 (time 1735100248.53 total 185.31 last 290.80) - /unlink 58737 (time 1735100258.53 total 195.32 last 270.24) - /unlink 60000 (time 1735100263.34 total 200.12 last 262.89) - /unlink 62785 (time 1735100273.34 total 210.12 last 278.48) total: 64768 /unlink in 217.06 seconds: 298.39 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1802 (time 1735100306.80 total 10.00 last 180.17) - open/close 3883 (time 1735100316.80 total 20.01 last 208.02) - open/close 5960 (time 1735100326.80 total 30.01 last 207.61) - open/close 7991 (time 1735100336.81 total 40.01 last 203.06) - open/close 10000 (time 1735100344.47 total 47.68 last 261.99) - open/close 13084 (time 1735100354.48 total 57.68 last 308.38) - open/close 16663 (time 1735100364.48 total 67.68 last 357.86) - open/close 20000 (time 1735100373.97 total 77.18 last 351.41) - open/close 23503 (time 1735100383.97 total 87.18 last 350.28) - open/close 26669 (time 1735100393.98 total 97.18 last 316.48) - open/close 30000 (time 1735100403.57 total 106.77 last 347.32) - open/close 33642 (time 1735100413.57 total 116.78 last 364.10) - open/close 37614 (time 1735100423.57 total 126.78 last 397.19) - open/close 40000 (time 1735100430.23 total 133.43 last 358.52) - open/close 43994 (time 1735100440.23 total 143.43 last 399.37) - open/close 47095 (time 1735100450.23 total 153.43 last 310.09) - open/close 50000 (time 1735100458.83 total 162.03 last 337.77) - open/close 54131 (time 1735100468.83 total 172.04 last 412.94) - open/close 57563 (time 1735100478.83 total 182.04 last 343.14) - open/close 60000 (time 1735100485.19 total 188.40 last 383.26) - open/close 63973 (time 1735100495.19 total 198.40 last 397.19) total: 64768 open/close in 200.91 seconds: 322.37 ops/second - /unlink 7914 (time 1735100508.60 total 10.00 last 791.34) - /unlink 10000 (time 1735100511.11 total 12.50 last 833.11) - /unlink 18957 (time 1735100521.11 total 22.51 last 895.59) - /unlink 27294 (time 1735100531.11 total 32.51 last 833.67) - /unlink 30000 (time 1735100534.08 total 35.47 last 911.84) - /unlink 38339 (time 1735100544.08 total 45.47 last 833.84) - /unlink 40000 (time 1735100546.60 total 47.99 last 659.48) - /unlink 46849 (time 1735100556.60 total 57.99 last 684.88) - /unlink 50000 (time 1735100561.25 total 62.65 last 677.20) - /unlink 57186 (time 1735100571.25 total 72.65 last 718.52) - /unlink 60000 (time 1735100574.46 total 75.85 last 877.94) total: 64768 /unlink in 82.13 seconds: 788.59 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4317 (time 1735100611.37 total 10.00 last 431.65) - open/close 9130 (time 1735100621.38 total 20.00 last 481.16) - open/close 14377 (time 1735100631.38 total 30.00 last 524.68) - open/close 19174 (time 1735100641.38 total 40.01 last 479.66) - open/close 24379 (time 1735100651.38 total 50.01 last 520.47) - open/close 29207 (time 1735100661.38 total 60.01 last 482.72) - open/close 33567 (time 1735100671.38 total 70.01 last 435.99) - open/close 38301 (time 1735100681.38 total 80.01 last 473.38) - open/close 40000 (time 1735100684.33 total 82.95 last 576.67) - open/close 45182 (time 1735100694.33 total 92.95 last 518.18) - open/close 50000 (time 1735100704.24 total 102.87 last 485.81) - open/close 54846 (time 1735100714.24 total 112.87 last 484.57) - open/close 58935 (time 1735100724.24 total 122.87 last 408.89) - open/close 60000 (time 1735100726.49 total 125.12 last 474.31) total: 64768 open/close in 134.49 seconds: 481.58 ops/second - /unlink 8126 (time 1735100746.71 total 10.00 last 812.53) - /unlink 10000 (time 1735100748.87 total 12.16 last 868.00) - /unlink 18178 (time 1735100758.87 total 22.16 last 817.78) - /unlink 20000 (time 1735100761.22 total 24.52 last 772.26) - /unlink 27623 (time 1735100771.23 total 34.52 last 762.28) - /unlink 30000 (time 1735100773.88 total 37.18 last 895.04) - /unlink 38649 (time 1735100783.88 total 47.18 last 864.87) - /unlink 46682 (time 1735100793.88 total 57.18 last 803.29) - /unlink 50000 (time 1735100797.56 total 60.86 last 901.07) - /unlink 58872 (time 1735100807.56 total 70.86 last 887.15) total: 64768 /unlink in 77.01 seconds: 840.98 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2