== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:32:09 (1771439529) Stopping clients: oleg440-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg440-client.virtnet /mnt/lustre2 (opts:-f) oleg440-server: oleg440-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg440-server: detected 4 online CPUs by sysfs oleg440-server: MODOPTS_LIBCFS= oleg440-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg440-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg440-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2021 (time 1771439588.29 total 10.00 last 202.07) - open/close 4032 (time 1771439598.29 total 20.00 last 201.09) - open/close 6192 (time 1771439608.29 total 30.00 last 215.96) - open/close 7926 (time 1771439618.29 total 40.01 last 173.31) - open/close 10000 (time 1771439628.14 total 49.86 last 210.63) - open/close 11625 (time 1771439638.14 total 59.86 last 162.45) - open/close 13041 (time 1771439648.15 total 69.86 last 141.53) - open/close 14423 (time 1771439658.15 total 79.86 last 138.19) - open/close 16057 (time 1771439668.15 total 89.87 last 163.36) - open/close 17472 (time 1771439678.15 total 99.87 last 141.47) - open/close 19557 (time 1771439688.16 total 109.87 last 208.45) - open/close 22061 (time 1771439698.16 total 119.87 last 250.35) - open/close 24419 (time 1771439708.16 total 129.88 last 235.72) - open/close 26574 (time 1771439718.16 total 139.88 last 215.49) - open/close 28600 (time 1771439728.16 total 149.88 last 202.58) - open/close 30000 (time 1771439736.56 total 158.27 last 166.74) - open/close 31518 (time 1771439746.57 total 168.28 last 151.68) - open/close 33324 (time 1771439756.57 total 178.29 last 180.54) - open/close 35462 (time 1771439766.57 total 188.29 last 213.78) - open/close 37280 (time 1771439776.58 total 198.29 last 181.67) - open/close 39479 (time 1771439786.58 total 208.30 last 219.87) - open/close 41965 (time 1771439796.58 total 218.30 last 248.53) - open/close 44562 (time 1771439806.58 total 228.30 last 259.69) - open/close 47643 (time 1771439816.58 total 238.30 last 308.06) - open/close 50000 (time 1771439825.74 total 247.46 last 257.35) - open/close 53278 (time 1771439835.74 total 257.46 last 327.72) - open/close 56369 (time 1771439845.75 total 267.46 last 309.02) - open/close 59507 (time 1771439855.75 total 277.47 last 313.74) - open/close 62969 (time 1771439865.75 total 287.47 last 346.04) total: 64768 open/close in 292.97 seconds: 221.08 ops/second - /unlink 5452 (time 1771439882.90 total 10.00 last 545.16) - /unlink 10000 (time 1771439891.18 total 18.28 last 549.54) - /unlink 13971 (time 1771439901.18 total 28.28 last 397.06) - /unlink 18937 (time 1771439911.18 total 38.28 last 496.59) - /unlink 20000 (time 1771439913.77 total 40.87 last 409.55) - /unlink 25014 (time 1771439923.77 total 50.87 last 501.33) - /unlink 29931 (time 1771439933.78 total 60.88 last 491.64) - /unlink 35989 (time 1771439943.78 total 70.88 last 605.79) - /unlink 40000 (time 1771439950.14 total 77.24 last 630.48) - /unlink 44857 (time 1771439960.14 total 87.24 last 485.66) - /unlink 49574 (time 1771439970.14 total 97.24 last 471.66) - /unlink 55389 (time 1771439980.14 total 107.24 last 581.50) - /unlink 60000 (time 1771439987.46 total 114.56 last 629.73) total: 64768 /unlink in 122.47 seconds: 528.83 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3469 (time 1771440024.85 total 10.00 last 346.86) - open/close 7097 (time 1771440034.86 total 20.00 last 362.69) - open/close 10000 (time 1771440043.52 total 28.67 last 335.07) - open/close 14055 (time 1771440053.52 total 38.67 last 405.41) - open/close 17882 (time 1771440063.52 total 48.67 last 382.63) - open/close 20000 (time 1771440069.70 total 54.85 last 342.75) - open/close 23803 (time 1771440079.70 total 64.85 last 380.28) - open/close 27113 (time 1771440089.71 total 74.85 last 330.96) - open/close 30000 (time 1771440098.62 total 83.76 last 324.01) - open/close 33938 (time 1771440108.62 total 93.76 last 393.78) - open/close 38056 (time 1771440118.62 total 103.77 last 411.76) - open/close 40000 (time 1771440123.09 total 108.23 last 435.02) - open/close 44103 (time 1771440133.09 total 118.24 last 410.22) - open/close 48659 (time 1771440143.09 total 128.24 last 455.50) - open/close 50000 (time 1771440145.96 total 131.11 last 466.98) - open/close 54720 (time 1771440155.96 total 141.11 last 471.98) - open/close 59226 (time 1771440165.96 total 151.11 last 450.54) - open/close 63868 (time 1771440175.96 total 161.11 last 464.17) total: 64768 open/close in 163.26 seconds: 396.72 ops/second - /unlink 7842 (time 1771440189.41 total 10.00 last 784.10) - /unlink 10000 (time 1771440191.99 total 12.58 last 836.34) - /unlink 17450 (time 1771440201.99 total 22.58 last 744.95) - /unlink 20000 (time 1771440205.72 total 26.31 last 683.94) - /unlink 27255 (time 1771440215.72 total 36.31 last 725.40) - /unlink 30000 (time 1771440219.73 total 40.33 last 683.99) - /unlink 37468 (time 1771440229.73 total 50.33 last 746.77) - /unlink 40000 (time 1771440232.69 total 53.28 last 856.08) - /unlink 47702 (time 1771440242.69 total 63.28 last 770.20) - /unlink 50000 (time 1771440246.05 total 66.64 last 683.75) - /unlink 57578 (time 1771440256.06 total 76.65 last 757.37) - /unlink 60000 (time 1771440259.01 total 79.60 last 821.61) total: 64768 /unlink in 85.86 seconds: 754.34 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4632 (time 1771440298.13 total 10.00 last 463.18) - open/close 9547 (time 1771440308.13 total 20.00 last 491.47) - open/close 14223 (time 1771440318.13 total 30.00 last 467.52) - open/close 18817 (time 1771440328.13 total 40.00 last 459.35) - open/close 20000 (time 1771440330.76 total 42.63 last 450.17) - open/close 24362 (time 1771440340.76 total 52.63 last 436.20) - open/close 29507 (time 1771440350.76 total 62.63 last 514.45) - open/close 34877 (time 1771440360.76 total 72.63 last 536.94) - open/close 39939 (time 1771440370.76 total 82.64 last 506.12) - open/close 45011 (time 1771440380.89 total 92.76 last 500.93) - open/close 49703 (time 1771440390.89 total 102.76 last 469.18) - open/close 54839 (time 1771440400.89 total 112.76 last 513.58) - open/close 59887 (time 1771440410.89 total 122.76 last 504.75) total: 64768 open/close in 132.06 seconds: 490.45 ops/second - /unlink 8107 (time 1771440431.49 total 10.00 last 810.67) - /unlink 10000 (time 1771440434.03 total 12.54 last 744.82) - /unlink 18454 (time 1771440444.03 total 22.54 last 845.36) - /unlink 26624 (time 1771440454.03 total 32.54 last 816.94) - /unlink 30000 (time 1771440458.35 total 36.86 last 781.32) - /unlink 37533 (time 1771440468.35 total 46.86 last 753.29) - /unlink 40000 (time 1771440471.68 total 50.19 last 740.81) - /unlink 47277 (time 1771440481.68 total 60.19 last 727.66) - /unlink 50000 (time 1771440485.48 total 63.99 last 717.28) - /unlink 58103 (time 1771440495.48 total 73.99 last 810.21) - /unlink 60000 (time 1771440497.62 total 76.13 last 885.34) total: 64768 /unlink in 81.69 seconds: 792.89 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1605 (time 1771440538.25 total 10.00 last 160.49) - open/close 3038 (time 1771440548.25 total 20.01 last 143.21) - open/close 4632 (time 1771440558.26 total 30.01 last 159.32) - open/close 5688 (time 1771440568.27 total 40.02 last 105.53) - open/close 6570 (time 1771440578.27 total 50.02 last 88.20) - open/close 7739 (time 1771440588.27 total 60.02 last 116.82) - open/close 8913 (time 1771440598.27 total 70.03 last 117.39) - open/close 10000 (time 1771440605.73 total 77.48 last 145.75) - open/close 11635 (time 1771440615.74 total 87.49 last 163.38) - open/close 13017 (time 1771440625.74 total 97.49 last 138.16) - open/close 14248 (time 1771440635.75 total 107.50 last 123.05) - open/close 15442 (time 1771440645.75 total 117.50 last 119.36) - open/close 16802 (time 1771440655.75 total 127.51 last 135.93) - open/close 18305 (time 1771440665.75 total 137.51 last 150.29) - open/close 20000 (time 1771440674.64 total 146.39 last 190.78) - open/close 21780 (time 1771440684.64 total 156.39 last 177.96) - open/close 23224 (time 1771440694.65 total 166.40 last 144.32) - open/close 25167 (time 1771440704.65 total 176.40 last 194.25) - open/close 26381 (time 1771440714.66 total 186.41 last 121.27) - open/close 27512 (time 1771440724.66 total 196.41 last 113.09) - open/close 28820 (time 1771440734.66 total 206.42 last 130.75) - open/close 29979 (time 1771440744.67 total 216.42 last 115.82) - open/close 31323 (time 1771440754.68 total 226.43 last 134.31) - open/close 32604 (time 1771440764.69 total 236.44 last 127.98) - open/close 33886 (time 1771440774.69 total 246.44 last 128.17) - open/close 35094 (time 1771440784.69 total 256.44 last 120.77) - open/close 36232 (time 1771440794.70 total 266.46 last 113.67) - open/close 37474 (time 1771440804.70 total 276.46 last 124.19) - open/close 38914 (time 1771440814.71 total 286.46 last 143.95) - open/close 40000 (time 1771440822.37 total 294.12 last 141.74) - open/close 41505 (time 1771440832.37 total 304.12 last 150.49) - open/close 43287 (time 1771440842.38 total 314.13 last 178.09) - open/close 44683 (time 1771440852.38 total 324.13 last 139.57) - open/close 46128 (time 1771440862.38 total 334.13 last 144.49) - open/close 47670 (time 1771440872.38 total 344.13 last 154.20) - open/close 49184 (time 1771440882.39 total 354.14 last 151.30) - open/close 50000 (time 1771440887.52 total 359.27 last 159.00) - open/close 51375 (time 1771440897.52 total 369.28 last 137.42) - open/close 52710 (time 1771440907.53 total 379.28 last 133.42) - open/close 53954 (time 1771440917.53 total 389.28 last 124.38) - open/close 55229 (time 1771440927.53 total 399.29 last 127.49) - open/close 56546 (time 1771440937.54 total 409.29 last 131.65) - open/close 57726 (time 1771440947.54 total 419.29 last 117.95) - open/close 59059 (time 1771440957.55 total 429.30 last 133.22) - open/close 60000 (time 1771440963.89 total 435.64 last 148.31) - open/close 61595 (time 1771440973.89 total 445.64 last 159.49) - open/close 63336 (time 1771440983.89 total 455.64 last 174.10) total: 64768 open/close in 463.58 seconds: 139.71 ops/second - /unlink 3605 (time 1771441003.45 total 10.00 last 360.46) - /unlink 6888 (time 1771441013.45 total 20.00 last 328.22) - /unlink 9696 (time 1771441023.46 total 30.01 last 280.68) - /unlink 12518 (time 1771441033.46 total 40.01 last 282.19) - /unlink 15451 (time 1771441043.46 total 50.01 last 293.17) - /unlink 18402 (time 1771441053.46 total 60.01 last 295.07) - /unlink 20000 (time 1771441058.31 total 64.86 last 329.72) - /unlink 23142 (time 1771441068.31 total 74.86 last 314.19) - /unlink 25853 (time 1771441078.31 total 84.86 last 271.02) - /unlink 28877 (time 1771441088.32 total 94.87 last 302.27) - /unlink 30000 (time 1771441091.51 total 98.07 last 351.09) - /unlink 33497 (time 1771441101.52 total 108.07 last 349.60) - /unlink 36665 (time 1771441111.52 total 118.07 last 316.76) - /unlink 39504 (time 1771441121.52 total 128.07 last 283.86) - /unlink 40000 (time 1771441123.71 total 130.27 last 225.97) - /unlink 42745 (time 1771441133.72 total 140.27 last 274.47) - /unlink 45655 (time 1771441143.72 total 150.27 last 290.97) - /unlink 49077 (time 1771441153.72 total 160.27 last 342.11) - /unlink 50000 (time 1771441156.22 total 162.77 last 368.83) - /unlink 53897 (time 1771441166.22 total 172.77 last 389.69) - /unlink 57203 (time 1771441176.22 total 182.78 last 330.54) - /unlink 60000 (time 1771441184.78 total 191.33 last 326.89) - /unlink 63354 (time 1771441194.78 total 201.33 last 335.39) total: 64768 /unlink in 206.16 seconds: 314.17 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1795 (time 1771441227.42 total 10.00 last 179.42) - open/close 3413 (time 1771441237.42 total 20.01 last 161.74) - open/close 5294 (time 1771441247.42 total 30.01 last 188.09) - open/close 7768 (time 1771441257.42 total 40.01 last 247.37) - open/close 10000 (time 1771441265.74 total 48.32 last 268.47) - open/close 12689 (time 1771441275.74 total 58.33 last 268.84) - open/close 15025 (time 1771441285.74 total 68.33 last 233.59) - open/close 17295 (time 1771441295.74 total 78.33 last 226.99) - open/close 19464 (time 1771441305.74 total 88.33 last 216.87) - open/close 22217 (time 1771441315.74 total 98.33 last 275.22) - open/close 24703 (time 1771441325.75 total 108.33 last 248.54) - open/close 27293 (time 1771441335.75 total 118.34 last 258.93) - open/close 29546 (time 1771441345.75 total 128.34 last 225.26) - open/close 30000 (time 1771441347.91 total 130.50 last 209.90) - open/close 32362 (time 1771441357.92 total 140.51 last 236.10) - open/close 34508 (time 1771441367.92 total 150.51 last 214.60) - open/close 36905 (time 1771441377.92 total 160.51 last 239.64) - open/close 39300 (time 1771441387.92 total 170.51 last 239.43) - open/close 40000 (time 1771441390.24 total 172.83 last 301.80) - open/close 43111 (time 1771441400.24 total 182.83 last 311.08) - open/close 45525 (time 1771441410.25 total 192.83 last 241.34) - open/close 48106 (time 1771441420.25 total 202.84 last 257.96) - open/close 50000 (time 1771441428.03 total 210.62 last 243.49) - open/close 52923 (time 1771441438.03 total 220.62 last 292.29) - open/close 56016 (time 1771441448.03 total 230.62 last 309.29) - open/close 58739 (time 1771441458.03 total 240.62 last 272.29) - open/close 60000 (time 1771441462.89 total 245.48 last 259.46) - open/close 62943 (time 1771441472.90 total 255.48 last 294.19) total: 64768 open/close in 261.99 seconds: 247.21 ops/second - /unlink 5627 (time 1771441490.94 total 10.00 last 562.66) - /unlink 10000 (time 1771441499.10 total 18.16 last 536.19) - /unlink 14980 (time 1771441509.10 total 28.16 last 497.98) - /unlink 20000 (time 1771441518.87 total 37.93 last 513.87) - /unlink 24861 (time 1771441528.87 total 47.93 last 486.06) - /unlink 30000 (time 1771441538.36 total 57.42 last 541.13) - /unlink 35121 (time 1771441548.36 total 67.42 last 512.10) - /unlink 40000 (time 1771441556.89 total 75.95 last 572.42) - /unlink 46216 (time 1771441566.89 total 85.95 last 621.58) - /unlink 50000 (time 1771441572.33 total 91.39 last 695.50) - /unlink 56497 (time 1771441582.33 total 101.39 last 649.64) - /unlink 60000 (time 1771441587.91 total 106.97 last 627.19) total: 64768 /unlink in 114.76 seconds: 564.37 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server unloading modules via unload_modules_local on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local oleg440-server: modules unloaded. pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2