== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:54:34 (1775166874) Stopping clients: oleg212-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg212-client.virtnet /mnt/lustre2 (opts:-f) oleg212-server: oleg212-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg212-server' oleg212-server: oleg212-server.virtnet: executing load_modules_local oleg212-server: Loading modules from /home/green/git/lustre-release/lustre oleg212-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg212-server: detected 4 online CPUs by sysfs oleg212-server: MODOPTS_LIBCFS= oleg212-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg212-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg212-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1039 (time 1775166949.49 total 10.01 last 103.81) - open/close 2246 (time 1775166959.49 total 20.01 last 120.67) - open/close 3504 (time 1775166969.49 total 30.01 last 125.80) - open/close 4684 (time 1775166979.49 total 40.01 last 118.00) - open/close 5641 (time 1775166989.50 total 50.02 last 95.65) - open/close 6686 (time 1775166999.50 total 60.02 last 104.45) - open/close 7725 (time 1775167009.51 total 70.03 last 103.84) - open/close 8789 (time 1775167019.51 total 80.03 last 106.35) - open/close 9793 (time 1775167029.52 total 90.04 last 100.35) - open/close 10000 (time 1775167031.84 total 92.36 last 89.18) - open/close 10928 (time 1775167041.86 total 102.38 last 92.62) - open/close 11733 (time 1775167051.87 total 112.39 last 80.41) - open/close 12666 (time 1775167061.88 total 122.40 last 93.22) - open/close 13637 (time 1775167071.88 total 132.40 last 97.08) - open/close 14637 (time 1775167081.88 total 142.40 last 99.95) - open/close 15510 (time 1775167091.89 total 152.41 last 87.24) - open/close 16525 (time 1775167101.90 total 162.42 last 101.45) - open/close 17621 (time 1775167111.90 total 172.42 last 109.52) - open/close 18800 (time 1775167121.91 total 182.43 last 117.84) - open/close 19828 (time 1775167131.91 total 192.43 last 102.78) - open/close 20766 (time 1775167141.91 total 202.43 last 93.77) - open/close 21821 (time 1775167151.92 total 212.44 last 105.48) - open/close 22947 (time 1775167161.92 total 222.44 last 112.59) - open/close 24461 (time 1775167171.92 total 232.44 last 151.39) - open/close 26491 (time 1775167181.92 total 242.44 last 202.99) - open/close 28209 (time 1775167191.92 total 252.44 last 171.74) - open/close 29861 (time 1775167201.92 total 262.44 last 165.15) - open/close 32177 (time 1775167211.93 total 272.45 last 231.55) - open/close 35557 (time 1775167221.93 total 282.45 last 337.97) - open/close 39245 (time 1775167231.93 total 292.45 last 368.73) - open/close 40000 (time 1775167234.21 total 294.73 last 330.45) - open/close 43082 (time 1775167244.22 total 304.74 last 308.00) - open/close 45573 (time 1775167254.22 total 314.74 last 249.04) - open/close 48139 (time 1775167264.23 total 324.74 last 256.55) - open/close 50000 (time 1775167272.29 total 332.81 last 230.80) - open/close 52472 (time 1775167282.29 total 342.81 last 247.18) - open/close 54629 (time 1775167292.29 total 352.81 last 215.60) - open/close 57230 (time 1775167302.30 total 362.82 last 259.93) - open/close 59959 (time 1775167312.30 total 372.82 last 272.89) - open/close 62585 (time 1775167322.30 total 382.82 last 262.56) - open/close 64712 (time 1775167332.31 total 392.82 last 212.65) total: 64768 open/close in 393.11 seconds: 164.76 ops/second - /unlink 3842 (time 1775167344.37 total 10.00 last 384.06) - /unlink 7487 (time 1775167354.37 total 20.01 last 364.38) - /unlink 10000 (time 1775167361.64 total 27.28 last 345.66) - /unlink 13429 (time 1775167371.65 total 37.28 last 342.81) - /unlink 17443 (time 1775167381.65 total 47.28 last 401.33) - /unlink 20000 (time 1775167387.17 total 52.80 last 463.33) - /unlink 24746 (time 1775167397.17 total 62.80 last 474.49) - /unlink 29338 (time 1775167407.17 total 72.80 last 459.16) - /unlink 34299 (time 1775167417.17 total 82.80 last 496.07) - /unlink 38598 (time 1775167427.17 total 92.81 last 429.84) - /unlink 40000 (time 1775167429.53 total 95.16 last 595.00) - /unlink 45024 (time 1775167439.53 total 105.16 last 502.40) - /unlink 50000 (time 1775167449.23 total 114.86 last 512.99) - /unlink 55494 (time 1775167459.23 total 124.86 last 549.36) - /unlink 60000 (time 1775167466.85 total 132.49 last 590.91) total: 64768 /unlink in 140.78 seconds: 460.07 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2890 (time 1775167507.52 total 10.01 last 288.85) - open/close 6812 (time 1775167517.52 total 20.01 last 392.17) - open/close 10000 (time 1775167526.24 total 28.73 last 365.44) - open/close 13938 (time 1775167536.24 total 38.73 last 393.74) - open/close 17509 (time 1775167546.25 total 48.73 last 357.03) - open/close 20000 (time 1775167552.57 total 55.05 last 394.06) - open/close 24028 (time 1775167562.57 total 65.06 last 402.74) - open/close 27180 (time 1775167572.57 total 75.06 last 315.15) - open/close 30000 (time 1775167581.16 total 83.64 last 328.53) - open/close 33548 (time 1775167591.16 total 93.64 last 354.71) - open/close 37011 (time 1775167601.16 total 103.65 last 346.21) - open/close 40000 (time 1775167608.36 total 110.85 last 414.98) - open/close 44106 (time 1775167618.42 total 120.90 last 408.41) - open/close 47820 (time 1775167628.42 total 130.90 last 371.39) - open/close 50000 (time 1775167635.02 total 137.50 last 330.29) - open/close 53663 (time 1775167645.02 total 147.50 last 366.29) - open/close 57658 (time 1775167655.02 total 157.50 last 399.47) - open/close 60000 (time 1775167660.53 total 163.02 last 424.76) - open/close 64170 (time 1775167670.53 total 173.02 last 416.94) total: 64768 open/close in 174.49 seconds: 371.19 ops/second - /unlink 6255 (time 1775167683.40 total 10.01 last 625.16) - /unlink 10000 (time 1775167689.90 total 16.51 last 576.13) - /unlink 16409 (time 1775167699.90 total 26.51 last 640.89) - /unlink 20000 (time 1775167704.69 total 31.30 last 749.36) - /unlink 26587 (time 1775167714.69 total 41.30 last 658.67) - /unlink 30000 (time 1775167719.43 total 46.03 last 720.58) - /unlink 37384 (time 1775167729.43 total 56.04 last 738.32) - /unlink 40000 (time 1775167733.15 total 59.76 last 702.67) - /unlink 46727 (time 1775167743.15 total 69.76 last 672.64) - /unlink 50000 (time 1775167748.61 total 75.22 last 599.65) - /unlink 57464 (time 1775167758.61 total 85.22 last 746.31) - /unlink 60000 (time 1775167761.85 total 88.46 last 782.61) total: 64768 /unlink in 94.94 seconds: 682.22 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4943 (time 1775167795.70 total 10.00 last 494.24) - open/close 9585 (time 1775167805.70 total 20.00 last 464.13) - open/close 14175 (time 1775167815.70 total 30.00 last 458.93) - open/close 19190 (time 1775167825.70 total 40.00 last 501.50) - open/close 24499 (time 1775167835.70 total 50.01 last 530.82) - open/close 29457 (time 1775167845.70 total 60.01 last 495.74) - open/close 33832 (time 1775167855.70 total 70.01 last 437.48) - open/close 38302 (time 1775167865.71 total 80.01 last 446.95) - open/close 40000 (time 1775167869.42 total 83.72 last 457.47) - open/close 44760 (time 1775167879.42 total 93.72 last 475.99) - open/close 49887 (time 1775167889.42 total 103.72 last 512.66) - open/close 54784 (time 1775167899.42 total 113.72 last 489.56) - open/close 59603 (time 1775167909.42 total 123.73 last 481.81) - open/close 64518 (time 1775167919.42 total 133.73 last 491.48) total: 64768 open/close in 134.40 seconds: 481.90 ops/second - /unlink 7984 (time 1775167931.42 total 10.00 last 798.34) - /unlink 10000 (time 1775167934.23 total 12.81 last 717.32) - /unlink 17491 (time 1775167944.23 total 22.81 last 749.03) - /unlink 20000 (time 1775167947.46 total 26.04 last 777.29) - /unlink 28037 (time 1775167957.46 total 36.04 last 803.65) - /unlink 30000 (time 1775167959.95 total 38.53 last 789.13) - /unlink 37761 (time 1775167969.95 total 48.53 last 776.08) - /unlink 40000 (time 1775167973.02 total 51.60 last 728.46) - /unlink 48081 (time 1775167983.02 total 61.60 last 808.09) - /unlink 50000 (time 1775167985.20 total 63.78 last 882.56) - /unlink 58544 (time 1775167995.20 total 73.78 last 854.32) total: 64768 /unlink in 82.15 seconds: 788.45 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5231 (time 1775168034.78 total 10.00 last 523.06) - open/close 10000 (time 1775168043.50 total 18.72 last 546.72) - open/close 15532 (time 1775168053.50 total 28.72 last 553.14) - open/close 20000 (time 1775168062.36 total 37.58 last 504.29) - open/close 24656 (time 1775168072.36 total 47.59 last 465.58) - open/close 29520 (time 1775168082.36 total 57.59 last 486.38) - open/close 34617 (time 1775168092.36 total 67.59 last 509.68) - open/close 39182 (time 1775168102.36 total 77.59 last 456.45) - open/close 41999 (time 1775168112.37 total 87.59 last 281.64) - open/close 43572 (time 1775168122.37 total 97.59 last 157.23) - open/close 45264 (time 1775168132.38 total 107.60 last 169.07) - open/close 46752 (time 1775168142.39 total 117.62 last 148.58) - open/close 47782 (time 1775168152.40 total 127.62 last 102.97) - open/close 49446 (time 1775168162.40 total 137.62 last 166.35) - open/close 50000 (time 1775168165.25 total 140.48 last 194.12) - open/close 52015 (time 1775168175.25 total 150.48 last 201.50) - open/close 54315 (time 1775168185.25 total 160.48 last 229.98) - open/close 56416 (time 1775168195.26 total 170.49 last 209.88) - open/close 58363 (time 1775168205.27 total 180.49 last 194.63) - open/close 60000 (time 1775168212.45 total 187.68 last 227.78) - open/close 62499 (time 1775168222.46 total 197.68 last 249.83) total: 64768 open/close in 206.60 seconds: 313.50 ops/second - /unlink 6192 (time 1775168242.99 total 10.00 last 619.17) - /unlink 10000 (time 1775168250.63 total 17.64 last 498.41) - /unlink 14326 (time 1775168260.64 total 27.65 last 432.32) - /unlink 17885 (time 1775168270.64 total 37.65 last 355.76) - /unlink 20000 (time 1775168276.52 total 43.53 last 359.60) - /unlink 23929 (time 1775168286.52 total 53.53 last 392.89) - /unlink 27540 (time 1775168296.52 total 63.53 last 361.06) - /unlink 30000 (time 1775168304.65 total 71.66 last 302.56) - /unlink 32628 (time 1775168314.65 total 81.67 last 262.75) - /unlink 34956 (time 1775168324.66 total 91.67 last 232.77) - /unlink 37824 (time 1775168334.66 total 101.67 last 286.79) - /unlink 40000 (time 1775168342.74 total 109.75 last 269.25) - /unlink 42808 (time 1775168352.74 total 119.75 last 280.76) - /unlink 45629 (time 1775168362.74 total 129.75 last 282.07) - /unlink 48743 (time 1775168372.74 total 139.76 last 311.30) - /unlink 50000 (time 1775168376.45 total 143.46 last 339.07) - /unlink 54146 (time 1775168386.45 total 153.46 last 414.55) - /unlink 57573 (time 1775168396.45 total 163.46 last 342.67) - /unlink 60000 (time 1775168403.41 total 170.42 last 348.75) - /unlink 63688 (time 1775168413.41 total 180.43 last 368.70) total: 64768 /unlink in 184.17 seconds: 351.68 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1949 (time 1775168448.62 total 10.00 last 194.88) - open/close 3854 (time 1775168458.62 total 20.00 last 190.47) - open/close 5564 (time 1775168468.63 total 30.00 last 170.98) - open/close 7198 (time 1775168478.63 total 40.00 last 163.39) - open/close 8890 (time 1775168488.63 total 50.01 last 169.16) - open/close 10000 (time 1775168494.79 total 56.17 last 180.21) - open/close 11597 (time 1775168504.79 total 66.17 last 159.61) - open/close 13261 (time 1775168514.79 total 76.17 last 166.39) - open/close 14990 (time 1775168524.80 total 86.18 last 172.81) - open/close 16936 (time 1775168534.80 total 96.18 last 194.59) - open/close 19281 (time 1775168544.81 total 106.18 last 234.38) - open/close 20000 (time 1775168548.00 total 109.38 last 225.28) - open/close 22206 (time 1775168558.00 total 119.38 last 220.59) - open/close 25084 (time 1775168568.00 total 129.38 last 287.79) - open/close 27691 (time 1775168578.00 total 139.38 last 260.61) - open/close 29836 (time 1775168588.00 total 149.38 last 214.47) - open/close 31876 (time 1775168598.01 total 159.39 last 203.92) - open/close 34051 (time 1775168608.01 total 169.39 last 217.44) - open/close 37225 (time 1775168618.01 total 179.39 last 317.27) - open/close 39676 (time 1775168628.01 total 189.39 last 245.08) - open/close 42756 (time 1775168638.02 total 199.39 last 307.95) - open/close 45542 (time 1775168648.02 total 209.39 last 278.58) - open/close 48240 (time 1775168658.02 total 219.40 last 269.75) - open/close 50000 (time 1775168664.79 total 226.17 last 259.73) - open/close 52284 (time 1775168674.80 total 236.17 last 228.37) - open/close 54742 (time 1775168684.80 total 246.18 last 245.74) - open/close 57096 (time 1775168694.80 total 256.18 last 235.34) - open/close 59101 (time 1775168704.81 total 266.18 last 200.39) - open/close 60000 (time 1775168708.27 total 269.65 last 259.31) - open/close 62868 (time 1775168718.28 total 279.66 last 286.68) total: 64768 open/close in 287.34 seconds: 225.40 ops/second - /unlink 4962 (time 1775168737.67 total 10.00 last 496.11) - /unlink 10000 (time 1775168747.27 total 19.60 last 524.73) - /unlink 14932 (time 1775168757.27 total 29.61 last 493.07) - /unlink 19404 (time 1775168767.27 total 39.61 last 447.13) - /unlink 23378 (time 1775168777.28 total 49.61 last 397.36) - /unlink 27361 (time 1775168787.28 total 59.61 last 398.29) - /unlink 30000 (time 1775168793.77 total 66.11 last 406.18) - /unlink 34746 (time 1775168803.77 total 76.11 last 474.56) - /unlink 38456 (time 1775168813.78 total 86.11 last 370.94) - /unlink 40000 (time 1775168818.11 total 90.44 last 356.62) - /unlink 43438 (time 1775168828.11 total 100.44 last 343.71) - /unlink 46936 (time 1775168838.11 total 110.44 last 349.73) - /unlink 50000 (time 1775168847.20 total 119.53 last 337.02) - /unlink 53618 (time 1775168857.20 total 129.53 last 361.76) - /unlink 56908 (time 1775168867.20 total 139.54 last 328.96) - /unlink 59801 (time 1775168877.20 total 149.54 last 289.29) - /unlink 62811 (time 1775168887.20 total 159.54 last 300.97) total: 64768 /unlink in 166.29 seconds: 389.48 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg212-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg212-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg212-server stop mds service on oleg212-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg212-server unloading modules via unload_modules_local on: 'oleg212-server' oleg212-server: oleg212-server.virtnet: executing unload_modules_local oleg212-server: modules unloaded. pdsh@oleg212-client: oleg212-client: ssh exited with exit code 2 pdsh@oleg212-client: oleg212-server: ssh exited with exit code 2 pdsh@oleg212-client: oleg212-client: ssh exited with exit code 2 pdsh@oleg212-client: oleg212-server: ssh exited with exit code 2