== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:41:46 (1772678506) Stopping clients: oleg330-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg330-client.virtnet /mnt/lustre2 (opts:-f) oleg330-server: oleg330-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing load_modules_local oleg330-server: Loading modules from /home/green/git/lustre-release/lustre oleg330-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg330-server: detected 4 online CPUs by sysfs oleg330-server: MODOPTS_LIBCFS= oleg330-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg330-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg330-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg330-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg330-server: oleg330-server.virtnet: executing set_default_debug -1 all pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg330-client.virtnet: -o user_xattr,flock 192.168.203.130@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3172 (time 1772678563.95 total 10.00 last 317.09) - open/close 6271 (time 1772678573.95 total 20.01 last 309.83) - open/close 8695 (time 1772678583.96 total 30.01 last 242.31) - open/close 10000 (time 1772678590.32 total 36.38 last 205.00) - open/close 11861 (time 1772678600.33 total 46.38 last 185.97) - open/close 13901 (time 1772678610.33 total 56.38 last 203.96) - open/close 16061 (time 1772678620.33 total 66.39 last 215.97) - open/close 18154 (time 1772678630.34 total 76.39 last 209.23) - open/close 20000 (time 1772678639.12 total 85.18 last 210.08) - open/close 22856 (time 1772678649.12 total 95.18 last 285.59) - open/close 25689 (time 1772678659.12 total 105.18 last 283.29) - open/close 28668 (time 1772678669.13 total 115.18 last 297.82) - open/close 30000 (time 1772678673.05 total 119.11 last 339.22) - open/close 33040 (time 1772678683.05 total 129.11 last 304.00) - open/close 35868 (time 1772678693.06 total 139.11 last 282.73) - open/close 37836 (time 1772678703.06 total 149.11 last 196.75) - open/close 40000 (time 1772678712.90 total 158.95 last 219.96) - open/close 42099 (time 1772678722.91 total 168.96 last 209.74) - open/close 44763 (time 1772678732.91 total 178.96 last 266.39) - open/close 46756 (time 1772678742.91 total 188.96 last 199.21) - open/close 48988 (time 1772678752.92 total 198.97 last 223.09) - open/close 50000 (time 1772678757.25 total 203.30 last 233.70) - open/close 52362 (time 1772678767.25 total 213.30 last 236.11) - open/close 55204 (time 1772678777.25 total 223.30 last 284.17) - open/close 57710 (time 1772678787.25 total 233.30 last 250.60) - open/close 59894 (time 1772678797.25 total 243.31 last 218.34) - open/close 61922 (time 1772678807.26 total 253.31 last 202.75) - open/close 64013 (time 1772678817.26 total 263.31 last 209.07) total: 64768 open/close in 266.26 seconds: 243.25 ops/second - /unlink 6046 (time 1772678831.84 total 10.00 last 604.53) - /unlink 10000 (time 1772678838.79 total 16.95 last 569.05) - /unlink 15017 (time 1772678848.79 total 26.95 last 501.54) - /unlink 20000 (time 1772678858.33 total 36.49 last 522.67) - /unlink 25026 (time 1772678868.33 total 46.49 last 502.52) - /unlink 30000 (time 1772678877.34 total 55.50 last 552.02) - /unlink 37295 (time 1772678887.34 total 65.50 last 729.48) - /unlink 40000 (time 1772678891.56 total 69.72 last 641.01) - /unlink 45746 (time 1772678901.56 total 79.72 last 574.53) - /unlink 50000 (time 1772678908.89 total 87.05 last 580.32) - /unlink 54749 (time 1772678918.89 total 97.05 last 474.90) - /unlink 58603 (time 1772678928.89 total 107.05 last 385.37) - /unlink 60000 (time 1772678931.25 total 109.41 last 593.43) total: 64768 /unlink in 118.77 seconds: 545.31 ops/second Waiting for MDT destroys to complete oleg330-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 - open/close 2447 (time 1772678964.08 total 10.00 last 244.69) - open/close 5914 (time 1772678974.08 total 20.00 last 346.66) - open/close 8446 (time 1772678984.08 total 30.00 last 253.19) - open/close 10000 (time 1772678989.23 total 35.15 last 301.84) - open/close 13034 (time 1772678999.23 total 45.15 last 303.32) - open/close 15967 (time 1772679009.24 total 55.16 last 293.21) - open/close 19406 (time 1772679019.24 total 65.16 last 343.75) - open/close 20000 (time 1772679021.73 total 67.65 last 238.62) - open/close 22379 (time 1772679031.74 total 77.65 last 237.77) - open/close 24948 (time 1772679041.74 total 87.66 last 256.90) - open/close 27675 (time 1772679051.74 total 97.66 last 272.69) - open/close 30000 (time 1772679060.60 total 106.52 last 262.20) - open/close 32400 (time 1772679070.60 total 116.52 last 240.00) - open/close 34944 (time 1772679080.61 total 126.53 last 254.31) - open/close 38127 (time 1772679090.61 total 136.53 last 318.22) - open/close 40000 (time 1772679098.36 total 144.28 last 241.56) - open/close 42547 (time 1772679108.36 total 154.28 last 254.69) - open/close 45595 (time 1772679118.37 total 164.28 last 304.78) - open/close 47638 (time 1772679128.37 total 174.29 last 204.17) - open/close 50000 (time 1772679138.01 total 183.93 last 244.94) - open/close 52834 (time 1772679148.02 total 193.93 last 283.39) - open/close 55196 (time 1772679158.02 total 203.94 last 236.13) - open/close 58620 (time 1772679168.02 total 213.94 last 342.37) - open/close 60000 (time 1772679173.05 total 218.97 last 274.35) - open/close 62206 (time 1772679183.06 total 228.98 last 220.30) - open/close 64232 (time 1772679193.07 total 238.98 last 202.53) total: 64768 open/close in 240.86 seconds: 268.90 ops/second - /unlink 5074 (time 1772679206.55 total 10.00 last 507.40) - /unlink 10000 (time 1772679214.47 total 17.92 last 621.80) - /unlink 16538 (time 1772679224.47 total 27.92 last 653.78) - /unlink 20000 (time 1772679230.26 total 33.71 last 597.96) - /unlink 25519 (time 1772679240.26 total 43.71 last 551.89) - /unlink 30000 (time 1772679249.39 total 52.84 last 490.93) - /unlink 37194 (time 1772679259.39 total 62.84 last 719.36) - /unlink 40000 (time 1772679265.10 total 68.55 last 491.43) - /unlink 43760 (time 1772679275.10 total 78.55 last 375.93) - /unlink 49476 (time 1772679285.10 total 88.55 last 571.56) - /unlink 55792 (time 1772679295.11 total 98.56 last 631.21) - /unlink 59569 (time 1772679305.11 total 108.56 last 377.63) total: 64768 /unlink in 117.89 seconds: 549.39 ops/second Waiting for MDT destroys to complete oleg330-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 - open/close 1745 (time 1772679333.27 total 10.00 last 174.45) - open/close 4360 (time 1772679343.27 total 20.00 last 261.46) - open/close 7652 (time 1772679353.27 total 30.01 last 329.14) - open/close 10000 (time 1772679361.60 total 38.33 last 282.18) - open/close 13062 (time 1772679371.60 total 48.33 last 306.20) - open/close 16088 (time 1772679381.60 total 58.33 last 302.58) - open/close 20000 (time 1772679391.55 total 68.29 last 392.86) - open/close 23641 (time 1772679401.56 total 78.29 last 364.03) - open/close 27337 (time 1772679411.56 total 88.29 last 369.55) - open/close 30000 (time 1772679417.01 total 93.74 last 488.55) - open/close 34639 (time 1772679427.01 total 103.74 last 463.89) - open/close 39068 (time 1772679437.01 total 113.74 last 442.84) - open/close 40000 (time 1772679439.38 total 116.12 last 392.45) - open/close 45001 (time 1772679449.39 total 126.12 last 500.02) - open/close 49715 (time 1772679459.39 total 136.12 last 471.40) - open/close 54227 (time 1772679469.39 total 146.12 last 451.12) - open/close 58983 (time 1772679479.39 total 156.12 last 475.59) - open/close 60000 (time 1772679481.69 total 158.42 last 442.58) - open/close 63786 (time 1772679491.69 total 168.42 last 378.56) total: 64768 open/close in 170.90 seconds: 378.99 ops/second - /unlink 7211 (time 1772679505.56 total 10.00 last 721.07) - /unlink 10000 (time 1772679510.18 total 14.61 last 604.45) - /unlink 16215 (time 1772679520.18 total 24.61 last 621.47) - /unlink 20000 (time 1772679524.97 total 29.41 last 789.53) - /unlink 26891 (time 1772679534.97 total 39.41 last 689.10) - /unlink 30000 (time 1772679539.42 total 43.86 last 698.27) - /unlink 37069 (time 1772679549.42 total 53.86 last 706.88) - /unlink 40000 (time 1772679553.29 total 57.73 last 758.34) - /unlink 46859 (time 1772679563.29 total 67.73 last 685.87) - /unlink 50000 (time 1772679567.67 total 72.11 last 716.50) - /unlink 57291 (time 1772679577.68 total 82.11 last 729.00) - /unlink 60000 (time 1772679581.97 total 86.40 last 631.15) total: 64768 /unlink in 91.91 seconds: 704.66 ops/second Waiting for MDT destroys to complete oleg330-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 - open/close 4737 (time 1772679606.76 total 10.00 last 473.70) - open/close 9446 (time 1772679616.76 total 20.00 last 470.85) - open/close 13007 (time 1772679626.77 total 30.00 last 356.02) - open/close 16734 (time 1772679636.77 total 40.00 last 372.69) - open/close 20000 (time 1772679644.31 total 47.55 last 432.77) - open/close 24352 (time 1772679654.31 total 57.55 last 435.14) - open/close 28963 (time 1772679664.32 total 67.55 last 461.03) - open/close 30000 (time 1772679666.75 total 69.99 last 425.32) - open/close 34656 (time 1772679676.76 total 79.99 last 465.50) - open/close 39270 (time 1772679686.76 total 89.99 last 461.40) - open/close 43871 (time 1772679696.76 total 100.00 last 460.00) - open/close 48400 (time 1772679706.76 total 110.00 last 452.81) - open/close 50000 (time 1772679710.01 total 113.25 last 491.67) - open/close 54736 (time 1772679720.02 total 123.25 last 473.52) - open/close 59377 (time 1772679730.02 total 133.26 last 463.98) - open/close 64155 (time 1772679740.02 total 143.26 last 477.72) total: 64768 open/close in 144.45 seconds: 448.37 ops/second - /unlink 9182 (time 1772679752.46 total 10.00 last 918.20) - /unlink 17108 (time 1772679762.46 total 20.00 last 792.53) - /unlink 20000 (time 1772679766.13 total 23.67 last 787.33) - /unlink 28003 (time 1772679776.14 total 33.67 last 800.25) - /unlink 30000 (time 1772679778.45 total 35.99 last 861.82) - /unlink 37763 (time 1772679788.45 total 45.99 last 776.20) - /unlink 40000 (time 1772679791.54 total 49.08 last 724.68) - /unlink 47558 (time 1772679801.54 total 59.08 last 755.73) - /unlink 50000 (time 1772679805.33 total 62.87 last 644.07) - /unlink 56918 (time 1772679815.33 total 72.87 last 691.79) - /unlink 60000 (time 1772679819.47 total 77.01 last 744.32) total: 64768 /unlink in 82.43 seconds: 785.75 ops/second Waiting for MDT destroys to complete oleg330-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 - open/close 4075 (time 1772679843.87 total 10.00 last 407.47) - open/close 8160 (time 1772679853.87 total 20.00 last 408.42) - open/close 10000 (time 1772679860.41 total 26.54 last 281.42) - open/close 13966 (time 1772679870.41 total 36.54 last 396.58) - open/close 18192 (time 1772679880.41 total 46.54 last 422.53) - open/close 20000 (time 1772679886.50 total 52.63 last 297.02) - open/close 23775 (time 1772679896.50 total 62.63 last 377.45) - open/close 28161 (time 1772679906.50 total 72.63 last 438.54) - open/close 30000 (time 1772679910.55 total 76.68 last 454.01) - open/close 34770 (time 1772679920.55 total 86.69 last 476.91) - open/close 38748 (time 1772679930.55 total 96.69 last 397.72) - open/close 40000 (time 1772679933.67 total 99.81 last 401.35) - open/close 43742 (time 1772679943.68 total 109.81 last 374.06) - open/close 47967 (time 1772679953.68 total 119.81 last 422.45) - open/close 50000 (time 1772679958.62 total 124.76 last 410.95) - open/close 54231 (time 1772679968.62 total 134.76 last 423.09) - open/close 58570 (time 1772679978.63 total 144.76 last 433.85) - open/close 60000 (time 1772679981.74 total 147.88 last 458.83) - open/close 64500 (time 1772679991.74 total 157.88 last 449.95) total: 64768 open/close in 158.39 seconds: 408.91 ops/second - /unlink 6789 (time 1772680003.63 total 10.00 last 678.88) - /unlink 10000 (time 1772680008.71 total 15.09 last 631.26) - /unlink 17669 (time 1772680018.71 total 25.09 last 766.87) - /unlink 20000 (time 1772680021.57 total 27.95 last 815.54) - /unlink 27156 (time 1772680031.57 total 37.95 last 715.54) - /unlink 30000 (time 1772680035.24 total 41.62 last 775.08) - /unlink 38567 (time 1772680045.24 total 51.62 last 856.42) - /unlink 47399 (time 1772680055.25 total 61.62 last 883.17) - /unlink 50000 (time 1772680058.43 total 64.81 last 815.82) - /unlink 58426 (time 1772680068.43 total 74.81 last 842.59) total: 64768 /unlink in 81.83 seconds: 791.51 ops/second Waiting for MDT destroys to complete oleg330-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg330-client: oleg330-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg330-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg330-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg330-server stop mds service on oleg330-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg330-server stop mds service on oleg330-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg330-server unloading modules via unload_modules_local on: 'oleg330-server' oleg330-server: oleg330-server.virtnet: executing unload_modules_local oleg330-server: modules unloaded. pdsh@oleg330-client: oleg330-client: ssh exited with exit code 2 pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2 pdsh@oleg330-client: oleg330-client: ssh exited with exit code 2 pdsh@oleg330-client: oleg330-server: ssh exited with exit code 2