== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:11:05 (1741626665) Stopping clients: oleg150-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg150-client.virtnet /mnt/lustre2 (opts:-f) oleg150-server: oleg150-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing load_modules_local oleg150-server: Loading modules from /home/green/git/lustre-release/lustre oleg150-server: detected 4 online CPUs by sysfs oleg150-server: MODOPTS_LIBCFS= oleg150-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg150-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg150-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg150-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4583 (time 1741626705.37 total 10.00 last 458.28) - open/close 8529 (time 1741626715.37 total 20.00 last 394.59) - open/close 10000 (time 1741626719.30 total 23.92 last 375.15) - open/close 13231 (time 1741626729.30 total 33.92 last 323.07) - open/close 17602 (time 1741626739.30 total 43.92 last 437.03) - open/close 20000 (time 1741626744.19 total 48.81 last 490.38) - open/close 24574 (time 1741626754.19 total 58.82 last 457.34) - open/close 28498 (time 1741626764.19 total 68.82 last 392.37) - open/close 30000 (time 1741626767.27 total 71.90 last 487.01) - open/close 33947 (time 1741626777.27 total 81.90 last 394.67) - open/close 38270 (time 1741626787.28 total 91.90 last 432.23) - open/close 40000 (time 1741626792.13 total 96.76 last 356.16) - open/close 43571 (time 1741626802.13 total 106.76 last 357.06) - open/close 46772 (time 1741626812.14 total 116.76 last 320.03) - open/close 49505 (time 1741626822.14 total 126.76 last 273.27) - open/close 52809 (time 1741626832.14 total 136.77 last 330.31) - open/close 55871 (time 1741626842.14 total 146.77 last 306.16) - open/close 58283 (time 1741626852.14 total 156.77 last 241.14) - open/close 60000 (time 1741626858.90 total 163.52 last 254.23) - open/close 63149 (time 1741626868.90 total 173.53 last 314.87) total: 64768 open/close in 178.08 seconds: 363.70 ops/second - /unlink 7269 (time 1741626884.42 total 10.00 last 726.84) - /unlink 10000 (time 1741626888.12 total 13.70 last 738.41) - /unlink 18423 (time 1741626898.12 total 23.70 last 842.25) - /unlink 20000 (time 1741626900.12 total 25.70 last 787.75) - /unlink 28334 (time 1741626910.12 total 35.70 last 833.32) - /unlink 30000 (time 1741626912.15 total 37.73 last 821.86) - /unlink 38846 (time 1741626922.15 total 47.73 last 884.57) - /unlink 47989 (time 1741626932.15 total 57.73 last 914.29) - /unlink 50000 (time 1741626934.37 total 59.95 last 904.27) - /unlink 59210 (time 1741626944.37 total 69.96 last 920.92) total: 64768 /unlink in 75.95 seconds: 852.72 ops/second Waiting for MDT destroys to complete oleg150-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg150-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 - open/close 5208 (time 1741627043.85 total 10.00 last 520.77) - open/close 10000 (time 1741627053.01 total 19.15 last 523.57) - open/close 15124 (time 1741627063.01 total 29.15 last 512.39) - open/close 20000 (time 1741627072.45 total 38.60 last 516.31) - open/close 25239 (time 1741627082.45 total 48.60 last 523.90) - open/close 30000 (time 1741627091.59 total 57.74 last 521.02) - open/close 35172 (time 1741627101.59 total 67.74 last 517.12) - open/close 40000 (time 1741627111.04 total 77.19 last 510.71) - open/close 45123 (time 1741627121.04 total 87.19 last 512.26) - open/close 50000 (time 1741627130.67 total 96.81 last 506.77) - open/close 55029 (time 1741627140.67 total 106.81 last 502.90) - open/close 60000 (time 1741627150.35 total 116.49 last 513.63) total: 64768 open/close in 126.09 seconds: 513.68 ops/second - /unlink 9232 (time 1741627170.81 total 10.00 last 923.16) - /unlink 18345 (time 1741627180.81 total 20.00 last 911.24) - /unlink 27679 (time 1741627190.81 total 30.00 last 933.36) - /unlink 30000 (time 1741627193.30 total 32.48 last 935.59) - /unlink 38931 (time 1741627203.30 total 42.48 last 893.04) - /unlink 47410 (time 1741627213.30 total 52.48 last 847.85) - /unlink 50000 (time 1741627216.24 total 55.42 last 880.90) - /unlink 59075 (time 1741627226.24 total 65.42 last 907.48) total: 64768 /unlink in 71.49 seconds: 905.96 ops/second Waiting for MDT destroys to complete oleg150-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg150-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 - open/close 5203 (time 1741627334.90 total 10.00 last 520.28) - open/close 10000 (time 1741627344.21 total 19.31 last 515.15) - open/close 15075 (time 1741627354.21 total 29.31 last 507.49) - open/close 20000 (time 1741627363.96 total 39.07 last 504.85) - open/close 25088 (time 1741627373.97 total 49.07 last 508.68) - open/close 30000 (time 1741627383.87 total 58.97 last 496.08) - open/close 34859 (time 1741627393.87 total 68.97 last 485.86) - open/close 39692 (time 1741627403.87 total 78.97 last 483.27) - open/close 44685 (time 1741627413.87 total 88.97 last 499.24) - open/close 49715 (time 1741627423.87 total 98.97 last 503.00) - open/close 54479 (time 1741627433.87 total 108.98 last 476.35) - open/close 59479 (time 1741627443.87 total 118.98 last 499.91) - open/close 64244 (time 1741627453.87 total 128.98 last 476.48) total: 64768 open/close in 130.01 seconds: 498.16 ops/second - /unlink 8498 (time 1741627465.77 total 10.00 last 849.72) - /unlink 17434 (time 1741627475.77 total 20.00 last 893.44) - /unlink 20000 (time 1741627479.52 total 23.75 last 684.76) - /unlink 29167 (time 1741627489.52 total 33.75 last 916.66) - /unlink 38538 (time 1741627499.52 total 43.75 last 937.01) - /unlink 48124 (time 1741627509.52 total 53.75 last 958.53) - /unlink 50000 (time 1741627511.53 total 55.77 last 931.71) - /unlink 59216 (time 1741627521.54 total 65.77 last 921.57) total: 64768 /unlink in 71.74 seconds: 902.86 ops/second Waiting for MDT destroys to complete oleg150-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg150-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 - open/close 3108 (time 1741627620.58 total 10.00 last 310.71) - open/close 6411 (time 1741627630.58 total 20.00 last 330.29) - open/close 10000 (time 1741627637.73 total 27.15 last 502.37) - open/close 15249 (time 1741627647.73 total 37.15 last 524.80) - open/close 18894 (time 1741627657.73 total 47.15 last 364.49) - open/close 20000 (time 1741627660.91 total 50.33 last 347.50) - open/close 23326 (time 1741627670.91 total 60.33 last 332.57) - open/close 26918 (time 1741627680.91 total 70.33 last 359.18) - open/close 30000 (time 1741627689.54 total 78.96 last 357.19) - open/close 33532 (time 1741627699.55 total 88.97 last 352.77) - open/close 37051 (time 1741627709.56 total 98.98 last 351.80) - open/close 40000 (time 1741627718.90 total 108.32 last 315.63) - open/close 42743 (time 1741627728.90 total 118.32 last 274.24) - open/close 45790 (time 1741627738.91 total 128.33 last 304.62) - open/close 48811 (time 1741627748.91 total 138.33 last 302.01) - open/close 50000 (time 1741627753.22 total 142.64 last 275.77) - open/close 53648 (time 1741627763.22 total 152.64 last 364.72) - open/close 57359 (time 1741627773.22 total 162.64 last 371.09) - open/close 60000 (time 1741627780.06 total 169.48 last 386.29) - open/close 64411 (time 1741627790.06 total 179.48 last 441.05) total: 64768 open/close in 180.32 seconds: 359.18 ops/second - /unlink 9157 (time 1741627801.78 total 10.00 last 915.64) - /unlink 18534 (time 1741627811.78 total 20.00 last 937.69) - /unlink 28055 (time 1741627821.78 total 30.00 last 952.06) - /unlink 37635 (time 1741627831.78 total 40.00 last 957.98) - /unlink 40000 (time 1741627834.24 total 42.46 last 961.78) - /unlink 49792 (time 1741627844.24 total 52.46 last 979.12) - /unlink 59450 (time 1741627854.24 total 62.46 last 965.77) total: 64768 /unlink in 67.94 seconds: 953.32 ops/second Waiting for MDT destroys to complete oleg150-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg150-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 - open/close 5374 (time 1741627966.48 total 10.00 last 537.36) - open/close 10000 (time 1741627975.16 total 18.69 last 532.45) - open/close 15297 (time 1741627985.17 total 28.69 last 529.67) - open/close 20000 (time 1741627994.03 total 37.55 last 530.72) - open/close 25197 (time 1741628004.03 total 47.55 last 519.59) - open/close 30000 (time 1741628013.35 total 56.88 last 515.20) - open/close 35240 (time 1741628023.35 total 66.88 last 523.90) - open/close 40000 (time 1741628032.32 total 75.85 last 530.66) - open/close 45274 (time 1741628042.32 total 85.85 last 527.37) - open/close 50000 (time 1741628051.35 total 94.88 last 523.36) - open/close 55194 (time 1741628061.36 total 104.88 last 519.33) - open/close 60000 (time 1741628070.58 total 114.11 last 520.89) total: 64768 open/close in 123.42 seconds: 524.78 ops/second - /unlink 8993 (time 1741628090.72 total 10.00 last 899.22) - /unlink 18107 (time 1741628100.72 total 20.00 last 911.36) - /unlink 20000 (time 1741628102.76 total 22.04 last 926.31) - /unlink 29457 (time 1741628112.76 total 32.05 last 945.68) - /unlink 38660 (time 1741628122.76 total 42.05 last 920.23) - /unlink 48315 (time 1741628132.76 total 52.05 last 965.44) - /unlink 56899 (time 1741628142.76 total 62.05 last 858.32) - /unlink 60000 (time 1741628145.92 total 65.20 last 983.50) total: 64768 /unlink in 70.10 seconds: 923.94 ops/second Waiting for MDT destroys to complete oleg150-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg150-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg150-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg150-server unloading modules via unload_modules_local on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg150-client: oleg150-client: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-client: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2