== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:04:18 (1745456658) Stopping clients: oleg311-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg311-client.virtnet /mnt/lustre2 (opts:-f) oleg311-server: oleg311-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing load_modules_local oleg311-server: Loading modules from /home/green/git/lustre-release/lustre oleg311-server: detected 4 online CPUs by sysfs oleg311-server: MODOPTS_LIBCFS= oleg311-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg311-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg311-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg311-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2456 (time 1745456707.20 total 10.00 last 245.55) - open/close 4824 (time 1745456717.20 total 20.00 last 236.75) - open/close 6754 (time 1745456727.20 total 30.01 last 192.96) - open/close 8916 (time 1745456737.20 total 40.01 last 216.18) - open/close 10000 (time 1745456740.80 total 43.61 last 301.01) - open/close 12682 (time 1745456750.80 total 53.61 last 268.19) - open/close 16737 (time 1745456760.81 total 63.61 last 405.47) - open/close 18985 (time 1745456770.81 total 73.61 last 224.75) - open/close 20000 (time 1745456774.17 total 76.97 last 302.29) - open/close 22846 (time 1745456784.17 total 86.97 last 284.58) - open/close 25162 (time 1745456794.17 total 96.97 last 231.60) - open/close 27176 (time 1745456804.17 total 106.98 last 201.26) - open/close 29442 (time 1745456814.18 total 116.98 last 226.55) - open/close 30000 (time 1745456816.90 total 119.70 last 205.05) - open/close 31838 (time 1745456826.90 total 129.70 last 183.78) - open/close 33346 (time 1745456836.90 total 139.71 last 150.73) - open/close 35000 (time 1745456846.91 total 149.71 last 165.31) - open/close 36910 (time 1745456856.91 total 159.71 last 190.95) - open/close 38595 (time 1745456866.91 total 169.72 last 168.49) - open/close 40000 (time 1745456876.52 total 179.32 last 146.25) - open/close 41719 (time 1745456886.52 total 189.33 last 171.80) - open/close 43210 (time 1745456896.52 total 199.33 last 149.10) - open/close 44887 (time 1745456906.54 total 209.34 last 167.46) - open/close 46857 (time 1745456916.54 total 219.35 last 196.95) - open/close 48873 (time 1745456926.54 total 229.35 last 201.54) - open/close 50000 (time 1745456931.79 total 234.60 last 214.62) - open/close 51860 (time 1745456941.80 total 244.60 last 185.96) - open/close 53694 (time 1745456951.80 total 254.60 last 183.39) - open/close 55556 (time 1745456961.80 total 264.61 last 186.11) - open/close 57618 (time 1745456971.81 total 274.61 last 206.11) - open/close 59066 (time 1745456981.82 total 284.62 last 144.67) - open/close 60000 (time 1745456988.33 total 291.14 last 143.30) - open/close 61976 (time 1745456998.34 total 301.15 last 197.49) - open/close 63624 (time 1745457008.35 total 311.15 last 164.64) total: 64768 open/close in 318.62 seconds: 203.28 ops/second - /unlink 2992 (time 1745457027.03 total 10.00 last 299.19) - /unlink 6507 (time 1745457037.03 total 20.00 last 351.43) - /unlink 10000 (time 1745457044.67 total 27.64 last 457.04) - /unlink 14827 (time 1745457054.67 total 37.65 last 482.69) - /unlink 19337 (time 1745457064.67 total 47.65 last 450.94) - /unlink 23408 (time 1745457074.67 total 57.65 last 407.09) - /unlink 28491 (time 1745457084.68 total 67.65 last 508.13) - /unlink 30000 (time 1745457088.28 total 71.25 last 419.34) - /unlink 34385 (time 1745457098.28 total 81.25 last 438.45) - /unlink 40000 (time 1745457108.20 total 91.18 last 565.60) - /unlink 44488 (time 1745457118.21 total 101.18 last 448.69) - /unlink 49511 (time 1745457128.21 total 111.18 last 502.19) - /unlink 54893 (time 1745457138.21 total 121.18 last 538.12) - /unlink 60000 (time 1745457147.48 total 130.45 last 550.98) total: 64768 /unlink in 138.68 seconds: 467.03 ops/second Waiting for MDT destroys to complete - open/close 3122 (time 1745457174.62 total 10.00 last 312.07) - open/close 6068 (time 1745457184.62 total 20.01 last 294.55) - open/close 9505 (time 1745457194.62 total 30.01 last 343.64) - open/close 12778 (time 1745457204.62 total 40.01 last 327.24) - open/close 16136 (time 1745457214.63 total 50.01 last 335.66) - open/close 19179 (time 1745457224.63 total 60.02 last 304.25) - open/close 20000 (time 1745457226.86 total 62.25 last 367.91) - open/close 23577 (time 1745457236.86 total 72.25 last 357.60) - open/close 26584 (time 1745457246.87 total 82.25 last 300.62) - open/close 30000 (time 1745457256.76 total 92.14 last 345.41) - open/close 34206 (time 1745457266.76 total 102.14 last 420.57) - open/close 38528 (time 1745457276.76 total 112.14 last 432.13) - open/close 40000 (time 1745457280.40 total 115.78 last 404.63) - open/close 44998 (time 1745457290.40 total 125.78 last 499.71) - open/close 49665 (time 1745457300.40 total 135.78 last 466.69) - open/close 53808 (time 1745457310.40 total 145.79 last 414.21) - open/close 57698 (time 1745457320.40 total 155.79 last 388.98) - open/close 60000 (time 1745457325.31 total 160.70 last 468.77) - open/close 64516 (time 1745457335.31 total 170.70 last 451.51) total: 64768 open/close in 171.48 seconds: 377.69 ops/second - /unlink 7189 (time 1745457347.07 total 10.00 last 718.86) - /unlink 10000 (time 1745457350.36 total 13.29 last 854.90) - /unlink 19304 (time 1745457360.36 total 23.29 last 930.40) - /unlink 28750 (time 1745457370.36 total 33.29 last 944.57) - /unlink 39108 (time 1745457380.36 total 43.29 last 1035.73) - /unlink 49717 (time 1745457390.36 total 53.29 last 1060.89) - /unlink 60000 (time 1745457400.17 total 63.10 last 1048.30) total: 64768 /unlink in 67.31 seconds: 962.26 ops/second Waiting for MDT destroys to complete - open/close 4520 (time 1745457510.43 total 10.00 last 451.96) - open/close 10000 (time 1745457520.14 total 19.72 last 564.04) - open/close 14903 (time 1745457530.14 total 29.72 last 490.26) - open/close 20000 (time 1745457538.40 total 37.98 last 617.08) - open/close 24630 (time 1745457548.40 total 47.98 last 462.92) - open/close 30000 (time 1745457558.18 total 57.76 last 549.10) - open/close 35607 (time 1745457568.19 total 67.76 last 560.64) - open/close 40000 (time 1745457576.08 total 75.65 last 556.70) - open/close 44864 (time 1745457586.08 total 85.65 last 486.34) - open/close 49921 (time 1745457596.08 total 95.65 last 505.65) - open/close 55936 (time 1745457606.08 total 105.65 last 601.45) - open/close 60000 (time 1745457612.90 total 112.47 last 595.88) total: 64768 open/close in 121.35 seconds: 533.72 ops/second - /unlink 10000 (time 1745457632.21 total 9.56 last 1046.33) - /unlink 18283 (time 1745457642.21 total 19.56 last 828.30) - /unlink 20000 (time 1745457644.68 total 22.03 last 694.60) - /unlink 28117 (time 1745457654.68 total 32.03 last 811.63) - /unlink 36320 (time 1745457664.68 total 42.03 last 820.29) - /unlink 40000 (time 1745457669.36 total 46.71 last 786.08) - /unlink 48902 (time 1745457679.36 total 56.71 last 890.12) - /unlink 58714 (time 1745457689.37 total 66.71 last 981.16) total: 64768 /unlink in 72.83 seconds: 889.36 ops/second Waiting for MDT destroys to complete - open/close 4891 (time 1745457785.75 total 10.00 last 489.06) - open/close 10000 (time 1745457795.53 total 19.78 last 522.58) - open/close 15081 (time 1745457805.53 total 29.78 last 508.10) - open/close 20000 (time 1745457815.07 total 39.31 last 515.76) - open/close 24709 (time 1745457825.07 total 49.32 last 470.85) - open/close 29007 (time 1745457835.07 total 59.32 last 429.79) - open/close 34117 (time 1745457845.07 total 69.32 last 510.99) - open/close 39795 (time 1745457855.07 total 79.32 last 567.76) - open/close 45402 (time 1745457865.07 total 89.32 last 560.63) - open/close 50000 (time 1745457874.63 total 98.88 last 480.84) - open/close 55083 (time 1745457884.63 total 108.88 last 508.26) - open/close 60000 (time 1745457894.35 total 118.60 last 505.84) total: 64768 open/close in 128.50 seconds: 504.04 ops/second - /unlink 9234 (time 1745457915.08 total 10.00 last 923.31) - /unlink 19262 (time 1745457925.09 total 20.01 last 1002.35) - /unlink 28382 (time 1745457935.09 total 30.01 last 911.95) - /unlink 38243 (time 1745457945.09 total 40.01 last 986.07) - /unlink 47421 (time 1745457955.09 total 50.01 last 917.75) - /unlink 50000 (time 1745457958.04 total 52.96 last 873.12) - /unlink 58551 (time 1745457968.04 total 62.96 last 855.05) total: 64768 /unlink in 69.93 seconds: 926.14 ops/second Waiting for MDT destroys to complete - open/close 5310 (time 1745458076.57 total 10.00 last 530.94) - open/close 10000 (time 1745458086.03 total 19.46 last 495.76) - open/close 15200 (time 1745458096.03 total 29.46 last 519.95) - open/close 20000 (time 1745458105.02 total 38.45 last 534.19) - open/close 25215 (time 1745458115.02 total 48.45 last 521.50) - open/close 30000 (time 1745458124.41 total 57.84 last 509.21) - open/close 35198 (time 1745458134.41 total 67.85 last 519.73) - open/close 40000 (time 1745458143.85 total 77.28 last 508.83) - open/close 44734 (time 1745458153.85 total 87.28 last 473.37) - open/close 49889 (time 1745458163.85 total 97.28 last 515.44) - open/close 54831 (time 1745458173.86 total 107.29 last 494.13) - open/close 59678 (time 1745458183.86 total 117.29 last 484.59) total: 64768 open/close in 127.16 seconds: 509.35 ops/second - /unlink 9500 (time 1745458204.58 total 10.00 last 949.93) - /unlink 18694 (time 1745458214.58 total 20.00 last 919.38) - /unlink 28302 (time 1745458224.58 total 30.00 last 960.76) - /unlink 38084 (time 1745458234.58 total 40.00 last 978.14) - /unlink 40000 (time 1745458236.78 total 42.20 last 870.02) - /unlink 49442 (time 1745458246.78 total 52.20 last 944.15) - /unlink 59218 (time 1745458256.78 total 62.21 last 977.51) total: 64768 /unlink in 67.70 seconds: 956.72 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg311-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg311-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg311-server stop mds service on oleg311-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg311-server stop mds service on oleg311-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg311-server unloading modules via unload_modules_local on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2