== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:35:21 (1733805321) Stopping clients: oleg204-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg204-client.virtnet /mnt/lustre2 (opts:-f) oleg204-server: oleg204-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing load_modules_local oleg204-server: Loading modules from /home/green/git/lustre-release/lustre oleg204-server: detected 4 online CPUs by sysfs oleg204-server: MODOPTS_LIBCFS= oleg204-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg204-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg204-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg204-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 275 79717 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 256 79736 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33059 531 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1453 (time 1733805382.66 total 10.01 last 145.22) - open/close 2792 (time 1733805392.66 total 20.01 last 133.78) - open/close 4224 (time 1733805402.67 total 30.02 last 143.07) - open/close 5725 (time 1733805412.67 total 40.03 last 150.09) - open/close 7277 (time 1733805422.68 total 50.03 last 155.14) - open/close 8751 (time 1733805432.68 total 60.03 last 147.37) - open/close 10000 (time 1733805441.00 total 68.35 last 150.09) - open/close 11511 (time 1733805451.01 total 78.36 last 151.02) - open/close 13078 (time 1733805461.01 total 88.36 last 156.66) - open/close 14514 (time 1733805471.01 total 98.36 last 143.56) - open/close 16160 (time 1733805481.01 total 108.36 last 164.60) - open/close 18022 (time 1733805491.01 total 118.36 last 186.20) - open/close 19708 (time 1733805501.01 total 128.36 last 168.59) - open/close 21415 (time 1733805511.01 total 138.36 last 170.70) - open/close 22926 (time 1733805521.02 total 148.37 last 151.07) - open/close 24517 (time 1733805531.02 total 158.37 last 159.09) - open/close 26117 (time 1733805541.02 total 168.37 last 160.00) - open/close 27773 (time 1733805551.02 total 178.37 last 165.51) - open/close 29250 (time 1733805561.03 total 188.38 last 147.61) - open/close 30000 (time 1733805565.82 total 193.17 last 156.67) - open/close 31650 (time 1733805575.82 total 203.17 last 164.94) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 208.48 seconds: 156.18 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32836 47156 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 256 79736 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33092 33092 0 100% /mnt/lustre - /unlink 2928 (time 1733805592.67 total 10.00 last 292.74) - /unlink 5944 (time 1733805602.67 total 20.01 last 301.51) - /unlink 9145 (time 1733805612.67 total 30.01 last 320.07) - /unlink 10000 (time 1733805615.34 total 32.67 last 320.48) - /unlink 13050 (time 1733805625.34 total 42.68 last 304.84) - /unlink 16084 (time 1733805635.34 total 52.68 last 303.37) - /unlink 19081 (time 1733805645.35 total 62.68 last 299.63) - /unlink 20000 (time 1733805648.60 total 65.94 last 282.24) - /unlink 22808 (time 1733805658.60 total 75.94 last 280.72) - /unlink 25826 (time 1733805668.61 total 85.94 last 301.74) - /unlink 28996 (time 1733805678.61 total 95.94 last 316.96) - /unlink 30000 (time 1733805681.71 total 99.05 last 323.38) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 106.91 seconds: 304.54 ops/second Waiting for MDT destroys to complete oleg204-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg204-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 - open/close 2065 (time 1733805709.87 total 10.01 last 206.37) - open/close 3708 (time 1733805719.87 total 20.01 last 164.25) - open/close 5497 (time 1733805729.87 total 30.01 last 178.84) - open/close 7322 (time 1733805739.88 total 40.02 last 182.43) - open/close 9180 (time 1733805749.88 total 50.02 last 185.76) - open/close 10000 (time 1733805754.15 total 54.29 last 192.15) - open/close 11821 (time 1733805764.15 total 64.29 last 182.06) - open/close 13522 (time 1733805774.15 total 74.29 last 170.02) - open/close 15116 (time 1733805784.16 total 84.30 last 159.33) - open/close 16639 (time 1733805794.16 total 94.30 last 152.29) - open/close 18184 (time 1733805804.17 total 104.31 last 154.28) - open/close 19600 (time 1733805814.17 total 114.31 last 141.58) - open/close 20000 (time 1733805816.74 total 116.88 last 155.65) - open/close 21533 (time 1733805826.75 total 126.89 last 153.24) - open/close 23008 (time 1733805836.75 total 136.89 last 147.47) - open/close 24429 (time 1733805846.75 total 146.89 last 142.08) - open/close 25895 (time 1733805856.76 total 156.90 last 146.53) - open/close 27448 (time 1733805866.76 total 166.90 last 155.26) - open/close 29083 (time 1733805876.76 total 176.90 last 163.45) - open/close 30000 (time 1733805883.25 total 183.39 last 141.35) - open/close 31531 (time 1733805893.25 total 193.39 last 153.09) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 199.20 seconds: 163.45 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32836 47156 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 256 79736 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33092 33092 0 100% /mnt/lustre - /unlink 3327 (time 1733805910.46 total 10.00 last 332.64) - /unlink 6156 (time 1733805920.47 total 20.00 last 282.84) - /unlink 8850 (time 1733805930.47 total 30.01 last 269.31) - /unlink 10000 (time 1733805934.47 total 34.01 last 287.36) - /unlink 12769 (time 1733805944.47 total 44.01 last 276.83) - /unlink 15506 (time 1733805954.48 total 54.01 last 273.64) - /unlink 18380 (time 1733805964.48 total 64.02 last 287.36) - /unlink 20000 (time 1733805969.81 total 69.34 last 304.05) - /unlink 23099 (time 1733805979.81 total 79.35 last 309.85) - /unlink 25959 (time 1733805989.81 total 89.35 last 285.97) - /unlink 29080 (time 1733805999.81 total 99.35 last 312.08) - /unlink 30000 (time 1733806002.70 total 102.24 last 318.34) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 109.57 seconds: 297.17 ops/second Waiting for MDT destroys to complete oleg204-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 oleg204-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 - open/close 2106 (time 1733806030.72 total 10.00 last 210.59) - open/close 3964 (time 1733806040.73 total 20.01 last 185.67) - open/close 5899 (time 1733806050.73 total 30.01 last 193.44) - open/close 7834 (time 1733806060.73 total 40.01 last 193.50) - open/close 9857 (time 1733806070.73 total 50.01 last 202.25) - open/close 11963 (time 1733806080.73 total 60.01 last 210.58) - open/close 13826 (time 1733806090.74 total 70.02 last 186.22) - open/close 15465 (time 1733806100.74 total 80.02 last 163.89) - open/close 17106 (time 1733806110.74 total 90.02 last 164.09) - open/close 18727 (time 1733806120.74 total 100.02 last 162.03) - open/close 20000 (time 1733806128.58 total 107.86 last 162.41) - open/close 21576 (time 1733806138.58 total 117.86 last 157.60) - open/close 23179 (time 1733806148.58 total 127.87 last 160.25) - open/close 24817 (time 1733806158.59 total 137.87 last 163.75) - open/close 26475 (time 1733806168.59 total 147.87 last 165.80) - open/close 28375 (time 1733806178.59 total 157.87 last 189.99) - open/close 30000 (time 1733806187.36 total 166.65 last 185.17) - open/close 31634 (time 1733806197.37 total 176.65 last 163.36) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 182.11 seconds: 178.79 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32836 47156 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 256 79736 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32880 16 100% /mnt/lustre[OST:0] filesystem_summary: 33108 33092 16 100% /mnt/lustre - /unlink 3029 (time 1733806214.63 total 10.00 last 302.77) - /unlink 6062 (time 1733806224.63 total 20.01 last 303.21) - /unlink 9294 (time 1733806234.63 total 30.01 last 323.15) - /unlink 12343 (time 1733806244.63 total 40.01 last 304.86) - /unlink 15472 (time 1733806254.63 total 50.01 last 312.86) - /unlink 18547 (time 1733806264.63 total 60.01 last 307.48) - /unlink 20000 (time 1733806269.09 total 64.47 last 326.00) - /unlink 22935 (time 1733806279.09 total 74.47 last 293.43) - /unlink 25880 (time 1733806289.09 total 84.47 last 294.45) - /unlink 29117 (time 1733806299.10 total 94.48 last 323.60) - /unlink 30000 (time 1733806301.71 total 97.09 last 338.26) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 104.55 seconds: 311.43 ops/second Waiting for MDT destroys to complete oleg204-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 oleg204-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 - open/close 2334 (time 1733806328.93 total 10.01 last 233.26) - open/close 4651 (time 1733806338.93 total 20.01 last 231.68) - open/close 6658 (time 1733806348.93 total 30.01 last 200.67) - open/close 8671 (time 1733806358.94 total 40.01 last 201.17) - open/close 10000 (time 1733806365.68 total 46.76 last 197.16) - open/close 11823 (time 1733806375.69 total 56.76 last 182.23) - open/close 13538 (time 1733806385.69 total 66.76 last 171.48) - open/close 15344 (time 1733806395.69 total 76.76 last 180.55) - open/close 17338 (time 1733806405.70 total 86.77 last 199.27) - open/close 19231 (time 1733806415.70 total 96.77 last 189.24) - open/close 20000 (time 1733806419.71 total 100.79 last 191.51) - open/close 22187 (time 1733806429.72 total 110.79 last 218.67) - open/close 24304 (time 1733806439.72 total 120.79 last 211.67) - open/close 26518 (time 1733806449.72 total 130.79 last 221.38) - open/close 28713 (time 1733806459.72 total 140.80 last 219.41) - open/close 30000 (time 1733806465.17 total 146.24 last 236.25) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 156.16 seconds: 208.51 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32836 47156 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 256 79736 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33092 33092 0 100% /mnt/lustre - /unlink 5499 (time 1733806486.10 total 10.00 last 549.86) - /unlink 10000 (time 1733806495.27 total 19.17 last 490.74) - /unlink 14406 (time 1733806505.27 total 29.17 last 440.51) - /unlink 18203 (time 1733806515.27 total 39.17 last 379.69) - /unlink 20000 (time 1733806520.00 total 43.90 last 380.00) - /unlink 24286 (time 1733806530.00 total 53.90 last 428.56) - /unlink 28149 (time 1733806540.00 total 63.91 last 386.25) - /unlink 30000 (time 1733806544.74 total 68.65 last 390.22) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 75.56 seconds: 430.91 ops/second Waiting for MDT destroys to complete oleg204-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 oleg204-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 - open/close 1807 (time 1733806573.94 total 10.00 last 180.62) - open/close 4053 (time 1733806583.94 total 20.01 last 224.55) - open/close 5823 (time 1733806593.94 total 30.01 last 176.98) - open/close 7658 (time 1733806603.94 total 40.01 last 183.49) - open/close 9445 (time 1733806613.95 total 50.01 last 178.69) - open/close 10000 (time 1733806617.17 total 53.24 last 171.93) - open/close 11654 (time 1733806627.18 total 63.24 last 165.31) - open/close 13327 (time 1733806637.18 total 73.24 last 167.28) - open/close 14874 (time 1733806647.19 total 83.26 last 154.49) - open/close 16405 (time 1733806657.19 total 93.26 last 153.07) - open/close 17968 (time 1733806667.20 total 103.27 last 156.14) - open/close 19552 (time 1733806677.21 total 113.27 last 158.33) - open/close 20000 (time 1733806680.00 total 116.07 last 160.45) - open/close 21536 (time 1733806690.01 total 126.07 last 153.46) - open/close 22913 (time 1733806700.02 total 136.08 last 137.64) - open/close 24378 (time 1733806710.03 total 146.09 last 146.31) - open/close 25784 (time 1733806720.03 total 156.10 last 140.56) - open/close 27232 (time 1733806730.03 total 166.10 last 144.76) - open/close 28603 (time 1733806740.04 total 176.10 last 137.02) - open/close 29791 (time 1733806750.05 total 186.11 last 118.74) - open/close 31209 (time 1733806760.05 total 196.11 last 141.75) - open/close 32557 (time 1733806770.06 total 206.12 last 134.71) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 206.15 seconds: 157.94 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32836 47156 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 256 79736 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33092 33092 0 100% /mnt/lustre - /unlink 2237 (time 1733806782.31 total 10.00 last 223.63) - /unlink 4521 (time 1733806792.32 total 20.01 last 228.33) - /unlink 6753 (time 1733806802.32 total 30.01 last 223.16) - /unlink 9095 (time 1733806812.32 total 40.01 last 234.16) - /unlink 10000 (time 1733806816.09 total 43.78 last 240.13) - /unlink 12371 (time 1733806826.09 total 53.78 last 237.06) - /unlink 14532 (time 1733806836.09 total 63.78 last 216.07) - /unlink 16569 (time 1733806846.09 total 73.78 last 203.68) - /unlink 18682 (time 1733806856.09 total 83.78 last 211.28) - /unlink 20000 (time 1733806862.29 total 89.98 last 212.75) - /unlink 21721 (time 1733806872.29 total 99.98 last 172.05) - /unlink 24072 (time 1733806882.29 total 109.98 last 235.09) - /unlink 26586 (time 1733806892.29 total 119.98 last 251.39) - /unlink 29083 (time 1733806902.30 total 129.99 last 249.63) - /unlink 30000 (time 1733806906.43 total 134.12 last 221.74) - /unlink 32383 (time 1733806916.43 total 144.12 last 238.24) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 144.75 seconds: 224.94 ops/second Waiting for MDT destroys to complete oleg204-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 oleg204-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg204-server unloading modules via unload_modules_local on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2