== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 03:11:19 (1711091479) Stopping clients: oleg427-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg427-client.virtnet /mnt/lustre2 (opts:-f) oleg427-server: oleg427-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing load_modules_local oleg427-server: Loading modules from /home/green/git/lustre-release/lustre oleg427-server: detected 4 online CPUs by sysfs oleg427-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg427-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg427-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg427-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg427-server: oleg427-server.virtnet: executing set_default_debug -1 all pdsh@oleg427-client: oleg427-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg427-client.virtnet: -o user_xattr,flock oleg427-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4220 (time 1711091519.99 total 10.00 last 421.97) - open/close 8141 (time 1711091530.00 total 20.00 last 391.98) - open/close 10000 (time 1711091535.78 total 25.78 last 321.65) - open/close 12742 (time 1711091545.78 total 35.78 last 274.18) - open/close 15166 (time 1711091555.78 total 45.79 last 242.33) - open/close 17396 (time 1711091565.78 total 55.79 last 222.93) - open/close 19265 (time 1711091575.79 total 65.79 last 186.81) - open/close 20000 (time 1711091579.87 total 69.88 last 180.00) - open/close 21397 (time 1711091589.88 total 79.88 last 139.65) - open/close 22594 (time 1711091599.88 total 89.89 last 119.60) - open/close 23732 (time 1711091609.89 total 99.90 last 113.73) - open/close 24814 (time 1711091619.90 total 109.91 last 108.10) - open/close 25696 (time 1711091629.91 total 119.92 last 88.11) - open/close 26709 (time 1711091639.91 total 129.92 last 101.29) - open/close 27614 (time 1711091649.91 total 139.92 last 90.49) - open/close 28526 (time 1711091659.92 total 149.93 last 91.11) - open/close 29405 (time 1711091669.92 total 159.93 last 87.88) - open/close 30000 (time 1711091677.66 total 167.66 last 76.93) - open/close 30884 (time 1711091687.66 total 177.67 last 88.37) - open/close 31719 (time 1711091697.68 total 187.68 last 83.38) - open/close 32449 (time 1711091707.68 total 197.69 last 72.95) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 199.18 seconds: 163.47 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 933 (time 1711091720.13 total 10.01 last 93.24) - /unlink 1941 (time 1711091730.14 total 20.02 last 100.70) - /unlink 3150 (time 1711091740.15 total 30.02 last 120.87) - /unlink 4420 (time 1711091750.15 total 40.02 last 126.96) - /unlink 5471 (time 1711091760.16 total 50.03 last 105.05) - /unlink 6801 (time 1711091770.16 total 60.03 last 132.96) - /unlink 8065 (time 1711091780.16 total 70.03 last 126.35) - /unlink 9541 (time 1711091790.17 total 80.04 last 147.52) - /unlink 10000 (time 1711091793.21 total 83.08 last 150.76) - /unlink 11684 (time 1711091803.22 total 93.09 last 168.34) - /unlink 13175 (time 1711091813.22 total 103.09 last 149.05) - /unlink 15131 (time 1711091823.22 total 113.09 last 195.54) - /unlink 17111 (time 1711091833.23 total 123.10 last 197.94) - /unlink 19592 (time 1711091843.23 total 133.10 last 248.03) - /unlink 22304 (time 1711091853.23 total 143.10 last 271.11) - /unlink 25703 (time 1711091863.23 total 153.10 last 339.84) - /unlink 30000 (time 1711091871.03 total 160.90 last 551.24) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 164.07 seconds: 198.45 ops/second Waiting for MDT destroys to complete - open/close 4344 (time 1711091891.71 total 10.00 last 434.35) - open/close 7672 (time 1711091901.71 total 20.00 last 332.77) - open/close 10000 (time 1711091911.39 total 29.68 last 240.63) - open/close 11964 (time 1711091921.39 total 39.68 last 196.34) - open/close 13701 (time 1711091931.39 total 49.68 last 173.63) - open/close 15374 (time 1711091941.40 total 59.68 last 167.30) - open/close 16896 (time 1711091951.40 total 69.69 last 152.11) - open/close 18460 (time 1711091961.41 total 79.70 last 156.31) - open/close 19727 (time 1711091971.41 total 89.70 last 126.63) - open/close 20000 (time 1711091973.85 total 92.13 last 112.23) - open/close 21169 (time 1711091983.85 total 102.14 last 116.88) - open/close 22339 (time 1711091993.85 total 112.14 last 116.93) - open/close 23418 (time 1711092003.86 total 122.15 last 107.79) - open/close 24592 (time 1711092013.87 total 132.16 last 117.32) - open/close 25778 (time 1711092023.87 total 142.16 last 118.57) - open/close 27062 (time 1711092033.87 total 152.16 last 128.39) - open/close 28330 (time 1711092043.88 total 162.17 last 126.75) - open/close 29617 (time 1711092053.88 total 172.17 last 128.66) - open/close 30000 (time 1711092056.56 total 174.84 last 143.19) - open/close 31224 (time 1711092066.56 total 184.85 last 122.34) - open/close 32458 (time 1711092076.57 total 194.86 last 123.31) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 196.31 seconds: 165.86 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 1120 (time 1711092088.92 total 10.01 last 111.92) - /unlink 2306 (time 1711092098.92 total 20.01 last 118.56) - /unlink 3599 (time 1711092108.93 total 30.01 last 129.26) - /unlink 5009 (time 1711092118.93 total 40.01 last 140.99) - /unlink 6657 (time 1711092128.93 total 50.01 last 164.80) - /unlink 8397 (time 1711092138.93 total 60.02 last 173.95) - /unlink 10000 (time 1711092147.55 total 68.63 last 186.07) - /unlink 11959 (time 1711092157.55 total 78.64 last 195.82) - /unlink 14172 (time 1711092167.55 total 88.64 last 221.24) - /unlink 16705 (time 1711092177.56 total 98.64 last 253.22) - /unlink 19669 (time 1711092187.56 total 108.64 last 296.33) - /unlink 23364 (time 1711092197.56 total 118.65 last 369.43) - /unlink 28247 (time 1711092207.56 total 128.65 last 488.22) - /unlink 30000 (time 1711092210.32 total 131.41 last 635.08) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 134.55 seconds: 241.99 ops/second Waiting for MDT destroys to complete - open/close 4583 (time 1711092233.87 total 10.00 last 458.25) - open/close 8135 (time 1711092243.87 total 20.00 last 355.10) - open/close 10000 (time 1711092250.15 total 26.29 last 296.91) - open/close 12840 (time 1711092260.15 total 36.29 last 283.98) - open/close 15052 (time 1711092270.16 total 46.29 last 221.10) - open/close 17095 (time 1711092280.16 total 56.29 last 204.28) - open/close 18874 (time 1711092290.16 total 66.29 last 177.87) - open/close 20000 (time 1711092296.76 total 72.89 last 170.57) - open/close 21522 (time 1711092306.77 total 82.90 last 152.08) - open/close 23014 (time 1711092316.77 total 92.90 last 149.17) - open/close 24432 (time 1711092326.77 total 102.91 last 141.75) - open/close 25755 (time 1711092336.78 total 112.91 last 132.25) - open/close 27066 (time 1711092346.78 total 122.92 last 131.02) - open/close 28267 (time 1711092356.79 total 132.92 last 120.06) - open/close 29456 (time 1711092366.80 total 142.93 last 118.80) - open/close 30000 (time 1711092371.27 total 147.41 last 121.53) - open/close 31098 (time 1711092381.27 total 157.41 last 109.78) - open/close 32218 (time 1711092391.28 total 167.41 last 111.95) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 170.60 seconds: 190.85 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 1171 (time 1711092405.33 total 10.00 last 117.06) - /unlink 2479 (time 1711092415.34 total 20.01 last 130.74) - /unlink 3842 (time 1711092425.34 total 30.01 last 136.22) - /unlink 5320 (time 1711092435.34 total 40.01 last 147.78) - /unlink 6796 (time 1711092445.34 total 50.02 last 147.57) - /unlink 8424 (time 1711092455.35 total 60.02 last 162.73) - /unlink 10000 (time 1711092464.90 total 69.57 last 164.97) - /unlink 11782 (time 1711092474.91 total 79.58 last 178.14) - /unlink 13856 (time 1711092484.91 total 89.58 last 207.36) - /unlink 16066 (time 1711092494.91 total 99.58 last 220.94) - /unlink 18664 (time 1711092504.91 total 109.59 last 259.72) - /unlink 20000 (time 1711092509.44 total 114.12 last 294.84) - /unlink 23451 (time 1711092519.45 total 124.12 last 345.05) - /unlink 28133 (time 1711092529.45 total 134.12 last 468.16) - /unlink 30000 (time 1711092532.45 total 137.12 last 621.80) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 140.29 seconds: 232.10 ops/second Waiting for MDT destroys to complete - open/close 4039 (time 1711092553.95 total 10.00 last 403.73) - open/close 7643 (time 1711092563.95 total 20.01 last 360.37) - open/close 10000 (time 1711092572.05 total 28.10 last 291.04) - open/close 12778 (time 1711092582.05 total 38.10 last 277.77) - open/close 15229 (time 1711092592.05 total 48.11 last 245.07) - open/close 17426 (time 1711092602.05 total 58.11 last 219.68) - open/close 19549 (time 1711092612.05 total 68.11 last 212.23) - open/close 20000 (time 1711092614.07 total 70.12 last 223.87) - open/close 21719 (time 1711092624.07 total 80.13 last 171.89) - open/close 23410 (time 1711092634.07 total 90.13 last 169.06) - open/close 24983 (time 1711092644.07 total 100.13 last 157.29) - open/close 26444 (time 1711092654.07 total 110.13 last 146.06) - open/close 27792 (time 1711092664.08 total 120.13 last 134.77) - open/close 29153 (time 1711092674.08 total 130.14 last 136.06) - open/close 30000 (time 1711092681.31 total 137.36 last 117.19) - open/close 31244 (time 1711092691.31 total 147.37 last 124.36) - open/close 32369 (time 1711092701.31 total 157.37 last 112.48) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 159.11 seconds: 204.63 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 1406 (time 1711092713.89 total 10.01 last 140.51) - /unlink 2845 (time 1711092723.89 total 20.01 last 143.87) - /unlink 4296 (time 1711092733.90 total 30.01 last 145.01) - /unlink 5760 (time 1711092743.90 total 40.02 last 146.35) - /unlink 7452 (time 1711092753.90 total 50.02 last 169.12) - /unlink 9210 (time 1711092763.90 total 60.02 last 175.80) - /unlink 10000 (time 1711092767.83 total 63.95 last 201.05) - /unlink 12290 (time 1711092777.83 total 73.95 last 228.98) - /unlink 14748 (time 1711092787.84 total 83.96 last 245.73) - /unlink 17614 (time 1711092797.84 total 93.96 last 286.56) - /unlink 20000 (time 1711092805.95 total 102.06 last 294.32) - /unlink 23162 (time 1711092815.95 total 112.06 last 316.19) - /unlink 26026 (time 1711092825.95 total 122.07 last 286.34) - /unlink 29165 (time 1711092835.95 total 132.07 last 313.84) - /unlink 30000 (time 1711092838.29 total 134.41 last 356.96) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 140.74 seconds: 231.34 ops/second Waiting for MDT destroys to complete - open/close 2908 (time 1711092864.91 total 10.00 last 290.74) - open/close 5213 (time 1711092874.91 total 20.00 last 230.46) - open/close 7366 (time 1711092884.91 total 30.01 last 215.22) - open/close 9208 (time 1711092894.91 total 40.01 last 184.15) - open/close 10000 (time 1711092899.49 total 44.59 last 173.06) - open/close 11692 (time 1711092909.49 total 54.59 last 169.19) - open/close 13298 (time 1711092919.50 total 64.59 last 160.52) - open/close 14806 (time 1711092929.50 total 74.59 last 150.77) - open/close 16307 (time 1711092939.50 total 84.60 last 150.04) - open/close 17851 (time 1711092949.50 total 94.60 last 154.37) - open/close 19646 (time 1711092959.51 total 104.60 last 179.43) - open/close 21173 (time 1711092969.51 total 114.61 last 152.63) - open/close 22676 (time 1711092979.52 total 124.61 last 150.22) - open/close 23915 (time 1711092989.53 total 134.62 last 123.81) - open/close 25117 (time 1711092999.53 total 144.63 last 120.14) - open/close 26531 (time 1711093009.53 total 154.63 last 141.39) - open/close 28498 (time 1711093019.54 total 164.63 last 196.55) - open/close 29740 (time 1711093029.54 total 174.64 last 124.16) - open/close 30000 (time 1711093031.98 total 177.07 last 106.77) - open/close 31159 (time 1711093041.98 total 187.08 last 115.87) - open/close 32214 (time 1711093051.99 total 197.09 last 105.40) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 200.19 seconds: 162.65 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 1297 (time 1711093066.21 total 10.00 last 129.65) - /unlink 2720 (time 1711093076.22 total 20.01 last 142.27) - /unlink 4166 (time 1711093086.22 total 30.01 last 144.59) - /unlink 5737 (time 1711093096.22 total 40.01 last 157.08) - /unlink 7138 (time 1711093106.22 total 50.01 last 140.03) - /unlink 8532 (time 1711093116.23 total 60.02 last 139.37) - /unlink 10000 (time 1711093122.85 total 66.64 last 221.53) - /unlink 11571 (time 1711093132.86 total 76.65 last 157.04) - /unlink 13205 (time 1711093142.86 total 86.65 last 163.37) - /unlink 15550 (time 1711093152.86 total 96.65 last 234.42) - /unlink 18325 (time 1711093162.86 total 106.65 last 277.43) - /unlink 20000 (time 1711093168.13 total 111.92 last 318.29) - /unlink 24088 (time 1711093178.13 total 121.92 last 408.76) - /unlink 29735 (time 1711093188.13 total 131.92 last 564.66) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 135.34 seconds: 240.58 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg427-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg427-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg427-server stop mds service on oleg427-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg427-server unloading modules on: 'oleg427-server' oleg427-server: oleg427-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-client: ssh exited with exit code 2 pdsh@oleg427-client: oleg427-server: ssh exited with exit code 2