== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:54:09 (1718398449) Stopping clients: oleg234-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg234-client.virtnet /mnt/lustre2 (opts:-f) oleg234-server: oleg234-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing load_modules_local oleg234-server: Loading modules from /home/green/git/lustre-release/lustre oleg234-server: detected 4 online CPUs by sysfs oleg234-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg234-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg234-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg234-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg234-client.virtnet: -o user_xattr,flock oleg234-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 3471 (time 1718398498.08 total 10.00 last 347.07) - open/close 6637 (time 1718398508.08 total 20.00 last 316.59) - open/close 9840 (time 1718398518.09 total 30.01 last 320.15) - open/close 12830 (time 1718398528.09 total 40.01 last 298.90) - open/close 15259 (time 1718398538.09 total 50.01 last 242.90) - open/close 18285 (time 1718398548.09 total 60.01 last 302.51) - open/close 20000 (time 1718398553.62 total 65.54 last 310.18) - open/close 22854 (time 1718398563.63 total 75.55 last 285.26) - open/close 25140 (time 1718398573.63 total 85.55 last 228.54) - open/close 27755 (time 1718398583.63 total 95.55 last 261.48) - open/close 30000 (time 1718398592.31 total 104.23 last 258.65) - open/close 32469 (time 1718398602.32 total 114.23 last 246.76) - open/close 34932 (time 1718398612.32 total 124.23 last 246.30) - open/close 37086 (time 1718398622.32 total 134.24 last 215.37) - open/close 39828 (time 1718398632.32 total 144.24 last 274.13) - open/close 42434 (time 1718398642.32 total 154.24 last 260.58) - open/close 45565 (time 1718398652.32 total 164.24 last 313.01) - open/close 48281 (time 1718398662.33 total 174.25 last 271.51) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 179.25 seconds: 277.25 ops/second - /unlink 5989 (time 1718398678.32 total 10.00 last 598.81) - /unlink 10000 (time 1718398686.51 total 18.20 last 489.42) - /unlink 15227 (time 1718398696.52 total 28.20 last 522.63) - /unlink 20000 (time 1718398706.16 total 37.84 last 494.80) - /unlink 25098 (time 1718398716.16 total 47.85 last 509.75) - /unlink 30000 (time 1718398724.88 total 56.57 last 562.11) - /unlink 35751 (time 1718398734.88 total 66.57 last 575.05) - /unlink 40000 (time 1718398742.36 total 74.04 last 568.58) - /unlink 45492 (time 1718398752.36 total 84.04 last 549.20) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 92.77 seconds: 535.71 ops/second - open/close 2732 (time 1718398772.19 total 10.00 last 273.16) - open/close 6010 (time 1718398782.19 total 20.00 last 327.71) - open/close 8798 (time 1718398792.19 total 30.01 last 278.76) - open/close 10000 (time 1718398795.94 total 33.75 last 320.97) - open/close 13322 (time 1718398805.94 total 43.75 last 332.17) - open/close 16839 (time 1718398815.94 total 53.75 last 351.69) - open/close 19982 (time 1718398825.94 total 63.75 last 314.26) - open/close 23075 (time 1718398835.94 total 73.75 last 309.27) - open/close 26118 (time 1718398845.94 total 83.76 last 304.23) - open/close 29036 (time 1718398855.95 total 93.76 last 291.73) - open/close 30000 (time 1718398859.07 total 96.88 last 308.92) - open/close 33122 (time 1718398869.07 total 106.88 last 312.12) - open/close 36274 (time 1718398879.07 total 116.88 last 315.12) - open/close 39903 (time 1718398889.07 total 126.88 last 362.88) - open/close 43639 (time 1718398899.07 total 136.89 last 373.58) - open/close 47300 (time 1718398909.07 total 146.89 last 366.08) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 153.37 seconds: 324.05 ops/second - /unlink 6936 (time 1718398926.53 total 10.00 last 693.59) - /unlink 10000 (time 1718398931.31 total 14.78 last 640.91) - /unlink 16154 (time 1718398941.31 total 24.78 last 615.34) - /unlink 20000 (time 1718398947.23 total 30.70 last 650.18) - /unlink 26925 (time 1718398957.23 total 40.70 last 692.47) - /unlink 30000 (time 1718398961.00 total 44.47 last 814.61) - /unlink 37025 (time 1718398971.00 total 54.47 last 702.43) - /unlink 40000 (time 1718398975.22 total 58.69 last 705.01) - /unlink 46298 (time 1718398985.22 total 68.70 last 629.65) - /unlink 48514 (time 1718398995.23 total 78.70 last 221.56) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 83.45 seconds: 595.55 ops/second - open/close 1194 (time 1718399011.84 total 10.00 last 119.38) - open/close 2354 (time 1718399021.84 total 20.00 last 115.99) - open/close 3854 (time 1718399031.84 total 30.00 last 149.99) - open/close 5567 (time 1718399041.84 total 40.00 last 171.30) - open/close 7374 (time 1718399051.84 total 50.01 last 180.65) - open/close 8919 (time 1718399061.86 total 60.02 last 154.30) - open/close 10000 (time 1718399069.62 total 67.78 last 139.26) - open/close 11375 (time 1718399079.62 total 77.78 last 137.45) - open/close 12840 (time 1718399089.63 total 87.79 last 146.42) - open/close 14353 (time 1718399099.63 total 97.79 last 151.26) - open/close 16305 (time 1718399109.63 total 107.79 last 195.16) - open/close 18042 (time 1718399119.64 total 117.80 last 173.65) - open/close 19572 (time 1718399129.64 total 127.80 last 153.00) - open/close 20000 (time 1718399132.34 total 130.51 last 158.00) - open/close 21663 (time 1718399142.35 total 140.51 last 166.27) - open/close 23130 (time 1718399152.36 total 150.52 last 146.54) - open/close 24618 (time 1718399162.36 total 160.52 last 148.75) - open/close 26436 (time 1718399172.36 total 170.52 last 181.79) - open/close 27988 (time 1718399182.36 total 180.52 last 155.18) - open/close 29922 (time 1718399192.36 total 190.53 last 193.37) - open/close 31628 (time 1718399202.37 total 200.53 last 170.47) - open/close 32888 (time 1718399212.38 total 210.54 last 125.94) - open/close 34097 (time 1718399222.38 total 220.54 last 120.89) - open/close 35316 (time 1718399232.38 total 230.54 last 121.89) - open/close 36569 (time 1718399242.38 total 240.54 last 125.25) - open/close 37893 (time 1718399252.39 total 250.55 last 132.30) - open/close 39375 (time 1718399262.39 total 260.55 last 148.19) - open/close 40000 (time 1718399268.05 total 266.21 last 110.47) - open/close 41181 (time 1718399278.06 total 276.22 last 118.01) - open/close 42403 (time 1718399288.06 total 286.22 last 122.13) - open/close 43728 (time 1718399298.07 total 296.23 last 132.41) - open/close 44953 (time 1718399308.07 total 306.23 last 122.45) - open/close 46072 (time 1718399318.07 total 316.24 last 111.87) - open/close 47415 (time 1718399328.10 total 326.26 last 133.96) - open/close 48782 (time 1718399338.10 total 336.26 last 136.69) open(/mnt/lustre/d106.conf-sanity/f-49698) error: No space left on device total: 49698 open/close in 341.54 seconds: 145.51 ops/second - /unlink 2991 (time 1718399355.19 total 10.00 last 298.96) - /unlink 7096 (time 1718399365.19 total 20.01 last 410.48) - /unlink 10000 (time 1718399373.83 total 28.65 last 336.09) - /unlink 13246 (time 1718399383.83 total 38.65 last 324.58) - /unlink 16144 (time 1718399393.83 total 48.65 last 289.75) - /unlink 18902 (time 1718399403.83 total 58.65 last 275.72) - /unlink 20000 (time 1718399408.65 total 63.47 last 227.77) - /unlink 22506 (time 1718399418.65 total 73.47 last 250.59) - /unlink 24873 (time 1718399428.65 total 83.47 last 236.68) - /unlink 27133 (time 1718399438.65 total 93.47 last 225.97) - /unlink 29405 (time 1718399448.66 total 103.48 last 227.16) - /unlink 30000 (time 1718399450.91 total 105.73 last 264.02) - /unlink 32428 (time 1718399460.91 total 115.73 last 242.74) - /unlink 34939 (time 1718399470.92 total 125.73 last 251.02) - /unlink 37149 (time 1718399480.92 total 135.73 last 221.00) - /unlink 39587 (time 1718399490.92 total 145.74 last 243.76) - /unlink 42229 (time 1718399500.92 total 155.74 last 264.19) - /unlink 44929 (time 1718399510.92 total 165.74 last 269.90) - /unlink 47694 (time 1718399520.92 total 175.74 last 276.40) unlink(/mnt/lustre/d106.conf-sanity/f-49698) error: No such file or directory total: 49698 /unlink in 181.01 seconds: 274.56 ops/second - open/close 2127 (time 1718399537.33 total 10.00 last 212.68) - open/close 4069 (time 1718399547.33 total 20.00 last 194.19) - open/close 5803 (time 1718399557.34 total 30.00 last 173.37) - open/close 7575 (time 1718399567.34 total 40.00 last 177.19) - open/close 9358 (time 1718399577.34 total 50.00 last 178.29) - open/close 10000 (time 1718399580.68 total 53.35 last 192.01) - open/close 11863 (time 1718399590.68 total 63.35 last 186.28) - open/close 13675 (time 1718399600.68 total 73.35 last 181.16) - open/close 15383 (time 1718399610.69 total 83.35 last 170.78) - open/close 17069 (time 1718399620.69 total 93.36 last 168.55) - open/close 19012 (time 1718399630.69 total 103.36 last 194.29) - open/close 20000 (time 1718399635.56 total 108.23 last 202.90) - open/close 22289 (time 1718399645.56 total 118.23 last 228.88) - open/close 25064 (time 1718399655.56 total 128.23 last 277.45) - open/close 29183 (time 1718399665.56 total 138.23 last 411.83) - open/close 33394 (time 1718399675.57 total 148.23 last 420.98) - open/close 37477 (time 1718399685.57 total 158.23 last 408.29) - open/close 40000 (time 1718399691.39 total 164.05 last 433.40) - open/close 44638 (time 1718399701.39 total 174.06 last 463.74) - open/close 48754 (time 1718399711.39 total 184.06 last 411.60) open(/mnt/lustre/d106.conf-sanity/f-49470) error: No space left on device total: 49470 open/close in 185.84 seconds: 266.20 ops/second - /unlink 8130 (time 1718399724.12 total 10.00 last 812.96) - /unlink 10000 (time 1718399726.35 total 12.24 last 836.31) - /unlink 18124 (time 1718399736.37 total 22.25 last 811.12) - /unlink 20000 (time 1718399738.49 total 24.37 last 885.75) - /unlink 29832 (time 1718399748.49 total 34.37 last 983.20) - /unlink 39519 (time 1718399758.49 total 44.37 last 968.61) - /unlink 49226 (time 1718399768.49 total 54.37 last 970.66) unlink(/mnt/lustre/d106.conf-sanity/f-49470) error: No such file or directory total: 49470 /unlink in 54.66 seconds: 904.99 ops/second - open/close 4705 (time 1718399779.72 total 10.00 last 470.45) - open/close 9044 (time 1718399789.72 total 20.00 last 433.87) - open/close 14215 (time 1718399799.72 total 30.00 last 517.03) - open/close 18831 (time 1718399809.72 total 40.00 last 461.52) - open/close 20000 (time 1718399812.11 total 42.40 last 488.91) - open/close 25612 (time 1718399822.11 total 52.40 last 561.13) - open/close 30000 (time 1718399830.35 total 60.63 last 532.89) - open/close 35469 (time 1718399840.35 total 70.63 last 546.86) - open/close 40000 (time 1718399848.50 total 78.78 last 555.99) - open/close 45995 (time 1718399858.50 total 88.78 last 599.44) open(/mnt/lustre/d106.conf-sanity/f-49470) error: No space left on device total: 49470 open/close in 94.39 seconds: 524.11 ops/second - /unlink 10000 (time 1718399873.97 total 9.04 last 1106.45) - /unlink 20000 (time 1718399883.12 total 18.19 last 1093.13) - /unlink 29863 (time 1718399893.12 total 28.19 last 986.29) - /unlink 39374 (time 1718399903.12 total 38.19 last 951.04) unlink(/mnt/lustre/d106.conf-sanity/f-49470) error: No such file or directory total: 49470 /unlink in 46.44 seconds: 1065.35 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg234-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg234-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg234-server LNET ready to unload unloading modules on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing unload_modules_local oleg234-server: LNET ready to unload modules unloaded. pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2