[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 368529704 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.442421] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.445544] pid_max: default: 32768 minimum: 301 [ 0.447136] Security Framework initialized [ 0.448545] SELinux: Initializing. [ 0.449838] SELinux: Starting in permissive mode [ 0.451294] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.455760] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.459257] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.461358] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.464656] Initializing cgroup subsys memory [ 0.466162] Initializing cgroup subsys devices [ 0.467632] Initializing cgroup subsys freezer [ 0.469272] Initializing cgroup subsys net_cls [ 0.470707] Initializing cgroup subsys blkio [ 0.472120] Initializing cgroup subsys perf_event [ 0.473618] Initializing cgroup subsys hugetlb [ 0.475528] Initializing cgroup subsys pids [ 0.476866] Initializing cgroup subsys net_prio [ 0.478497] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.481202] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.483204] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.484833] tlb_flushall_shift: 6 [ 0.485905] FEATURE SPEC_CTRL Present [ 0.487365] FEATURE IBPB_SUPPORT Present [ 0.489442] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.491858] Spectre V2 : Vulnerable [ 0.493760] Speculative Store Bypass: Vulnerable [ 0.495765] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.503848] ACPI: Core revision 20130517 [ 0.506963] ACPI: All ACPI Tables successfully acquired [ 0.508901] ftrace: allocating 30294 entries in 119 pages [ 0.565522] Enabling x2apic [ 0.566431] Enabled x2apic [ 0.567663] Switched APIC routing to physical x2apic. [ 0.570860] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.572670] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.575837] TSC deadline timer enabled [ 0.576041] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.578444] ... version: 2 [ 0.579562] ... bit width: 48 [ 0.580697] ... generic registers: 4 [ 0.581978] ... value mask: 0000ffffffffffff [ 0.583271] ... max period: 00007fffffffffff [ 0.584880] ... fixed-purpose events: 3 [ 0.586156] ... event mask: 000000070000000f [ 0.588090] KVM setup paravirtual spinlock [ 0.592714] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.595886] KVM setup async PF for cpu 1 [ 0.596875] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.600039] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.603995] KVM setup async PF for cpu 2 [ 0.591414] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.605123] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.606606] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.608919] Brought up 4 CPUs [ 0.608989] KVM setup async PF for cpu 3 [ 0.608996] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.611889] smpboot: Max logical packages: 1 [ 0.613306] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.618488] devtmpfs: initialized [ 0.619469] x86/mm: Memory block size: 128MB [ 0.624038] EVM: security.selinux [ 0.624823] EVM: security.ima [ 0.625777] EVM: security.capability [ 0.628951] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.631633] NET: Registered protocol family 16 [ 0.633254] cpuidle: using governor haltpoll [ 0.635478] ACPI: bus type PCI registered [ 0.636640] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.638780] PCI: Using configuration type 1 for base access [ 0.640471] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.651054] ACPI: Added _OSI(Module Device) [ 0.652737] ACPI: Added _OSI(Processor Device) [ 0.654272] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.658248] ACPI: Added _OSI(Processor Aggregator Device) [ 0.661456] ACPI: Added _OSI(Linux-Dell-Video) [ 0.664228] ACPI: EC: Look up EC in DSDT [ 0.667639] ACPI: Interpreter enabled [ 0.669347] ACPI: (supports S0 S3 S4 S5) [ 0.670724] ACPI: Using IOAPIC for interrupt routing [ 0.672941] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.676770] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.685007] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.687207] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.689184] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.691033] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.695383] acpiphp: Slot [2] registered [ 0.696668] acpiphp: Slot [3] registered [ 0.698694] acpiphp: Slot [4] registered [ 0.700137] acpiphp: Slot [5] registered [ 0.701348] acpiphp: Slot [6] registered [ 0.703991] acpiphp: Slot [7] registered [ 0.706128] acpiphp: Slot [8] registered [ 0.707763] acpiphp: Slot [9] registered [ 0.709200] acpiphp: Slot [10] registered [ 0.710436] acpiphp: Slot [11] registered [ 0.711570] acpiphp: Slot [12] registered [ 0.712807] acpiphp: Slot [13] registered [ 0.714460] acpiphp: Slot [14] registered [ 0.715749] acpiphp: Slot [15] registered [ 0.717393] acpiphp: Slot [16] registered [ 0.718581] acpiphp: Slot [17] registered [ 0.719613] acpiphp: Slot [18] registered [ 0.720777] acpiphp: Slot [19] registered [ 0.722241] acpiphp: Slot [20] registered [ 0.723358] acpiphp: Slot [21] registered [ 0.724429] acpiphp: Slot [22] registered [ 0.725754] acpiphp: Slot [23] registered [ 0.726823] acpiphp: Slot [24] registered [ 0.727730] acpiphp: Slot [25] registered [ 0.729193] acpiphp: Slot [26] registered [ 0.730471] acpiphp: Slot [27] registered [ 0.731277] acpiphp: Slot [28] registered [ 0.732599] acpiphp: Slot [29] registered [ 0.733807] acpiphp: Slot [30] registered [ 0.735172] acpiphp: Slot [31] registered [ 0.736440] PCI host bridge to bus 0000:00 [ 0.737496] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.739533] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.741174] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.743102] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.745191] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.747584] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.749666] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.750398] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.751416] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.758498] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.762155] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.765343] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.767575] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.770603] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.774520] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.775391] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.779368] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.782498] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.785061] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.796118] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.798087] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.799211] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.804291] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.808755] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.830747] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.839414] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.845169] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.849927] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.869989] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.881707] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.889346] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.893984] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.909619] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.914089] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.916670] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.921451] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.941963] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.948214] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.952254] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.959268] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.972867] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.977596] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.984448] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.989036] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.008973] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 1.018231] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 1.022011] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 1.024257] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 1.026205] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 1.028665] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 1.032441] vgaarb: loaded [ 1.034135] SCSI subsystem initialized [ 1.035878] ACPI: bus type USB registered [ 1.038601] usbcore: registered new interface driver usbfs [ 1.040937] usbcore: registered new interface driver hub [ 1.042985] usbcore: registered new device driver usb [ 1.045541] PCI: Using ACPI for IRQ routing [ 1.046836] PCI: pci_cache_line_size set to 64 bytes [ 1.047121] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 1.047127] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 1.047129] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 1.047490] NetLabel: Initializing [ 1.048619] NetLabel: domain hash size = 128 [ 1.049812] NetLabel: protocols = UNLABELED CIPSOv4 [ 1.051600] NetLabel: unlabeled traffic allowed by default [ 1.055284] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.057287] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.061927] amd_nb: Cannot enumerate AMD northbridges [ 1.072212] Switched to clocksource kvm-clock [ 1.097083] pnp: PnP ACPI init [ 1.098150] ACPI: bus type PNP registered [ 1.099555] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 1.099643] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 1.099687] pnp 00:02: [dma 2] [ 1.099721] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 1.099802] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.099879] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.099956] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.100373] pnp: PnP ACPI: found 6 devices [ 1.101800] ACPI: bus type PNP unregistered [ 1.119585] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.119591] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.119595] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.119597] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.119600] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 1.119676] NET: Registered protocol family 2 [ 1.121600] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 1.123946] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.126741] TCP: Hash tables configured (established 32768 bind 32768) [ 1.128565] TCP: reno registered [ 1.129563] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.131848] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.134262] NET: Registered protocol family 1 [ 1.136323] RPC: Registered named UNIX socket transport module. [ 1.138507] RPC: Registered udp transport module. [ 1.140351] RPC: Registered tcp transport module. [ 1.141985] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.146044] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.148203] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.149620] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.151459] PCI: CLS 0 bytes, default 64 [ 1.151626] Unpacking initramfs... [ 2.541314] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 2.545365] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.547368] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 2.550367] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 2.552428] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 2.554162] RAPL PMU: hw unit of domain package 2^-0 Joules [ 2.556104] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 2.560230] cryptomgr_test (50) used greatest stack depth: 14480 bytes left [ 2.560961] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 2.561033] Initialise system trusted keyring [ 2.592431] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 2.595038] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 2.601476] zpool: loaded [ 2.602541] zbud: loaded [ 2.603791] VFS: Disk quotas dquot_6.6.0 [ 2.605390] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.608367] NFS: Registering the id_resolver key type [ 2.610423] Key type id_resolver registered [ 2.611876] Key type id_legacy registered [ 2.613450] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 2.616279] Key type big_key registered [ 2.617683] SELinux: Registering netfilter hooks [ 2.621302] cryptomgr_test (57) used greatest stack depth: 14048 bytes left [ 2.624236] cryptomgr_test (58) used greatest stack depth: 13968 bytes left [ 2.629079] NET: Registered protocol family 38 [ 2.630657] Key type asymmetric registered [ 2.631926] Asymmetric key parser 'x509' registered [ 2.634025] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 2.636882] io scheduler noop registered [ 2.638348] io scheduler deadline registered (default) [ 2.640170] io scheduler cfq registered [ 2.641454] io scheduler mq-deadline registered [ 2.643084] io scheduler kyber registered [ 2.646191] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 2.648053] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 2.650397] intel_idle: does not run on family 6 model 62 [ 2.650547] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.653172] ACPI: Power Button [PWRF] [ 2.654961] GHES: HEST is not enabled! [ 2.715443] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 2.779451] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 2.905710] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 2.964879] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 3.083360] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.113559] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 3.143714] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 3.149198] Non-volatile memory driver v1.3 [ 3.150584] Linux agpgart interface v0.103 [ 3.153192] crash memory driver: version 1.1 [ 3.155387] nbd: registered device at major 43 [ 3.160296] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 3.160343] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 3.160378] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 3.160421] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 3.160463] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 3.167613] virtio_blk virtio1: [vda] 53408 512-byte logical blocks (27.3 MB/26.0 MiB) [ 3.175014] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 3.175049] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 3.175106] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 3.175149] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 3.175192] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 3.181526] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.189118] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 3.189178] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 3.189218] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 3.189261] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 3.189307] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 3.195560] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.206002] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 3.206048] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 3.206099] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 3.206135] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 3.206182] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 3.216078] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.223462] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 3.223519] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 3.223566] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 3.223608] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 3.223659] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 3.230816] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.237275] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 3.237317] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 3.237361] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 3.237402] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 3.237438] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 3.243735] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.249950] rdac: device handler registered [ 3.250953] hp_sw: device handler registered [ 3.252083] emc: device handler registered [ 3.253377] libphy: Fixed MDIO Bus: probed [ 3.257787] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 3.259437] ehci-pci: EHCI PCI platform driver [ 3.261426] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 3.263451] ohci-pci: OHCI PCI platform driver [ 3.265251] uhci_hcd: USB Universal Host Controller Interface driver [ 3.267822] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.271676] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.273497] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.275745] mousedev: PS/2 mouse device common for all mice [ 3.278373] rtc_cmos 00:05: RTC can wake from S4 [ 3.280828] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 3.283252] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.283291] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.289749] hidraw: raw HID events driver (C) Jiri Kosina [ 3.292136] usbcore: registered new interface driver usbhid [ 3.293852] usbhid: USB HID core driver [ 3.295122] drop_monitor: Initializing network drop monitor service [ 3.298718] Netfilter messages via NETLINK v0.30. [ 3.300836] TCP: cubic registered [ 3.302017] Initializing XFRM netlink socket [ 3.303605] NET: Registered protocol family 10 [ 3.306007] NET: Registered protocol family 17 [ 3.307548] Key type dns_resolver registered [ 3.309841] mce: Using 10 MCE banks [ 3.311868] Loading compiled-in X.509 certificates [ 3.314368] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 3.318348] registered taskstats version 1 [ 3.322517] modprobe (72) used greatest stack depth: 13456 bytes left [ 3.327423] Key type trusted registered [ 3.331943] Key type encrypted registered [ 3.333684] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 3.337651] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 3.342136] rtc_cmos 00:05: setting system clock to 2025-01-15 23:12:27 UTC (1736982747) [ 3.345483] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 3.349000] Write protecting the kernel read-only data: 12288k [ 3.351721] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 3.354438] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 3.365821] random: systemd: uninitialized urandom read (16 bytes read) [ 3.369407] random: systemd: uninitialized urandom read (16 bytes read) [ 3.371884] random: systemd: uninitialized urandom read (16 bytes read) [ 3.376418] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 3.382533] systemd[1]: Detected virtualization kvm. [ 3.384051] systemd[1]: Detected architecture x86-64. [ 3.385689] systemd[1]: Running in initial RAM disk. [ 3.390209] systemd[1]: No hostname configured. [ 3.392098] systemd[1]: Set hostname to . [ 3.393824] random: systemd: uninitialized urandom read (16 bytes read) [ 3.395948] systemd[1]: Initializing machine ID from random generator. [ 3.445928] dracut-rootfs-g (86) used greatest stack depth: 13264 bytes left [ 3.451480] random: systemd: uninitialized urandom read (16 bytes read) [ 3.455212] random: systemd: uninitialized urandom read (16 bytes read) [ 3.458296] random: systemd: uninitialized urandom read (16 bytes read) [ 3.460806] random: systemd: uninitialized urandom read (16 bytes read) [ 3.464578] random: systemd: uninitialized urandom read (16 bytes read) [ 3.467499] random: systemd: uninitialized urandom read (16 bytes read) [ 3.478555] systemd[1]: Reached target Swap. [ 3.483047] systemd[1]: Created slice Root Slice. [ 3.488023] systemd[1]: Listening on udev Kernel Socket. [ 3.493156] systemd[1]: Created slice System Slice. [ 3.497662] systemd[1]: Listening on Journal Socket. [ 3.503636] systemd[1]: Starting Load Kernel Modules... [ 3.507853] systemd[1]: Listening on udev Control Socket. [ 3.512268] systemd[1]: Reached target Timers. [ 3.516113] systemd[1]: Reached target Slices. [ 3.523100] systemd[1]: Starting Journal Service... [ 3.528259] systemd[1]: Starting Setup Virtual Console... [ 3.532780] systemd[1]: Starting dracut cmdline hook... [ 3.536960] systemd[1]: Reached target Sockets. [ 3.540382] systemd[1]: Reached target Local File Systems. [ 3.546127] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 3.554490] systemd[1]: Started Load Kernel Modules. [ 3.560035] tsc: Refined TSC clocksource calibration: 2399.993 MHz [ 3.566080] systemd[1]: Started Setup Virtual Console. [ 3.571877] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 3.577885] systemd[1]: Started Journal Service. [ 3.723211] random: fast init done [ 3.913993] rpc.statd (219) used greatest stack depth: 13008 bytes left [ 4.136030] libata version 3.00 loaded. [ 4.139828] ata_piix 0000:00:01.1: version 2.13 [ 4.162150] scsi host0: ata_piix [ 4.164864] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 4.175153] scsi host1: ata_piix [ 4.177474] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.183285] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 4.472818] ip (343) used greatest stack depth: 12280 bytes left [ 10.784349] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 11.413699] systemd-journald[102]: Received SIGTERM from PID 1 (systemd). [ 11.778633] SELinux: Disabled at runtime. [ 11.781680] SELinux: Unregistering netfilter hooks [ 11.892145] ip_tables: (C) 2000-2006 Netfilter Core Team [ 11.898018] systemd[1]: Inserted module 'ip_tables' [ 12.811659] systemd-journald[566]: Received request to flush runtime journal from PID 1 [ 13.082701] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 13.113257] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.195018] cryptd: max_cpu_qlen set to 1000 [ 13.261671] AVX version of gcm_enc/dec engaged. [ 13.263334] AES CTR mode by8 optimization enabled [ 13.314477] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 13.328592] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 13.339914] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 13.344382] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 13.495051] EDAC MC: Ver: 3.0.0 [ 13.509899] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 13.509908] EDAC sbridge: Ver: 1.1.2 [ 16.631565] mount.nfs (771) used greatest stack depth: 10704 bytes left [ 25.472454] device-mapper: uevent: version 1.0.3 [ 25.475386] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 35.905570] libcfs: loading out-of-tree module taints kernel. [ 35.907798] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 35.932936] alg: No test for adler32 (adler32-zlib) [ 36.727309] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_hostid [ 45.197743] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 45.748412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 46.032862] Lustre: Lustre: Build Version: 2.15.6_2_gda5530f [ 46.355607] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 46.357761] LNet: Accept secure, port 988 [ 47.964096] Key type lgssc registered [ 48.706280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 54.554363] icp: module license 'CDDL' taints kernel. [ 54.560904] Disabling lock debugging due to kernel taint [ 56.666171] ZFS: Loaded module v2.1.5-1, ZFS pool version 5000, ZFS filesystem version 5 [ 61.211608] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 69.236002] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 73.472474] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 73.484578] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.778031] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 77.790232] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 83.786482] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 89.181801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 89.225377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 90.344914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 90.360335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 90.395844] Lustre: lustre-MDT0000: new disk, initializing [ 90.442026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 90.454306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 90.526288] mount.lustre (6849) used greatest stack depth: 10064 bytes left [ 92.124825] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 98.934665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 98.984591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 99.028662] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 99.045839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 99.048929] Lustre: Skipped 1 previous similar message [ 99.092489] Lustre: lustre-MDT0001: new disk, initializing [ 99.123152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 99.139013] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 99.143366] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 100.760555] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 107.295743] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 107.306944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 107.343059] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 107.351655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 107.355722] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 107.442268] Lustre: lustre-OST0000: new disk, initializing [ 107.449446] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 107.474091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 109.214917] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 114.365174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 114.385427] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 114.394029] random: crng init done [ 116.022054] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 116.031666] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 116.070853] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 116.077509] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 116.081664] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 116.123892] Lustre: lustre-OST0001: new disk, initializing [ 116.128461] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 116.163338] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 117.678155] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 119.976143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 119.983750] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 123.423840] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 127.946436] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 134.292931] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing check_logdir /tmp/testlogs/ [ 136.262650] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing yml_node [ 138.442688] Lustre: DEBUG MARKER: Client: 2.15.6.2 [ 139.652661] Lustre: DEBUG MARKER: MDS: 2.15.6.2 [ 140.770543] Lustre: DEBUG MARKER: OSS: 2.15.6.2 [ 141.594515] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity ============----- Wed Jan 15 18:14:45 EST 2025 [ 144.729079] Lustre: DEBUG MARKER: excepting tests: 225 255 256 400a 42a 42b 42c 407 118c 118d [ 145.368935] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 60i 64b 68 71 115 135 136 230d 300o [ 147.611195] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing check_config_client /mnt/lustre [ 155.263476] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 156.449947] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 157.505864] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 160.340092] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 18:15:04 (1736982904) [ 161.829514] Lustre: DEBUG MARKER: SKIP: sanity test_60a missing subtest run-llog.sh [ 162.668168] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 18:15:06 (1736982906) [ 166.759016] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ============ 18:15:10 (1736982910) [ 224.896094] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 18:16:08 (1736982968) [ 228.131515] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================================== 18:16:11 (1736982971) [ 228.729108] Lustre: *** cfs_fail_loc=15b, val=0*** [ 228.731252] Lustre: *** cfs_fail_loc=15b, val=0*** [ 231.664830] Lustre: DEBUG MARKER: == sanity test 60f: change debug_path works ============== 18:16:15 (1736982975) [ 234.929856] Lustre: DEBUG MARKER: == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 18:16:18 (1736982978) [ 235.508766] Lustre: *** cfs_fail_loc=19a, val=0*** [ 235.514035] LustreError: 8103:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0001-osd: fail to cancel 1 llog-records: rc = -5 [ 235.521824] LustreError: 8103:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0001-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 236.331692] Lustre: *** cfs_fail_loc=19a, val=0*** [ 236.333269] Lustre: Skipped 1 previous similar message [ 236.337380] LustreError: 6900:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osd: fail to cancel 1 llog-records: rc = -5 [ 236.345495] LustreError: 6900:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message [ 236.352837] LustreError: 6900:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 236.362728] LustreError: 6900:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1 previous similar message [ 237.599771] Lustre: *** cfs_fail_loc=19a, val=0*** [ 237.602169] Lustre: Skipped 2 previous similar messages [ 237.603998] LustreError: 6900:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osd: fail to cancel 1 llog-records: rc = -5 [ 237.613643] LustreError: 6900:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message [ 237.620779] LustreError: 6900:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 237.629811] LustreError: 6900:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1 previous similar message [ 239.842606] Lustre: *** cfs_fail_loc=19a, val=0*** [ 239.844494] Lustre: Skipped 4 previous similar messages [ 239.849343] LustreError: 6900:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osd: fail to cancel 1 llog-records: rc = -5 [ 239.861160] LustreError: 6900:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message [ 239.869828] LustreError: 6900:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 239.885052] LustreError: 6900:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 1 previous similar message [ 243.908640] Lustre: *** cfs_fail_loc=19a, val=0*** [ 243.910686] Lustre: Skipped 8 previous similar messages [ 244.807269] LustreError: 6900:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osd: fail to cancel 1 llog-records: rc = -5 [ 244.812040] LustreError: 6900:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 2 previous similar messages [ 244.818750] LustreError: 6900:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 244.824762] LustreError: 6900:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 2 previous similar messages [ 252.073488] Lustre: *** cfs_fail_loc=19a, val=0*** [ 252.075083] Lustre: Skipped 18 previous similar messages [ 253.315183] LustreError: 6900:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 253.320184] LustreError: 6900:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 9 previous similar messages [ 253.324405] LustreError: 6900:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 253.332313] LustreError: 6900:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 9 previous similar messages [ 268.220063] Lustre: *** cfs_fail_loc=19a, val=0*** [ 268.226745] Lustre: Skipped 39 previous similar messages [ 269.907493] LustreError: 6900:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) lustre-MDT0000-osd: fail to cancel 1 llog-records: rc = -5 [ 269.913229] LustreError: 6900:0:(llog_cat.c:753:llog_cat_cancel_arr_rec()) Skipped 13 previous similar messages [ 269.922892] LustreError: 6900:0:(llog_cat.c:789:llog_cat_cancel_records()) lustre-MDT0000-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 269.933919] LustreError: 6900:0:(llog_cat.c:789:llog_cat_cancel_records()) Skipped 13 previous similar messages [ 283.336356] Lustre: DEBUG MARKER: == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 18:17:07 (1736983027) [ 283.756332] Lustre: *** cfs_fail_loc=188, val=0*** [ 285.131615] Lustre: *** cfs_fail_loc=189, val=0*** [ 288.546636] Lustre: DEBUG MARKER: SKIP: sanity test_60i skipping SLOW test 60i [ 289.224405] Lustre: DEBUG MARKER: == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 18:17:12 (1736983032) [ 292.401947] Lustre: DEBUG MARKER: == sanity test 61b: mmap() of unstriped file is successful ========================================================== 18:17:15 (1736983035) [ 295.294911] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 18:17:18 (1736983038) [ 360.101264] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ============================================================= 18:18:23 (1736983103) [ 367.987544] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 18:18:31 (1736983111) [ 372.392768] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 373.510684] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ================== 18:18:36 (1736983116) [ 377.432156] Lustre: DEBUG MARKER: == sanity test 64d: check grant limit exceed ============= 18:18:40 (1736983120) [ 413.024728] Lustre: DEBUG MARKER: == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 18:19:16 (1736983156) [ 414.910891] Lustre: *** cfs_fail_loc=725, val=0*** [ 417.240736] Lustre: *** cfs_fail_loc=725, val=0*** [ 421.470708] Lustre: DEBUG MARKER: == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 18:19:24 (1736983164) [ 427.066933] Lustre: DEBUG MARKER: == sanity test 64g: grant shrink on MDT ================== 18:19:30 (1736983170) [ 467.820287] Lustre: DEBUG MARKER: == sanity test 64h: grant shrink on read ================= 18:20:11 (1736983211) [ 480.563934] Lustre: DEBUG MARKER: == sanity test 64i: shrink on reconnect ================== 18:20:24 (1736983224) [ 483.356641] Lustre: *** cfs_fail_loc=513, val=0*** [ 483.358357] LustreError: 15817:0:(service.c:2115:ptlrpc_server_handle_req_in()) drop incoming rpc opc 17, x1821358460074048 [ 484.016635] Lustre: Failing over lustre-OST0000 [ 484.042372] Lustre: server umount lustre-OST0000 complete [ 485.030858] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 485.037274] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.043556] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.796839] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.803876] LustreError: Skipped 1 previous similar message [ 490.584865] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.595017] LustreError: Skipped 1 previous similar message [ 493.591649] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 493.597528] LustreError: Skipped 1 previous similar message [ 497.054550] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 497.059694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 497.138158] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 498.336826] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 498.750968] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 499.594727] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.102@tcp (at 0@lo) [ 499.594733] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 499.597336] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:3 to 0x280000400:33 [ 499.610736] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2519 to 0x0:2561 [ 502.082864] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 502.602621] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 506.876687] Lustre: DEBUG MARKER: == sanity test 65a: directory with no stripe info ======== 18:20:50 (1736983250) [ 509.723359] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 18:20:53 (1736983253) [ 512.710854] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 18:20:56 (1736983256) [ 515.366270] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 18:20:59 (1736983259) [ 517.983304] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========= 18:21:01 (1736983261) [ 520.939322] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ============================================================= 18:21:04 (1736983264) [ 523.897036] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =============== 18:21:07 (1736983267) [ 527.378698] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ============================================================================== 18:21:11 (1736983271) [ 530.124071] Lustre: DEBUG MARKER: == sanity test 65i: various tests to set root directory striping ========================================================== 18:21:13 (1736983273) [ 533.435430] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 18:21:17 (1736983277) [ 537.346714] Lustre: DEBUG MARKER: == sanity test 65k: validate manual striping works properly with deactivated OSCs ========================================================== 18:21:20 (1736983280) [ 538.268564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 538.275670] Lustre: Skipped 1 previous similar message [ 538.281080] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 538.287100] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.102@tcp (at 0@lo) [ 538.287459] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:3 to 0x280000400:65 [ 538.303298] Lustre: Skipped 1 previous similar message [ 538.705820] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2564 to 0x0:2593 [ 539.073257] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 539.078320] Lustre: Skipped 1 previous similar message [ 539.082042] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:3 to 0x2c0000400:33 [ 539.447166] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.452323] Lustre: Skipped 2 previous similar messages [ 539.455710] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 192.168.203.102@tcp (at 0@lo) [ 539.457612] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2514 to 0x0:2529 [ 539.463132] Lustre: Skipped 2 previous similar messages [ 547.557436] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 553.572923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.579328] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 553.582350] Lustre: Skipped 1 previous similar message [ 553.584721] LustreError: 167-0: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 553.589714] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.102@tcp (at 0@lo) [ 553.590903] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:3 to 0x280000400:97 [ 554.878187] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 554.956848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 556.250873] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 556.407162] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 562.610743] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 568.174896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 568.180824] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 568.183788] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 568.188270] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.102@tcp (at 0@lo) [ 568.188462] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3594 to 0x0:3617 [ 569.423520] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 [ 569.488134] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 570.719835] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 [ 570.778313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 577.591876] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 582.506952] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 582.512004] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 582.516050] LustreError: 167-0: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 582.522206] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 192.168.203.102@tcp (at 0@lo) [ 582.523401] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:35 to 0x2c0000400:65 [ 583.660811] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 583.727900] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 584.845132] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 584.905200] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 590.056005] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 595.260058] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 595.263712] LustreError: 167-0: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 595.269878] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3532 to 0x0:3553 [ 596.474642] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 40 [ 596.541726] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 597.818963] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 40 [ 597.892920] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 599.998583] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 18:22:23 (1736983343) [ 601.950600] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 18:22:25 (1736983345) [ 603.986248] Lustre: DEBUG MARKER: == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 18:22:27 (1736983347) [ 618.483925] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client ========================================================================= 18:22:42 (1736983362) [ 622.119535] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 18:22:45 (1736983365) [ 622.735008] Lustre: *** cfs_fail_loc=217, val=0*** [ 623.774093] Lustre: *** cfs_fail_loc=217, val=0*** [ 623.776245] Lustre: Skipped 1 previous similar message [ 626.014482] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 626.549318] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 18:22:50 (1736983370) [ 628.713277] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 18:22:52 (1736983372) [ 631.060793] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 18:22:54 (1736983374) [ 659.155458] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 18:23:23 (1736983403) [ 661.156443] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 18:23:25 (1736983405) [ 663.271668] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 18:23:27 (1736983407) [ 665.092900] Lustre: DEBUG MARKER: == sanity test 76a: confirm clients recycle inodes properly ============================================================== 18:23:29 (1736983409) [ 680.031457] Lustre: DEBUG MARKER: == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 18:23:43 (1736983423) [ 692.359545] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ========================================================== 18:23:56 (1736983436) [ 694.992354] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ========================================================== 18:23:58 (1736983438) [ 695.217684] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.2@tcp inode [0x200000407:0xc15:0x0] object 0x0:3883 extent [0-4194303]: client csum 61eee5, server csum 61eee4 [ 697.035176] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 698.196190] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.2@tcp inode [0x200000407:0xc15:0x0] object 0x0:3883 extent [0-4194303], client returned csum 11be8f5e (type 1), server csum cf5a74ae (type 1) [ 699.117147] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 700.274008] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.2@tcp inode [0x200000407:0xc15:0x0] object 0x0:3883 extent [0-4194303], client returned csum 39179d4d (type 2), server csum 9b259dac (type 2) [ 701.086430] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 702.219742] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.2@tcp inode [0x200000407:0xc15:0x0] object 0x0:3883 extent [0-4194303], client returned csum 14f35887 (type 4), server csum 56573633 (type 4) [ 702.990715] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 704.950147] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 706.082716] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.2@tcp inode [0x200000407:0xc15:0x0] object 0x0:3883 extent [0-4194303], client returned csum 77ee44 (type 20), server csum 61eee4 (type 20) [ 706.090688] LustreError: Skipped 1 previous similar message [ 706.868228] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 708.824192] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 710.686775] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 712.754939] Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read with debug ========================================================== 18:24:16 (1736983456) [ 715.044478] Lustre: 25105:0:(tgt_handler.c:1903:dump_all_bulk_pages()) dumping checksum data to /tmp/lustre-log-checksum_dump-ost-[0x200000407:0xc16:0x0]:[0-1048575]-c7c0f5a1-77df641 [ 715.050518] LustreError: dumping log to /tmp/lustre-log.1736983459.25105 [ 715.069422] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.2@tcp inode [0x200000407:0xc16:0x0] object 0x0:3884 extent [0-1048575], client returned csum c7c0f5a1 (type 20), server csum 77df641 (type 20) [ 715.077166] LustreError: Skipped 2 previous similar messages [ 724.905582] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================================== 18:24:28 (1736983468) [ 725.060817] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-192.168.203.2@tcp inode [0x200000407:0xc18:0x0] object 0x0:3817 extent [0-4194303]: client csum fec05011, server csum fec05010 [ 727.340631] LustreError: 132-0: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 192.168.203.2@tcp inode [0x200000407:0xc18:0x0] object 0x0:3817 extent [0-4194303], client returned csum 872950a1 (type 20), server csum fec05010 (type 20) [ 729.398585] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 18:24:33 (1736983473) [ 729.894465] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 730.041302] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.2@tcp inode [0x200000407:0xc19:0x0] object 0x0:3885 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 733.209032] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.2@tcp inode [0x200000407:0xc19:0x0] object 0x0:3885 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f [ 733.217349] LustreError: Skipped 4 previous similar messages [ 740.333768] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.2@tcp inode [0x200000407:0xc19:0x0] object 0x0:3885 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 740.346000] LustreError: Skipped 2 previous similar messages [ 751.478435] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.2@tcp inode [0x200000407:0xc19:0x0] object 0x0:3885 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 751.486991] LustreError: Skipped 3 previous similar messages [ 769.244879] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.2@tcp inode [0x200000407:0xc19:0x0] object 0x0:3885 extent [4194304-8388607]: client csum 8b19b060, server csum 8b19b05f [ 769.258177] LustreError: Skipped 4 previous similar messages [ 775.811452] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 803.294654] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.2@tcp inode [0x200000407:0xc19:0x0] object 0x0:3885 extent [4194304-8388607]: client csum 7d33b9a0, server csum 7d33b99f [ 803.301160] LustreError: Skipped 17 previous similar messages [ 821.868080] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 867.466320] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.2@tcp inode [0x200000407:0xc19:0x0] object 0x0:3885 extent [4194304-8388607]: client csum de2bf73f, server csum de2bf73e [ 867.478325] LustreError: Skipped 27 previous similar messages [ 868.034388] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 914.212322] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 960.069150] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 999.656429] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.2@tcp inode [0x200000407:0xc19:0x0] object 0x0:3885 extent [4194304-8388607]: client csum a30822f0, server csum a30822ef [ 999.667687] LustreError: Skipped 62 previous similar messages [ 1006.247032] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1052.637782] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1054.634467] Lustre: DEBUG MARKER: == sanity test 77g: checksum error on OST write, read ==== 18:29:58 (1736983798) [ 1055.127941] Lustre: *** cfs_fail_loc=21a, val=0*** [ 1057.194550] Lustre: *** cfs_fail_loc=21b, val=0*** [ 1060.643476] Lustre: DEBUG MARKER: == sanity test 77k: enable/disable checksum correctly ==== 18:30:04 (1736983804) [ 1060.912279] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 1061.507935] Lustre: Modifying parameter lustre.osc.lustre*.checksums in log params [ 1064.185001] Lustre: Disabling parameter lustre.osc.lustre*.checksums in log params [ 1066.808182] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 1068.341257] Lustre: DEBUG MARKER: == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 18:30:12 (1736983812) [ 1068.807128] Lustre: DEBUG MARKER: set checksum type to invalid, rc = 22 [ 1069.256617] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 1070.780238] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid 40 [ 1086.566097] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid in IDLE state after 15 sec [ 1088.333711] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid 40 [ 1088.813647] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid in FULL state after 0 sec [ 1089.349575] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 1091.218891] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid 40 [ 1111.020609] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid in IDLE state after 19 sec [ 1112.556368] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid 40 [ 1112.966397] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid in FULL state after 0 sec [ 1113.417538] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 1115.034291] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid 40 [ 1135.807182] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid in IDLE state after 20 sec [ 1137.340630] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid 40 [ 1137.738747] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid in FULL state after 0 sec [ 1138.175148] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1139.786291] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid 40 [ 1161.551861] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid in IDLE state after 21 sec [ 1163.056530] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid 40 [ 1163.432566] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid in FULL state after 0 sec [ 1163.842409] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1165.293329] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid 40 [ 1186.013120] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid in IDLE state after 20 sec [ 1187.522002] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid 40 [ 1187.908574] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid in FULL state after 0 sec [ 1188.338662] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1189.901399] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid 40 [ 1211.782158] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid in IDLE state after 21 sec [ 1213.415619] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid 40 [ 1213.845334] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid in FULL state after 0 sec [ 1214.296576] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1215.888737] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid 40 [ 1236.653898] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid in IDLE state after 20 sec [ 1238.215014] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid 40 [ 1238.606104] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800aa5aa000.ost_server_uuid in FULL state after 0 sec [ 1240.348615] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1240.832380] Lustre: DEBUG MARKER: == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 18:33:04 (1736983984) [ 1242.614278] Lustre: DEBUG MARKER: == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 18:33:06 (1736983986) [ 1243.137523] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 1243.555952] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1243.977392] Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0 [ 1244.407941] Lustre: DEBUG MARKER: set checksum type to t10crc4K, rc = 0 [ 1246.135735] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 1246.593545] Lustre: DEBUG MARKER: == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 18:33:10 (1736983990) [ 1249.733577] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 18:33:13 (1736983993) [ 1252.024871] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check ================================================================================= 18:33:15 (1736983995) [ 1264.428023] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 18:33:28 (1736984008) [ 1267.412033] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 18:33:31 (1736984011) [ 1267.764684] Lustre: *** cfs_fail_loc=228, val=0*** [ 1269.745707] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================================================= 18:33:33 (1736984013) [ 1270.135020] Lustre: *** cfs_fail_loc=228, val=0*** [ 1272.114500] Lustre: DEBUG MARKER: == sanity test 99: cvs strange file/directory operations ========================================================== 18:33:35 (1736984015) [ 1278.939477] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ===================================================================== 18:33:42 (1736984022) [ 1282.146383] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads === 18:33:45 (1736984025) [ 1311.720789] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =========================================================================== 18:34:15 (1736984055) [ 1316.640840] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ========================================================== 18:34:20 (1736984060) [ 1327.476883] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled ========================================================== 18:34:31 (1736984071) [ 1380.011676] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ========================================================== 18:35:23 (1736984123) [ 1388.676666] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ========= 18:35:32 (1736984132) [ 1391.499397] Lustre: DEBUG MARKER: == sanity test 101g: Big bulk(4/16 MiB) readahead ======== 18:35:35 (1736984135) [ 1400.533253] Lustre: DEBUG MARKER: == sanity test 101h: Readahead should cover current read window ========================================================== 18:35:44 (1736984144) [ 1403.134166] Lustre: DEBUG MARKER: == sanity test 101i: allow current readahead to exceed reservation ========================================================== 18:35:47 (1736984147) [ 1405.263797] Lustre: DEBUG MARKER: == sanity test 101j: A complete read block should be submitted when no RA ========================================================== 18:35:49 (1736984149) [ 1414.793188] Lustre: DEBUG MARKER: == sanity test 102a: user xattr test ============================================================================================ 18:35:58 (1736984158) [ 1416.731342] Lustre: DEBUG MARKER: == sanity test 102b: getfattr/setfattr for trusted.lov EAs ========================================================== 18:36:00 (1736984160) [ 1419.957925] Lustre: DEBUG MARKER: == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ===================================================================== 18:36:03 (1736984163) [ 1421.905796] Lustre: DEBUG MARKER: == sanity test 102d: tar restore stripe info from tarfile,not keep osts ========================================================== 18:36:05 (1736984165) [ 1425.789931] Lustre: DEBUG MARKER: == sanity test 102f: tar copy files, not keep osts ======= 18:36:09 (1736984169) [ 1429.585568] Lustre: DEBUG MARKER: == sanity test 102h: grow xattr from inside inode to external block ========================================================== 18:36:13 (1736984173) [ 1430.048163] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102h.sanity [ 1430.509194] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102h.sanity [ 1430.944538] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102h.sanity [ 1431.397225] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 1433.335711] Lustre: DEBUG MARKER: == sanity test 102ha: grow xattr from inside inode to external inode ========================================================== 18:36:17 (1736984177) [ 1434.180621] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 1434.742815] Lustre: DEBUG MARKER: save trusted.sml on /mnt/lustre/f102ha.sanity [ 1435.239809] Lustre: DEBUG MARKER: grow trusted.sml on /mnt/lustre/f102ha.sanity [ 1435.759667] Lustre: DEBUG MARKER: trusted.big still valid after growing trusted.sml [ 1436.305739] Lustre: DEBUG MARKER: save trusted.big on /mnt/lustre/f102ha.sanity [ 1438.250764] Lustre: DEBUG MARKER: == sanity test 102i: lgetxattr test on symbolic link ====================================================================== 18:36:22 (1736984182) [ 1440.195117] Lustre: DEBUG MARKER: == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ============================================================= 18:36:24 (1736984184) [ 1444.012595] Lustre: DEBUG MARKER: == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ========================================================== 18:36:27 (1736984187) [ 1446.133694] Lustre: DEBUG MARKER: == sanity test 102l: listxattr size test ============================================================================================ 18:36:29 (1736984189) [ 1448.111829] Lustre: DEBUG MARKER: == sanity test 102m: Ensure listxattr fails on small bufffer ================================================================== 18:36:32 (1736984192) [ 1449.963306] Lustre: DEBUG MARKER: == sanity test 102n: silently ignore setxattr on internal trusted xattrs ========================================================== 18:36:33 (1736984193) [ 1452.148598] Lustre: DEBUG MARKER: == sanity test 102p: check setxattr(2) correctly fails without permission ========================================================== 18:36:36 (1736984196) [ 1454.239679] Lustre: DEBUG MARKER: == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ========================================================== 18:36:38 (1736984198) [ 1456.254683] Lustre: DEBUG MARKER: == sanity test 102r: set EAs with empty values =========== 18:36:40 (1736984200) [ 1458.243066] Lustre: DEBUG MARKER: == sanity test 102s: getting nonexistent xattrs should fail ========================================================== 18:36:42 (1736984202) [ 1460.263192] Lustre: DEBUG MARKER: == sanity test 102t: zero length xattr values handled correctly ========================================================== 18:36:44 (1736984204) [ 1462.219921] Lustre: DEBUG MARKER: == sanity test 103a: acl test ============================ 18:36:46 (1736984206) [ 1495.684715] Lustre: DEBUG MARKER: == sanity test 103b: umask lfs setstripe ================= 18:37:19 (1736984239) [ 1511.307696] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ======== 18:37:35 (1736984255) [ 1513.295185] Lustre: DEBUG MARKER: == sanity test 103e: inheritance of big amount of default ACLs ========================================================== 18:37:37 (1736984257) [ 1781.562816] Lustre: 6872:0:(osd_handler.c:1950:osd_trans_start()) lustre-MDT0000: credits 64189 > trans_max 3200 [ 1781.568923] Lustre: 6872:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 1000/4000/0, destroy: 1/4/0 [ 1781.572923] Lustre: 6872:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 1004/148/0 [ 1781.576554] Lustre: 6872:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 5001/43010/0, punch: 0/0/0, quota 0/0/0 [ 1781.579713] Lustre: 6872:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 1001/17016/0, delete: 2/5/0 [ 1781.584154] Lustre: 6872:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 1781.587718] Pid: 6872, comm: mdt00_002 3.10.0-7.9-debug #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 1781.592149] Call Trace: [ 1781.593735] [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [ 1781.595730] [<0>] libcfs_debug_dumpstack+0x26/0x30 [libcfs] [ 1781.597765] [<0>] osd_trans_start+0x5ba/0x5e0 [osd_ldiskfs] [ 1781.599902] [<0>] top_trans_start+0x763/0xa10 [ptlrpc] [ 1781.601714] [<0>] lod_trans_start+0x34/0x40 [lod] [ 1781.603379] [<0>] mdd_trans_start+0x14/0x20 [mdd] [ 1781.605393] [<0>] mdd_unlink+0x5e3/0xdb0 [mdd] [ 1781.606927] [<0>] mdt_reint_unlink+0xe32/0x1df0 [mdt] [ 1781.608825] [<0>] mdt_reint_rec+0x87/0x240 [mdt] [ 1781.610640] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [ 1781.612776] [<0>] mdt_reint+0x67/0x150 [mdt] [ 1781.614663] [<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc] [ 1781.617099] [<0>] ptlrpc_server_handle_request+0x250/0xc30 [ptlrpc] [ 1781.619299] [<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc] [ 1781.621634] [<0>] kthread+0xe4/0xf0 [ 1781.623050] [<0>] ret_from_fork_nospec_begin+0x7/0x21 [ 1781.624594] [<0>] 0xfffffffffffffffe [ 1781.626311] Lustre: 6872:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 3199 < left 43010, rollback = 7 [ 1783.747896] Lustre: DEBUG MARKER: == sanity test 103f: changelog doesn't interfere with default ACLs buffers ========================================================== 18:42:07 (1736984527) [ 1784.858691] Lustre: lustre-MDD0000: changelog on [ 1785.810460] Lustre: lustre-MDD0001: changelog on [ 1788.327142] Lustre: lustre-MDD0001: changelog off [ 1789.382684] Lustre: lustre-MDD0000: changelog off [ 1790.278745] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test =================================================================================== 18:42:14 (1736984534) [ 1790.391079] Lustre: lustre-OST0000: Client 862af9c6-8aec-45ec-b9e0-8520290e31cc (at 192.168.203.2@tcp) reconnecting [ 1792.215899] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff880077cb2800.ost_server_uuid 40 [ 1792.695238] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880077cb2800.ost_server_uuid in FULL state after 0 sec [ 1794.893273] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ============================================================================== 18:42:18 (1736984538) [ 1796.944352] Lustre: DEBUG MARKER: == sanity test 104c: Verify df vs lfs_df stays same after recordsize change ========================================================== 18:42:20 (1736984540) [ 1797.384800] Lustre: DEBUG MARKER: SKIP: sanity test_104c zfs only test [ 1797.943666] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ================================================================== 18:42:21 (1736984541) [ 1799.995220] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ================================================================== 18:42:23 (1736984543) [ 1802.354541] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ========================================================== 18:42:26 (1736984546) [ 1804.544785] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================================== 18:42:28 (1736984548) [ 1816.569364] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ========================================================== 18:42:40 (1736984560) [ 1818.518088] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ========================================================== 18:42:42 (1736984562) [ 1820.626354] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ====================== 18:42:44 (1736984564) [ 1824.318377] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ============= 18:42:48 (1736984568) [ 1826.567140] Lustre: DEBUG MARKER: SKIP: sanity test_115 skipping SLOW test 115 [ 1827.092275] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ============================================================================= 18:42:50 (1736984570) [ 1893.783844] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ========================================================== 18:43:57 (1736984637) [ 1894.753241] Lustre: *** cfs_fail_loc=147, val=0*** [ 1897.706543] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ==================== 18:44:01 (1736984641) [ 1900.158227] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ================= 18:44:03 (1736984643) [ 1902.850534] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ==================================================================== 18:44:06 (1736984646) [ 1903.479222] Lustre: *** cfs_fail_loc=217, val=0*** [ 1903.482142] Lustre: Skipped 22 previous similar messages [ 1906.505690] Lustre: DEBUG MARKER: SKIP: sanity test_118c skipping ALWAYS excluded test 118c [ 1907.175784] Lustre: DEBUG MARKER: SKIP: sanity test_118d skipping ALWAYS excluded test 118d [ 1907.767551] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error ==================================================================== 18:44:11 (1736984651) [ 1910.209369] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ==================================================================== 18:44:14 (1736984654) [ 1912.801032] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ==================================================================== 18:44:16 (1736984656) [ 1913.416463] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1914.441542] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1916.363286] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1919.368266] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1923.382341] Lustre: *** cfs_fail_loc=20e, val=0*** [ 1925.877700] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ==================================================================== 18:44:29 (1736984669) [ 1934.779268] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error ==================================================================== 18:44:38 (1736984678) [ 1935.347806] Lustre: *** cfs_fail_loc=220, val=0*** [ 1935.349151] Lustre: Skipped 3 previous similar messages [ 1938.215604] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =================================================================== 18:44:41 (1736984681) [ 1948.922199] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir =========================== 18:44:52 (1736984692) [ 1950.972520] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir ======================= 18:44:54 (1736984694) [ 1952.947231] Lustre: DEBUG MARKER: == sanity test 118n: statfs() sends OST_STATFS requests in parallel ========================================================== 18:44:56 (1736984696) [ 1955.415018] LustreError: 9296:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 242 sleeping for 10000ms [ 1955.924139] LustreError: 9296:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 1957.798530] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ========================================================== 18:45:01 (1736984701) [ 1959.724279] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ========================================================== 18:45:03 (1736984703) [ 1961.687702] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ========================================================== 18:45:05 (1736984705) [ 1963.551296] Lustre: DEBUG MARKER: == sanity test 119d: The DIO path should try to send a new rpc once one is completed ========================================================== 18:45:07 (1736984707) [ 1965.031947] Lustre: DEBUG MARKER: the DIO writes have completed, now wait for the reads (should not block very long) [ 1968.913924] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test ======= 18:45:12 (1736984712) [ 1971.750588] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ====== 18:45:15 (1736984715) [ 1974.184457] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ======== 18:45:18 (1736984718) [ 1977.045528] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ===== 18:45:20 (1736984720) [ 1979.892626] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ====== 18:45:23 (1736984723) [ 1989.804857] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ====== 18:45:33 (1736984733) [ 1999.570668] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ========================================================== 18:45:43 (1736984743) [ 2094.399561] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ===================== 18:47:18 (1736984838) [ 2096.475213] Lustre: DEBUG MARKER: == sanity test 123aa: verify statahead work ============== 18:47:20 (1736984840) [ 2098.034353] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 0 sec [ 2098.779303] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 0 sec [ 2107.940527] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 4 sec [ 2109.147221] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 0 sec [ 2246.678949] Lustre: DEBUG MARKER: ls -l 10000 files without statahead: 70 sec [ 2258.145700] Lustre: DEBUG MARKER: ls -l 10000 files with statahead: 9 sec [ 2258.932704] Lustre: DEBUG MARKER: ls -l done [ 2305.804862] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123aa.sanity/: 46 seconds [ 2306.657425] Lustre: DEBUG MARKER: rm done [ 2310.829948] Lustre: DEBUG MARKER: == sanity test 123ab: verify statahead work by using statx ========================================================== 18:50:54 (1736985054) [ 2311.760581] Lustre: DEBUG MARKER: SKIP: sanity test_123ab Test must be statx() syscall supported [ 2313.051137] Lustre: DEBUG MARKER: == sanity test 123ac: verify statahead work by using statx without glimpse RPCs ========================================================== 18:50:56 (1736985056) [ 2313.897509] Lustre: DEBUG MARKER: SKIP: sanity test_123ac Test must be statx() syscall supported [ 2314.862679] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) ========================================================== 18:50:58 (1736985058) [ 2333.492849] Lustre: DEBUG MARKER: ls done [ 2342.868746] Lustre: DEBUG MARKER: == sanity test 123c: Can not initialize inode warning on DNE statahead ========================================================== 18:51:26 (1736985086) [ 2348.649720] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ================================================================================================= 18:51:32 (1736985092) [ 2349.599574] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [ 2370.857430] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a9525800 [ 2371.862346] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a9525800 [ 2372.795708] Lustre: DEBUG MARKER: LRU=1004 [ 2373.730959] Lustre: DEBUG MARKER: LIMIT=46624 [ 2374.621203] Lustre: DEBUG MARKER: LVF=5572500 [ 2375.689173] Lustre: DEBUG MARKER: OLD_LVF=100 [ 2376.726194] Lustre: DEBUG MARKER: Sleep 50 sec [ 2427.943079] Lustre: DEBUG MARKER: Dropped 414 locks in 50s [ 2428.881693] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [ 2445.060538] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) ================================================================================= 18:53:08 (1736985188) [ 2500.016352] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [ 2555.229804] Lustre: DEBUG MARKER: ls -la time: 54 seconds [ 2555.923663] Lustre: DEBUG MARKER: lru_size = 400 [ 2654.443206] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [ 2663.216345] Lustre: DEBUG MARKER: ls -la time: 7 seconds [ 2663.874761] Lustre: DEBUG MARKER: lru_size = 4006 [ 2664.473447] Lustre: DEBUG MARKER: ls -la is 87% faster with lru resize enabled [ 2695.376380] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ========= 18:57:19 (1736985439) [ 2719.327618] Lustre: DEBUG MARKER: == sanity test 124d: cancel very aged locks if lru-resize diasbaled ========================================================== 18:57:43 (1736985463) [ 2743.795449] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs ========================================================== 18:58:07 (1736985487) [ 2746.821797] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ========================================================== 18:58:10 (1736985490) [ 2749.558582] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ==== 18:58:13 (1736985493) [ 2752.028086] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================================== 18:58:15 (1736985495) [ 2754.755015] Lustre: DEBUG MARKER: == sanity test 127c: test llite extent stats with regular [ 2761.173237] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================================== 18:58:24 (1736985504) [ 2763.741729] Lustre: DEBUG MARKER: == sanity test 129: test directory size limit ================================================================================== 18:58:27 (1736985507) [ 2770.196440] Lustre: 9316:0:(osd_handler.c:586:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 1805, FID: [0x20000040b:0x23d8:0x0]) is approaching max size limit [ 2775.003559] Lustre: 9316:0:(osd_handler.c:582:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 1805, FID: [0x20000040b:0x23d8:0x0]) has reached max size limit [ 2787.672943] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ============== 18:58:51 (1736985531) [ 2790.276549] Lustre: DEBUG MARKER: == sanity test 130b: FIEMAP (2-stripe file) ============== 18:58:54 (1736985534) [ 2792.947763] Lustre: DEBUG MARKER: == sanity test 130c: FIEMAP (2-stripe file with hole) ==== 18:58:56 (1736985536) [ 2795.589405] Lustre: DEBUG MARKER: == sanity test 130d: FIEMAP (N-stripe file) ============== 18:58:59 (1736985539) [ 2796.125408] Lustre: DEBUG MARKER: SKIP: sanity test_130d needs >= 3 OSTs [ 2796.767197] Lustre: DEBUG MARKER: == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================================== 18:59:00 (1736985540) [ 2810.000852] Lustre: DEBUG MARKER: == sanity test 130f: FIEMAP (unstriped file) ============= 18:59:13 (1736985553) [ 2812.544930] Lustre: DEBUG MARKER: == sanity test 130g: FIEMAP (overstripe file) ============ 18:59:16 (1736985556) [ 2817.304483] Lustre: 6443:0:(osd_handler.c:1950:osd_trans_start()) lustre-MDT0000: credits 12989 > trans_max 3200 [ 2817.307276] Lustre: 6443:0:(osd_handler.c:1879:osd_trans_dump_creds()) create: 200/800/0, destroy: 1/4/0 [ 2817.309548] Lustre: 6443:0:(osd_handler.c:1879:osd_trans_dump_creds()) Skipped 4001 previous similar messages [ 2817.313893] Lustre: 6443:0:(osd_handler.c:1886:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 204/148/0 [ 2817.317653] Lustre: 6443:0:(osd_handler.c:1886:osd_trans_dump_creds()) Skipped 4001 previous similar messages [ 2817.321079] Lustre: 6443:0:(osd_handler.c:1896:osd_trans_dump_creds()) write: 1001/8610/0, punch: 0/0/0, quota 0/0/0 [ 2817.324589] Lustre: 6443:0:(osd_handler.c:1896:osd_trans_dump_creds()) Skipped 4001 previous similar messages [ 2817.328708] Lustre: 6443:0:(osd_handler.c:1903:osd_trans_dump_creds()) insert: 201/3416/0, delete: 2/5/0 [ 2817.331972] Lustre: 6443:0:(osd_handler.c:1903:osd_trans_dump_creds()) Skipped 4001 previous similar messages [ 2817.335654] Lustre: 6443:0:(osd_handler.c:1910:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/0 [ 2817.339044] Lustre: 6443:0:(osd_handler.c:1910:osd_trans_dump_creds()) Skipped 4001 previous similar messages [ 2817.341706] Pid: 6443, comm: mdt00_006 3.10.0-7.9-debug #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 2817.343941] Call Trace: [ 2817.345000] [<0>] libcfs_call_trace+0x90/0xf0 [libcfs] [ 2817.346301] [<0>] libcfs_debug_dumpstack+0x26/0x30 [libcfs] [ 2817.347884] [<0>] osd_trans_start+0x5ba/0x5e0 [osd_ldiskfs] [ 2817.349696] [<0>] top_trans_start+0x763/0xa10 [ptlrpc] [ 2817.351547] [<0>] lod_trans_start+0x34/0x40 [lod] [ 2817.353284] [<0>] mdd_trans_start+0x14/0x20 [mdd] [ 2817.354685] [<0>] mdd_unlink+0x5e3/0xdb0 [mdd] [ 2817.356134] [<0>] mdt_reint_unlink+0xe32/0x1df0 [mdt] [ 2817.357696] [<0>] mdt_reint_rec+0x87/0x240 [mdt] [ 2817.359658] [<0>] mdt_reint_internal+0x76c/0xb50 [mdt] [ 2817.361953] [<0>] mdt_reint+0x67/0x150 [mdt] [ 2817.363816] [<0>] tgt_request_handle+0x93a/0x19c0 [ptlrpc] [ 2817.365991] [<0>] ptlrpc_server_handle_request+0x250/0xc30 [ptlrpc] [ 2817.368254] [<0>] ptlrpc_main+0xbd9/0x15f0 [ptlrpc] [ 2817.370477] [<0>] kthread+0xe4/0xf0 [ 2817.372378] [<0>] ret_from_fork_nospec_begin+0x7/0x21 [ 2817.373988] [<0>] 0xfffffffffffffffe [ 2817.376986] Lustre: 6443:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-MDT0000: opcode 7: before 3200 < left 8610, rollback = 7 [ 2817.380309] Lustre: 6443:0:(osd_internal.h:1333:osd_trans_exec_op()) Skipped 4000 previous similar messages [ 2818.122952] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ========================================================== 18:59:21 (1736985561) [ 2820.758484] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ================== 18:59:24 (1736985564) [ 2823.502218] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ========================================================== 18:59:27 (1736985567) [ 2825.836073] Lustre: DEBUG MARKER: == sanity test 131d: test short read ===================== 18:59:29 (1736985569) [ 2828.246858] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ============== 18:59:32 (1736985572) [ 2830.782351] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ================================================================================================== 18:59:34 (1736985574) [ 2838.387733] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ============================================================================================ 18:59:42 (1736985582) [ 2843.147618] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ================================================================================================== 18:59:46 (1736985586) [ 2857.504268] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ================================================================================================== 19:00:01 (1736985601) [ 2868.614572] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =========================================================================== 19:00:12 (1736985612) [ 2872.533216] Lustre: DEBUG MARKER: == sanity test 133f: Check reads/writes of client lustre proc files with bad area io ========================================================== 19:00:16 (1736985616) [ 2878.038631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2878.039171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2878.046208] Lustre: Skipped 3 previous similar messages [ 2882.601117] Lustre: server umount lustre-MDT0000 complete [ 2883.046871] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2883.053589] LustreError: Skipped 4 previous similar messages [ 2885.395943] LustreError: 6858:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1736985629 with bad export cookie 13988562637162244389 [ 2885.397138] LustreError: 166-1: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2885.405153] LustreError: 6858:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2885.518597] Lustre: server umount lustre-MDT0001 complete [ 2894.369461] Lustre: 9241:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736985632/real 1736985632] req@ffff88006a334280 x1821358473617216/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1736985638 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2894.404899] Lustre: server umount lustre-OST0000 complete [ 2895.054394] Lustre: 3333:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736985632/real 1736985632] req@ffff88006a332600 x1821358473616960/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1736985639 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 2895.072468] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2895.080865] Lustre: Skipped 1 previous similar message [ 2897.054567] Lustre: 3330:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736985634/real 1736985634] req@ffff88006a336d40 x1821358473617280/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1736985641 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 2897.510439] Lustre: server umount lustre-OST0001 complete [ 2899.861997] device-mapper: core: cleaned up [ 2902.849254] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 2903.481841] Key type lgssc unregistered [ 2903.551096] LNet: 10782:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2903.556554] LNet: Removed LNI 192.168.203.102@tcp [ 2908.612355] alg: No test for adler32 (adler32-zlib) [ 2909.395221] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 2909.614832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2909.698388] Lustre: Lustre: Build Version: 2.15.6_2_gda5530f [ 2909.755763] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 2909.758499] LNet: Accept secure, port 988 [ 2911.299087] Key type lgssc registered [ 2911.568560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2915.978502] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 2917.362347] device-mapper: uevent: version 1.0.3 [ 2917.364438] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 2919.932055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2921.125877] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2921.168264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2922.597312] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2926.163624] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2926.682755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2926.833960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2928.002361] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2931.545367] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2931.551016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2931.651308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2932.649822] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2932.652278] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11066 to 0x280000400:11105 [ 2932.712935] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2936.088673] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 2936.093580] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2936.154721] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2936.169311] mount.lustre (16561) used greatest stack depth: 10048 bytes left [ 2936.277914] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:10887 to 0x2c0000400:10913 [ 2937.161732] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2937.278987] Lustre: lustre-OST0001: deleting orphan objects from 0x0:16545 to 0x0:16577 [ 2937.281053] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16738 to 0x0:16769 [ 2941.052324] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2942.137902] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2949.384339] Lustre: DEBUG MARKER: == sanity test 133g: Check reads/writes of server lustre proc files with bad area io ========================================================== 19:01:33 (1736985693) [ 2971.879232] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2971.884610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2971.890461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2975.377214] Lustre: server umount lustre-MDT0000 complete [ 2977.334898] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2977.342198] LustreError: Skipped 3 previous similar messages [ 2978.273363] LustreError: 13145:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1736985722 with bad export cookie 13401270410748792114 [ 2978.274607] LustreError: 166-1: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2978.284409] LustreError: 13145:0:(ldlm_lockd.c:2521:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2978.407916] Lustre: server umount lustre-MDT0001 complete [ 2987.149158] Lustre: 21606:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736985725/real 1736985725] req@ffff8801372fa140 x1821361468127488/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1736985731 ref 2 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'umount.0' [ 2987.178084] Lustre: server umount lustre-OST0000 complete [ 2989.342006] Lustre: 11303:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736985726/real 1736985726] req@ffff8800945697c0 x1821361468127616/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1736985733 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 2989.351902] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2989.356662] Lustre: Skipped 3 previous similar messages [ 2989.954141] Lustre: server umount lustre-OST0001 complete [ 2992.007404] device-mapper: core: cleaned up [ 2995.082052] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing unload_modules_local [ 2995.756733] Key type lgssc unregistered [ 2995.823838] LNet: 23136:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2995.827686] LNet: Removed LNI 192.168.203.102@tcp [ 3000.740332] alg: No test for adler32 (adler32-zlib) [ 3001.517611] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 3001.676393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3001.780465] Lustre: Lustre: Build Version: 2.15.6_2_gda5530f [ 3001.835033] LNet: Added LNI 192.168.203.102@tcp [8/256/0/180] [ 3001.836968] LNet: Accept secure, port 988 [ 3003.376124] Key type lgssc registered [ 3003.636778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3008.137191] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_modules_local [ 3009.477690] device-mapper: uevent: version 1.0.3 [ 3009.481403] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 3011.771218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3012.942067] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3012.973121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3014.088556] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3017.916671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3017.966678] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3018.054079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3019.285140] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3023.338274] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 3023.343624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3023.462536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3024.702757] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3028.459468] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3028.469096] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16738 to 0x0:16801 [ 3028.632612] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 3028.639401] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3028.704112] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3029.748241] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3032.489209] Lustre: lustre-OST0001: deleting orphan objects from 0x0:16545 to 0x0:16609 [ 3033.489710] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:10887 to 0x2c0000400:10945 [ 3033.490590] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11066 to 0x280000400:11137 [ 3037.359050] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3038.480741] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 3045.418142] Lustre: DEBUG MARKER: == sanity test 133h: Proc files should end with newlines ========================================================== 19:03:09 (1736985789) [ 3251.809192] Lustre: DEBUG MARKER: == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ========================================================== 19:06:35 (1736985995) [ 3255.965116] Lustre: *** cfs_fail_loc=327, val=0*** [ 3270.141845] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================================================== 19:06:54 (1736986014) [ 3271.734667] Lustre: *** cfs_fail_loc=328, val=0*** [ 3271.736225] Lustre: Skipped 512 previous similar messages [ 3272.742769] Lustre: *** cfs_fail_loc=328, val=0*** [ 3275.312217] Lustre: *** cfs_fail_loc=328, val=0*** [ 3275.313870] Lustre: Skipped 252 previous similar messages [ 3279.621171] Lustre: *** cfs_fail_loc=328, val=0*** [ 3279.623039] Lustre: Skipped 125 previous similar messages [ 3287.796249] Lustre: *** cfs_fail_loc=328, val=0*** [ 3287.798014] Lustre: Skipped 65 previous similar messages [ 3291.898347] LustreError: 17748:0:(ldlm_resource.c:133:seq_watermark_write()) Failed to set lock_reclaim_threshold_mb, rc = -22. [ 3295.800663] Lustre: DEBUG MARKER: SKIP: sanity test_135 skipping SLOW test 135 [ 3296.248110] Lustre: DEBUG MARKER: SKIP: sanity test_136 skipping SLOW test 136 [ 3296.709104] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ============================================================== 19:07:20 (1736986040) [ 3304.783246] Lustre: DEBUG MARKER: == sanity test 150a: truncate/append tests =============== 19:07:28 (1736986048) [ 3315.849554] Lustre: DEBUG MARKER: == sanity test 150b: Verify fallocate (prealloc) functionality ========================================================== 19:07:39 (1736986059) [ 3326.428603] Lustre: DEBUG MARKER: == sanity test 150bb: Verify fallocate modes both zero space ========================================================== 19:07:50 (1736986070) [ 3335.143679] Lustre: DEBUG MARKER: == sanity test 150c: Verify fallocate Size and Blocks ==== 19:07:59 (1736986079) [ 3346.153037] Lustre: DEBUG MARKER: == sanity test 150d: Verify fallocate Size and Blocks - Non zero start ========================================================== 19:08:10 (1736986090) [ 3349.942011] Lustre: DEBUG MARKER: == sanity test 150e: Verify 60% of available OST space consumed by fallocate ========================================================== 19:08:13 (1736986093) [ 3364.591287] Lustre: DEBUG MARKER: == sanity test 150f: Verify fallocate punch functionality ========================================================== 19:08:28 (1736986108) [ 3375.502264] Lustre: DEBUG MARKER: == sanity test 150g: Verify fallocate punch on large range ========================================================== 19:08:39 (1736986119) [ 3386.581222] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ========================================================================================= 19:08:50 (1736986130) [ 3391.136957] bash (26378): drop_caches: 1 [ 3394.302339] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================================================== 19:08:58 (1736986138) [ 3395.981578] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================================================= 19:08:59 (1736986139) [ 3397.729210] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================================== 19:09:01 (1736986141) [ 3399.566499] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ==== 19:09:03 (1736986143) [ 3401.376744] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ========================= 19:09:05 (1736986145) [ 3401.659619] LustreError: 580:0:(fld_handler.c:263:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 3403.997810] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ==== 19:09:07 (1736986147) [ 3404.257840] LustreError: 580:0:(fld_handler.c:263:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 3404.262420] LustreError: 580:0:(fld_handler.c:263:fld_server_lookup()) Skipped 3 previous similar messages [ 3406.432594] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ========================================================== 19:09:10 (1736986150) [ 3408.422122] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ================ 19:09:12 (1736986152) [ 3410.697380] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir == 19:09:14 (1736986154) [ 3412.735290] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea == 19:09:16 (1736986156) [ 3415.040359] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ============= 19:09:18 (1736986158) [ 3508.381421] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ========================================================== 19:10:52 (1736986252) [ 3511.849721] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ========================================================== 19:10:55 (1736986255) [ 3515.468932] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ========================================================== 19:10:59 (1736986259) [ 3519.225686] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off ========================================================== 19:11:03 (1736986263) [ 3522.777241] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ========================================================== 19:11:06 (1736986266) [ 3535.584640] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ========================================================== 19:11:19 (1736986279) [ 3551.220309] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ========================================================== 19:11:35 (1736986295) [ 3565.466138] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ========================================================== 19:11:49 (1736986309) [ 3575.832616] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ============= 19:11:59 (1736986319) [ 3578.214948] Lustre: DEBUG MARKER: Turn on read and write cache [ 3579.272954] Lustre: DEBUG MARKER: Write data and read it back. [ 3579.669667] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 3580.767921] Lustre: DEBUG MARKER: cache hits: before: 65581, after: 65584 [ 3581.152140] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 3581.901554] Lustre: DEBUG MARKER: cache hits:: before: 65584, after: 65587 [ 3582.306969] Lustre: DEBUG MARKER: Turn off the read cache and turn on the write cache [ 3583.456750] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 3584.556462] Lustre: DEBUG MARKER: cache hits:: before: 65587, after: 65590 [ 3584.946440] Lustre: DEBUG MARKER: Write data and read it back. [ 3585.341945] Lustre: DEBUG MARKER: Read should be satisfied from the cache. [ 3586.521854] Lustre: DEBUG MARKER: cache hits:: before: 65590, after: 65593 [ 3586.945280] Lustre: DEBUG MARKER: Turn off read and write cache [ 3588.093585] Lustre: DEBUG MARKER: Write data and read it back [ 3588.503542] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 3589.725159] Lustre: DEBUG MARKER: cache hits:: before: 65593, after: 65593 [ 3590.152806] Lustre: DEBUG MARKER: Turn on the read cache and turn off the write cache [ 3591.351554] Lustre: DEBUG MARKER: Write data and read it back [ 3591.790912] Lustre: DEBUG MARKER: It should not be satisfied from the cache. [ 3593.002569] Lustre: DEBUG MARKER: cache hits:: before: 65593, after: 65593 [ 3593.409073] Lustre: DEBUG MARKER: Read again; it should be satisfied from the cache. [ 3594.600612] Lustre: DEBUG MARKER: cache hits:: before: 65593, after: 65596 [ 3596.945545] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ==================== 19:12:20 (1736986340) [ 3597.741617] Lustre: lustre-MDD0000: changelog on [ 3598.495485] Lustre: lustre-MDD0001: changelog on [ 3602.588832] Lustre: Failing over lustre-MDT0000 [ 3602.652898] Lustre: server umount lustre-MDT0000 complete [ 3603.958257] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3603.960884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3603.965754] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3605.019924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3605.083541] LustreError: 11-0: MGC192.168.203.102@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 3605.083698] LustreError: 166-1: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3605.084234] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x158aa7dbbe8d39c4 to 0x158aa7dbbe95f3ce [ 3605.094638] Lustre: MGC192.168.203.102@tcp: Connection restored to 192.168.203.102@tcp (at 0@lo) [ 3605.161835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3605.170512] Lustre: lustre-MDD0000: changelog on [ 3605.174406] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3605.626738] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3606.266605] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3610.168569] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) [ 3610.196369] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3610.212871] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17627 to 0x0:17665 [ 3610.213155] Lustre: lustre-OST0001: deleting orphan objects from 0x0:17442 to 0x0:17473 [ 3610.448509] Lustre: lustre-MDD0000: changelog off [ 3610.967800] Lustre: lustre-MDD0001: changelog off [ 3614.705653] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================================================== 19:12:38 (1736986358) [ 3615.611488] Lustre: lustre-MDD0000: changelog on [ 3618.976862] Lustre: lustre-MDD0001: changelog off [ 3620.734560] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ========================================================== 19:12:44 (1736986364) [ 3621.629089] Lustre: lustre-MDD0000: changelog on [ 3621.631273] Lustre: Skipped 1 previous similar message [ 3625.282562] Lustre: lustre-MDD0001: changelog off [ 3625.284398] Lustre: Skipped 1 previous similar message [ 3626.856612] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event ========================================================== 19:12:50 (1736986370) [ 3630.504930] Lustre: lustre-MDD0001: changelog off [ 3630.506423] Lustre: Skipped 1 previous similar message [ 3631.937822] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing (should return errors) ========================================================== 19:12:55 (1736986375) [ 3632.712284] Lustre: lustre-MDD0000: changelog on [ 3632.713460] Lustre: Skipped 3 previous similar messages [ 3637.112641] Lustre: DEBUG MARKER: == sanity test 160f: changelog garbage collect (timestamped users) ========================================================== 19:13:01 (1736986381) [ 3640.485066] Lustre: DEBUG MARKER: 1736986384: creating first files [ 3655.845426] Lustre: *** cfs_fail_loc=1313, val=0*** [ 3655.847098] Lustre: 25510:0:(mdd_dir.c:895:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 3655.850787] Lustre: 18206:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl7 idle for 17s with 4 unprocessed records [ 3661.249538] Lustre: lustre-MDD0001: changelog off [ 3661.251006] Lustre: Skipped 3 previous similar messages [ 3662.653734] Lustre: DEBUG MARKER: == sanity test 160g: changelog garbage collect on idle records ========================================================== 19:13:26 (1736986406) [ 3663.413282] Lustre: lustre-MDD0000: changelog on [ 3663.414523] Lustre: Skipped 3 previous similar messages [ 3668.964460] Lustre: 26575:0:(mdd_dir.c:895:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 3668.967749] Lustre: 26575:0:(mdd_dir.c:895:mdd_changelog_store()) Skipped 1 previous similar message [ 3668.970782] Lustre: 21493:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl9 idle for 5s with 4 unprocessed records [ 3668.975790] Lustre: 21493:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [ 3675.439637] Lustre: DEBUG MARKER: == sanity test 160h: changelog gc thread stop upon umount, orphan records delete ========================================================== 19:13:39 (1736986419) [ 3693.034790] Lustre: *** cfs_fail_loc=1316, val=0*** [ 3693.036371] Lustre: 25509:0:(mdd_dir.c:895:mdd_changelog_store()) lustre-MDD0001: simulate starting changelog garbage collection [ 3693.039246] Lustre: 25509:0:(mdd_dir.c:895:mdd_changelog_store()) Skipped 1 previous similar message [ 3693.042268] Lustre: 24798:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) lustre-MDD0001: force deregister of changelog user cl11 idle for 15s with 3 unprocessed records [ 3693.046848] Lustre: 24798:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [ 3693.616342] Lustre: Failing over lustre-MDT0001 [ 3693.648827] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_disconnect to node 0@lo failed: rc = -107 [ 3695.302363] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3695.302511] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3695.308491] Lustre: Skipped 5 previous similar messages [ 3696.110607] Lustre: server umount lustre-MDT0001 complete [ 3698.266831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3698.292384] LustreError: 166-1: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3698.297046] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x158aa7dbbe95f3ce to 0x158aa7dbbe960498 [ 3698.302005] Lustre: MGC192.168.203.102@tcp: Connection restored to (at 0@lo) [ 3698.303616] Lustre: Skipped 3 previous similar messages [ 3698.372136] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3698.378880] LustreError: Skipped 7 previous similar messages [ 3698.385521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3698.393864] Lustre: lustre-MDD0000: changelog on [ 3698.395044] Lustre: Skipped 3 previous similar messages [ 3698.396249] Lustre: 26555:0:(mdd_device.c:618:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 31 to index 32, being cleared now [ 3698.402258] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3699.229954] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3700.758170] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3701.356161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3701.442854] Lustre: 27517:0:(mdd_device.c:618:mdd_changelog_llog_init()) lustre-MDD0001 : orphan changelog records found, starting from index 15 to index 16, being cleared now [ 3701.448292] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 3701.778615] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3702.283671] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3703.460993] Lustre: lustre-MDT0001: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3703.461062] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to (at 0@lo) [ 3703.482794] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11143 to 0x280000400:11169 [ 3703.483071] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:10950 to 0x2c0000400:10977 [ 3703.483721] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17627 to 0x0:17697 [ 3703.483759] Lustre: lustre-OST0001: deleting orphan objects from 0x0:17475 to 0x0:17505 [ 3707.675430] Lustre: lustre-MDD0001: changelog off [ 3707.676418] Lustre: Skipped 3 previous similar messages [ 3709.027877] Lustre: DEBUG MARKER: == sanity test 160i: changelog user register/unregister race ========================================================== 19:14:12 (1736986452) [ 3709.439566] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 3709.441838] Lustre: Skipped 1 previous similar message [ 3711.549847] LustreError: 30143:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 1315 sleeping [ 3713.784827] LustreError: 30312:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 1315 waking [ 3713.787067] LustreError: 30143:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 1315 awake: rc=2766 [ 3714.532576] LustreError: 30648:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 1315 waking [ 3714.535705] LustreError: 30648:0:(libcfs_fail.h:180:cfs_race()) Skipped 1 previous similar message [ 3720.246284] Lustre: DEBUG MARKER: == sanity test 160j: client can be umounted while its chanangelog is being used ========================================================== 19:14:24 (1736986464) [ 3726.245872] Lustre: DEBUG MARKER: == sanity test 160k: Verify that changelog records are not lost ========================================================== 19:14:30 (1736986470) [ 3728.418504] LustreError: 28678:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 15d sleeping for 3000ms [ 3731.422063] LustreError: 28678:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 15d awake [ 3738.663272] Lustre: DEBUG MARKER: == sanity test 160l: Verify that MTIME changelog records contain the parent FID ========================================================== 19:14:42 (1736986482) [ 3744.543186] Lustre: lustre-MDD0001: changelog off [ 3744.544424] Lustre: Skipped 9 previous similar messages [ 3746.041224] Lustre: DEBUG MARKER: == sanity test 160m: Changelog clear race ================ 19:14:49 (1736986489) [ 3750.059651] LustreError: 28678:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 15f sleeping [ 3752.060832] LustreError: 26564:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15f waking [ 3752.063492] LustreError: 28678:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 15f awake: rc=2999 [ 3756.456854] Lustre: DEBUG MARKER: == sanity test 160n: Changelog destroy race ============== 19:15:00 (1736986500) [ 4024.488853] LustreError: 27352:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 16c sleeping [ 4026.491245] LustreError: 1271:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 16c waking [ 4026.495177] LustreError: 27352:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 16c awake: rc=2996 [ 4028.716266] Lustre: lustre-MDD0001: changelog off [ 4028.717977] Lustre: Skipped 3 previous similar messages [ 4030.079556] Lustre: DEBUG MARKER: == sanity test 160o: changelog user name and mask ======== 19:19:34 (1736986774) [ 4030.826692] Lustre: lustre-MDD0000: changelog on [ 4030.827981] Lustre: Skipped 15 previous similar messages [ 4031.771611] LustreError: 8241:0:(mdd_device.c:1704:mdd_changelog_name_check()) lustre-MDD0000: wrong char '#' in name 'Tt3_-#': rc = -22 [ 4032.014201] Lustre: 8306:0:(mdd_device.c:1721:mdd_changelog_name_check()) lustre-MDD0000: changelog name test_160o exists already: rc = -17 [ 4032.240531] LustreError: 8371:0:(mdd_device.c:1713:mdd_changelog_name_check()) lustre-MDD0000: name 'test_160toolongname' is over 16 symbols limit: rc = -36 [ 4040.017543] Lustre: DEBUG MARKER: == sanity test 160p: Changelog orphan cleanup with no users ========================================================== 19:19:43 (1736986783) [ 4042.296673] Lustre: Failing over lustre-MDT0000 [ 4042.297814] Lustre: Skipped 1 previous similar message [ 4042.358987] Lustre: server umount lustre-MDT0000 complete [ 4042.360404] Lustre: Skipped 1 previous similar message [ 4044.022277] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4044.024839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4044.029533] Lustre: Skipped 1 previous similar message [ 4044.031059] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4044.035149] LustreError: Skipped 1 previous similar message [ 4044.819738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4044.848230] LustreError: 166-1: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4044.853462] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x158aa7dbbe960498 to 0x158aa7dbbed97b0b [ 4044.856450] Lustre: MGC192.168.203.102@tcp: Connection restored to (at 0@lo) [ 4044.858917] Lustre: Skipped 3 previous similar messages [ 4044.909428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4044.911845] Lustre: Skipped 1 previous similar message [ 4044.959530] Lustre: 11350:0:(mdd_device.c:618:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 90155 to index 18446744073709551615, being cleared now [ 4044.967603] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4045.841380] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4049.158490] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4049.930952] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4049.933798] Lustre: Skipped 1 previous similar message [ 4049.946404] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17699 to 0x0:17729 [ 4049.946406] Lustre: lustre-OST0001: deleting orphan objects from 0x0:17507 to 0x0:17537 [ 4052.755230] Lustre: DEBUG MARKER: == sanity test 160q: changelog effective mask is DEFMASK if not set ========================================================== 19:19:56 (1736986796) [ 4055.543218] Lustre: DEBUG MARKER: == sanity test 160s: changelog garbage collect on idle records anaconda-ks.cfg stress.sh time ========================================================== 19:19:59 (1736986799) [ 4062.040514] Lustre: 26565:0:(mdd_dir.c:895:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 4062.043618] Lustre: 26565:0:(mdd_dir.c:895:mdd_changelog_store()) Skipped 1 previous similar message [ 4062.047438] Lustre: 13991:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl2 idle for 864006s with 500000004 unprocessed records [ 4062.057352] Lustre: 13991:0:(mdd_trans.c:160:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [ 4067.682957] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ====================== 19:20:11 (1736986811) [ 4075.380813] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory ========================================================== 19:20:19 (1736986819) [ 4082.820552] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags ========================================================== 19:20:26 (1736986826) [ 4088.584375] Lustre: DEBUG MARKER: == sanity test 161d: create with concurrent .lustre/fid access ========================================================== 19:20:32 (1736986832) [ 4095.304603] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ================== 19:20:39 (1736986839) [ 4097.137727] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ========================================================== 19:20:41 (1736986841) [ 4099.080576] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================================================== 19:20:42 (1736986842) [ 4110.961727] Lustre: DEBUG MARKER: == sanity test 165a: ofd access log discovery ============ 19:20:54 (1736986854) [ 4116.900257] Lustre: Failing over lustre-OST0000 [ 4116.910069] Lustre: server umount lustre-OST0000 complete [ 4117.094340] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4117.096893] LustreError: Skipped 1 previous similar message [ 4117.098250] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4117.101575] Lustre: Skipped 3 previous similar messages [ 4117.103037] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4117.106946] LustreError: Skipped 5 previous similar messages [ 4121.512001] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4121.515597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4121.553866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4121.557734] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4122.414363] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4122.744579] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4123.327108] Lustre: DEBUG MARKER: == sanity test 165b: ofd access log entries are produced and consumed ========================================================== 19:21:07 (1736986867) [ 4123.404157] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4123.404197] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.102@tcp (at 0@lo) [ 4123.404199] Lustre: Skipped 4 previous similar messages [ 4123.405082] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17735 to 0x0:17761 [ 4123.406012] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11174 to 0x280000400:11201 [ 4146.181371] Lustre: Failing over lustre-OST0000 [ 4146.193090] Lustre: server umount lustre-OST0000 complete [ 4146.598452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4146.601861] Lustre: Skipped 1 previous similar message [ 4148.327657] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4148.331076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4148.373663] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4149.221682] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4149.323060] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4149.825718] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4149.827007] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11174 to 0x280000400:11233 [ 4149.827711] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17763 to 0x0:17793 [ 4150.166280] Lustre: DEBUG MARKER: == sanity test 165c: full ofd access logs do not block IOs ========================================================== 19:21:34 (1736986894) [ 4159.527451] Lustre: Failing over lustre-OST0000 [ 4159.542257] Lustre: server umount lustre-OST0000 complete [ 4159.846226] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4159.846237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4159.846238] Lustre: Skipped 1 previous similar message [ 4159.846411] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4159.846412] LustreError: Skipped 5 previous similar messages [ 4159.861387] LustreError: Skipped 1 previous similar message [ 4161.699706] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4161.702890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4162.584251] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4163.282164] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4163.282205] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.102@tcp (at 0@lo) [ 4163.282207] Lustre: Skipped 3 previous similar messages [ 4163.283455] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17861 to 0x0:17889 [ 4163.285112] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11295 to 0x280000400:11329 [ 4163.564487] Lustre: DEBUG MARKER: == sanity test 165d: ofd_access_log mask works =========== 19:21:47 (1736986907) [ 4183.833523] Lustre: Failing over lustre-OST0000 [ 4183.845223] Lustre: server umount lustre-OST0000 complete [ 4186.084261] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4186.088917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4186.127830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4186.131801] Lustre: Skipped 1 previous similar message [ 4186.135801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4186.139036] Lustre: Skipped 2 previous similar messages [ 4186.142186] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4186.144416] Lustre: Skipped 1 previous similar message [ 4187.064122] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4187.271926] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4187.275202] Lustre: Skipped 1 previous similar message [ 4187.992807] Lustre: DEBUG MARKER: == sanity test 165e: ofd_access_log MDT index filter works ========================================================== 19:22:11 (1736986931) [ 4188.008031] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4188.009176] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17891 to 0x0:17921 [ 4188.009721] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11295 to 0x280000400:11361 [ 4200.727797] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4200.731338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4201.665376] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4202.428716] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11363 to 0x280000400:11393 [ 4202.429148] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17923 to 0x0:17953 [ 4202.608737] Lustre: DEBUG MARKER: == sanity test 165f: ofd_access_log_reader --exit-on-close works ========================================================== 19:22:26 (1736986946) [ 4212.850412] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4212.853942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4213.818816] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4214.735030] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock ========================================================== 19:22:38 (1736986958) [ 4214.762517] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17923 to 0x0:17985 [ 4214.765080] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:11363 to 0x280000400:11425 [ 4215.119898] Lustre: DEBUG MARKER: creating a 10 Mb file [ 4215.558307] Lustre: DEBUG MARKER: starting reads [ 4215.944286] Lustre: DEBUG MARKER: truncating the file [ 4216.342328] Lustre: DEBUG MARKER: killing dd [ 4216.714579] Lustre: DEBUG MARKER: removing the temporary file [ 4218.332440] Lustre: DEBUG MARKER: == sanity test 170: test lctl df to handle corrupted log =============================================================================== 19:22:42 (1736986962) [ 4220.084942] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ================================================================ 19:22:44 (1736986964) [ 4224.779562] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ================= 19:22:48 (1736986968) [ 4225.184651] Lustre: DEBUG MARKER: SKIP: sanity test_180a obdecho on osc is no longer supported [ 4225.658648] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter == 19:22:49 (1736986969) [ 4226.374859] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_module obdecho/obdecho [ 4230.849413] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT ========================================================== 19:22:54 (1736986974) [ 4231.533600] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing load_module obdecho/obdecho [ 4231.570623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4236.558293] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir ================================================================================== 19:23:00 (1736986980) [ 4251.031605] Lustre: DEBUG MARKER: == sanity test 182: Test parallel modify metadata operations ========================================================================== 19:23:14 (1736986994) [ 4261.068731] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ================================================================== 19:23:25 (1736987005) [ 4261.371007] Lustre: *** cfs_fail_loc=148, val=0*** [ 4263.317898] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =================== 19:23:27 (1736987007) [ 4265.858158] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ========================================================== 19:23:29 (1736987009) [ 4267.566669] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ==== 19:23:31 (1736987011) [ 4274.709366] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap ======= 19:23:38 (1736987018) [ 4278.397364] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps ========================================================== 19:23:42 (1736987022) [ 4281.962733] Lustre: DEBUG MARKER: == sanity test 184f: IOC_MDC_GETFILEINFO for files with long names but no striping ========================================================== 19:23:45 (1736987025) [ 4283.633655] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ================ 19:23:47 (1736987027) [ 4286.340080] Lustre: DEBUG MARKER: == sanity test 185a: Volatile file creation in .lustre/fid/ ========================================================== 19:23:50 (1736987030) [ 4292.036252] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ============ 19:23:55 (1736987035) [ 4293.904111] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ========================================================== 19:23:57 (1736987037) [ 4295.531101] Lustre: DEBUG MARKER: == sanity test 200: OST pools ============================ 19:23:59 (1736987039) [ 4310.841900] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes ===== 19:24:14 (1736987054) [ 4312.513399] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset ========================================================== 19:24:16 (1736987056) [ 4314.124518] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset ========================================================== 19:24:18 (1736987058) [ 4315.800768] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size ========================================================== 19:24:19 (1736987059) [ 4317.469882] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes ========= 19:24:21 (1736987061) [ 4319.137841] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset ==== 19:24:23 (1736987063) [ 4320.797960] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset === 19:24:24 (1736987064) [ 4322.436198] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size ===== 19:24:26 (1736987066) [ 4324.160773] Lustre: DEBUG MARKER: == sanity test 205a: Verify job stats ==================== 19:24:28 (1736987068) [ 4327.342296] Lustre: lustre-MDD0000: changelog on [ 4327.344251] Lustre: Skipped 10 previous similar messages [ 4328.986903] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity [ 4329.377695] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.28919 [ 4330.024968] Lustre: DEBUG MARKER: Test: rmdir /mnt/lustre/d205a.sanity [ 4330.426255] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rmdir.32487 [ 4331.079337] Lustre: DEBUG MARKER: Test: lfs mkdir -i 1 /mnt/lustre/d205a.sanity.remote [ 4331.490643] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.7903 [ 4332.141294] Lustre: DEBUG MARKER: Test: mknod /mnt/lustre/f205a.sanity c 1 3 [ 4332.520912] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mknod.25257 [ 4333.153349] Lustre: DEBUG MARKER: Test: rm -f /mnt/lustre/f205a.sanity [ 4333.534180] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rm.1257 [ 4334.179194] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/f205a.sanity [ 4334.582226] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.26085 [ 4335.326513] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 4335.738107] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.touch.13427 [ 4336.658087] Lustre: DEBUG MARKER: Test: dd if=/dev/zero of=/mnt/lustre/f205a.sanity bs=1M count=1 oflag=sync [ 4337.058641] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.8017 [ 4337.751321] Lustre: DEBUG MARKER: Test: dd if=/mnt/lustre/f205a.sanity of=/dev/null bs=1M count=1 iflag=direct [ 4338.170588] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.3940 [ 4338.860678] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/tests/truncate /mnt/lustre/f205a.sanity 0 [ 4339.273996] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.truncate.7492 [ 4340.164761] Lustre: DEBUG MARKER: Test: mv -f /mnt/lustre/f205a.sanity /mnt/lustre/d205a.sanity.rename [ 4340.563138] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mv.24315 [ 4341.214353] Lustre: DEBUG MARKER: Test: /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity.expire [ 4341.602130] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.23476 [ 4345.031110] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 4345.428061] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg302-client.v [ 4346.092209] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 4346.486919] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.oleg302-client.E [ 4347.143299] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 4347.548454] Lustre: DEBUG MARKER: Using JobID environment session=S.root.touch.0.oleg302-client.v [ 4349.805784] Lustre: lustre-MDD0001: changelog off [ 4349.807096] Lustre: Skipped 11 previous similar messages [ 4351.467484] Lustre: DEBUG MARKER: == sanity test 205b: Verify job stats jobid and output format ========================================================== 19:24:55 (1736987095) [ 4354.423983] Lustre: DEBUG MARKER: == sanity test 205c: Verify client stats format ========== 19:24:58 (1736987098) [ 4356.064798] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug === 19:25:00 (1736987100) [ 4357.748589] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse ======= 19:25:01 (1736987101) [ 4359.388745] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open ========== 19:25:03 (1736987103) [ 4361.090510] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open ======================= 19:25:05 (1736987105) [ 4367.686840] Lustre: Failing over lustre-MDT0000 [ 4367.688066] Lustre: Skipped 2 previous similar messages [ 4367.763136] Lustre: server umount lustre-MDT0000 complete [ 4367.764770] Lustre: Skipped 2 previous similar messages [ 4368.150422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4368.150626] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4368.150628] LustreError: Skipped 6 previous similar messages [ 4368.164268] Lustre: Skipped 7 previous similar messages [ 4380.096864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4380.128946] LustreError: 166-1: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4380.129284] LustreError: 11-0: MGC192.168.203.102@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 4380.136212] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x158aa7dbbed97b0b to 0x158aa7dbbeee15a7 [ 4380.140416] Lustre: MGC192.168.203.102@tcp: Connection restored to 192.168.203.102@tcp (at 0@lo) [ 4380.143724] Lustre: Skipped 7 previous similar messages [ 4380.200044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4380.204356] Lustre: Skipped 2 previous similar messages [ 4380.215025] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4380.217404] Lustre: Skipped 2 previous similar messages [ 4380.694188] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4380.698327] Lustre: Skipped 2 previous similar messages [ 4381.136998] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4385.212248] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4385.219628] Lustre: Skipped 2 previous similar messages [ 4385.237871] Lustre: lustre-OST0001: deleting orphan objects from 0x0:21108 to 0x0:21153 [ 4385.237873] Lustre: lustre-OST0000: deleting orphan objects from 0x0:21443 to 0x0:21473 [ 4386.641624] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4387.038708] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4402.230071] Lustre: 23654:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736987139/real 1736987139] req@ffff88011a79d580 x1821361566213504/t0(0) o400->MGC192.168.203.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1736987146 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 4402.235934] LustreError: 166-1: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4403.249017] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4408.254812] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x158aa7dbbeee15a7 to 0x158aa7dbbeee1a77 [ 4409.182632] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4413.350647] Lustre: lustre-OST0000: deleting orphan objects from 0x0:21443 to 0x0:21505 [ 4413.351001] Lustre: lustre-OST0001: deleting orphan objects from 0x0:21108 to 0x0:21185 [ 4414.674241] Lustre: DEBUG MARKER: oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4415.033889] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4417.898545] Lustre: DEBUG MARKER: == sanity test 209: read-only open/close requests should be freed promptly ========================================================== 19:26:01 (1736987161) [ 4426.030252] Lustre: DEBUG MARKER: == sanity test 210: lfs getstripe does not break leases == 19:26:10 (1736987170) [ 4429.700907] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ====================================================================================================== 19:26:13 (1736987173) [ 4431.604356] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ========================================================== 19:26:15 (1736987175) [ 4443.430820] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================================== 19:26:27 (1736987187) [ 4447.690224] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ========================================================== 19:26:31 (1736987191) [ 4449.541117] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write updates file size and kms correctly ========================================================== 19:26:33 (1736987193) [ 4455.065291] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with hiphen ('-') ========================================================== 19:26:39 (1736987199) [ 4457.081814] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock ========================================================== 19:26:41 (1736987201) [ 4457.469717] Lustre: DEBUG MARKER: creating a 10 Mb file [ 4457.897948] Lustre: DEBUG MARKER: starting reads [ 4458.310193] Lustre: DEBUG MARKER: truncating the file [ 4458.734159] Lustre: DEBUG MARKER: killing dd [ 4459.138536] Lustre: DEBUG MARKER: removing the temporary file [ 4460.831561] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND ========================================================== 19:26:44 (1736987204) [ 4462.531702] Lustre: DEBUG MARKER: == sanity test 220: preallocated MDS objects still used if ENOSPC from OST ========================================================== 19:26:46 (1736987206) [ 4465.158329] Lustre: *** cfs_fail_loc=229, val=0*** [ 4467.654004] Lustre: *** cfs_fail_loc=229, val=0*** [ 4475.188051] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM ========================================================== 19:26:59 (1736987219) [ 4477.230182] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ========================================================== 19:27:01 (1736987221) [ 4478.961715] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ========================================================== 19:27:02 (1736987222) [ 4480.719176] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ================================================================================= 19:27:04 (1736987224) [ 4482.491183] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure ====== 19:27:06 (1736987226) [ 4482.542179] Lustre: lustre-OST0001: Client 541083a5-cc36-42b1-9962-e6a136b7f6eb (at 192.168.203.2@tcp) reconnecting [ 4485.242532] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure ====== 19:27:09 (1736987229) [ 4486.472597] LustreError: 23169:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 515 sleeping for 3000ms [ 4489.475068] LustreError: 23169:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 515 awake [ 4491.733920] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC ========================================================== 19:27:15 (1736987235) [ 4496.599759] Lustre: *** cfs_fail_loc=520, val=57344*** [ 4496.601702] LNet: *** cfs_fail_loc=e000, val=2147483648*** [ 4501.609420] Lustre: lustre-MDT0000: Client 541083a5-cc36-42b1-9962-e6a136b7f6eb (at 192.168.203.2@tcp) reconnecting [ 4501.617542] Lustre: 29907:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@ffff88011bb9b850 x1821361589433600/t34359738608(0) o35->541083a5-cc36-42b1-9962-e6a136b7f6eb@192.168.203.2@tcp:0/0 lens 392/456 e 0 to 0 dl 1736987250 ref 1 fl Interpret:/2/0 rc 0/0 job:'' [ 4507.711463] Lustre: DEBUG MARKER: == sanity test 224d: Don't corrupt data on bulk IO timeout ========================================================== 19:27:31 (1736987251) [ 4508.913884] LustreError: 23169:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 515 sleeping for 22000ms [ 4529.661370] Lustre: lustre-OST0000: Client 541083a5-cc36-42b1-9962-e6a136b7f6eb (at 192.168.203.2@tcp) reconnecting [ 4530.916012] LustreError: 23169:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 4530.919602] LustreError: 23169:0:(ldlm_lib.c:3493:target_bulk_io()) @@@ bulk READ failed: rc = -107 req@ffff880084c24280 x1821361589439744/t0(0) o3->541083a5-cc36-42b1-9962-e6a136b7f6eb@192.168.203.2@tcp:0/0 lens 488/440 e 0 to 0 dl 1736987273 ref 1 fl Interpret:/0/0 rc 0/0 job:'' [ 4530.928265] Lustre: 23169:0:(service.c:2333:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/2s); client may timeout req@ffff880084c24280 x1821361589439744/t0(0) o3->541083a5-cc36-42b1-9962-e6a136b7f6eb@192.168.203.2@tcp:0/0 lens 488/440 e 0 to 0 dl 1736987273 ref 1 fl Complete:/0/0 rc 0/0 job:'' [ 4532.941063] Lustre: DEBUG MARKER: SKIP: sanity test_225a skipping excluded test 225a (base 225) [ 4533.387944] Lustre: DEBUG MARKER: SKIP: sanity test_225b skipping excluded test 225b (base 225) [ 4533.804097] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type ========================================================== 19:27:57 (1736987277) [ 4535.543086] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ========================================================== 19:27:59 (1736987279) [ 4537.312328] Lustre: DEBUG MARKER: == sanity test 226c: call path2fid and fid2path under remote dir with subdir mount ========================================================== 19:28:01 (1736987281) [ 4537.420594] Lustre: lustre-MDT0000: subdir mount '/d226c.sanity' is remote and may be slow [ 4539.094144] Lustre: DEBUG MARKER: == sanity test 227: running truncated executable does not cause OOM ========================================================== 19:28:03 (1736987283) [ 4540.804737] Lustre: DEBUG MARKER: == sanity test 228a: try to reuse idle OI blocks ========= 19:28:04 (1736987284) [ 4588.226333] Lustre: DEBUG MARKER: == sanity test 228b: idle OI blocks can be reused after MDT restart ========================================================== 19:28:52 (1736987332) [ 4624.452700] Lustre: Failing over lustre-MDT0000 [ 4624.454368] Lustre: Skipped 1 previous similar message [ 4624.519587] LustreError: 23655:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88007bfb9c80 x1821361567325568/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'osp-syn-0-0.0' [ 4624.583419] LustreError: 3207:0:(ldlm_resource.c:1126:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200003ab1:0x204f:0x0].0x0 (ffff880077961d00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4624.683258] Lustre: server umount lustre-MDT0000 complete [ 4624.684771] Lustre: Skipped 1 previous similar message [ 4625.686277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4625.686573] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4625.686575] LustreError: Skipped 42 previous similar messages [ 4625.696039] Lustre: Skipped 7 previous similar messages [ 4626.844883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4626.876079] LustreError: 166-1: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4626.876104] LustreError: 11-0: MGC192.168.203.102@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 4626.876106] LustreError: Skipped 1 previous similar message [ 4626.882999] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x158aa7dbbeee1a77 to 0x158aa7dbbf048ed5 [ 4626.886305] Lustre: MGC192.168.203.102@tcp: Connection restored to (at 0@lo) [ 4626.888527] Lustre: Skipped 9 previous similar messages [ 4626.960562] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4626.964508] Lustre: Skipped 1 previous similar message [ 4627.584047] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4627.586166] Lustre: Skipped 1 previous similar message [ 4628.025227] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4631.964314] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4631.968337] Lustre: Skipped 1 previous similar message [ 4631.982291] Lustre: lustre-OST0000: deleting orphan objects from 0x0:26557 to 0x0:26593 [ 4631.982292] Lustre: lustre-OST0001: deleting orphan objects from 0x0:26192 to 0x0:26209 [ 4644.863550] Lustre: DEBUG MARKER: == sanity test 228c: NOT shrink the last entry in OI index node to recycle idle leaf ========================================================== 19:29:48 (1736987388) [ 4719.366688] Lustre: DEBUG MARKER: == sanity test 229: getstripe/stat/rm/attr changes work on released files ========================================================== 19:31:03 (1736987463) [ 4721.051497] Lustre: DEBUG MARKER: == sanity test 230a: Create remote directory and files under the remote directory ========================================================== 19:31:04 (1736987464) [ 4722.879603] Lustre: DEBUG MARKER: == sanity test 230b: migrate directory =================== 19:31:06 (1736987466) [ 4732.000708] Lustre: DEBUG MARKER: == sanity test 230c: check directory accessiblity if migration failed ========================================================== 19:31:15 (1736987475) [ 4732.455459] Lustre: *** cfs_fail_loc=1801, val=0*** [ 4732.457614] Lustre: Skipped 1 previous similar message [ 4732.710246] LustreError: 28678:0:(mdd_dir.c:4234:mdd_migrate_cmd_check()) lustre-MDD0000: 'migrate_dir' migration was interrupted, run 'lfs migrate -m 1 -c 1 -H crush migrate_dir' to finish migration. [ 4734.536228] Lustre: DEBUG MARKER: SKIP: sanity test_230d skipping SLOW test 230d [ 4734.966328] Lustre: DEBUG MARKER: == sanity test 230e: migrate mulitple local link files === 19:31:18 (1736987478) [ 4736.832675] Lustre: DEBUG MARKER: == sanity test 230f: migrate mulitple remote link files == 19:31:20 (1736987480) [ 4738.781717] Lustre: DEBUG MARKER: == sanity test 230g: migrate dir to non-exist MDT ======== 19:31:22 (1736987482) [ 4740.426414] Lustre: DEBUG MARKER: == sanity test 230h: migrate .. and root ================= 19:31:24 (1736987484) [ 4742.068531] Lustre: DEBUG MARKER: == sanity test 230i: lfs migrate -m tolerates trailing slashes ========================================================== 19:31:26 (1736987486) [ 4743.858020] Lustre: DEBUG MARKER: == sanity test 230j: DoM file data not changed after dir migration ========================================================== 19:31:27 (1736987487) [ 4745.652305] Lustre: DEBUG MARKER: == sanity test 230k: file data not changed after dir migration ========================================================== 19:31:29 (1736987489) [ 4746.043319] Lustre: DEBUG MARKER: SKIP: sanity test_230k needs >= 4 MDTs [ 4746.527851] Lustre: DEBUG MARKER: == sanity test 230l: readdir between MDTs won't crash ==== 19:31:30 (1736987490) [ 4767.144165] Lustre: DEBUG MARKER: == sanity test 230m: xattrs not changed after dir migration ========================================================== 19:31:51 (1736987511) [ 4769.465257] Lustre: DEBUG MARKER: == sanity test 230n: Dir migration with mirrored file ==== 19:31:53 (1736987513) [ 4771.295668] Lustre: DEBUG MARKER: == sanity test 230o: dir split =========================== 19:31:55 (1736987515) [ 4777.175157] Lustre: DEBUG MARKER: == sanity test 230p: dir merge =========================== 19:32:01 (1736987521) [ 4782.936284] Lustre: DEBUG MARKER: == sanity test 230q: dir auto split ====================== 19:32:06 (1736987526) [ 4792.549900] Lustre: DEBUG MARKER: == sanity test 230r: migrate with too many local locks === 19:32:16 (1736987536) [ 4794.494249] Lustre: DEBUG MARKER: == sanity test 230s: lfs mkdir should return -EEXIST if target exists ========================================================== 19:32:18 (1736987538) [ 4797.426339] Lustre: DEBUG MARKER: == sanity test 230t: migrate directory with project ID set ========================================================== 19:32:21 (1736987541) [ 4799.353331] Lustre: DEBUG MARKER: == sanity test 230u: migrate directory by QOS ============ 19:32:23 (1736987543) [ 4799.754650] Lustre: DEBUG MARKER: SKIP: sanity test_230u needs >= 4 MDTs [ 4800.217846] Lustre: DEBUG MARKER: == sanity test 230v: subdir migrated to the MDT where its parent is located ========================================================== 19:32:24 (1736987544) [ 4800.612661] Lustre: DEBUG MARKER: SKIP: sanity test_230v needs >= 4 MDTs [ 4801.044752] Lustre: DEBUG MARKER: == sanity test 230w: non-recursive mode dir migration ==== 19:32:25 (1736987545) [ 4802.757545] Lustre: DEBUG MARKER: == sanity test 230y: unlink dir with bad hash type ======= 19:32:26 (1736987546) [ 4804.189230] Lustre: *** cfs_fail_loc=1802, val=0*** [ 4804.700198] Lustre: *** cfs_fail_loc=1802, val=0*** [ 4804.701408] Lustre: Skipped 46 previous similar messages [ 4807.857822] Lustre: DEBUG MARKER: == sanity test 230z: resume dir migration with bad hash type ========================================================== 19:32:31 (1736987551) [ 4809.272769] Lustre: *** cfs_fail_loc=1802, val=0*** [ 4809.274435] Lustre: Skipped 21 previous similar messages [ 4816.255405] Lustre: DEBUG MARKER: == sanity test 231a: checking that reading/writing of BRW RPC size results in one RPC ========================================================== 19:32:40 (1736987560) [ 4819.320475] Lustre: DEBUG MARKER: == sanity test 231b: must not assert on fully utilized OST request buffer ========================================================== 19:32:43 (1736987563) [ 4823.856285] Lustre: DEBUG MARKER: == sanity test 232a: failed lock should not block umount ========================================================== 19:32:47 (1736987567) [ 4824.124590] Lustre: *** cfs_fail_loc=31c, val=0*** [ 4825.211544] Lustre: Failing over lustre-OST0000 [ 4825.235619] Lustre: server umount lustre-OST0000 complete [ 4825.734340] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4825.737132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4825.741937] Lustre: Skipped 1 previous similar message [ 4827.573436] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4827.577020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4827.623880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4827.627209] Lustre: Skipped 2 previous similar messages [ 4827.630124] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4828.297808] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4828.636194] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4828.819626] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4828.821246] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28154 to 0x0:28193 [ 4828.827700] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:32106 to 0x280000400:32129 [ 4830.941466] Lustre: DEBUG MARKER: == sanity test 232b: failed data version lock should not block umount ========================================================== 19:32:54 (1736987574) [ 4831.254919] Lustre: *** cfs_fail_loc=31c, val=0*** [ 4831.256614] LustreError: 29407:0:(ldlm_request.c:490:ldlm_cli_enqueue_local()) ### delayed lvb init failed (rc -12) ns: filter-lustre-OST0000_UUID lock: ffff880064ce33c0/0x158aa7dbbf23792e lrc: 2/0,0 mode: --/PR res: [0x6e22:0x0:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x40000000000000 nid: local remote: 0x0 expref: -99 pid: 29407 timeout: 0 lvb_type: 0 [ 4834.551331] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 4834.554468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4835.497809] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4836.335497] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:32106 to 0x280000400:32161 [ 4836.338786] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28195 to 0x0:28225 [ 4837.671726] Lustre: DEBUG MARKER: == sanity test 233a: checking that OBF of the FS root succeeds ========================================================== 19:33:01 (1736987581) [ 4839.408609] Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds ========================================================== 19:33:03 (1736987583) [ 4841.091239] Lustre: DEBUG MARKER: == sanity test 234: xattr cache should not crash on ENOMEM ========================================================== 19:33:05 (1736987585) [ 4842.868502] Lustre: DEBUG MARKER: == sanity test 235: LU-1715: flock deadlock detection does not work properly ========================================================== 19:33:06 (1736987586) [ 4846.583798] Lustre: DEBUG MARKER: == sanity test 236: Layout swap on open unlinked file ==== 19:33:10 (1736987590) [ 4848.550447] Lustre: DEBUG MARKER: == sanity test 238: Verify linkea consistency ============ 19:33:12 (1736987592) [ 4850.447135] Lustre: DEBUG MARKER: == sanity test 239A: osp_sync test ======================= 19:33:14 (1736987594) [ 4868.736900] Lustre: DEBUG MARKER: == sanity test 239a: process invalid osp sync record correctly ========================================================== 19:33:32 (1736987612) [ 4869.650355] Lustre: *** cfs_fail_loc=2100, val=0*** [ 4876.944624] Lustre: DEBUG MARKER: == sanity test 239b: process osp sync record with ENOMEM error correctly ========================================================== 19:33:40 (1736987620) [ 4882.301265] Lustre: DEBUG MARKER: == sanity test 240: race between ldlm enqueue and the connection RPC (no ASSERT) ========================================================== 19:33:46 (1736987626) [ 4882.881118] Lustre: *** cfs_fail_loc=713, val=0*** [ 4882.882796] Lustre: Skipped 1 previous similar message [ 4887.506136] Lustre: DEBUG MARKER: == sanity test 241a: bio vs dio ========================== 19:33:51 (1736987631) [ 4904.384854] Lustre: DEBUG MARKER: == sanity test 241b: dio vs dio ========================== 19:34:08 (1736987648) [ 4910.648808] Lustre: DEBUG MARKER: == sanity test 242: mdt_readpage failure should not cause directory unreadable ========================================================== 19:34:14 (1736987654) [ 4910.961823] Lustre: *** cfs_fail_loc=105, val=0*** [ 4912.898295] Lustre: DEBUG MARKER: == sanity test 243: various group lock tests ============= 19:34:16 (1736987656) [ 4924.112373] Lustre: DEBUG MARKER: == sanity test 244a: sendfile with group lock tests ====== 19:34:28 (1736987668) [ 4935.215318] Lustre: DEBUG MARKER: == sanity test 244b: multi-threaded write with group lock ========================================================== 19:34:39 (1736987679) [ 4937.090043] Lustre: DEBUG MARKER: == sanity test 245: check mdc connection flag/data: multiple modify RPCs ========================================================== 19:34:41 (1736987681) [ 4938.830334] Lustre: DEBUG MARKER: == sanity test 247a: mount subdir as fileset ============= 19:34:42 (1736987682) [ 4941.080517] Lustre: DEBUG MARKER: == sanity test 247b: mount subdir that dose not exist ==== 19:34:45 (1736987685) [ 4942.955110] Lustre: DEBUG MARKER: == sanity test 247c: running fid2path outside subdirectory root ========================================================== 19:34:46 (1736987686) [ 4943.226757] Lustre: lustre-MDT0000: subdir mount '/d247c.sanity' is remote and may be slow [ 4945.234521] Lustre: DEBUG MARKER: == sanity test 247d: running fid2path inside subdirectory root ========================================================== 19:34:49 (1736987689) [ 4945.464856] Lustre: lustre-MDT0000: subdir mount '/d247d.sanity' is remote and may be slow [ 4947.350211] Lustre: DEBUG MARKER: == sanity test 247e: mount .. as fileset ================= 19:34:51 (1736987691) [ 4949.178271] Lustre: DEBUG MARKER: == sanity test 247f: mount striped or remote directory as fileset ========================================================== 19:34:53 (1736987693) [ 4950.220374] Lustre: lustre-MDT0000: subdir mount '/d247f.sanity/remote' refused because 'enable_remote_subdir_mount=0': rc = -66 [ 4952.480543] Lustre: lustre-MDT0000: subdir mount '/d247f.sanity/remote' is remote and may be slow [ 4954.649078] Lustre: DEBUG MARKER: == sanity test 247g: mount striped directory as fileset caches ROOT lookup lock ========================================================== 19:34:58 (1736987698) [ 4955.047790] Lustre: DEBUG MARKER: SKIP: sanity test_247g needs >= 4 MDTs [ 4955.485420] Lustre: DEBUG MARKER: == sanity test 248a: fast read verification ============== 19:34:59 (1736987699) [ 4962.460659] Lustre: DEBUG MARKER: == sanity test 248b: test short_io read and write for both small and large sizes ========================================================== 19:35:06 (1736987706) [ 4969.475616] Lustre: DEBUG MARKER: == sanity test 249: Write above 2T file size ============= 19:35:13 (1736987713) [ 4971.178276] Lustre: DEBUG MARKER: == sanity test 250: Write above 16T limit ================ 19:35:15 (1736987715) [ 4972.868877] Lustre: DEBUG MARKER: == sanity test 251: Handling short read and write correctly ========================================================== 19:35:16 (1736987716) [ 4974.548690] Lustre: DEBUG MARKER: == sanity test 252: check lr_reader tool ================= 19:35:18 (1736987718) [ 4977.295949] Lustre: DEBUG MARKER: == sanity test 253: Check object allocation limit ======== 19:35:21 (1736987721) [ 4988.502440] Lustre: lustre-OST0001: haven't heard from client 25ecb275-3cb2-44bf-9cfa-493b2863ca4d (at 192.168.203.2@tcp) in 47 seconds. I think it's dead, and I am evicting it. exp ffff880079b53000, cur 1736987732 expire 1736987702 last 1736987685 [ 5025.599095] LustreError: 5102:0:(lod_qos.c:1362:lod_ost_alloc_specific()) can't lstripe objid [0x200004283:0x20:0x0]: have 0 want 1 [ 5069.038572] Lustre: DEBUG MARKER: == sanity test 254: Check changelog size ================= 19:36:52 (1736987812) [ 5070.065941] Lustre: lustre-MDD0000: changelog on [ 5070.066992] Lustre: Skipped 1 previous similar message [ 5073.389658] Lustre: lustre-MDD0001: changelog off [ 5073.390992] Lustre: Skipped 1 previous similar message [ 5074.799602] Lustre: DEBUG MARKER: SKIP: sanity test_255a skipping excluded test 255a (base 255) [ 5075.275815] Lustre: DEBUG MARKER: SKIP: sanity test_255b skipping excluded test 255b (base 255) [ 5075.783022] Lustre: DEBUG MARKER: SKIP: sanity test_255c skipping excluded test 255c (base 255) [ 5076.442293] Lustre: DEBUG MARKER: SKIP: sanity test_256 skipping excluded test 256 [ 5076.894166] Lustre: DEBUG MARKER: == sanity test 257: xattr locks are not lost ============= 19:37:00 (1736987820) [ 5077.299277] Lustre: *** cfs_fail_loc=161, val=0*** [ 5079.990910] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5079.995009] Lustre: Skipped 6 previous similar messages [ 5082.708090] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.2@tcp (stopping) [ 5083.400464] Lustre: server umount lustre-MDT0001 complete [ 5083.402298] Lustre: Skipped 1 previous similar message [ 5084.998560] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5085.002298] LustreError: Skipped 13 previous similar messages [ 5086.162805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5086.244126] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 5086.251564] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5086.256593] Lustre: Skipped 1 previous similar message [ 5087.109109] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5087.702143] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5087.705493] Lustre: Skipped 1 previous similar message [ 5088.262543] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 5091.255750] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 5091.258445] Lustre: Skipped 8 previous similar messages [ 5091.264383] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5091.266647] Lustre: Skipped 1 previous similar message [ 5091.279546] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34669 to 0x280000400:34689 [ 5091.279548] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:34214 to 0x2c0000400:34241 [ 5094.924118] Lustre: DEBUG MARKER: == sanity test 258a: verify i_mutex security behavior when suid attributes is set ========================================================== 19:37:18 (1736987838) [ 5096.583985] Lustre: DEBUG MARKER: == sanity test 258b: verify i_mutex security behavior ==== 19:37:20 (1736987840) [ 5098.310008] Lustre: DEBUG MARKER: == sanity test 259: crash at delayed truncate ============ 19:37:22 (1736987842) [ 5108.525563] Lustre: *** cfs_fail_loc=2301, val=0*** [ 5109.259498] Lustre: Failing over lustre-OST0000 [ 5109.261225] Lustre: Skipped 2 previous similar messages [ 5110.038374] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5110.040928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5110.044085] Lustre: Skipped 6 previous similar messages [ 5111.848996] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5111.854109] LDISKFS-fs (dm-2): 1 truncate cleaned up [ 5111.855365] LDISKFS-fs (dm-2): recovery complete [ 5111.858235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5111.901185] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 5112.766840] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5113.475117] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28246 to 0x0:28289 [ 5113.475944] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34669 to 0x280000400:34721 [ 5117.204990] Lustre: DEBUG MARKER: == sanity test 260: Check mdc_close fail ================= 19:37:41 (1736987861) [ 5118.856745] Lustre: DEBUG MARKER: == sanity test 270a: DoM: basic functionality tests ====== 19:37:42 (1736987862) [ 5122.124202] Lustre: DEBUG MARKER: == sanity test 270b: DoM: maximum size overflow checks for DoM-only file ========================================================== 19:37:46 (1736987866) [ 5123.866868] Lustre: DEBUG MARKER: == sanity test 270c: DoM: DoM EA inheritance tests ======= 19:37:47 (1736987867) [ 5125.585394] Lustre: DEBUG MARKER: == sanity test 270d: DoM: change striping from DoM to RAID0 ========================================================== 19:37:49 (1736987869) [ 5127.307103] Lustre: DEBUG MARKER: == sanity test 270e: DoM: lfs find with DoM files test === 19:37:51 (1736987871) [ 5129.265883] Lustre: DEBUG MARKER: == sanity test 270f: DoM: maximum DoM stripe size checks ========================================================== 19:37:53 (1736987873) [ 5130.828088] Lustre: Increasing provided stripe size to a minimum value 64 [ 5134.035459] Lustre: DEBUG MARKER: == sanity test 270g: DoM: default DoM stripe size depends on free space ========================================================== 19:37:57 (1736987877) [ 5135.948865] Lustre: *** cfs_fail_loc=168, val=0*** [ 5136.896401] Lustre: *** cfs_fail_loc=168, val=0*** [ 5136.897735] Lustre: Skipped 5 previous similar messages [ 5138.338777] Lustre: *** cfs_fail_loc=168, val=0*** [ 5138.340248] Lustre: Skipped 8 previous similar messages [ 5140.813899] Lustre: DEBUG MARKER: == sanity test 270h: DoM: DoM stripe removal when disabled on server ========================================================== 19:38:04 (1736987884) [ 5143.197767] Lustre: DEBUG MARKER: == sanity test 270i: DoM: setting invalid DoM striping should fail ========================================================== 19:38:07 (1736987887) [ 5144.788229] Lustre: DEBUG MARKER: == sanity test 271a: DoM: data is cached for read after write ========================================================== 19:38:08 (1736987888) [ 5146.383767] Lustre: DEBUG MARKER: == sanity test 271b: DoM: no glimpse RPC for stat (DoM only file) ========================================================== 19:38:10 (1736987890) [ 5148.096525] Lustre: DEBUG MARKER: == sanity test 271ba: DoM: no glimpse RPC for stat (combined file) ========================================================== 19:38:12 (1736987892) [ 5149.772868] Lustre: DEBUG MARKER: == sanity test 271c: DoM: IO lock at open saves enqueue RPCs ========================================================== 19:38:13 (1736987893) [ 5168.622893] Lustre: DEBUG MARKER: == sanity test 271d: DoM: read on open (1K file in reply buffer) ========================================================== 19:38:32 (1736987912) [ 5170.451712] Lustre: DEBUG MARKER: == sanity test 271f: DoM: read on open (200K file and read tail) ========================================================== 19:38:34 (1736987914) [ 5172.201130] Lustre: DEBUG MARKER: == sanity test 271g: Discard DoM data vs client flush race ========================================================== 19:38:36 (1736987916) [ 5174.938262] Lustre: DEBUG MARKER: == sanity test 272a: DoM migration: new layout with the same DOM component ========================================================== 19:38:38 (1736987918) [ 5176.770174] Lustre: DEBUG MARKER: == sanity test 272b: DoM migration: DOM file to the OST-striped file (plain) ========================================================== 19:38:40 (1736987920) [ 5179.443080] Lustre: DEBUG MARKER: == sanity test 272c: DoM migration: DOM file to the OST-striped file (composite) ========================================================== 19:38:43 (1736987923) [ 5181.733294] Lustre: DEBUG MARKER: == sanity test 272d: DoM mirroring: OST-striped mirror to DOM file ========================================================== 19:38:45 (1736987925) [ 5183.915862] Lustre: DEBUG MARKER: == sanity test 272e: DoM mirroring: DOM mirror to the OST-striped file ========================================================== 19:38:47 (1736987927) [ 5185.748258] Lustre: DEBUG MARKER: == sanity test 272f: DoM migration: OST-striped file to DOM file ========================================================== 19:38:49 (1736987929) [ 5187.600375] Lustre: DEBUG MARKER: == sanity test 273a: DoM: layout swapping should fail with DOM ========================================================== 19:38:51 (1736987931) [ 5189.347485] Lustre: DEBUG MARKER: == sanity test 273b: DoM: race writeback and object destroy ========================================================== 19:38:53 (1736987933) [ 5189.642814] LustreError: 12787:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 16b sleeping for 2000ms [ 5191.646120] LustreError: 12787:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 16b awake [ 5193.449793] Lustre: DEBUG MARKER: == sanity test 275: Read on a canceled duplicate lock ==== 19:38:57 (1736987937) [ 5196.497220] Lustre: DEBUG MARKER: == sanity test 276: Race between mount and obd_statfs ==== 19:39:00 (1736987940) [ 5199.190151] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5199.193408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5199.219878] LustreError: 18563:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 5199.232388] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 5200.156007] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5200.942715] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34740 to 0x280000400:34785 [ 5200.943387] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28302 to 0x0:28321 [ 5201.168864] LustreError: 19784:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 5203.695296] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5203.699518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5203.738865] LustreError: 21182:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 5203.756333] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 5204.629169] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5205.463433] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28302 to 0x0:28353 [ 5205.464453] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34740 to 0x280000400:34817 [ 5207.760844] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5207.763829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5207.805056] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 5208.649293] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5209.399237] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34740 to 0x280000400:34849 [ 5209.400136] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28302 to 0x0:28385 [ 5209.606336] LustreError: 25024:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 5209.607983] LustreError: 25024:0:(obd_class.h:1061:obd_statfs()) Skipped 1 previous similar message [ 5211.788475] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5211.791407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5212.653549] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5213.143294] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28302 to 0x0:28417 [ 5213.144266] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34740 to 0x280000400:34881 [ 5213.656163] LustreError: 27591:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 5215.797821] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5215.800931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5215.841805] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 5215.844598] Lustre: Skipped 1 previous similar message [ 5216.690532] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5217.617273] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34740 to 0x280000400:34913 [ 5217.620384] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28302 to 0x0:28449 [ 5219.767556] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5219.770479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5220.659208] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5221.304020] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34740 to 0x280000400:34945 [ 5221.306425] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28302 to 0x0:28481 [ 5223.785879] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5223.791513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5223.832222] LustreError: 1694:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 5223.834560] LustreError: 1694:0:(obd_class.h:1061:obd_statfs()) Skipped 4 previous similar messages [ 5224.811167] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5225.282562] Lustre: lustre-OST0000: deleting orphan objects from 0x0:28302 to 0x0:28513 [ 5225.283460] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34740 to 0x280000400:34977 [ 5227.857202] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5227.860304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5228.774152] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5229.719116] LustreError: 5590:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery [ 5229.723615] Lustre: 4358:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5229.727301] Lustre: 4358:0:(ofd_obd.c:554:ofd_postrecov()) lustre-OST0000: auto trigger paused LFSCK failed: rc = -6 [ 5231.899542] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5231.902603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5231.945174] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 5231.947951] Lustre: Skipped 3 previous similar messages [ 5232.765174] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5233.697015] LustreError: 8259:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery [ 5233.699761] Lustre: 7032:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5233.702321] Lustre: 7032:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages [ 5233.705199] Lustre: 7032:0:(ofd_obd.c:554:ofd_postrecov()) lustre-OST0000: auto trigger paused LFSCK failed: rc = -6 [ 5235.860423] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5235.864114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5236.730283] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5237.654126] LustreError: 10913:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery [ 5237.656454] Lustre: 9707:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5237.658584] Lustre: 9707:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages [ 5237.661048] Lustre: 9707:0:(ofd_obd.c:554:ofd_postrecov()) lustre-OST0000: auto trigger paused LFSCK failed: rc = -6 [ 5239.751025] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5239.754162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5240.668064] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5241.627064] LustreError: 13510:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 5241.627305] LustreError: 13503:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery [ 5241.627319] Lustre: 12288:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5241.627321] Lustre: 12288:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages [ 5241.628106] Lustre: 12288:0:(ofd_obd.c:554:ofd_postrecov()) lustre-OST0000: auto trigger paused LFSCK failed: rc = -6 [ 5241.637164] LustreError: 13510:0:(obd_class.h:1061:obd_statfs()) Skipped 15 previous similar messages [ 5243.764628] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5243.768089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5244.654871] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5247.790242] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5247.793535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5248.636416] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5249.557799] LustreError: 18732:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery [ 5249.560219] LustreError: 18732:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) Skipped 1 previous similar message [ 5249.562896] Lustre: 17506:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5249.565186] Lustre: 17506:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 5 previous similar messages [ 5249.567881] Lustre: 17506:0:(ofd_obd.c:554:ofd_postrecov()) lustre-OST0000: auto trigger paused LFSCK failed: rc = -6 [ 5249.570091] Lustre: 17506:0:(ofd_obd.c:554:ofd_postrecov()) Skipped 1 previous similar message [ 5251.651022] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5251.653819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5252.502535] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5255.617868] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5255.620709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5256.493031] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5259.606831] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5259.610226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5260.508280] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5261.660988] LustreError: 26554:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery [ 5261.663650] LustreError: 26554:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) Skipped 2 previous similar messages [ 5261.666339] Lustre: 25269:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5261.669425] Lustre: 25269:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 8 previous similar messages [ 5261.674317] Lustre: 25269:0:(ofd_obd.c:554:ofd_postrecov()) lustre-OST0000: auto trigger paused LFSCK failed: rc = -6 [ 5261.678085] Lustre: 25269:0:(ofd_obd.c:554:ofd_postrecov()) Skipped 2 previous similar messages [ 5264.115327] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5264.118795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5264.160759] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 5264.164274] Lustre: Skipped 7 previous similar messages [ 5265.042089] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5268.122852] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5268.126107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5269.053145] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5272.339349] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5272.344410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5273.375666] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5274.407483] LustreError: 2062:0:(obd_class.h:1061:obd_statfs()) Device 22 not setup [ 5274.410892] LustreError: 2062:0:(obd_class.h:1061:obd_statfs()) Skipped 20 previous similar messages [ 5275.407053] Lustre: 23652:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1736987977/real 1736987977] req@ffff88009e03bdc0 x1821361572724032/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1736988019 ref 1 fl Rpc:XQr/c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' [ 5276.664924] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 5276.669967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5277.657554] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5280.781584] Lustre: DEBUG MARKER: == sanity test 277: Direct IO shall drop page cache ====== 19:40:24 (1736988024) [ 5283.199654] Lustre: DEBUG MARKER: == sanity test 278: Race starting MDS between MDTs stop/start ========================================================== 19:40:26 (1736988026) [ 5284.172103] LustreError: 23652:0:(import.c:692:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5284.202082] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:34 [ 5284.219202] LustreError: 5476:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 60c sleeping [ 5286.976367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5287.024287] LustreError: 6055:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 60c waking [ 5287.025292] LustreError: 166-1: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5287.025652] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x158aa7dbbf048ed5 to 0x158aa7dbbf2e5611 [ 5287.034534] LustreError: 5476:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 60c awake: rc=2186 [ 5289.113474] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 5289.116368] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 3 known clients (1 recovered, 0 in progress, and 1 evicted) to recover in 1:29 [ 5290.188224] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5292.575886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5292.651885] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 3 known clients (1 recovered, 0 in progress, and 1 evicted) to recover in 1:25 [ 5292.654914] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 5292.660231] Lustre: Skipped 1 previous similar message [ 5293.527709] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5294.638196] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 [ 5297.671205] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 3 known clients (0 recovered, 0 in progress, and 2 evicted) to recover in 1:20 [ 5297.677757] Lustre: Skipped 1 previous similar message [ 5297.687907] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:34251 to 0x2c0000400:34273 [ 5297.694186] Lustre: lustre-OST0001: deleting orphan objects from 0x0:27794 to 0x0:27809 [ 5301.699072] Lustre: DEBUG MARKER: == sanity test 280: Race between MGS umount and client llog processing ========================================================== 19:40:45 (1736988045) [ 5302.218049] LustreError: 6062:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 15e sleeping [ 5302.678560] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0000-mdtlov_UUID (at 0@lo), waiting for 3 known clients (0 recovered, 0 in progress, and 2 evicted) to recover in 1:15 [ 5302.682480] Lustre: Skipped 1 previous similar message [ 5303.693441] LustreError: 8405:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 5303.696259] LustreError: 6062:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 15e awake: rc=3523 [ 5304.701599] LustreError: 8405:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 5306.953862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5306.979570] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 3 known clients (0 recovered, 0 in progress, and 2 evicted) to recover in 1:11 [ 5306.979623] LustreError: 166-1: MGC192.168.203.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5306.979887] Lustre: Evicted from MGS (at 192.168.203.102@tcp) after server handle changed from 0x158aa7dbbf2e5611 to 0x158aa7dbbf2e5a4e [ 5306.990891] LustreError: 9006:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 5308.036888] LustreError: 9007:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 5309.040913] LustreError: 9007:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 5311.046677] LustreError: 9007:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 5311.049030] LustreError: 9007:0:(libcfs_fail.h:180:cfs_race()) Skipped 1 previous similar message [ 5315.058456] LustreError: 9005:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 5315.060630] LustreError: 9005:0:(libcfs_fail.h:180:cfs_race()) Skipped 3 previous similar messages [ 5317.960668] Lustre: DEBUG MARKER: oleg302-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5319.062560] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 3 known clients (0 recovered, 0 in progress, and 2 evicted) to recover in 0:59 [ 5319.067100] Lustre: Skipped 7 previous similar messages [ 5319.078113] Lustre: lustre-OST0001: deleting orphan objects from 0x0:27794 to 0x0:27841 [ 5323.524238] LustreError: 9005:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 15e waking [ 5323.526659] LustreError: 9005:0:(libcfs_fail.h:180:cfs_race()) Skipped 6 previous similar messages [ 5326.384334] Lustre: DEBUG MARKER: == sanity test 300a: basic striped dir sanity test ======= 19:41:10 (1736988070) [ 5328.477451] Lustre: DEBUG MARKER: == sanity test 300b: check ctime/mtime for striped dir === 19:41:12 (1736988072) [ 5339.094509] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 3 known clients (0 recovered, 0 in progress, and 2 evicted) to recover in 0:39 [ 5339.097945] Lustre: Skipped 10 previous similar messages [ 5350.365495] Lustre: DEBUG MARKER: == sanity test 300c: chown [ 5368.761507] Lustre: DEBUG MARKER: == sanity test 300d: check default stripe under striped directory ========================================================== 19:41:52 (1736988112) [ 5369.142504] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 5369.144784] LustreError: Skipped 35 previous similar messages [ 5369.412644] Lustre: DEBUG MARKER: sanity test_300d: @@@@@@ FAIL: wrong stripe 1 for /mnt/lustre/d300d.sanity/remote_striped_dir/f6