[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 343436056 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.443529] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.446287] pid_max: default: 32768 minimum: 301 [ 0.447685] Security Framework initialized [ 0.448868] SELinux: Initializing. [ 0.449942] SELinux: Starting in permissive mode [ 0.451422] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.455349] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.458009] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.460047] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.462763] Initializing cgroup subsys memory [ 0.464139] Initializing cgroup subsys devices [ 0.465584] Initializing cgroup subsys freezer [ 0.466914] Initializing cgroup subsys net_cls [ 0.468311] Initializing cgroup subsys blkio [ 0.469656] Initializing cgroup subsys perf_event [ 0.471273] Initializing cgroup subsys hugetlb [ 0.472646] Initializing cgroup subsys pids [ 0.474074] Initializing cgroup subsys net_prio [ 0.475710] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.478702] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.480381] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.481965] tlb_flushall_shift: 6 [ 0.483160] FEATURE SPEC_CTRL Present [ 0.484279] FEATURE IBPB_SUPPORT Present [ 0.485542] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.487542] Spectre V2 : Vulnerable [ 0.488629] Speculative Store Bypass: Vulnerable [ 0.491070] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.499288] ACPI: Core revision 20130517 [ 0.502246] ACPI: All ACPI Tables successfully acquired [ 0.504027] ftrace: allocating 30294 entries in 119 pages [ 0.560519] Enabling x2apic [ 0.561430] Enabled x2apic [ 0.562685] Switched APIC routing to physical x2apic. [ 0.566214] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.568296] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.571475] TSC deadline timer enabled [ 0.571660] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.574357] ... version: 2 [ 0.575549] ... bit width: 48 [ 0.576736] ... generic registers: 4 [ 0.577520] ... value mask: 0000ffffffffffff [ 0.579193] ... max period: 00007fffffffffff [ 0.580623] ... fixed-purpose events: 3 [ 0.581580] ... event mask: 000000070000000f [ 0.582956] KVM setup paravirtual spinlock [ 0.587867] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.591153] KVM setup async PF for cpu 1 [ 0.592494] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.595138] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.598168] KVM setup async PF for cpu 2 [ 0.586272] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.598636] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.600613] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.601682] Brought up 4 CPUs [ 0.601714] KVM setup async PF for cpu 3 [ 0.601721] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.603928] smpboot: Max logical packages: 1 [ 0.604802] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.608218] devtmpfs: initialized [ 0.609141] x86/mm: Memory block size: 128MB [ 0.613104] EVM: security.selinux [ 0.614262] EVM: security.ima [ 0.614946] EVM: security.capability [ 0.616963] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.618484] NET: Registered protocol family 16 [ 0.619508] cpuidle: using governor haltpoll [ 0.620922] ACPI: bus type PCI registered [ 0.621907] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.623236] PCI: Using configuration type 1 for base access [ 0.624387] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.631405] ACPI: Added _OSI(Module Device) [ 0.632749] ACPI: Added _OSI(Processor Device) [ 0.634116] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.635576] ACPI: Added _OSI(Processor Aggregator Device) [ 0.637261] ACPI: Added _OSI(Linux-Dell-Video) [ 0.639466] ACPI: EC: Look up EC in DSDT [ 0.641977] ACPI: Interpreter enabled [ 0.643184] ACPI: (supports S0 S3 S4 S5) [ 0.644381] ACPI: Using IOAPIC for interrupt routing [ 0.645967] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.648895] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.655932] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.657852] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.659947] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.661997] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.665714] acpiphp: Slot [2] registered [ 0.666688] acpiphp: Slot [3] registered [ 0.667939] acpiphp: Slot [4] registered [ 0.668833] acpiphp: Slot [5] registered [ 0.669548] acpiphp: Slot [6] registered [ 0.670801] acpiphp: Slot [7] registered [ 0.671756] acpiphp: Slot [8] registered [ 0.672678] acpiphp: Slot [9] registered [ 0.673420] acpiphp: Slot [10] registered [ 0.674362] acpiphp: Slot [11] registered [ 0.675189] acpiphp: Slot [12] registered [ 0.675940] acpiphp: Slot [13] registered [ 0.677133] acpiphp: Slot [14] registered [ 0.677937] acpiphp: Slot [15] registered [ 0.678983] acpiphp: Slot [16] registered [ 0.680115] acpiphp: Slot [17] registered [ 0.680988] acpiphp: Slot [18] registered [ 0.681821] acpiphp: Slot [19] registered [ 0.682972] acpiphp: Slot [20] registered [ 0.683860] acpiphp: Slot [21] registered [ 0.684627] acpiphp: Slot [22] registered [ 0.685633] acpiphp: Slot [23] registered [ 0.686860] acpiphp: Slot [24] registered [ 0.687697] acpiphp: Slot [25] registered [ 0.688966] acpiphp: Slot [26] registered [ 0.690324] acpiphp: Slot [27] registered [ 0.691760] acpiphp: Slot [28] registered [ 0.693112] acpiphp: Slot [29] registered [ 0.694439] acpiphp: Slot [30] registered [ 0.695749] acpiphp: Slot [31] registered [ 0.696579] PCI host bridge to bus 0000:00 [ 0.697744] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.699232] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.701024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.702987] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.705200] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.707382] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.709282] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.710079] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.711051] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.715601] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.717030] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.718813] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.720014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.721800] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.724180] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.724590] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.725816] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.727446] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.728943] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.736745] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.738261] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.739315] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.742044] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.744737] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.753798] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.760829] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.763636] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.766167] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.775930] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.779534] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.784637] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.799989] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.814545] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.821621] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.825738] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.829439] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.843927] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.853114] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.857497] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.863558] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.880909] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.887773] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.892708] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.897147] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.913561] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.920282] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.923335] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.925690] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.927964] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.929996] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.933755] vgaarb: loaded [ 0.934668] SCSI subsystem initialized [ 0.935558] ACPI: bus type USB registered [ 0.936904] usbcore: registered new interface driver usbfs [ 0.938559] usbcore: registered new interface driver hub [ 0.940288] usbcore: registered new device driver usb [ 0.942344] PCI: Using ACPI for IRQ routing [ 0.943088] PCI: pci_cache_line_size set to 64 bytes [ 0.943427] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 0.943433] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 0.943435] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 0.943778] NetLabel: Initializing [ 0.944567] NetLabel: domain hash size = 128 [ 0.945730] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.946946] NetLabel: unlabeled traffic allowed by default [ 0.948870] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.950287] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.956528] amd_nb: Cannot enumerate AMD northbridges [ 0.958216] Switched to clocksource kvm-clock [ 0.976916] pnp: PnP ACPI init [ 0.977684] ACPI: bus type PNP registered [ 0.979146] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 0.979249] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 0.979303] pnp 00:02: [dma 2] [ 0.979338] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 0.979426] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.979506] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.979590] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.980014] pnp: PnP ACPI: found 6 devices [ 0.981443] ACPI: bus type PNP unregistered [ 0.994498] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.994504] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.994507] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.994510] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.994512] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 0.994579] NET: Registered protocol family 2 [ 0.996291] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 0.998759] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.001160] TCP: Hash tables configured (established 32768 bind 32768) [ 1.002625] TCP: reno registered [ 1.003426] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.004840] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.006824] NET: Registered protocol family 1 [ 1.008092] RPC: Registered named UNIX socket transport module. [ 1.010310] RPC: Registered udp transport module. [ 1.011864] RPC: Registered tcp transport module. [ 1.013421] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.015250] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.016389] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.017857] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.019557] PCI: CLS 0 bytes, default 64 [ 1.020539] Unpacking initramfs... [ 2.316502] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 2.320153] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.321913] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 2.325021] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 2.327525] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 2.329261] RAPL PMU: hw unit of domain package 2^-0 Joules [ 2.330984] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 2.336152] cryptomgr_test (51) used greatest stack depth: 14128 bytes left [ 2.336759] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 2.336819] Initialise system trusted keyring [ 2.368067] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 2.370053] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 2.376288] zpool: loaded [ 2.376845] zbud: loaded [ 2.378051] VFS: Disk quotas dquot_6.6.0 [ 2.379233] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.381293] NFS: Registering the id_resolver key type [ 2.383397] Key type id_resolver registered [ 2.384790] Key type id_legacy registered [ 2.386070] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 2.388711] Key type big_key registered [ 2.390248] SELinux: Registering netfilter hooks [ 2.391636] cryptomgr_test (57) used greatest stack depth: 13968 bytes left [ 2.394780] NET: Registered protocol family 38 [ 2.397214] Key type asymmetric registered [ 2.398727] Asymmetric key parser 'x509' registered [ 2.400379] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 2.402970] io scheduler noop registered [ 2.404081] io scheduler deadline registered (default) [ 2.405510] io scheduler cfq registered [ 2.406552] io scheduler mq-deadline registered [ 2.407778] io scheduler kyber registered [ 2.412013] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 2.413644] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 2.415685] intel_idle: does not run on family 6 model 62 [ 2.415817] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.417904] ACPI: Power Button [PWRF] [ 2.419440] GHES: HEST is not enabled! [ 2.479233] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 2.539093] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 2.659463] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 2.725575] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 2.865252] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.894674] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 2.922825] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 2.925745] Non-volatile memory driver v1.3 [ 2.927417] Linux agpgart interface v0.103 [ 2.928628] crash memory driver: version 1.1 [ 2.930447] nbd: registered device at major 43 [ 2.934863] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 2.934905] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 2.934945] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 2.934987] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 2.935036] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 2.941465] virtio_blk virtio1: [vda] 115920 512-byte logical blocks (59.3 MB/56.6 MiB) [ 2.947598] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 2.947654] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 2.947699] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 2.947755] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 2.947800] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 2.955015] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.962736] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 2.962774] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 2.962807] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 2.962845] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 2.962878] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 2.968063] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.976015] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 2.976061] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 2.976093] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 2.976131] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 2.976155] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 2.982076] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.987601] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 2.987653] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 2.987703] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 2.987744] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 2.987786] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 2.992987] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.999161] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 2.999198] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 2.999264] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 2.999307] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 2.999347] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 3.004803] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.013593] rdac: device handler registered [ 3.014748] hp_sw: device handler registered [ 3.015701] emc: device handler registered [ 3.016566] libphy: Fixed MDIO Bus: probed [ 3.020034] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 3.021595] ehci-pci: EHCI PCI platform driver [ 3.023030] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 3.024417] ohci-pci: OHCI PCI platform driver [ 3.025340] uhci_hcd: USB Universal Host Controller Interface driver [ 3.026802] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.030111] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.031633] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.033290] mousedev: PS/2 mouse device common for all mice [ 3.035756] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.035792] rtc_cmos 00:05: RTC can wake from S4 [ 3.036975] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 3.037448] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.041867] hidraw: raw HID events driver (C) Jiri Kosina [ 3.042127] usbcore: registered new interface driver usbhid [ 3.042128] usbhid: USB HID core driver [ 3.042189] drop_monitor: Initializing network drop monitor service [ 3.042251] Netfilter messages via NETLINK v0.30. [ 3.042320] TCP: cubic registered [ 3.042326] Initializing XFRM netlink socket [ 3.042609] NET: Registered protocol family 10 [ 3.043187] NET: Registered protocol family 17 [ 3.043765] Key type dns_resolver registered [ 3.044149] mce: Using 10 MCE banks [ 3.044421] Loading compiled-in X.509 certificates [ 3.045396] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 3.045423] registered taskstats version 1 [ 3.047863] modprobe (72) used greatest stack depth: 13456 bytes left [ 3.050323] Key type trusted registered [ 3.055114] Key type encrypted registered [ 3.055164] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 3.058351] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 3.059023] rtc_cmos 00:05: setting system clock to 2025-05-07 13:44:25 UTC (1746625465) [ 3.082961] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 3.085387] Write protecting the kernel read-only data: 12288k [ 3.087673] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 3.090469] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 3.099625] random: systemd: uninitialized urandom read (16 bytes read) [ 3.103476] random: systemd: uninitialized urandom read (16 bytes read) [ 3.105535] random: systemd: uninitialized urandom read (16 bytes read) [ 3.109624] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 3.114755] systemd[1]: Detected virtualization kvm. [ 3.116703] systemd[1]: Detected architecture x86-64. [ 3.118679] systemd[1]: Running in initial RAM disk. [ 3.123630] systemd[1]: No hostname configured. [ 3.125235] systemd[1]: Set hostname to . [ 3.127115] random: systemd: uninitialized urandom read (16 bytes read) [ 3.128702] systemd[1]: Initializing machine ID from random generator. [ 3.172460] dracut-rootfs-g (86) used greatest stack depth: 13264 bytes left [ 3.175703] random: systemd: uninitialized urandom read (16 bytes read) [ 3.180468] random: systemd: uninitialized urandom read (16 bytes read) [ 3.182187] random: systemd: uninitialized urandom read (16 bytes read) [ 3.184434] random: systemd: uninitialized urandom read (16 bytes read) [ 3.187779] random: systemd: uninitialized urandom read (16 bytes read) [ 3.189978] random: systemd: uninitialized urandom read (16 bytes read) [ 3.201515] systemd[1]: Created slice Root Slice. [ 3.204614] systemd[1]: Reached target Swap. [ 3.207745] systemd[1]: Reached target Timers. [ 3.211173] systemd[1]: Listening on udev Kernel Socket. [ 3.215471] systemd[1]: Listening on Journal Socket. [ 3.219956] systemd[1]: Created slice System Slice. [ 3.225531] systemd[1]: Starting Journal Service... [ 3.229283] systemd[1]: Reached target Slices. [ 3.233671] systemd[1]: Starting Setup Virtual Console... [ 3.239365] systemd[1]: Starting Load Kernel Modules... [ 3.244323] systemd[1]: Starting dracut cmdline hook... [ 3.249374] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 3.255283] systemd[1]: Reached target Local File Systems. [ 3.259713] systemd[1]: Listening on udev Control Socket. [ 3.263715] systemd[1]: Reached target Sockets. [ 3.268028] systemd[1]: Started Journal Service. [ 3.334343] tsc: Refined TSC clocksource calibration: 2399.978 MHz [ 3.447288] random: fast init done [ 3.719867] libata version 3.00 loaded. [ 3.722338] ata_piix 0000:00:01.1: version 2.13 [ 3.726057] scsi host0: ata_piix [ 3.728013] scsi host1: ata_piix [ 3.729967] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 3.732787] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 3.903815] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 3.960479] ip (321) used greatest stack depth: 13080 bytes left [ 4.003607] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 4.005508] ip (344) used greatest stack depth: 12464 bytes left [ 4.020249] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 7.138898] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 7.587824] systemd-journald[100]: Received SIGTERM from PID 1 (systemd). [ 7.925186] SELinux: Disabled at runtime. [ 7.927338] SELinux: Unregistering netfilter hooks [ 8.026114] ip_tables: (C) 2000-2006 Netfilter Core Team [ 8.030135] systemd[1]: Inserted module 'ip_tables' [ 8.919594] systemd-journald[565]: Received request to flush runtime journal from PID 1 [ 9.158988] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 9.204434] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 9.241870] cryptd: max_cpu_qlen set to 1000 [ 9.283636] AVX version of gcm_enc/dec engaged. [ 9.284620] AES CTR mode by8 optimization enabled [ 9.348330] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 9.372170] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 9.377087] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 9.382375] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 9.566768] EDAC MC: Ver: 3.0.0 [ 9.593556] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 9.593570] EDAC sbridge: Ver: 1.1.2 [ 12.677436] mount.nfs (768) used greatest stack depth: 10704 bytes left [ 22.438699] device-mapper: uevent: version 1.0.3 [ 22.440227] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 26.814115] libcfs: loading out-of-tree module taints kernel. [ 26.816578] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 26.844953] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_hostid [ 31.881979] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing load_modules_local [ 32.074635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 32.078427] alg: No test for adler32 (adler32-zlib) [ 32.970727] Lustre: Lustre: Build Version: 2.16.53_74_g832d326 [ 33.160260] LNet: Added LNI 192.168.203.140@tcp [8/256/0/180] [ 33.161844] LNet: Accept secure, port 988 [ 34.709468] Key type lgssc registered [ 35.070352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 38.112787] icp: module license 'CDDL' taints kernel. [ 38.114451] Disabling lock debugging due to kernel taint [ 39.608695] ZFS: Loaded module v2.1.5-1, ZFS pool version 5000, ZFS filesystem version 5 [ 42.793621] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 47.191270] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 49.086110] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 49.091152] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 51.018054] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 51.020450] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 53.957447] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing load_modules_local [ 56.071208] random: crng init done [ 56.863181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 56.883068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 56.889007] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 57.964193] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 57.974827] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 58.009387] Lustre: lustre-MDT0000: new disk, initializing [ 58.028810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 58.036042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 58.063967] mount.lustre (7186) used greatest stack depth: 10112 bytes left [ 58.836896] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 63.037777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 63.062594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 63.095806] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 63.099775] Lustre: 8343:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.140@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 63.106839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 63.108356] Lustre: Skipped 1 previous similar message [ 63.136083] Lustre: lustre-MDT0001: new disk, initializing [ 63.147950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 63.156064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 63.164423] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 63.858245] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 65.807697] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 68.123693] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 68.127248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 68.144005] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 68.147267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 68.205713] Lustre: 9544:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.140@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 68.215463] Lustre: lustre-OST0000: new disk, initializing [ 68.218516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 68.229634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 68.625248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 68.629847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 68.648472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 69.478265] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 73.899830] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 73.903339] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 73.921759] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 73.926519] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 73.954139] Lustre: 10885:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.140@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 73.962897] Lustre: lustre-OST0001: new disk, initializing [ 73.964577] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 73.977860] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 75.292867] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 77.067699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 77.070811] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 77.088070] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 80.567164] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 86.815968] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 93.002143] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing check_logdir /tmp/testlogs/ [ 93.864640] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing yml_node [ 94.851916] Lustre: DEBUG MARKER: Client: 2.16.53.74 [ 95.497720] Lustre: DEBUG MARKER: MDS: 2.16.53.74 [ 96.101617] Lustre: DEBUG MARKER: OSS: 2.16.53.74 [ 96.466284] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-scrub ============----- Wed May 7 09:45:59 EDT 2025 [ 100.735531] Lustre: DEBUG MARKER: excepting tests: [ 102.426784] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing load_modules_local [ 107.113070] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 107.113466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 107.119399] Lustre: Skipped 3 previous similar messages [ 109.831503] Lustre: server umount lustre-MDT0000 complete [ 111.972769] LustreError: 7195:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746625574 with bad export cookie 18317602469857461280 [ 111.973764] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 111.979109] LustreError: 7195:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 112.088724] Lustre: server umount lustre-MDT0001 complete [ 124.323853] Lustre: server umount lustre-OST0000 complete [ 136.555103] Lustre: server umount lustre-OST0001 complete [ 138.324185] device-mapper: core: cleaned up [ 140.965271] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing unload_modules_local [ 141.595164] Key type lgssc unregistered [ 141.680629] LNet: 17282:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 142.686502] LNet: Removed LNI 192.168.203.140@tcp [ 144.157678] device-mapper: uevent: version 1.0.3 [ 144.159609] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 148.418261] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_hostid [ 152.724226] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing load_modules_local [ 152.915137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 152.919719] alg: No test for adler32 (adler32-zlib) [ 153.735321] Lustre: Lustre: Build Version: 2.16.53_74_g832d326 [ 153.785225] LNet: Added LNI 192.168.203.140@tcp [8/256/0/180] [ 153.787802] LNet: Accept secure, port 988 [ 155.324355] Key type lgssc registered [ 155.547843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 158.315803] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 160.659173] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 162.694861] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 162.699095] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 164.748841] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 164.753622] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 168.192166] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing load_modules_local [ 171.506382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 171.524109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 171.530006] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.601761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 172.614778] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 172.649197] Lustre: lustre-MDT0000: new disk, initializing [ 172.677186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.685225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 172.712317] mount.lustre (22526) used greatest stack depth: 9968 bytes left [ 173.528969] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.884060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 177.905917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 177.925083] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 177.930334] Lustre: 23684:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.140@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 177.937947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 177.940590] Lustre: Skipped 1 previous similar message [ 177.975156] Lustre: lustre-MDT0001: new disk, initializing [ 177.989529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 177.998069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 178.001383] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 178.789892] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.818595] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 183.105671] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 183.109067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 183.124427] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 183.127184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 183.180131] Lustre: 24890:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.140@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 183.188469] Lustre: lustre-OST0000: new disk, initializing [ 183.190282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 183.200812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.337169] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.241603] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 188.244782] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 188.265349] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 188.268529] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 188.286225] Lustre: 26154:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.140@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 188.293947] Lustre: lustre-OST0001: new disk, initializing [ 188.295957] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 188.307327] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 189.620822] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.654869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 192.658256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 192.663454] Lustre: Skipped 1 previous similar message [ 192.669900] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 192.677864] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 199.422984] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 201.552271] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 208.657971] Lustre: DEBUG MARKER: === sanity-scrub: finish setup 09:47:51 (1746625671) === [ 209.231411] Lustre: DEBUG MARKER: == sanity-scrub test 0: Do not auto trigger OI scrub for non-backup/restore case ========================================================== 09:47:51 (1746625671) [ 221.327624] Lustre: Failing over lustre-MDT0000 [ 221.448762] Lustre: server umount lustre-MDT0000 complete [ 222.712867] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 222.718110] Lustre: Skipped 2 previous similar messages [ 222.925196] LustreError: 23698:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746625685 with bad export cookie 424414070867599164 [ 222.925280] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 222.927256] Lustre: Failing over lustre-MDT0001 [ 222.936954] LustreError: 23698:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 1 previous similar message [ 223.062094] Lustre: server umount lustre-MDT0001 complete [ 226.991468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 232.724545] LustreError: 19123:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff880098f7dc00 x1831469702031488/t0(0) o250->MGC192.168.203.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 232.814846] LustreError: 24908:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 232.824270] LustreError: 24908:0:(ldlm_lib.c:1103:target_handle_connect()) Skipped 5 previous similar messages [ 232.831392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 233.676324] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.216005] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 236.277965] Lustre: 29598:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.140@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 236.331629] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 236.331666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 236.342436] Lustre: Skipped 2 previous similar messages [ 236.376448] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:42 to 0x280000400:65) [ 236.377533] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:41 to 0x2c0000400:65) [ 237.349608] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.107857] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 239.111952] Lustre: lustre-MDT0000: Denying connection for new client 89239480-54d7-4ec6-b5ae-af219afb418f (at 192.168.203.40@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 241.370185] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [ 241.386878] Lustre: lustre-MDT0000: Recovery over after 0:02, of 1 clients 1 recovered and 0 were evicted. [ 241.404819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:41 to 0x280000401:65) [ 241.404908] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:42 to 0x2c0000401:65) [ 243.381955] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 243.384914] Lustre: Skipped 2 previous similar messages [ 244.379305] Lustre: 19127:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746625690/real 1746625690] req@ffff8800a6107b80 x1831469702031360/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746625706 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 246.085362] Lustre: DEBUG MARKER: == sanity-scrub test 1a: Auto trigger initial OI scrub when server mounts ========================================================== 09:48:28 (1746625708) [ 250.216283] Lustre: 19124:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746625695/real 1746625695] req@ffff880098f7ca80 x1831469702031744/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746625711 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 250.230751] Lustre: 19124:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 254.787631] Lustre: Failing over lustre-MDT0000 [ 254.886792] Lustre: server umount lustre-MDT0000 complete [ 256.125473] LustreError: 22535:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746625718 with bad export cookie 424414070867615418 [ 256.130913] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 256.132080] Lustre: Failing over lustre-MDT0001 [ 256.270131] Lustre: server umount lustre-MDT0001 complete [ 259.584301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 266.142054] LustreError: 19123:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88008e87c380 x1831469702126848/t0(0) o250->MGC192.168.203.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 266.290711] LustreError: 28743:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 266.335698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 266.340141] Lustre: Skipped 1 previous similar message [ 267.537261] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 270.957567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 271.008155] Lustre: 32549:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.140@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 271.075994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 271.076723] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 271.098490] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.203.140@tcp (at 0@lo) [ 271.103758] Lustre: Skipped 1 previous similar message [ 271.144783] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:106 to 0x2c0000400:129) [ 271.145035] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:105 to 0x280000400:129) [ 271.163973] mount.lustre (32549) used greatest stack depth: 9864 bytes left [ 272.509415] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 273.606906] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 273.610542] Lustre: lustre-MDT0000: Denying connection for new client 0753017b-fc0e-4126-9bcd-9b587d17d42b (at 192.168.203.40@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 274.118360] Lustre: 19124:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746625720/real 1746625720] req@ffff88011d9e5180 x1831469702126208/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746625736 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 274.131574] Lustre: 19124:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 276.138658] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 276.141186] Lustre: Skipped 1 previous similar message [ 276.156288] Lustre: lustre-MDT0000: Recovery over after 0:03, of 1 clients 1 recovered and 0 were evicted. [ 276.181731] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:106 to 0x2c0000401:129) [ 276.186893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:105 to 0x280000401:129) [ 277.137700] Lustre: 19125:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746625723/real 1746625723] req@ffff88012d402d80 x1831469702126336/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746625739 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 279.002841] Lustre: *** cfs_fail_loc=193, val=0*** [ 280.296650] Lustre: Failing over lustre-MDT0000 [ 280.383702] Lustre: server umount lustre-MDT0000 complete [ 281.145139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 281.145326] LustreError: 32558:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 281.156920] Lustre: Skipped 6 previous similar messages [ 282.144782] Lustre: 19127:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746625728/real 1746625728] req@ffff88008e87ed80 x1831469702127232/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746625744 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 282.155531] Lustre: 19127:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 283.448873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 283.506844] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 283.598578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 283.602620] Lustre: Skipped 1 previous similar message [ 284.578713] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 287.247716] Lustre: DEBUG MARKER: == sanity-scrub test 1b: Trigger OI scrub when MDT mounts for OI files remove/recreate case ========================================================== 09:49:10 (1746625750) [ 288.601607] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 288.602748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 288.602750] Lustre: Skipped 2 previous similar messages [ 288.614410] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 288.633434] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:106 to 0x2c0000401:161) [ 288.633436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:105 to 0x280000401:161) [ 290.635965] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing load_modules_local [ 295.408031] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 302.101240] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 311.916589] Lustre: *** cfs_fail_loc=198, val=0*** [ 317.222376] Lustre: Failing over lustre-MDT0000 [ 317.302196] Lustre: server umount lustre-MDT0000 complete [ 318.273956] LustreError: 22535:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746625780 with bad export cookie 424414070867644391 [ 318.278786] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 318.282803] Lustre: Failing over lustre-MDT0001 [ 318.403135] Lustre: server umount lustre-MDT0001 complete [ 319.751550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 321.181753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 324.015358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 328.287517] Lustre: Evicted from MGS (at 192.168.203.140@tcp) after server handle changed from 0x0 to 0x5e3d2594bbeba5a [ 328.292159] Lustre: MGC192.168.203.140@tcp: Connection restored to (at 0@lo) [ 328.294887] Lustre: Skipped 3 previous similar messages [ 328.423731] LustreError: 30280:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 328.430798] LustreError: 30280:0:(ldlm_lib.c:1103:target_handle_connect()) Skipped 3 previous similar messages [ 328.465565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 329.479441] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 332.290239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 332.332783] Lustre: 8269:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.140@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 332.385860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 332.386043] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.438814] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:170 to 0x280000400:193) [ 332.439266] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:170 to 0x2c0000400:193) [ 333.378546] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 334.750492] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 334.755347] Lustre: lustre-MDT0000: Denying connection for new client 6098587c-27dc-4c1f-8615-fa6a2c243e76 (at 192.168.203.40@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 335.385630] Lustre: 19126:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746625781/real 1746625781] req@ffff88009e93a300 x1831469702270336/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746625797 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 335.396175] Lustre: 19126:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 337.452544] Lustre: lustre-MDT0000: Recovery over after 0:02, of 1 clients 1 recovered and 0 were evicted. [ 337.480185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:202 to 0x280000401:225) [ 337.480230] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:202 to 0x2c0000401:225) [ 342.621517] Lustre: DEBUG MARKER: == sanity-scrub test 1c: Auto detect kinds of OI file(s) removed/recreated cases ========================================================== 09:50:05 (1746625805) [ 352.782647] Lustre: Failing over lustre-MDT0000 [ 352.868272] Lustre: server umount lustre-MDT0000 complete [ 354.086655] LustreError: 22535:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746625816 with bad export cookie 424414070867671642 [ 354.087811] Lustre: Failing over lustre-MDT0001 [ 354.088085] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 354.100305] LustreError: 22535:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 1 previous similar message [ 354.230628] Lustre: server umount lustre-MDT0001 complete [ 355.753933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 357.215454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 360.106160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 360.132639] Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64 [ 364.093955] Lustre: Evicted from MGS (at 192.168.203.140@tcp) after server handle changed from 0x0 to 0x5e3d2594bbf2441 [ 364.098039] Lustre: MGC192.168.203.140@tcp: Connection restored to (at 0@lo) [ 364.100380] Lustre: Skipped 5 previous similar messages [ 364.189773] LustreError: 24908:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.104390] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 367.536242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 367.555183] Lustre: lustre-MDT0001: invalid oi count 63, remove them, then set it to 64 [ 367.586270] Lustre: 11870:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.140@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 367.634430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 367.635240] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.635242] Lustre: Skipped 3 previous similar messages [ 367.675896] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:234 to 0x2c0000400:257) [ 367.680968] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:233 to 0x280000400:257) [ 368.543736] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 372.654357] Lustre: 19127:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746625819/real 1746625819] req@ffff8800a5bac380 x1831469702376320/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746625835 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 372.663345] Lustre: 19127:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 10 previous similar messages [ 372.665469] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 372.675277] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 372.693709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:265 to 0x280000401:289) [ 372.693711] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:266 to 0x2c0000401:289) [ 375.084681] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing load_modules_local [ 380.316236] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 380.318585] Lustre: Skipped 1 previous similar message [ 387.435237] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 402.820518] Lustre: Failing over lustre-MDT0000 [ 402.909915] Lustre: server umount lustre-MDT0000 complete [ 404.048811] LustreError: 22534:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746625866 with bad export cookie 424414070867698753 [ 404.050694] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 404.050828] Lustre: Failing over lustre-MDT0001 [ 404.062374] LustreError: 22534:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 404.179358] Lustre: server umount lustre-MDT0001 complete [ 405.522332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 407.899841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 411.451817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 411.483633] Lustre: lustre-MDT0000: invalid oi count 58, remove them, then set it to 64 [ 414.056107] Lustre: Evicted from MGS (at 192.168.203.140@tcp) after server handle changed from 0x0 to 0x5e3d2594bbf8ffd [ 414.060193] Lustre: MGC192.168.203.140@tcp: Connection restored to (at 0@lo) [ 414.063594] Lustre: Skipped 5 previous similar messages [ 414.152773] LustreError: 24907:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 414.183827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 414.186768] Lustre: Skipped 3 previous similar messages [ 415.090079] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 417.645064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 417.664864] Lustre: lustre-MDT0001: invalid oi count 58, remove them, then set it to 64 [ 417.699175] Lustre: 19260:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.140@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 417.749074] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 417.749132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 417.758736] Lustre: Skipped 5 previous similar messages [ 417.789019] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:297 to 0x280000400:321) [ 417.789021] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:298 to 0x2c0000400:321) [ 418.684867] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 422.776945] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 422.783826] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 422.802438] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:330 to 0x2c0000401:353) [ 422.803015] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:329 to 0x280000401:353) [ 423.779383] Lustre: 19127:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746625870/real 1746625870] req@ffff88009e93b100 x1831469702507264/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746625886 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 423.790464] Lustre: 19127:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [ 424.907373] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing load_modules_local [ 429.570650] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 429.573207] Lustre: Skipped 1 previous similar message [ 436.742841] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 450.995128] Lustre: Failing over lustre-MDT0000 [ 451.065618] Lustre: server umount lustre-MDT0000 complete [ 452.114971] LustreError: 22536:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746625914 with bad export cookie 424414070867726333 [ 452.118515] Lustre: Failing over lustre-MDT0001 [ 452.118831] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 452.126247] LustreError: 22536:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 452.257804] Lustre: server umount lustre-MDT0001 complete [ 453.749322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 455.590695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 458.510626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 458.528864] Lustre: lustre-MDT0000: invalid oi count 56, remove them, then set it to 64 [ 462.131576] LustreError: 19123:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8800a7877100 x1831469702617728/t0(0) o250->MGC192.168.203.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 462.238187] LustreError: 30280:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 463.095969] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 465.608592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 465.716258] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.716359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 465.730163] Lustre: Skipped 3 previous similar messages [ 465.759760] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:361 to 0x2c0000400:385) [ 465.759765] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:362 to 0x280000400:385) [ 466.603184] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 470.760986] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 470.775973] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 470.798162] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:393 to 0x2c0000401:417) [ 470.798164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:394 to 0x280000401:417) [ 471.131612] Lustre: DEBUG MARKER: == sanity-scrub test 2: Trigger OI scrub when MDT mounts for backup/restore case ========================================================== 09:52:13 (1746625933) [ 474.284318] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing load_modules_local [ 479.042921] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 479.045552] Lustre: Skipped 1 previous similar message [ 486.447061] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 501.472506] Lustre: Failing over lustre-MDT0000 [ 501.572461] Lustre: server umount lustre-MDT0000 complete [ 502.760499] LustreError: 23698:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746625965 with bad export cookie 424414070867753913 [ 502.762641] Lustre: Failing over lustre-MDT0001 [ 502.763024] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 502.773671] LustreError: 23698:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 502.931483] Lustre: server umount lustre-MDT0001 complete [ 504.779227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 507.901594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 508.209465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 510.815768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 513.715816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 513.998384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 517.303232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 517.310474] Lustre: lustre-MDT0000: reset Object Index mappings [ 521.784315] Lustre: 19124:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1746625968/real 1746625968] req@ffff88006e9e4700 x1831469702746624/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1746625984 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 521.794006] Lustre: 19124:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 25 previous similar messages [ 527.797476] LustreError: 19123:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88006e9e6d80 x1831469702749184/t0(0) o250->MGC192.168.203.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 527.909787] LustreError: 24907:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.783487] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 531.164856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 531.173182] Lustre: lustre-MDT0001: reset Object Index mappings [ 531.207385] Lustre: 2731:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.140@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 531.212057] Lustre: 2731:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 1 previous similar message [ 531.256683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 531.293104] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:426 to 0x2c0000400:449) [ 531.293110] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:425 to 0x280000400:449) [ 532.157633] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 532.282128] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 532.283121] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) [ 532.283123] Lustre: Skipped 10 previous similar messages [ 532.297324] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 532.317752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:458 to 0x280000401:481) [ 532.317753] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:457 to 0x2c0000401:481) [ 535.151495] Lustre: DEBUG MARKER: == sanity-scrub test 4a: Auto trigger OI scrub if bad OI mapping was found (1) ========================================================== 09:53:17 (1746625997) [ 543.191679] Lustre: Failing over lustre-MDT0000 [ 543.272609] Lustre: server umount lustre-MDT0000 complete [ 544.335827] LustreError: 22535:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746626006 with bad export cookie 424414070867781493 [ 544.337488] Lustre: Failing over lustre-MDT0001 [ 544.342796] LustreError: 22535:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 544.479374] Lustre: server umount lustre-MDT0001 complete [ 546.266633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 549.302450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 549.612570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 552.109489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 554.681891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 554.940518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 557.988096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 557.996016] Lustre: lustre-MDT0000: reset Object Index mappings [ 563.352372] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 563.357379] Lustre: Skipped 8 previous similar messages [ 569.359635] LustreError: 19123:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88012bdaea00 x1831469702843776/t0(0) o250->MGC192.168.203.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 569.477263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 569.479858] Lustre: Skipped 5 previous similar messages [ 570.291367] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 572.620328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 572.625720] Lustre: lustre-MDT0001: reset Object Index mappings [ 572.703494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 572.734991] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:489 to 0x2c0000400:513) [ 572.739778] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:490 to 0x280000400:513) [ 573.575536] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 573.743195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:522 to 0x280000401:545) [ 573.743206] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:521 to 0x2c0000401:545) [ 573.746246] LustreError: 8118:0:(mdt_handler.c:8352:mdt_trash_setup_thread()) lustre-MDT0000: Trash dir setup failed: rc = -78 [ 574.247232] LustreError: 8118:0:(mdt_handler.c:8352:mdt_trash_setup_thread()) lustre-MDT0000: Trash dir setup failed: rc = -78 [ 574.250876] LustreError: 8118:0:(mdt_handler.c:8352:mdt_trash_setup_thread()) Skipped 46072 previous similar messages [ 575.247231] LustreError: 8118:0:(mdt_handler.c:8352:mdt_trash_setup_thread()) lustre-MDT0000: Trash dir setup failed: rc = -78 [ 575.250655] LustreError: 8118:0:(mdt_handler.c:8352:mdt_trash_setup_thread()) Skipped 87100 previous similar messages [ 577.247228] LustreError: 8118:0:(mdt_handler.c:8352:mdt_trash_setup_thread()) lustre-MDT0000: Trash dir setup failed: rc = -78 [ 577.250580] LustreError: 8118:0:(mdt_handler.c:8352:mdt_trash_setup_thread()) Skipped 167209 previous similar messages [ 577.827283] Lustre: lustre-MDT0000: trigger OI scrub by RPC for [0x200000400:0x2:0x0]/0 with flags 0x4a: rc = 0 [ 588.883741] Lustre: DEBUG MARKER: == sanity-scrub test 4b: Auto trigger OI scrub if bad OI mapping was found (2) ========================================================== 09:54:11 (1746626051) [ 596.316907] Lustre: Failing over lustre-MDT0000 [ 596.384498] Lustre: server umount lustre-MDT0000 complete [ 597.377934] LustreError: MGC192.168.203.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 597.378846] Lustre: Failing over lustre-MDT0001 [ 597.383583] LustreError: Skipped 1 previous similar message [ 597.490815] Lustre: server umount lustre-MDT0001 complete [ 598.946539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 601.576230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 601.854633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 604.330941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 607.094699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 607.379881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 610.723015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 610.731104] Lustre: lustre-MDT0000: reset Object Index mappings [ 622.393221] LustreError: 19123:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8800a006aa00 x1831469702943616/t0(0) o250->MGC192.168.203.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 622.520754] LustreError: 28743:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 622.534171] LustreError: 28743:0:(ldlm_lib.c:1103:target_handle_connect()) Skipped 1 previous similar message [ 622.560171] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 623.502059] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 626.276230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 626.283189] Lustre: lustre-MDT0001: reset Object Index mappings [ 626.408351] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:554 to 0x280000400:577) [ 626.408353] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:553 to 0x2c0000400:577) [ 627.314556] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 628.585241] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 628.587713] Lustre: Skipped 1 previous similar message [ 628.589040] Lustre: lustre-MDT0000: Denying connection for new client 4d753a2a-5140-40d0-bb9e-28ee14c3afea (at 192.168.203.40@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 631.408779] Lustre: lustre-MDT0000: Recovery over after 0:03, of 1 clients 1 recovered and 0 were evicted. [ 631.413128] Lustre: Skipped 1 previous similar message [ 631.428670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:585 to 0x280000401:609) [ 631.428678] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:586 to 0x2c0000401:609) [ 631.428814] LustreError: 14080:0:(mdt_handler.c:8352:mdt_trash_setup_thread()) lustre-MDT0000: Trash dir setup failed: rc = -78 [ 631.428816] LustreError: 14080:0:(mdt_handler.c:8352:mdt_trash_setup_thread()) Skipped 51334 previous similar messages [ 634.004261] Lustre: lustre-MDT0000: trigger OI scrub by RPC for [0x200000400:0x2:0x0]/0 with flags 0x4a: rc = 0 [ 634.008556] Lustre: Skipped 1 previous similar message [ 645.744970] Lustre: DEBUG MARKER: sanity-scrub test_4b: @@@@@@ FAIL: (12) Auto trigger full scrub unexpectedly [ 648.026966] Lustre: DEBUG MARKER: == sanity-scrub test 4c: Auto trigger OI scrub if bad OI mapping was found (3) ========================================================== 09:55:10 (1746626110) [ 690.129168] Lustre: Failing over lustre-MDT0000 [ 690.226890] Lustre: server umount lustre-MDT0000 complete [ 691.405364] LustreError: 24913:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1746626153 with bad export cookie 424414070867836380 [ 691.407110] Lustre: Failing over lustre-MDT0001 [ 691.413592] LustreError: 24913:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 691.496626] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 691.496627] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 691.496631] Lustre: Skipped 4 previous similar messages [ 691.508541] Lustre: Skipped 2 previous similar messages [ 691.563831] Lustre: server umount lustre-MDT0001 complete [ 693.190982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 696.119689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 696.398804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 698.968702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 702.084170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 702.396778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 705.879235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc [ 705.887297] Lustre: lustre-MDT0000: reset Object Index mappings [ 716.504905] LustreError: 19123:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88007b097100 x1831469703070336/t0(0) o250->MGC192.168.203.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 716.620390] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 717.397417] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 719.659737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 719.693507] Lustre: 20201:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.140@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 719.698583] Lustre: 20201:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 2 previous similar messages [ 719.748700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 719.753653] LustreError: Skipped 1 previous similar message [ 719.779061] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:617 to 0x280000400:641) [ 719.779063] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:618 to 0x2c0000400:641) [ 720.591431] Lustre: DEBUG MARKER: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 721.651563] Lustre: lustre-MDT0000: Denying connection for new client 86e6e915-e757-4b6a-aa55-d5896b7951f7 (at 192.168.203.40@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 724.786135] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 724.788677] Lustre: Skipped 14 previous similar messages [ 724.799122] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:649 to 0x2c0000401:673) [ 724.799323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:650 to 0x280000401:673) [ 724.800387] LustreError: 20754:0:(mdt_handler.c:8352:mdt_trash_setup_thread()) lustre-MDT0000: Trash dir setup failed: rc = -78 [ 724.800388] LustreError: 20754:0:(mdt_handler.c:8352:mdt_trash_setup_thread()) Skipped 223037 previous similar messages [ 727.020955] Lustre: lustre-MDT0000: trigger OI scrub by RPC for [0x200000400:0x2:0x0]/0 with flags 0x4a: rc = 0 [ 727.026333] Lustre: Skipped 1 previous similar message [ 739.615739] Lustre: DEBUG MARKER: sanity-scrub test_4c: @@@@@@ FAIL: (12) Auto trigger full scrub unexpectedly