[ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-7.9-debug (green@centos7-base) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Sat Mar 26 23:28:42 EDT 2022 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013edfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x13ee00 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] e820: last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] mapped at [ffffffffff200b30] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x023c1000, 0x023c1fff] PGTABLE [ 0.000000] BRK [0x023c2000, 0x023c2fff] PGTABLE [ 0.000000] BRK [0x023c3000, 0x023c3fff] PGTABLE [ 0.000000] BRK [0x023c4000, 0x023c4fff] PGTABLE [ 0.000000] BRK [0x023c5000, 0x023c5fff] PGTABLE [ 0.000000] BRK [0x023c6000, 0x023c6fff] PGTABLE [ 0.000000] RAMDISK: [mem 0xbc2e2000-0xbffbffff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f5950 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1bb7 00034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe1a53 00074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffe0040 01A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffe0000 00040 [ 0.000000] ACPI: APIC 00000000bffe1ac7 00090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 00000000bffe1b57 00038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 00000000bffe1b8f 00028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013edfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x13e5e3000-0x13e609fff] [ 0.000000] Reserving 128MB of memory at 768MB for crashkernel (System RAM: 4077MB) [ 0.000000] kvm-clock: cpu 0, msr 1:3e592001, primary cpu clock [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 440831820 cycles [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13edfffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffcdfff] [ 0.000000] node 0: [mem 0x100000000-0x13edfffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x13edfffff] [ 0.000000] On node 0 totalpages: 1043820 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782286 pages, LIFO batch:31 [ 0.000000] Normal zone: 4024 pages used for memmap [ 0.000000] Normal zone: 257536 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 38 pages/cpu s115176 r8192 d32280 u524288 [ 0.000000] pcpu-alloc: s115176 r8192 d32280 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13e2135c0 [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes) [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1027487 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:centos7:ext4:ro:-p,-b4096 ro crashkernel=128M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] audit: disabled (until reboot) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3820268k/5224448k available (8172k kernel code, 1049168k absent, 355012k reserved, 5773k data, 2532k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=4. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-3. [ 0.000000] NR_IRQS:327936 nr_irqs:456 0 [ 0.000000] Console: colour *CGA 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] allocated 25165824 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ODEBUG: 73 of 73 active objects replaced [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.486027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.490010] pid_max: default: 32768 minimum: 301 [ 0.491932] Security Framework initialized [ 0.493528] SELinux: Initializing. [ 0.495160] SELinux: Starting in permissive mode [ 0.497376] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.501825] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.513993] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.519568] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes) [ 0.526481] Initializing cgroup subsys memory [ 0.528217] Initializing cgroup subsys devices [ 0.532524] Initializing cgroup subsys freezer [ 0.535561] Initializing cgroup subsys net_cls [ 0.540785] Initializing cgroup subsys blkio [ 0.542028] Initializing cgroup subsys perf_event [ 0.543422] Initializing cgroup subsys hugetlb [ 0.545667] Initializing cgroup subsys pids [ 0.546980] Initializing cgroup subsys net_prio [ 0.548472] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.559012] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.561936] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.565584] tlb_flushall_shift: 6 [ 0.567194] FEATURE SPEC_CTRL Present [ 0.569813] FEATURE IBPB_SUPPORT Present [ 0.570852] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.580665] Spectre V2 : Vulnerable [ 0.584659] Speculative Store Bypass: Vulnerable [ 0.588619] debug: unmapping init [mem 0xffffffff82019000-0xffffffff8201ffff] [ 0.604267] ACPI: Core revision 20130517 [ 0.622598] ACPI: All ACPI Tables successfully acquired [ 0.633718] ftrace: allocating 30294 entries in 119 pages [ 0.729250] Enabling x2apic [ 0.731596] Enabled x2apic [ 0.733148] Switched APIC routing to physical x2apic. [ 0.738895] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.741764] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (fam: 06, model: 3e, stepping: 04) [ 0.745467] TSC deadline timer enabled [ 0.745672] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.748880] ... version: 2 [ 0.750263] ... bit width: 48 [ 0.751670] ... generic registers: 4 [ 0.753073] ... value mask: 0000ffffffffffff [ 0.755197] ... max period: 00007fffffffffff [ 0.756974] ... fixed-purpose events: 3 [ 0.761015] ... event mask: 000000070000000f [ 0.764010] KVM setup paravirtual spinlock [ 0.772205] kvm-clock: cpu 1, msr 1:3e592041, secondary cpu clock [ 0.786318] KVM setup async PF for cpu 1 [ 0.787698] kvm-stealtime: cpu 1, msr 13e2935c0 [ 0.791867] kvm-clock: cpu 2, msr 1:3e592081, secondary cpu clock [ 0.769408] smpboot: Booting Node 0, Processors #1 #2 #3 OK [ 0.797912] KVM setup async PF for cpu 2 [ 0.798446] kvm-clock: cpu 3, msr 1:3e5920c1, secondary cpu clock [ 0.801873] kvm-stealtime: cpu 2, msr 13e3135c0 [ 0.803721] Brought up 4 CPUs [ 0.803786] KVM setup async PF for cpu 3 [ 0.803794] kvm-stealtime: cpu 3, msr 13e3935c0 [ 0.808076] smpboot: Max logical packages: 1 [ 0.809709] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.819537] devtmpfs: initialized [ 0.822094] x86/mm: Memory block size: 128MB [ 0.833902] EVM: security.selinux [ 0.836245] EVM: security.ima [ 0.836999] EVM: security.capability [ 0.840410] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.842806] NET: Registered protocol family 16 [ 0.851749] cpuidle: using governor haltpoll [ 0.858778] ACPI: bus type PCI registered [ 0.860196] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.863998] PCI: Using configuration type 1 for base access [ 0.865888] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.882565] ACPI: Added _OSI(Module Device) [ 0.884108] ACPI: Added _OSI(Processor Device) [ 0.885878] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.887355] ACPI: Added _OSI(Processor Aggregator Device) [ 0.888842] ACPI: Added _OSI(Linux-Dell-Video) [ 0.892391] ACPI: EC: Look up EC in DSDT [ 0.897106] ACPI: Interpreter enabled [ 0.898228] ACPI: (supports S0 S3 S4 S5) [ 0.899528] ACPI: Using IOAPIC for interrupt routing [ 0.900962] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.904042] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.919633] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.922176] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.925662] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.927522] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.931064] acpiphp: Slot [2] registered [ 0.932791] acpiphp: Slot [3] registered [ 0.934275] acpiphp: Slot [4] registered [ 0.935801] acpiphp: Slot [5] registered [ 0.937829] acpiphp: Slot [6] registered [ 0.939828] acpiphp: Slot [7] registered [ 0.942207] acpiphp: Slot [8] registered [ 0.944046] acpiphp: Slot [9] registered [ 0.945379] acpiphp: Slot [10] registered [ 0.947112] acpiphp: Slot [11] registered [ 0.948679] acpiphp: Slot [12] registered [ 0.950269] acpiphp: Slot [13] registered [ 0.951832] acpiphp: Slot [14] registered [ 0.953376] acpiphp: Slot [15] registered [ 0.954930] acpiphp: Slot [16] registered [ 0.957042] acpiphp: Slot [17] registered [ 0.958672] acpiphp: Slot [18] registered [ 0.960113] acpiphp: Slot [19] registered [ 0.964009] acpiphp: Slot [20] registered [ 0.965574] acpiphp: Slot [21] registered [ 0.967065] acpiphp: Slot [22] registered [ 0.968681] acpiphp: Slot [23] registered [ 0.970198] acpiphp: Slot [24] registered [ 0.971645] acpiphp: Slot [25] registered [ 0.972961] acpiphp: Slot [26] registered [ 0.974245] acpiphp: Slot [27] registered [ 0.975570] acpiphp: Slot [28] registered [ 0.976809] acpiphp: Slot [29] registered [ 0.978138] acpiphp: Slot [30] registered [ 0.979391] acpiphp: Slot [31] registered [ 0.980615] PCI host bridge to bus 0000:00 [ 0.981862] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.983917] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.986265] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.988593] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.990751] pci_bus 0000:00: root bus resource [mem 0x140000000-0x1bfffffff window] [ 0.993015] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.998480] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.999277] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 1.000577] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 1.007590] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 1.011544] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 1.013824] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 1.015027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 1.017259] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 1.020297] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 1.020983] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 1.023567] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 1.026712] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 1.028894] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 1.038304] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 1.040026] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 1.041244] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 1.046395] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 1.053980] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 1.067549] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 1.078533] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 1.081392] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 1.091092] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 1.108311] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 1.114883] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 1.120515] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 1.123863] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 1.136447] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 1.141299] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.148148] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.151322] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.166866] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 1.177070] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.180403] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.182988] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.193261] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 1.197538] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.201040] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.205064] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.216417] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 1.220911] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 1.223118] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 1.227228] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 1.229789] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 1.232214] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 1.236060] vgaarb: loaded [ 1.238427] SCSI subsystem initialized [ 1.241928] ACPI: bus type USB registered [ 1.252084] usbcore: registered new interface driver usbfs [ 1.263654] usbcore: registered new interface driver hub [ 1.272277] usbcore: registered new device driver usb [ 1.279001] PCI: Using ACPI for IRQ routing [ 1.280835] PCI: pci_cache_line_size set to 64 bytes [ 1.281115] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 1.281124] e820: reserve RAM buffer [mem 0xbffce000-0xbfffffff] [ 1.281126] e820: reserve RAM buffer [mem 0x13ee00000-0x13fffffff] [ 1.281520] NetLabel: Initializing [ 1.282405] NetLabel: domain hash size = 128 [ 1.283577] NetLabel: protocols = UNLABELED CIPSOv4 [ 1.284900] NetLabel: unlabeled traffic allowed by default [ 1.292778] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.294219] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.297772] amd_nb: Cannot enumerate AMD northbridges [ 1.304887] Switched to clocksource kvm-clock [ 1.328252] pnp: PnP ACPI init [ 1.329808] ACPI: bus type PNP registered [ 1.331508] pnp 00:00: Plug and Play ACPI device, IDs PNP0303 (active) [ 1.331591] pnp 00:01: Plug and Play ACPI device, IDs PNP0f13 (active) [ 1.331664] pnp 00:02: [dma 2] [ 1.331697] pnp 00:02: Plug and Play ACPI device, IDs PNP0700 (active) [ 1.331797] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.331959] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 1.332200] pnp 00:05: Plug and Play ACPI device, IDs PNP0b00 (active) [ 1.333033] pnp: PnP ACPI: found 6 devices [ 1.334641] ACPI: bus type PNP unregistered [ 1.352045] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.352052] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.352055] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.352058] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.352060] pci_bus 0000:00: resource 8 [mem 0x140000000-0x1bfffffff window] [ 1.352178] NET: Registered protocol family 2 [ 1.354489] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 1.357875] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes) [ 1.361353] TCP: Hash tables configured (established 32768 bind 32768) [ 1.363982] TCP: reno registered [ 1.369872] UDP hash table entries: 2048 (order: 5, 196608 bytes) [ 1.372299] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes) [ 1.387527] NET: Registered protocol family 1 [ 1.391931] RPC: Registered named UNIX socket transport module. [ 1.395966] RPC: Registered udp transport module. [ 1.400244] RPC: Registered tcp transport module. [ 1.402782] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.408619] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.411284] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.413755] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.416394] PCI: CLS 0 bytes, default 64 [ 1.419048] Unpacking initramfs... [ 3.796269] hrtimer: interrupt took 27622564 ns [ 6.454084] debug: unmapping init [mem 0xffff8800bc2e2000-0xffff8800bffbffff] [ 6.474407] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 6.482198] software IO TLB [mem 0xb82e2000-0xbc2e2000] (64MB) mapped at [ffff8800b82e2000-ffff8800bc2e1fff] [ 6.500552] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 10737418240 ms ovfl timer [ 6.508381] RAPL PMU: hw unit of domain pp0-core 2^-0 Joules [ 6.516956] RAPL PMU: hw unit of domain package 2^-0 Joules [ 6.518694] RAPL PMU: hw unit of domain dram 2^-0 Joules [ 6.540042] cryptomgr_test (52) used greatest stack depth: 14480 bytes left [ 6.543346] futex hash table entries: 1024 (order: 4, 65536 bytes) [ 6.553682] Initialise system trusted keyring [ 6.664001] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 6.673292] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 6.710331] zpool: loaded [ 6.714129] zbud: loaded [ 6.720459] VFS: Disk quotas dquot_6.6.0 [ 6.727149] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 6.735340] NFS: Registering the id_resolver key type [ 6.744306] Key type id_resolver registered [ 6.749947] Key type id_legacy registered [ 6.755754] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 6.761307] Key type big_key registered [ 6.762437] SELinux: Registering netfilter hooks [ 6.794515] cryptomgr_test (58) used greatest stack depth: 14048 bytes left [ 6.797861] cryptomgr_test (63) used greatest stack depth: 13984 bytes left [ 6.798472] NET: Registered protocol family 38 [ 6.798484] Key type asymmetric registered [ 6.798487] Asymmetric key parser 'x509' registered [ 6.798667] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 6.798781] io scheduler noop registered [ 6.798786] io scheduler deadline registered (default) [ 6.798915] io scheduler cfq registered [ 6.798921] io scheduler mq-deadline registered [ 6.798929] io scheduler kyber registered [ 6.801866] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 6.801876] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 6.871432] intel_idle: does not run on family 6 model 62 [ 6.871572] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.877386] ACPI: Power Button [PWRF] [ 6.879769] GHES: HEST is not enabled! [ 7.075966] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 7.330762] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 [ 7.538723] tsc: Refined TSC clocksource calibration: 2399.979 MHz [ 7.570367] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 7.749000] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 10 [ 8.058748] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 8.091016] 00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 8.125787] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 8.133426] Non-volatile memory driver v1.3 [ 8.135197] Linux agpgart interface v0.103 [ 8.138504] crash memory driver: version 1.1 [ 8.141897] nbd: registered device at major 43 [ 8.156349] virtio-pci 0000:00:05.0: irq 24 for MSI/MSI-X [ 8.156398] virtio-pci 0000:00:05.0: irq 25 for MSI/MSI-X [ 8.156442] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X [ 8.156494] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X [ 8.156533] virtio-pci 0000:00:05.0: irq 28 for MSI/MSI-X [ 8.171467] virtio_blk virtio1: [vda] 113176 512-byte logical blocks (57.9 MB/55.2 MiB) [ 8.191410] virtio-pci 0000:00:06.0: irq 29 for MSI/MSI-X [ 8.191450] virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X [ 8.191483] virtio-pci 0000:00:06.0: irq 31 for MSI/MSI-X [ 8.191719] virtio-pci 0000:00:06.0: irq 32 for MSI/MSI-X [ 8.191753] virtio-pci 0000:00:06.0: irq 33 for MSI/MSI-X [ 8.209083] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 8.221098] virtio-pci 0000:00:07.0: irq 34 for MSI/MSI-X [ 8.221162] virtio-pci 0000:00:07.0: irq 35 for MSI/MSI-X [ 8.221197] virtio-pci 0000:00:07.0: irq 36 for MSI/MSI-X [ 8.221230] virtio-pci 0000:00:07.0: irq 37 for MSI/MSI-X [ 8.221265] virtio-pci 0000:00:07.0: irq 38 for MSI/MSI-X [ 8.231086] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.241393] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 8.241440] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 8.241476] virtio-pci 0000:00:08.0: irq 41 for MSI/MSI-X [ 8.241517] virtio-pci 0000:00:08.0: irq 42 for MSI/MSI-X [ 8.241564] virtio-pci 0000:00:08.0: irq 43 for MSI/MSI-X [ 8.249879] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.261508] virtio-pci 0000:00:09.0: irq 44 for MSI/MSI-X [ 8.261548] virtio-pci 0000:00:09.0: irq 45 for MSI/MSI-X [ 8.261591] virtio-pci 0000:00:09.0: irq 46 for MSI/MSI-X [ 8.261641] virtio-pci 0000:00:09.0: irq 47 for MSI/MSI-X [ 8.261683] virtio-pci 0000:00:09.0: irq 48 for MSI/MSI-X [ 8.274131] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.282767] virtio-pci 0000:00:0a.0: irq 49 for MSI/MSI-X [ 8.282822] virtio-pci 0000:00:0a.0: irq 50 for MSI/MSI-X [ 8.282863] virtio-pci 0000:00:0a.0: irq 51 for MSI/MSI-X [ 8.282900] virtio-pci 0000:00:0a.0: irq 52 for MSI/MSI-X [ 8.282938] virtio-pci 0000:00:0a.0: irq 53 for MSI/MSI-X [ 8.293156] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.301630] rdac: device handler registered [ 8.303704] hp_sw: device handler registered [ 8.305669] emc: device handler registered [ 8.307276] libphy: Fixed MDIO Bus: probed [ 8.322732] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 8.325722] ehci-pci: EHCI PCI platform driver [ 8.328435] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 8.330523] ohci-pci: OHCI PCI platform driver [ 8.332675] uhci_hcd: USB Universal Host Controller Interface driver [ 8.335325] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.339134] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.340721] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.344162] mousedev: PS/2 mouse device common for all mice [ 8.348257] rtc_cmos 00:05: RTC can wake from S4 [ 8.353946] rtc_cmos 00:05: rtc core: registered rtc_cmos as rtc0 [ 8.361388] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.362397] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.376652] hidraw: raw HID events driver (C) Jiri Kosina [ 8.379511] usbcore: registered new interface driver usbhid [ 8.382341] usbhid: USB HID core driver [ 8.387760] drop_monitor: Initializing network drop monitor service [ 8.390299] Netfilter messages via NETLINK v0.30. [ 8.395025] TCP: cubic registered [ 8.396399] Initializing XFRM netlink socket [ 8.397796] NET: Registered protocol family 10 [ 8.405199] NET: Registered protocol family 17 [ 8.406639] Key type dns_resolver registered [ 8.409014] mce: Using 10 MCE banks [ 8.410437] Loading compiled-in X.509 certificates [ 8.414109] Loaded X.509 cert 'Magrathea: Glacier signing key: e34d0e1b7fcf5b414cce75d36d8482945c781ed6' [ 8.420804] registered taskstats version 1 [ 8.428504] modprobe (72) used greatest stack depth: 13456 bytes left [ 8.446081] Key type trusted registered [ 8.472363] Key type encrypted registered [ 8.475975] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 8.485401] BERT: Boot Error Record Table support is disabled. Enable it by using bert_enable as kernel parameter. [ 8.505550] rtc_cmos 00:05: setting system clock to 2024-12-15 09:29:14 UTC (1734254954) [ 8.510391] debug: unmapping init [mem 0xffffffff81da0000-0xffffffff82018fff] [ 8.516155] Write protecting the kernel read-only data: 12288k [ 8.519096] debug: unmapping init [mem 0xffff8800017fe000-0xffff8800017fffff] [ 8.523736] debug: unmapping init [mem 0xffff880001b9b000-0xffff880001bfffff] [ 8.537665] random: systemd: uninitialized urandom read (16 bytes read) [ 8.544328] random: systemd: uninitialized urandom read (16 bytes read) [ 8.546771] random: systemd: uninitialized urandom read (16 bytes read) [ 8.551175] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 8.557297] systemd[1]: Detected virtualization kvm. [ 8.560538] systemd[1]: Detected architecture x86-64. [ 8.562381] systemd[1]: Running in initial RAM disk. [ 8.571399] systemd[1]: No hostname configured. [ 8.573677] systemd[1]: Set hostname to . [ 8.575568] random: systemd: uninitialized urandom read (16 bytes read) [ 8.578256] systemd[1]: Initializing machine ID from random generator. [ 8.697943] dracut-rootfs-g (86) used greatest stack depth: 13264 bytes left [ 8.702241] random: systemd: uninitialized urandom read (16 bytes read) [ 8.704136] random: systemd: uninitialized urandom read (16 bytes read) [ 8.706854] random: systemd: uninitialized urandom read (16 bytes read) [ 8.710949] random: systemd: uninitialized urandom read (16 bytes read) [ 8.715702] random: systemd: uninitialized urandom read (16 bytes read) [ 8.718964] random: systemd: uninitialized urandom read (16 bytes read) [ 8.733572] systemd[1]: Reached target Timers. [ 8.739876] systemd[1]: Reached target Local File Systems. [ 8.749125] systemd[1]: Created slice Root Slice. [ 8.758886] systemd[1]: Listening on Journal Socket. [ 8.774924] systemd[1]: Listening on udev Kernel Socket. [ 8.786910] systemd[1]: Created slice System Slice. [ 8.799202] systemd[1]: Starting Setup Virtual Console... [ 8.818448] systemd[1]: Starting Journal Service... [ 8.828106] systemd[1]: Reached target Slices. [ 8.843796] systemd[1]: Starting Load Kernel Modules... [ 8.852832] systemd[1]: Reached target Swap. [ 8.866692] systemd[1]: Listening on udev Control Socket. [ 8.875273] systemd[1]: Reached target Sockets. [ 8.882366] systemd[1]: Starting dracut cmdline hook... [ 8.904974] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 8.914083] systemd[1]: Started Journal Service. [ 9.304662] random: fast init done [ 9.335921] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 10.404591] libata version 3.00 loaded. [ 10.442901] ata_piix 0000:00:01.1: version 2.13 [ 10.464182] scsi host0: ata_piix [ 10.471984] scsi host1: ata_piix [ 10.473573] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.478132] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.964457] ip (322) used greatest stack depth: 13080 bytes left [ 11.178078] ip (345) used greatest stack depth: 11392 bytes left [ 19.005108] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 20.513997] systemd-journald[102]: Received SIGTERM from PID 1 (systemd). [ 21.831402] SELinux: Disabled at runtime. [ 21.834754] SELinux: Unregistering netfilter hooks [ 22.234096] ip_tables: (C) 2000-2006 Netfilter Core Team [ 22.247734] systemd[1]: Inserted module 'ip_tables' [ 23.615558] systemd-journald[570]: Received request to flush runtime journal from PID 1 [ 24.305072] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 24.414459] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.552364] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 24.559152] cryptd: max_cpu_qlen set to 1000 [ 24.839857] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ 24.880200] AVX version of gcm_enc/dec engaged. [ 24.883318] AES CTR mode by8 optimization enabled [ 24.916255] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 24.936510] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 25.311997] EDAC MC: Ver: 3.0.0 [ 25.369956] EDAC sbridge: Seeking for: PCI ID 8086:0ea0 [ 25.369970] EDAC sbridge: Ver: 1.1.2 [ 32.615317] mount.nfs (780) used greatest stack depth: 10704 bytes left [ 54.405560] device-mapper: uevent: version 1.0.3 [ 54.408039] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 62.658135] libcfs: loading out-of-tree module taints kernel. [ 62.660313] libcfs: module verification failed: signature and/or required key missing - tainting kernel [ 62.740820] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 70.469279] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 70.965586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 70.972579] alg: No test for adler32 (adler32-zlib) [ 71.958956] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 72.231306] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 72.236291] LNet: Accept secure, port 988 [ 73.804027] Key type lgssc registered [ 74.379438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 80.444400] icp: module license 'CDDL' taints kernel. [ 80.451292] Disabling lock debugging due to kernel taint [ 82.893197] ZFS: Loaded module v2.1.5-1, ZFS pool version 5000, ZFS filesystem version 5 [ 87.714215] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 95.698957] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.839726] LDISKFS-fs (vde): file extents enabled, maximum tree depth=5 [ 99.847886] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 104.207678] LDISKFS-fs (vdf): file extents enabled, maximum tree depth=5 [ 104.217530] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 111.054317] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 117.394533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 117.438263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 117.463330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 118.591308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 118.615468] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 118.657202] Lustre: lustre-MDT0000: new disk, initializing [ 118.701533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 118.710932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 118.770924] mount.lustre (7232) used greatest stack depth: 10224 bytes left [ 120.302280] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 126.866279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 126.909964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 126.956437] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 126.968017] Lustre: 8392:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 126.982920] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 126.985228] Lustre: Skipped 1 previous similar message [ 127.029951] Lustre: lustre-MDT0001: new disk, initializing [ 127.066602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 127.079851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 127.086349] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 128.216331] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 130.739411] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 134.292066] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 134.297502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 134.319484] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 134.325392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 134.395026] Lustre: 9595:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 134.407059] Lustre: lustre-OST0000: new disk, initializing [ 134.410384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 134.432941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 135.644387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 135.648089] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 135.682252] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 136.109016] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 139.843961] random: crng init done [ 143.049592] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 143.067381] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 143.123799] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 143.133792] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 143.187737] Lustre: 10871:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 143.204348] Lustre: lustre-OST0001: new disk, initializing [ 143.209656] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 143.234313] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 144.050921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 144.055962] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 145.630291] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 149.073251] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 152.134119] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 155.663478] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 162.211934] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing check_logdir /tmp/testlogs/ [ 163.166740] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing yml_node [ 164.352773] Lustre: DEBUG MARKER: Client: 2.16.50.90 [ 165.001408] Lustre: DEBUG MARKER: MDS: 2.16.50.90 [ 165.623765] Lustre: DEBUG MARKER: OSS: 2.16.50.90 [ 166.008345] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Dec 15 04:31:52 EST 2024 [ 170.567076] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 110 [ 170.938875] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 111 114 [ 184.125175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 184.125767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 184.139853] Lustre: Skipped 1 previous similar message [ 188.416557] Lustre: server umount lustre-MDT0000 complete [ 190.363115] LustreError: 7241:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255136 with bad export cookie 5091060887544338235 [ 190.363585] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 190.383916] LustreError: 7241:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 194.142730] LustreError: 10887:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 194.145957] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 194.145958] Lustre: Skipped 2 previous similar messages [ 194.146595] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 194.146596] Lustre: Skipped 3 previous similar messages [ 194.167543] LustreError: 10887:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 6 previous similar messages [ 195.149064] LustreError: 7256:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 195.149078] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 195.149079] Lustre: Skipped 1 previous similar message [ 195.166871] LustreError: 7256:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 196.550286] Lustre: server umount lustre-MDT0001 complete [ 204.415151] Lustre: server umount lustre-OST0000 complete [ 212.032163] Lustre: server umount lustre-OST0001 complete [ 214.445647] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 217.248450] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 220.975446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 224.130421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 227.293864] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 227.301993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 230.339815] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 [ 230.345922] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 233.421889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 233.454671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 233.585720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 233.604341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 233.650229] Lustre: lustre-MDT0000: new disk, initializing [ 233.697733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 233.714575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 234.931484] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 239.236972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 239.262644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 239.303740] Lustre: 18613:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 239.413321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 239.437528] mount.lustre (18613) used greatest stack depth: 9520 bytes left [ 240.327455] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 243.102276] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 244.074953] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 246.313979] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 246.321133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 246.347038] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 246.352657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 246.441959] Lustre: 19812:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 246.463550] Lustre: lustre-OST0000: new disk, initializing [ 246.465704] Lustre: Skipped 1 previous similar message [ 246.472240] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 246.476069] Lustre: Skipped 2 previous similar messages [ 246.513005] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 246.516567] Lustre: Skipped 1 previous similar message [ 247.509730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 247.517228] Lustre: Skipped 1 previous similar message [ 247.521285] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 247.556576] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 248.484264] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 251.868939] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 253.487733] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 253.600740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 254.960743] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 255.042751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 257.549578] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 257.552179] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 257.552181] Lustre: Skipped 1 previous similar message [ 257.552647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 257.552648] Lustre: Skipped 1 previous similar message [ 257.947994] Lustre: server umount lustre-OST0000 complete [ 259.421799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 262.558358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 262.559067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 262.559069] Lustre: Skipped 2 previous similar messages [ 262.568697] Lustre: Skipped 3 previous similar messages [ 265.219288] Lustre: server umount lustre-MDT0000 complete [ 266.383612] LustreError: 19836:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255212 with bad export cookie 5091060887544339754 [ 266.384889] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 266.392254] LustreError: 19836:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 266.514285] Lustre: server umount lustre-MDT0001 complete [ 267.460384] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 04:33:33 (1734255213) [ 269.442370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 269.626607] LustreError: 22432:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 269.657095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 270.579135] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 273.157974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 273.232192] Lustre: 23292:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 274.269061] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 275.750402] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 276.689792] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 278.499785] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 278.503227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 279.861188] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 281.188375] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 288.605416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 288.609196] Lustre: Skipped 1 previous similar message [ 288.609403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 288.613299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 293.147552] Lustre: server umount lustre-OST0000 complete [ 298.349842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 298.353368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.355195] Lustre: server umount lustre-MDT0000 complete [ 301.558760] LustreError: 22414:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255247 with bad export cookie 5091060887544340636 [ 301.560590] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 301.566739] LustreError: 22414:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 301.683368] Lustre: server umount lustre-MDT0001 complete [ 305.319110] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 306.103560] Key type lgssc unregistered [ 306.201989] LNet: 26277:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 307.211030] LNet: Removed LNI 192.168.202.128@tcp [ 316.641001] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 316.842080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 316.850582] alg: No test for adler32 (adler32-zlib) [ 317.732210] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 317.845242] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 317.852082] LNet: Accept secure, port 988 [ 319.408825] Key type lgssc registered [ 319.886037] Lustre: Echo OBD driver; http://www.lustre.org/ [ 323.387760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 323.402463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 324.661744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 326.170381] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 329.815941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 329.872329] Lustre: 28655:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 329.982262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 331.114188] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 333.338517] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 334.734189] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 337.201266] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 337.207452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 337.315973] Lustre: 29764:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 337.365188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 339.064539] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 340.744263] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 349.372574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 359.389372] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 359.389971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.390304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 360.821010] Lustre: server umount lustre-OST0000 complete [ 364.397419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.397844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 364.397846] Lustre: Skipped 1 previous similar message [ 364.405343] Lustre: Skipped 2 previous similar messages [ 368.268551] Lustre: server umount lustre-MDT0000 complete [ 369.353065] LustreError: 27776:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255315 with bad export cookie 16407310136895114983 [ 369.353829] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 369.361693] LustreError: 27776:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 369.456980] Lustre: server umount lustre-MDT0001 complete [ 372.651236] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 373.282074] Key type lgssc unregistered [ 373.368899] LNet: 32232:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.373959] LNet: Removed LNI 192.168.202.128@tcp [ 381.449582] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 381.626814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 381.654908] alg: No test for adler32 (adler32-zlib) [ 382.480536] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 382.534864] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 382.537238] LNet: Accept secure, port 988 [ 384.071262] Key type lgssc registered [ 384.338602] Lustre: Echo OBD driver; http://www.lustre.org/ [ 386.744470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 386.753141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 387.977937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 389.143240] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 392.315757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 392.375067] Lustre: 2176:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 392.474664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 393.575710] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 395.347836] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 396.499505] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 400.595650] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 400.600467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 400.672400] Lustre: 3746:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 400.717687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 402.158825] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 403.664819] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 412.719104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 417.725706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 417.728431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 417.741753] Lustre: Skipped 1 previous similar message [ 421.425220] Lustre: server umount lustre-OST0000 complete [ 427.504183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 427.507429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 427.513054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 427.515395] Lustre: Skipped 1 previous similar message [ 429.153297] Lustre: server umount lustre-MDT0000 complete [ 430.402995] LustreError: 1299:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255376 with bad export cookie 4756859805019018730 [ 430.405417] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 430.412255] LustreError: 1299:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 430.535916] Lustre: server umount lustre-MDT0001 complete [ 433.973479] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 434.656702] Key type lgssc unregistered [ 434.746258] LNet: 5639:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 435.756092] LNet: Removed LNI 192.168.202.128@tcp [ 442.244774] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 442.408604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 442.423268] alg: No test for adler32 (adler32-zlib) [ 443.234911] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 443.289847] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 443.291819] LNet: Accept secure, port 988 [ 444.825841] Key type lgssc registered [ 445.111504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 447.334178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 447.340530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 448.516082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.427426] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 451.916412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 451.967538] Lustre: 8013:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 452.067060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 452.961385] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 454.406493] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 455.302067] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 457.305594] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 457.312243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 457.389264] Lustre: 9123:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 457.445455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 458.445349] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 458.911527] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 460.380309] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 470.528496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 470.533553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 470.540597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 472.461864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 472.462724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 472.469258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.889235] Lustre: server umount lustre-OST0000 complete [ 477.085727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 477.090367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 477.095992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 477.098976] Lustre: Skipped 1 previous similar message [ 482.177458] Lustre: server umount lustre-MDT0000 complete [ 482.477224] LustreError: 7144:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 482.484132] LustreError: 7144:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 3 previous similar messages [ 483.235261] LustreError: 7173:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255429 with bad export cookie 3848444019531198910 [ 483.236293] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 483.243821] LustreError: 7173:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 483.347877] Lustre: server umount lustre-MDT0001 complete [ 486.386061] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 487.099754] Key type lgssc unregistered [ 487.188163] LNet: 11003:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 488.195012] LNet: Removed LNI 192.168.202.128@tcp [ 494.565983] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 494.687901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 494.715582] alg: No test for adler32 (adler32-zlib) [ 495.531696] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 495.596215] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 495.598609] LNet: Accept secure, port 988 [ 497.137868] Key type lgssc registered [ 497.400203] Lustre: Echo OBD driver; http://www.lustre.org/ [ 499.683078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 499.691121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.895240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 501.836310] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 504.190561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 504.234858] Lustre: 13373:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 504.313773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 505.180480] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 506.795995] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 507.766529] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 509.746958] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 509.751497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 509.827666] Lustre: 14485:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 509.878273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 510.876488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 511.356294] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 513.037482] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 519.885606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 519.885795] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 519.886285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 521.806202] Lustre: server umount lustre-OST0000 complete [ 534.350132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 534.354122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 534.359127] Lustre: Skipped 1 previous similar message [ 534.362412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 534.365025] Lustre: Skipped 1 previous similar message [ 539.914901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 539.920161] Lustre: Skipped 3 previous similar messages [ 540.201641] Lustre: server umount lustre-MDT0000 complete [ 541.517915] LustreError: 15283:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255487 with bad export cookie 6903541308131321909 [ 541.519361] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 541.530988] LustreError: 15283:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 1 previous similar message [ 547.666762] Lustre: server umount lustre-MDT0001 complete [ 551.094753] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 551.735573] Key type lgssc unregistered [ 551.824198] LNet: 16570:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 552.833240] LNet: Removed LNI 192.168.202.128@tcp [ 560.007968] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 560.143254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 560.148922] alg: No test for adler32 (adler32-zlib) [ 560.987736] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 561.055187] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 561.059416] LNet: Accept secure, port 988 [ 562.600875] Key type lgssc registered [ 562.867412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 565.163016] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 565.171067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 566.344857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 567.342937] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 570.338366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 570.406972] Lustre: 18940:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 570.532888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 571.567594] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 573.233391] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 574.201799] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 576.208315] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 576.214867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 576.291597] Lustre: 20051:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 576.344043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 577.345520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 577.890188] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 579.330535] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 590.365742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 590.366088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 590.372002] Lustre: Skipped 2 previous similar messages [ 593.284342] Lustre: server umount lustre-MDT0000 complete [ 594.235484] LustreError: 20855:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255540 with bad export cookie 14753552657924961445 [ 594.236844] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 594.242431] LustreError: 20855:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 595.373207] LustreError: 18075:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.373344] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 595.373648] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 595.373650] Lustre: Skipped 3 previous similar messages [ 595.386065] LustreError: 18075:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 599.381319] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 599.381327] LustreError: 20292:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 599.381330] LustreError: 20292:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 599.391719] Lustre: Skipped 1 previous similar message [ 600.334084] Lustre: server umount lustre-MDT0001 complete [ 610.859135] LustreError: 18062:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.28@tcp arrived at 1734255556 with bad export cookie 14753552657924962320 [ 610.859145] LustreError: 20855:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.28@tcp arrived at 1734255556 with bad export cookie 14753552657924962320 [ 610.870468] LustreError: 18062:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 1 previous similar message [ 617.437905] Lustre: server umount lustre-OST0000 complete [ 622.232220] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 623.021385] Key type lgssc unregistered [ 623.117404] LNet: 22347:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 624.123307] LNet: Removed LNI 192.168.202.128@tcp [ 632.823215] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 633.184742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 633.194108] alg: No test for adler32 (adler32-zlib) [ 634.040354] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 634.127976] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 634.132892] LNet: Accept secure, port 988 [ 635.686860] Key type lgssc registered [ 636.000852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 638.841413] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 638.847224] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 638.853204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 655.951840] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 661.954140] LustreError: 23836:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.128@tcp: failed processing log, type 4: rc = -110 [ 691.984974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 694.067885] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 695.827148] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 859.169519] Lustre: server umount lustre-OST0000 complete [ 867.586870] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 868.951369] Key type lgssc unregistered [ 869.116375] LNet: 25747:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.123900] LNet: Removed LNI 192.168.202.128@tcp [ 879.665304] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 879.891415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 879.902868] alg: No test for adler32 (adler32-zlib) [ 880.764445] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 880.865788] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 880.868218] LNet: Accept secure, port 988 [ 882.432221] Key type lgssc registered [ 883.008589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 886.881349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 886.894094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.176061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 889.574445] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 893.183476] LustreError: 27262:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 893.190158] LustreError: 27262:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 893.280557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 893.331767] Lustre: 28125:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 893.470073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 894.997951] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 897.255272] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 898.506799] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 901.163255] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 901.169463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 901.295586] Lustre: 29237:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 901.346174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 903.477266] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 905.375012] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 908.350301] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 908.351002] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 908.351329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 912.234521] Lustre: server umount lustre-OST0000 complete [ 918.369570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 918.375227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 918.382618] Lustre: Skipped 1 previous similar message [ 920.166844] Lustre: server umount lustre-MDT0000 complete [ 922.270104] LustreError: 27246:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255868 with bad export cookie 6085756764902340528 [ 922.272963] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 922.283092] LustreError: 27246:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 922.517396] Lustre: server umount lustre-MDT0001 complete [ 927.350681] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 928.394347] Key type lgssc unregistered [ 928.488951] LNet: 31107:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.496363] LNet: Removed LNI 192.168.202.128@tcp [ 938.102771] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 938.416906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 938.423360] alg: No test for adler32 (adler32-zlib) [ 939.295833] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 939.388304] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 939.391068] LNet: Accept secure, port 988 [ 940.950064] Key type lgssc registered [ 941.292483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 944.601962] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 944.609202] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 944.618061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 961.740523] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 967.744936] LustreError: 32598:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.128@tcp: failed processing log, type 4: rc = -110 [ 997.779357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 999.386092] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1000.958492] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1003.279015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1003.495700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1003.523585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1004.452988] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1007.511952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1007.570742] Lustre: 2213:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1007.669619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1008.738207] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1010.478598] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1011.534971] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1012.653540] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1012.656872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1012.663016] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1013.486219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1013.495669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1013.501718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1014.287934] Lustre: server umount lustre-OST0000 complete [ 1036.701375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1036.701809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1036.707850] Lustre: Skipped 1 previous similar message [ 1038.955618] Lustre: server umount lustre-MDT0000 complete [ 1040.307806] LustreError: 32605:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734255986 with bad export cookie 17029737526995331713 [ 1040.309082] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1040.315990] LustreError: 32605:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1046.534239] Lustre: server umount lustre-MDT0001 complete [ 1051.786335] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1052.970527] Key type lgssc unregistered [ 1053.120229] LNet: 4297:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.130001] LNet: Removed LNI 192.168.202.128@tcp [ 1064.291667] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1064.540237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1064.559731] alg: No test for adler32 (adler32-zlib) [ 1065.466377] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 1065.554643] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1065.559003] LNet: Accept secure, port 988 [ 1067.122141] Key type lgssc registered [ 1067.561662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1071.650212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1071.660057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1072.940861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1074.261490] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1078.013660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1078.071768] Lustre: 6684:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1078.212609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1079.610147] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1081.764410] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1083.216337] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1086.039105] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1086.047833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1086.162739] Lustre: 7799:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1086.239981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1088.448100] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1090.778745] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1097.245149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1107.262697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1107.266343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1107.275950] Lustre: Skipped 1 previous similar message [ 1109.563065] Lustre: server umount lustre-OST0000 complete [ 1112.269778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1112.280204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1112.286212] Lustre: Skipped 2 previous similar messages [ 1117.181727] Lustre: server umount lustre-MDT0000 complete [ 1117.277124] LustreError: 5823:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1117.283723] LustreError: 5823:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 3 previous similar messages [ 1118.702690] LustreError: 5807:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256064 with bad export cookie 17142321603951280880 [ 1118.706534] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1118.714053] LustreError: 5807:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1118.842748] Lustre: server umount lustre-MDT0001 complete [ 1124.183071] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1125.476648] Key type lgssc unregistered [ 1125.615773] LNet: 9683:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1126.626258] LNet: Removed LNI 192.168.202.128@tcp [ 1139.059556] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1139.390093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1139.407054] alg: No test for adler32 (adler32-zlib) [ 1140.293251] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 1140.389477] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1140.392898] LNet: Accept secure, port 988 [ 1141.950931] Key type lgssc registered [ 1142.449223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1146.121761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1146.130746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1147.434443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1148.678957] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1151.794207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1151.849533] Lustre: 12323:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1151.985606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1153.127263] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1154.899519] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1156.080074] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1158.544950] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1158.550416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1158.623150] Lustre: 13433:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1158.668473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1160.155559] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1161.819527] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1172.684981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1179.185041] Lustre: Failing over lustre-MDT0000 [ 1179.283982] Lustre: server umount lustre-MDT0000 complete [ 1181.998912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1182.003364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1182.010245] LustreError: 11463:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1182.022955] LustreError: 11463:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 1182.702546] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1182.706230] LustreError: 11459:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1182.714527] LustreError: 11459:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 1183.456425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1183.521236] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1183.590194] Lustre: *** cfs_fail_loc=135, val=0*** [ 1183.647210] LustreError: 14912:0:(obd_config.c:797:class_setup()) setup lustre-MDT0000 failed (-2) [ 1183.650822] LustreError: 14912:0:(obd_config.c:2030:class_config_llog_handler()) MGC192.168.202.128@tcp: cfg command failed: rc = -2 [ 1183.656589] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1183.666541] LustreError: MGC192.168.202.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1183.678751] LustreError: 14901:0:(tgt_mount.c:1493:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1183.681746] LustreError: 14901:0:(tgt_mount.c:2222:server_fill_super()) Unable to start targets: -2 [ 1183.691536] LustreError: 14901:0:(obd_config.c:850:class_cleanup()) Device 5 not setup [ 1183.713776] Lustre: server umount lustre-MDT0000 complete [ 1183.715505] LustreError: 14901:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1186.934896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1187.023614] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1187.076991] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.28@tcp (not set up) [ 1187.150682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1187.173958] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1188.834532] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1192.086353] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1192.148558] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 1192.176859] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1192.213211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1196.750683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1196.753964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.759502] Lustre: Skipped 1 previous similar message [ 1196.762701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1199.495565] Lustre: server umount lustre-OST0000 complete [ 1202.157695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1202.160637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1202.160639] Lustre: Skipped 2 previous similar messages [ 1202.167095] Lustre: Skipped 2 previous similar messages [ 1206.870158] Lustre: server umount lustre-MDT0000 complete [ 1207.165487] LustreError: 13140:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1207.175093] LustreError: 13140:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 1208.244867] LustreError: 11445:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256154 with bad export cookie 1760912859804627808 [ 1208.245434] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1208.258776] LustreError: 11445:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1208.408425] Lustre: server umount lustre-MDT0001 complete [ 1212.332433] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1213.280727] Key type lgssc unregistered [ 1213.368089] LNet: 16989:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1214.377337] LNet: Removed LNI 192.168.202.128@tcp [ 1223.487335] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1223.827109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1223.838583] alg: No test for adler32 (adler32-zlib) [ 1224.730246] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 1224.866343] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1224.871711] LNet: Accept secure, port 988 [ 1226.477731] Key type lgssc registered [ 1226.920371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1229.928675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1229.939969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1231.140762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1232.434404] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1235.890405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1235.944091] Lustre: 19363:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1236.052426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1237.280891] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1238.925120] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1239.951896] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1242.004010] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1242.009593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1242.094203] Lustre: 20476:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1242.152398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1243.756045] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1245.208273] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1256.157646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1262.498690] Lustre: Failing over lustre-MDT0000 [ 1262.595963] Lustre: server umount lustre-MDT0000 complete [ 1265.408739] LustreError: 18497:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1265.415916] LustreError: 18497:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 1266.061265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1266.064322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1266.076129] LustreError: 18502:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1266.090013] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1266.172295] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1266.177601] LustreError: 17738:0:(client.c:1310:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff880136725180 x1818499397734016/t0(0) o250->MGC192.168.202.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 1266.257159] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1266.308430] LustreError: 21955:0:(obd_config.c:797:class_setup()) setup lustre-MDT0000 failed (-9) [ 1266.311732] LustreError: 21955:0:(obd_config.c:2030:class_config_llog_handler()) MGC192.168.202.128@tcp: cfg command failed: rc = -9 [ 1266.317100] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1266.323365] LustreError: MGC192.168.202.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1266.329584] LustreError: 21943:0:(tgt_mount.c:1493:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1266.335444] LustreError: 21943:0:(tgt_mount.c:2222:server_fill_super()) Unable to start targets: -9 [ 1266.338883] LustreError: 21943:0:(obd_config.c:850:class_cleanup()) Device 5 not setup [ 1266.342884] LustreError: 21284:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256212 with bad export cookie 5016655666344017020 [ 1266.347282] Lustre: server umount lustre-MDT0000 complete [ 1266.347293] LustreError: 21943:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1266.358792] LustreError: 21284:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1269.234776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1270.405460] LustreError: 18498:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1270.418326] LustreError: 18498:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 1275.412561] LustreError: 18496:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1275.421445] LustreError: 18496:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 3 previous similar messages [ 1276.340775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1276.381267] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1277.857920] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1280.421258] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1281.346266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 1281.366372] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1281.389212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1286.221660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1286.226476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.231553] Lustre: Skipped 2 previous similar messages [ 1286.234178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1288.628362] Lustre: server umount lustre-OST0000 complete [ 1291.357692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1291.361672] Lustre: Skipped 2 previous similar messages [ 1291.363431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1291.363432] Lustre: Skipped 1 previous similar message [ 1295.865027] Lustre: server umount lustre-MDT0000 complete [ 1296.366058] LustreError: 18502:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1296.373480] LustreError: 18502:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 4 previous similar messages [ 1297.023500] LustreError: 18483:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256243 with bad export cookie 5016655666344017202 [ 1297.026167] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1297.026168] LustreError: Skipped 1 previous similar message [ 1297.035727] LustreError: 18483:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1297.172120] Lustre: server umount lustre-MDT0001 complete [ 1301.345245] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1302.162774] Key type lgssc unregistered [ 1302.292785] LNet: 24041:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1303.300187] LNet: Removed LNI 192.168.202.128@tcp [ 1306.393578] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1313.850313] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1314.021406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1314.050649] alg: No test for adler32 (adler32-zlib) [ 1314.898890] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 1314.965496] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1314.973108] LNet: Accept secure, port 988 [ 1316.509556] Key type lgssc registered [ 1316.856543] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1319.740506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1319.757189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1320.969542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1322.144130] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1324.929341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1324.987365] Lustre: 26939:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1325.088488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1326.217983] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1328.136054] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1329.369117] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1332.173543] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1332.180335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1332.277221] Lustre: 28051:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1332.328597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1334.028080] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1336.006312] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1345.333870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1350.334093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1350.335157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1350.343740] Lustre: Skipped 1 previous similar message [ 1354.739479] Lustre: server umount lustre-OST0000 complete [ 1360.109949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1360.113369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1360.118988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1360.121640] Lustre: Skipped 1 previous similar message [ 1362.080444] Lustre: server umount lustre-MDT0000 complete [ 1363.267912] LustreError: 29129:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256309 with bad export cookie 4184154112555761453 [ 1363.269861] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1363.279233] LustreError: 29129:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1363.392469] Lustre: server umount lustre-MDT0001 complete [ 1366.871381] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1367.577946] Key type lgssc unregistered [ 1367.660227] LNet: 29934:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1368.668411] LNet: Removed LNI 192.168.202.128@tcp [ 1375.970972] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1376.175299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1376.180795] alg: No test for adler32 (adler32-zlib) [ 1377.028390] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 1377.100478] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1377.102793] LNet: Accept secure, port 988 [ 1378.641748] Key type lgssc registered [ 1378.943382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1381.447132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1381.455186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1382.647654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1383.700714] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1386.664376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1386.718841] Lustre: 32305:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1386.822743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1387.985091] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1389.879707] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1391.051773] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1393.534094] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1393.543770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1393.630584] Lustre: 990:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1393.694926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1395.441824] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1397.059143] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1407.693172] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1416.733380] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1416.736712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1416.742488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1417.709310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1417.709795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1417.709815] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1417.720337] Lustre: Skipped 1 previous similar message [ 1420.084073] Lustre: server umount lustre-OST0000 complete [ 1421.853546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1421.858191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1421.864334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1427.233228] Lustre: server umount lustre-MDT0000 complete [ 1427.725165] LustreError: 31446:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1427.733874] LustreError: 31446:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 3 previous similar messages [ 1428.440692] LustreError: 31427:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256374 with bad export cookie 11628968903391918787 [ 1428.441764] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1428.448097] LustreError: 31427:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1428.574536] Lustre: server umount lustre-MDT0001 complete [ 1431.935022] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1432.631703] Key type lgssc unregistered [ 1432.732302] LNet: 2872:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1433.738159] LNet: Removed LNI 192.168.202.128@tcp [ 1440.467383] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1440.642600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1440.646882] alg: No test for adler32 (adler32-zlib) [ 1441.478479] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 1441.539272] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1441.541724] LNet: Accept secure, port 988 [ 1443.075759] Key type lgssc registered [ 1443.364341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1446.128426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1446.136026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1447.332421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1448.336240] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1450.994851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1451.042800] Lustre: 5254:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1451.124078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1452.084387] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1453.683008] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1454.767505] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1456.837747] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1456.842548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1456.916195] Lustre: 6366:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1456.961248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1457.963753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1458.505714] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1460.456337] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1471.537771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1471.541419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1471.549664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1475.323519] Lustre: server umount lustre-OST0000 complete [ 1476.990284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1476.992451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1476.992452] Lustre: Skipped 2 previous similar messages [ 1477.008222] Lustre: Skipped 2 previous similar messages [ 1481.998212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1482.001451] Lustre: Skipped 2 previous similar messages [ 1482.597096] Lustre: server umount lustre-MDT0000 complete [ 1483.969435] LustreError: 6389:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256429 with bad export cookie 17687443780152158779 [ 1483.970226] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1483.980295] LustreError: 6389:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1484.132789] Lustre: server umount lustre-MDT0001 complete [ 1487.703680] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1488.514501] Key type lgssc unregistered [ 1488.626345] LNet: 8251:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1489.635980] LNet: Removed LNI 192.168.202.128@tcp [ 1497.225336] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1497.352214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1497.358469] alg: No test for adler32 (adler32-zlib) [ 1498.191736] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 1498.244687] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1498.247051] LNet: Accept secure, port 988 [ 1499.777925] Key type lgssc registered [ 1500.057079] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1502.793285] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1502.799592] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1502.812048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1519.924916] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1525.929193] LustreError: 9741:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.128@tcp: failed processing log, type 4: rc = -110 [ 1555.953761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1557.435391] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1558.988776] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1560.875130] Lustre: server umount lustre-OST0000 complete [ 1563.635179] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 04:55:10 (1734256510) [ 1565.885495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1566.033646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1567.006591] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1569.917199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1569.961134] Lustre: 12580:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1570.047605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1571.000572] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1572.917890] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1574.155415] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1576.550887] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1576.567424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1576.677526] Lustre: 13693:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1576.752398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1578.692377] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1580.696497] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1591.748916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1600.797991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1600.801431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1600.808298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1601.759775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1601.760463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1601.778011] Lustre: Skipped 1 previous similar message [ 1604.761922] Lustre: server umount lustre-OST0000 complete [ 1606.768344] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1606.774873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1606.791130] Lustre: Skipped 1 previous similar message [ 1611.773657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1611.775750] Lustre: Skipped 2 previous similar messages [ 1612.359484] Lustre: server umount lustre-MDT0000 complete [ 1613.720408] LustreError: 11700:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256559 with bad export cookie 12351382869500914119 [ 1613.720724] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1613.728290] LustreError: 11700:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1613.843125] Lustre: server umount lustre-MDT0001 complete [ 1617.040211] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1617.680035] Key type lgssc unregistered [ 1617.764130] LNet: 15743:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1618.773109] LNet: Removed LNI 192.168.202.128@tcp [ 1625.580300] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1625.787119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1625.793252] alg: No test for adler32 (adler32-zlib) [ 1626.631826] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 1626.699566] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1626.702149] LNet: Accept secure, port 988 [ 1628.240808] Key type lgssc registered [ 1628.546160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1630.751549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1630.762528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1631.940995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1632.920338] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1636.168912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1636.235401] Lustre: 18112:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1636.373484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1637.626460] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1639.535288] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1640.745773] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1646.333321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1646.338021] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1646.343739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1647.382748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1647.385781] Lustre: Skipped 2 previous similar messages [ 1647.766494] Lustre: server umount lustre-MDT0000 complete [ 1649.046718] LustreError: 17236:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256595 with bad export cookie 16500055199075814952 [ 1649.050263] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1649.054160] LustreError: 17236:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1649.184534] Lustre: server umount lustre-MDT0001 complete [ 1651.770101] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 04:56:38 (1734256598) [ 1654.129765] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1654.139447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1671.261105] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1677.265007] LustreError: 20023:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.128@tcp: failed processing log, type 4: rc = -110 [ 1706.305456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1708.035882] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1709.864861] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1710.851364] Lustre: server umount lustre-OST0000 complete [ 1714.640094] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 04:57:40 (1734256660) [ 1717.320159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1717.566333] LustreError: 21744:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1717.575587] LustreError: 21744:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 1717.607389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1718.985188] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1721.656706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1721.696547] Lustre: 22604:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1721.783655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1722.749058] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1724.254185] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1725.300589] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1727.436742] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1727.442653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1727.538219] Lustre: 23715:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1728.619212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1729.354602] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1731.144688] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1736.623759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1736.630255] Lustre: Skipped 1 previous similar message [ 1736.631034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1739.567792] Lustre: server umount lustre-MDT0000 complete [ 1740.986700] LustreError: 21724:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256686 with bad export cookie 16500055199075815603 [ 1740.987607] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1740.996381] LustreError: 21724:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1741.103383] Lustre: server umount lustre-MDT0001 complete [ 1752.470726] Lustre: server umount lustre-OST0000 complete [ 1754.960745] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 04:58:21 (1734256701) [ 1757.010187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1757.190491] LustreError: 25936:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1757.228312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1757.231863] Lustre: Skipped 1 previous similar message [ 1758.105760] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1760.499790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1760.529970] Lustre: 26795:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1761.526618] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1763.149004] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1764.098473] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1766.110147] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1766.114905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1766.188861] Lustre: 27906:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1767.240266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:548 to 0x280000401:577) [ 1767.677607] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1769.215739] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1770.047869] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1770.094869] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1770.970513] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1771.030329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1772.253422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1772.256939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1772.261986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1772.264051] Lustre: Skipped 2 previous similar messages [ 1775.245369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1775.251358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1777.626600] Lustre: server umount lustre-OST0000 complete [ 1780.253246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.253814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1780.253815] Lustre: Skipped 2 previous similar messages [ 1780.262711] Lustre: Skipped 1 previous similar message [ 1784.874047] Lustre: server umount lustre-MDT0000 complete [ 1785.262898] LustreError: 25936:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1785.272829] LustreError: 25936:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 1786.282732] LustreError: 25919:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256732 with bad export cookie 16500055199075816835 [ 1786.285756] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1786.290638] LustreError: 25919:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1786.441949] Lustre: server umount lustre-MDT0001 complete [ 1789.218074] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 04:58:55 (1734256735) [ 1791.676710] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1791.688175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1808.800587] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1814.808081] LustreError: 30621:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.128@tcp: failed processing log, type 4: rc = -110 [ 1843.863947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1843.867957] Lustre: Skipped 2 previous similar messages [ 1845.438196] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1847.391935] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1849.878272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1850.071901] LustreError: 30648:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1850.127229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:548 to 0x280000401:609) [ 1851.086438] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1854.278745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1854.337828] Lustre: 32666:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1855.905209] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1858.138516] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1859.589963] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1861.129234] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1861.250999] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1862.585274] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1862.664327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1864.479440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1864.483201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1864.494551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1864.498232] Lustre: Skipped 1 previous similar message [ 1869.682025] Lustre: server umount lustre-MDT0000 complete [ 1871.447891] LustreError: 30629:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256817 with bad export cookie 16500055199075817850 [ 1871.451533] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1871.467112] LustreError: 30629:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1871.644736] Lustre: server umount lustre-MDT0001 complete [ 1883.471856] Lustre: server umount lustre-OST0000 complete [ 1886.678467] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 05:00:33 (1734256833) [ 1889.413227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1889.607341] LustreError: 2978:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1889.637896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1889.641306] Lustre: Skipped 2 previous similar messages [ 1890.908296] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1894.516661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1894.574748] Lustre: 3839:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1894.637795] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1894.644161] Lustre: Skipped 3 previous similar messages [ 1896.127832] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1898.495252] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1899.886438] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1902.636285] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1902.643492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1904.551234] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1906.310085] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1907.450517] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1914.788254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:548 to 0x280000401:641) [ 1915.677958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 1916.648015] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1916.701915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1919.789555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1919.789998] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1919.790000] Lustre: Skipped 2 previous similar messages [ 1923.431876] Lustre: server umount lustre-OST0000 complete [ 1936.699137] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1936.705111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1936.792512] Lustre: 6950:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1936.796729] Lustre: 6950:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [ 1938.391882] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1939.843163] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1939.850659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 1939.852085] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 1940.100968] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1941.251973] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1941.313322] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1942.208957] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1942.270029] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1943.212211] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b65ac000.ost_server_uuid 50 [ 1943.629446] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8800b65ac000.ost_server_uuid in FULL state after 0 sec [ 1943.675255] LustreError: 2972:0:(ldlm_lockd.c:759:ldlm_handle_ast_error()) ### client (nid 192.168.202.28@tcp) returned error from blocking AST (req@ffff88008b5c1500 x1818499819344000 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: ffff88011e388d80/0xe4fbf8c8534060dc lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.28@tcp remote: 0xbc087cacff2e7911 expref: 5 pid: 2974 timeout: 2042 lvb_type: 0 [ 1943.693009] LustreError: lustre-MDT0000: A client on nid 192.168.202.28@tcp was evicted due to a lock blocking callback time out: rc -107 [ 1943.697253] LustreError: 2964:0:(ldlm_lockd.c:241:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.28@tcp ns: mdt-lustre-MDT0000_UUID lock: ffff88011e388d80/0xe4fbf8c8534060dc lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.28@tcp remote: 0xbc087cacff2e7911 expref: 6 pid: 2974 timeout: 0 lvb_type: 0 [ 1944.846091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1944.848430] Lustre: Skipped 3 previous similar messages [ 1946.614856] Lustre: server umount lustre-OST0000 complete [ 1949.773281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1953.827749] Lustre: server umount lustre-MDT0000 complete [ 1954.861343] LustreError: 8470:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1954.868625] LustreError: 8470:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 1 previous similar message [ 1954.927398] LustreError: 2960:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256900 with bad export cookie 16500055199075818704 [ 1954.928757] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1954.936090] LustreError: 2960:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1961.049598] Lustre: server umount lustre-MDT0001 complete [ 1964.868343] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1965.883734] Key type lgssc unregistered [ 1965.992111] LNet: 9550:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1967.002858] LNet: Removed LNI 192.168.202.128@tcp [ 1975.078484] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1975.232254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1975.255984] alg: No test for adler32 (adler32-zlib) [ 1976.107521] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 1976.179954] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1976.182028] LNet: Accept secure, port 988 [ 1977.718107] Key type lgssc registered [ 1978.029338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1980.678742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1980.688221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1981.928578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1983.123643] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1985.987148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1986.043771] Lustre: 11926:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1986.141944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1987.271589] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1988.988582] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1990.162352] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1992.788386] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 1992.799936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1992.900034] Lustre: 13035:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1992.956442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1994.893128] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1996.915014] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2006.963528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:643 to 0x280000401:673) [ 2009.317076] Lustre: Failing over lustre-MDT0000 [ 2009.396859] Lustre: server umount lustre-MDT0000 complete [ 2011.149733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2011.154083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.162300] LustreError: 11066:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2011.170295] LustreError: 11066:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 2011.965923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.966088] LustreError: 11066:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.975755] LustreError: 11065:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.982359] LustreError: 11065:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 4 previous similar messages [ 2020.108483] LustreError: 11047:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.28@tcp arrived at 1734256966 with bad export cookie 16944369221221753813 [ 2020.123399] LustreError: 11047:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2021.983768] LustreError: 11065:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2021.990764] LustreError: 11065:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 2022.266696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2022.330787] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2022.442829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2022.467695] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2023.678323] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2024.520354] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2024.525823] Lustre: lustre-MDT0000: Denying connection for new client 0f9a0cac-1332-4d02-bb89-8a8bd32c21f7 (at 192.168.202.28@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2027.459646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2029.540369] Lustre: lustre-MDT0000: Denying connection for new client 0f9a0cac-1332-4d02-bb89-8a8bd32c21f7 (at 192.168.202.28@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 2032.014118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2032.017745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2032.023985] Lustre: Skipped 1 previous similar message [ 2032.027361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2037.428706] Lustre: server umount lustre-OST0000 complete [ 2038.653765] LustreError: 15253:0:(ldlm_lib.c:2900:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2038.657460] Lustre: 14467:0:(ldlm_lib.c:2283:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2038.660867] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2038.668018] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2038.672954] LustreError: 14467:0:(client.c:1300:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88012c945f80 x1818500186277248/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 2038.680079] LustreError: 14467:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2038.684583] LustreError: 14467:0:(fid_request.c:315:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2038.690001] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 2038.706230] Lustre: 14467:0:(mdt_handler.c:8090:mdt_postrecov()) lustre-MDT0000: auto trigger paused LFSCK failed: rc = -6 [ 2038.774685] Lustre: server umount lustre-MDT0000 complete [ 2040.035819] LustreError: 13839:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734256986 with bad export cookie 16944369221221754142 [ 2040.038969] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2040.041091] LustreError: 11066:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2040.051221] LustreError: 13839:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2046.305838] Lustre: server umount lustre-MDT0001 complete [ 2050.658699] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2051.534105] Key type lgssc unregistered [ 2051.644317] LNet: 16061:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2052.655865] LNet: Removed LNI 192.168.202.128@tcp [ 2060.724996] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2060.965345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2060.975310] alg: No test for adler32 (adler32-zlib) [ 2061.811477] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 2061.881502] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2061.885179] LNet: Accept secure, port 988 [ 2063.439239] Key type lgssc registered [ 2063.739745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2066.461672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2066.468320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.620456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2068.502942] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2070.813536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2070.850456] Lustre: 18434:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2070.933719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2071.817684] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2073.330675] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2074.408414] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2076.328071] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2076.331896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2076.407146] Lustre: 19543:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2076.459897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2077.462206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:643 to 0x280000401:705) [ 2077.978799] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2079.341558] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2087.485408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2087.491670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2087.501318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2090.481292] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2090.481337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2090.496550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2092.709365] Lustre: server umount lustre-OST0000 complete [ 2095.485413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2095.488912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2095.488913] Lustre: Skipped 1 previous similar message [ 2095.495126] Lustre: Skipped 1 previous similar message [ 2100.495525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2100.497446] Lustre: Skipped 1 previous similar message [ 2100.623941] Lustre: server umount lustre-MDT0000 complete [ 2102.236855] LustreError: 17554:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257048 with bad export cookie 15133551528681173245 [ 2102.237652] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2102.255383] LustreError: 17554:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2102.463760] Lustre: server umount lustre-MDT0001 complete [ 2106.497405] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2107.459645] Key type lgssc unregistered [ 2107.564307] LNet: 21424:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2108.577251] LNet: Removed LNI 192.168.202.128@tcp [ 2121.062612] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2121.303825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2121.310299] alg: No test for adler32 (adler32-zlib) [ 2122.153601] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 2122.224514] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2122.226467] LNet: Accept secure, port 988 [ 2123.775473] Key type lgssc registered [ 2124.104536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2126.998064] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2127.011850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2128.255212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2129.575851] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2133.316834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2133.379888] Lustre: 24185:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2133.496828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2134.871281] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2136.833714] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2138.162292] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2141.128437] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2141.135928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2141.240426] Lustre: 25299:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2141.307339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2143.329726] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2145.538455] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2153.310431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:643 to 0x280000401:737) [ 2158.318209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2158.318567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2158.318724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2158.328866] Lustre: Skipped 1 previous similar message [ 2158.879813] Lustre: server umount lustre-OST0000 complete [ 2162.054299] Lustre: server umount lustre-MDT0000 complete [ 2163.152677] LustreError: 23307:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257109 with bad export cookie 6499640158376272014 [ 2163.154543] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2163.161659] LustreError: 23307:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2163.275648] Lustre: server umount lustre-MDT0001 complete [ 2166.572076] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2167.172323] Key type lgssc unregistered [ 2167.251946] LNet: 27171:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2168.256913] LNet: Removed LNI 192.168.202.128@tcp [ 2174.594724] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2174.726726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2174.755635] alg: No test for adler32 (adler32-zlib) [ 2175.593709] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 2175.652281] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2175.654353] LNet: Accept secure, port 988 [ 2177.185797] Key type lgssc registered [ 2177.494983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2182.161496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2182.174389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2183.375392] Lustre: *** cfs_fail_loc=135, val=0*** [ 2183.419329] LustreError: 28827:0:(obd_config.c:797:class_setup()) setup lustre-MDT0000 failed (-2) [ 2183.427228] LustreError: 28827:0:(obd_config.c:2030:class_config_llog_handler()) MGC192.168.202.128@tcp: cfg command failed: rc = -2 [ 2183.431471] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2183.441475] LustreError: MGC192.168.202.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2183.450110] LustreError: 28789:0:(tgt_mount.c:1493:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2183.456835] LustreError: 28789:0:(tgt_mount.c:2222:server_fill_super()) Unable to start targets: -2 [ 2183.460502] LustreError: 28789:0:(obd_config.c:850:class_cleanup()) Device 5 not setup [ 2183.598878] Lustre: server umount lustre-MDT0000 complete [ 2183.600119] LustreError: 28789:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2186.489421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2186.741349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2188.115095] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2191.746319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2191.796954] Lustre: 30172:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2191.888024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2193.060911] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2194.881387] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2196.188499] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2201.163837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2201.167719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2201.180138] Lustre: Skipped 1 previous similar message [ 2203.250033] Lustre: server umount lustre-MDT0000 complete [ 2204.721606] LustreError: 29294:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257150 with bad export cookie 12228703222232122668 [ 2204.723867] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2204.731051] LustreError: 29294:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2204.872474] Lustre: server umount lustre-MDT0001 complete [ 2208.155189] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2209.129836] Key type lgssc unregistered [ 2209.281354] LNet: 31764:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2210.296683] LNet: Removed LNI 192.168.202.128@tcp [ 2217.939001] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2224.338017] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2224.583358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2224.589618] alg: No test for adler32 (adler32-zlib) [ 2225.458316] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 2225.528788] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2225.534603] LNet: Accept secure, port 988 [ 2227.116984] Key type lgssc registered [ 2227.485742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2230.788193] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2230.794520] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2230.801520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2247.954911] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2253.957857] LustreError: 1690:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.128@tcp: failed processing log, type 4: rc = -110 [ 2283.976854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2285.630201] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2287.304835] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2289.484374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2289.687593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2289.708343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:739 to 0x280000401:769) [ 2290.685889] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2293.656176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2293.717122] Lustre: 3736:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2293.853544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2294.936966] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2296.837358] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2298.126179] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2299.237024] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 2302.862775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2302.863630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2302.873180] Lustre: Skipped 1 previous similar message [ 2306.353777] Lustre: server umount lustre-OST0000 complete [ 2308.845913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2308.851432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2308.858663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2308.862361] Lustre: Skipped 1 previous similar message [ 2312.877765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2312.878631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2312.887351] Lustre: Skipped 1 previous similar message [ 2314.063112] Lustre: server umount lustre-MDT0000 complete [ 2315.632941] LustreError: 4596:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257261 with bad export cookie 15347207277348088682 [ 2315.639030] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2315.798963] Lustre: server umount lustre-MDT0001 complete [ 2319.421849] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2320.551820] Key type lgssc unregistered [ 2320.659407] LNet: 5881:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2321.669682] LNet: Removed LNI 192.168.202.128@tcp [ 2331.400018] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2331.679525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2331.687417] alg: No test for adler32 (adler32-zlib) [ 2332.560466] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 2332.649815] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2332.653126] LNet: Accept secure, port 988 [ 2334.202752] Key type lgssc registered [ 2334.587003] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2337.778400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2337.794809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2339.014413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2340.301519] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2344.289134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2344.349410] Lustre: 8259:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2344.514280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2345.907085] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2347.920476] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2349.190536] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2351.771525] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2351.779438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2351.865421] Lustre: 9372:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2351.915221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2353.938267] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2356.160526] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2363.921279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:739 to 0x280000401:801) [ 2367.417839] Lustre: Failing over lustre-MDT0000 [ 2367.496905] Lustre: server umount lustre-MDT0000 complete [ 2368.926635] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2368.928658] LustreError: 7396:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2368.928661] LustreError: 7396:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 2368.942112] Lustre: Skipped 2 previous similar messages [ 2371.359948] LustreError: 7391:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2371.369995] LustreError: 7391:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 2373.933877] LustreError: 10065:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2373.940162] LustreError: 10065:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 2 previous similar messages [ 2376.355096] LustreError: 7391:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.684515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2380.751864] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2380.855655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2380.877325] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2381.361413] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2381.931611] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2383.609874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire in log lustre-MDT0000 [ 2385.855406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2385.862688] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2385.883419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:739 to 0x280000401:833) [ 2386.899044] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight in log lustre-client [ 2390.862936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2390.864737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2390.875847] Lustre: Skipped 1 previous similar message [ 2392.234292] Lustre: server umount lustre-OST0000 complete [ 2395.870706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2395.873710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2395.873711] Lustre: Skipped 1 previous similar message [ 2395.891610] Lustre: Skipped 1 previous similar message [ 2399.757266] Lustre: server umount lustre-MDT0000 complete [ 2400.877492] LustreError: 7396:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2400.884413] LustreError: 7396:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 4 previous similar messages [ 2401.228898] LustreError: 7380:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257347 with bad export cookie 13048625338382892281 [ 2401.229968] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2401.237821] LustreError: 7380:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2401.356561] Lustre: server umount lustre-MDT0001 complete [ 2405.689454] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2406.598525] Key type lgssc unregistered [ 2406.692032] LNet: 12865:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2407.704741] LNet: Removed LNI 192.168.202.128@tcp [ 2417.584993] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2417.917196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2417.947686] alg: No test for adler32 (adler32-zlib) [ 2418.826042] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 2418.913734] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2418.916583] LNet: Accept secure, port 988 [ 2420.493778] Key type lgssc registered [ 2420.887318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2424.558338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2424.571150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2425.790055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2427.089363] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2430.530695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2430.586895] Lustre: 15240:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2430.715248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2432.282975] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2434.581608] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2436.006173] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2439.363717] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2439.373920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2439.496951] Lustre: 16353:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2439.576966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2441.448035] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2443.416259] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2450.581208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:865) [ 2454.066649] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 2459.662975] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 2469.703532] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 2480.625179] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2480.632326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2480.635237] Lustre: Skipped 1 previous similar message [ 2485.077348] Lustre: server umount lustre-OST0000 complete [ 2490.640502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2490.641150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2490.650549] Lustre: Skipped 2 previous similar messages [ 2492.812252] Lustre: server umount lustre-MDT0000 complete [ 2494.654800] LustreError: 17095:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257440 with bad export cookie 5393841208905741192 [ 2494.659560] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2494.667448] LustreError: 17095:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2494.854583] Lustre: server umount lustre-MDT0001 complete [ 2499.537882] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2500.603449] Key type lgssc unregistered [ 2500.759182] LNet: 18465:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2501.768522] LNet: Removed LNI 192.168.202.128@tcp [ 2510.892656] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2511.128790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2511.136004] alg: No test for adler32 (adler32-zlib) [ 2511.999101] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 2512.093691] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2512.101778] LNet: Accept secure, port 988 [ 2513.674955] Key type lgssc registered [ 2514.060787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2517.373128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2517.384993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2518.618310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2520.197913] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2524.816802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2524.874026] Lustre: 20844:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2525.025181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2527.021840] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2529.799186] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2531.948962] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2535.534005] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2535.541926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2535.639044] Lustre: 21955:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2535.727614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2538.177894] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2540.533310] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2543.724857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:897) [ 2548.227448] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 2551.258923] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds in log lustre-OST0000 [ 2560.312559] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub in log lustre-OST0000 [ 2566.455919] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub in log lustre-OST0000 [ 2574.766378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2574.772170] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2574.773171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2574.788925] Lustre: Skipped 1 previous similar message [ 2579.774963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2579.777918] Lustre: Skipped 2 previous similar messages [ 2580.271718] Lustre: server umount lustre-OST0000 complete [ 2584.784456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2584.786719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2584.798893] Lustre: Skipped 1 previous similar message [ 2588.229613] Lustre: server umount lustre-MDT0000 complete [ 2589.790353] LustreError: 19980:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.797911] LustreError: 19980:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 3 previous similar messages [ 2590.056816] LustreError: 19963:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1734257536 with bad export cookie 6491733183800029089 [ 2590.057749] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2590.066588] LustreError: 19963:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2590.264730] Lustre: server umount lustre-MDT0001 complete [ 2594.849709] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2596.118620] Key type lgssc unregistered [ 2596.320775] LNet: 25886:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2597.333422] LNet: Removed LNI 192.168.202.128@tcp [ 2607.900014] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2608.161162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2608.191638] alg: No test for adler32 (adler32-zlib) [ 2609.043912] Lustre: Lustre: Build Version: 2.16.50_90_gf50dd6a [ 2609.114361] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2609.117172] LNet: Accept secure, port 988 [ 2610.663449] Key type lgssc registered [ 2611.139538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2614.701660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2614.713682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2616.018533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2617.744373] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2621.916044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2621.973891] Lustre: 28261:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2622.096395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2623.493386] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2626.133049] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2628.387096] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2631.784261] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [ 2631.797911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2631.926154] Lustre: 29379:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.202.128@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2632.029672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2634.803751] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2637.657446] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2640.037348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:929) [ 2643.481259] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 2645.047516] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 2652.891592] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 2662.866989] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 2668.539297] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 2678.945897] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 2678.949286] Lustre: Skipped 1 previous similar message [ 2696.519441] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 2696.522794] Lustre: Skipped 1 previous similar message [ 2731.953347] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client [ 2731.958915] Lustre: Skipped 4 previous similar messages [ 2794.031995] Lustre: DEBUG MARKER: conf-sanity test_30a: @@@@@@ FAIL: Deleted value=64 -ne orig=61