[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 477559501 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003154] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.007985] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010134] LSM: Security Framework initializing [ 0.011059] Yama: becoming mindful. [ 0.012045] SELinux: Initializing. [ 0.013060] *** VALIDATE selinux *** [ 0.021481] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026277] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027158] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030072] *** VALIDATE tmpfs *** [ 0.031451] *** VALIDATE proc *** [ 0.033198] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.036270] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038100] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040028] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.044392] debug: unmapping init [mem 0xffffffff9ee59000-0xffffffff9ee60fff] [ 0.046185] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047623] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048023] ... version: 2 [ 0.049009] ... bit width: 48 [ 0.049936] ... generic registers: 4 [ 0.050016] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052014] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054324] rcu: Hierarchical SRCU implementation. [ 0.056481] smp: Bringing up secondary CPUs ... [ 0.057606] x86: Booting SMP configuration: [ 0.058026] .... node #0, CPUs: #1 #2 #3 [ 0.062027] smp: Brought up 1 node, 4 CPUs [ 0.064036] smpboot: Max logical packages: 1 [ 0.065022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.244846] node 0 deferred pages initialised in 177ms [ 0.247092] devtmpfs: initialized [ 0.248172] x86/mm: Memory block size: 128MB [ 0.252451] gcov: version magic: 0x41383552 [ 0.255511] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.260116] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.262494] pinctrl core: initialized pinctrl subsystem [ 0.265237] [ 0.265912] ************************************************************* [ 0.269019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.271018] ** ** [ 0.274021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.277020] ** ** [ 0.280021] ** This means that this kernel is built to expose internal ** [ 0.283020] ** IOMMU data structures, which may compromise security on ** [ 0.286026] ** your system. ** [ 0.288017] ** ** [ 0.291026] ** If you see this message and you are not debugging the ** [ 0.294017] ** kernel, report this immediately to your vendor! ** [ 0.296020] ** ** [ 0.299020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.302020] ************************************************************* [ 0.305833] NET: Registered protocol family 16 [ 0.307461] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.310066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.313071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.317037] cpuidle: using governor menu [ 0.318672] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.321579] PCI: Using configuration type 1 for base access [ 0.324123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.333165] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.336058] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.341040] cryptd: max_cpu_qlen set to 1000 [ 0.344055] ACPI: Added _OSI(Module Device) [ 0.346021] ACPI: Added _OSI(Processor Device) [ 0.349021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.351022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.356234] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.362494] ACPI: Interpreter enabled [ 0.364082] ACPI: PM: (supports S0 S3 S4 S5) [ 0.366020] ACPI: Using IOAPIC for interrupt routing [ 0.368146] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.373461] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.383866] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.386048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.389022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.393092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.398542] acpiphp: Slot [2] registered [ 0.400212] acpiphp: Slot [5] registered [ 0.402136] acpiphp: Slot [6] registered [ 0.403141] acpiphp: Slot [7] registered [ 0.405216] acpiphp: Slot [8] registered [ 0.406136] acpiphp: Slot [9] registered [ 0.408137] acpiphp: Slot [10] registered [ 0.410177] acpiphp: Slot [3] registered [ 0.412150] acpiphp: Slot [4] registered [ 0.413097] acpiphp: Slot [11] registered [ 0.414136] acpiphp: Slot [12] registered [ 0.416104] acpiphp: Slot [13] registered [ 0.417077] acpiphp: Slot [14] registered [ 0.418098] acpiphp: Slot [15] registered [ 0.419134] acpiphp: Slot [16] registered [ 0.421108] acpiphp: Slot [17] registered [ 0.423113] acpiphp: Slot [18] registered [ 0.424114] acpiphp: Slot [19] registered [ 0.426198] acpiphp: Slot [20] registered [ 0.428130] acpiphp: Slot [21] registered [ 0.429105] acpiphp: Slot [22] registered [ 0.431154] acpiphp: Slot [23] registered [ 0.432110] acpiphp: Slot [24] registered [ 0.435138] acpiphp: Slot [25] registered [ 0.436108] acpiphp: Slot [26] registered [ 0.438107] acpiphp: Slot [27] registered [ 0.439136] acpiphp: Slot [28] registered [ 0.441115] acpiphp: Slot [29] registered [ 0.443127] acpiphp: Slot [30] registered [ 0.444124] acpiphp: Slot [31] registered [ 0.446065] PCI host bridge to bus 0000:00 [ 0.447021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.450035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.452040] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.455037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.458039] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.462051] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.464239] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.469534] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.473479] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.485017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.490019] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.492022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.495024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.497023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.502092] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.505934] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.510069] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.514938] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.521022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.535023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.541025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.547898] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.556022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.563019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.578020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.589490] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.598018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.604033] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.620018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.631424] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.643023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.650023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.678024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.691033] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.697017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.702016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.717016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.726939] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.733018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.739019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.751018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.759000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.767019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.773017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.786018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.797812] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.800420] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.802460] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.803352] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.805283] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.810139] iommu: Default domain type: Passthrough [ 0.812509] SCSI subsystem initialized [ 0.814245] ACPI: bus type USB registered [ 0.816150] usbcore: registered new interface driver usbfs [ 0.818112] usbcore: registered new interface driver hub [ 0.820151] usbcore: registered new device driver usb [ 0.822212] pps_core: LinuxPPS API ver. 1 registered [ 0.824016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.827274] PTP clock support registered [ 0.829111] EDAC MC: Ver: 3.0.0 [ 0.831048] PCI: Using ACPI for IRQ routing [ 0.832914] NetLabel: Initializing [ 0.833011] NetLabel: domain hash size = 128 [ 0.834011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.835159] NetLabel: unlabeled traffic allowed by default [ 0.837177] vgaarb: loaded [ 0.839300] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.840018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.844407] clocksource: Switched to clocksource kvm-clock [ 0.954516] VFS: Disk quotas dquot_6.6.0 [ 0.955988] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.957797] *** VALIDATE ramfs *** [ 0.958956] *** VALIDATE hugetlbfs *** [ 0.959948] pnp: PnP ACPI init [ 0.961793] pnp: PnP ACPI: found 6 devices [ 0.985299] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.988901] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.992541] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.994709] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.997307] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.000151] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.003044] NET: Registered protocol family 2 [ 1.006291] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.012483] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.016509] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.021791] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.025029] TCP: Hash tables configured (established 65536 bind 65536) [ 1.028422] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.031865] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.035271] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.038289] NET: Registered protocol family 1 [ 1.041665] RPC: Registered named UNIX socket transport module. [ 1.044452] RPC: Registered udp transport module. [ 1.046605] RPC: Registered tcp transport module. [ 1.048708] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.053112] NET: Registered protocol family 44 [ 1.055587] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.058756] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.061906] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.064838] PCI: CLS 0 bytes, default 64 [ 1.066792] Unpacking initramfs... [ 2.485735] debug: unmapping init [mem 0xffff900d3cc54000-0xffff900d3ffbffff] [ 2.491182] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.494987] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.499448] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.031492] Initialise system trusted keyrings [ 3.036596] Key type blacklist registered [ 3.039151] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.052071] zbud: loaded [ 3.055215] *** VALIDATE nfs *** [ 3.056711] *** VALIDATE nfs4 *** [ 3.058953] pstore: using deflate compression [ 3.063060] Platform Keyring initialized [ 3.175507] NET: Registered protocol family 38 [ 3.177235] Key type asymmetric registered [ 3.179128] Asymmetric key parser 'x509' registered [ 3.181290] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.184820] io scheduler mq-deadline registered [ 3.186272] io scheduler kyber registered [ 3.187585] io scheduler bfq registered [ 3.189425] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.192783] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.196355] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.200371] ACPI: Power Button [PWRF] [ 3.206279] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.214537] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.226606] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.237461] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.262394] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.291952] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.320242] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.325514] Non-volatile memory driver v1.3 [ 3.327356] Linux agpgart interface v0.103 [ 3.362144] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.365366] vda: detected capacity change from 0 to 74698752 [ 3.379196] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.382494] vdb: detected capacity change from 0 to 1073741824 [ 3.399794] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.402590] vdc: detected capacity change from 0 to 2621440000 [ 3.418678] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.421779] vdd: detected capacity change from 0 to 2621440000 [ 3.441149] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.444655] vde: detected capacity change from 0 to 4294967296 [ 3.462572] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.465774] vdf: detected capacity change from 0 to 4294967296 [ 3.474891] libphy: Fixed MDIO Bus: probed [ 3.489712] usbcore: registered new interface driver usbserial_generic [ 3.491723] usbserial: USB Serial support registered for generic [ 3.493633] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.497258] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.499313] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.501937] mousedev: PS/2 mouse device common for all mice [ 3.506073] rtc_cmos 00:05: RTC can wake from S4 [ 3.511301] rtc_cmos 00:05: registered as rtc0 [ 3.513237] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.516461] intel_pstate: CPU model not supported [ 3.516567] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.523540] hid: raw HID events driver (C) Jiri Kosina [ 3.524975] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.527382] usbcore: registered new interface driver usbhid [ 3.532460] usbhid: USB HID core driver [ 3.534578] drop_monitor: Initializing network drop monitor service [ 3.534776] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.537477] Initializing XFRM netlink socket [ 3.541916] NET: Registered protocol family 10 [ 3.545152] Segment Routing with IPv6 [ 3.546821] NET: Registered protocol family 17 [ 3.549581] mpls_gso: MPLS GSO support [ 3.557203] RAS: Correctable Errors collector initialized. [ 3.559728] AVX version of gcm_enc/dec engaged. [ 3.561695] AES CTR mode by8 optimization enabled [ 3.647714] sched_clock: Marking stable (3647680471, 0)->(4577463084, -929782613) [ 3.651990] registered taskstats version 1 [ 3.654698] Loading compiled-in X.509 certificates [ 3.657539] zswap: loaded using pool lzo/zbud [ 3.693276] Key type big_key registered [ 3.704609] Key type encrypted registered [ 3.706604] ima: No TPM chip found, activating TPM-bypass! [ 3.708997] ima: Allocated hash algorithm: sha1 [ 3.711057] ima: No architecture policies found [ 3.712806] evm: Initialising EVM extended attributes: [ 3.714767] evm: security.selinux [ 3.715984] evm: security.ima [ 3.717016] evm: security.capability [ 3.718196] evm: HMAC attrs: 0x1 [ 3.721075] rtc_cmos 00:05: setting system clock to 2026-09-03 13:27:24 UTC (1788442044) [ 3.727571] debug: unmapping init [mem 0xffffffff9fe03000-0xffffffff9fffffff] [ 3.730838] debug: unmapping init [mem 0xffffffff9eb82000-0xffffffff9ee58fff] [ 3.739082] Write protecting the kernel read-only data: 28672k [ 3.744233] debug: unmapping init [mem 0xffffffff9d203000-0xffffffff9d3fffff] [ 3.747149] debug: unmapping init [mem 0xffffffff9db14000-0xffffffff9dbfffff] [ 3.780714] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.790944] systemd[1]: Detected virtualization kvm. [ 3.793250] systemd[1]: Detected architecture x86-64. [ 3.795479] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.819441] systemd[1]: No hostname configured. [ 3.821395] systemd[1]: Set hostname to . [ 3.823673] random: systemd: uninitialized urandom read (16 bytes read) [ 3.826300] systemd[1]: Initializing machine ID from random generator. [ 3.945859] random: systemd: uninitialized urandom read (16 bytes read) [ 3.949126] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.954034] random: systemd: uninitialized urandom read (16 bytes read) [ 3.956773] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.960968] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.589648] device-mapper: uevent: version 1.0.3 [ 4.591763] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.300835] random: fast init done [ 5.327493] virtio_net virtio0 ens2: renamed from eth0 [ 5.409913] scsi host0: ata_piix [ 5.423898] scsi host1: ata_piix [ 5.425439] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.430880] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.289842] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.223539] random: crng init done [ 10.225278] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.767753] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.909480] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.187454] SELinux: Disabled at runtime. [ 12.255266] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.263243] systemd[1]: Detected virtualization kvm. [ 12.264670] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.780824] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.783983] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.788889] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.793505] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.797401] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.805824] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.823902] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ 12.896815] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.239324] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.518776] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.519814] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.644497] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.663949] EDAC sbridge: Ver: 1.1.2 [ 15.552625] Key type dns_resolver registered [ 15.857349] NFS: Registering the id_resolver key type [ 15.859587] Key type id_resolver registered [ 15.861509] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg402-server login: [ 42.684498] libcfs: loading out-of-tree module taints kernel. [ 42.710822] Key type ._llcrypt registered [ 42.712696] Key type .llcrypt registered [ 42.775711] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 52.124886] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 52.955147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.964221] alg: No test for adler32 (adler32-zlib) [ 54.566938] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 55.624055] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 57.464183] Key type lgssc registered [ 59.520274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 61.094041] hrtimer: interrupt took 5030660 ns [ 76.779933] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 124.917616] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 139.178860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 139.258354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 140.710534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 140.753490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 140.971707] Lustre: lustre-MDT0000: new disk, initializing [ 141.151928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 141.171503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 146.536287] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.807236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.965395] Lustre: 6487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 161.026619] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 161.036555] Lustre: Skipped 1 previous similar message [ 161.183578] Lustre: lustre-MDT0001: new disk, initializing [ 161.276399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 161.324947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 161.337792] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 165.795287] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.524310] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 183.617105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 183.908513] Lustre: lustre-OST0000: new disk, initializing [ 183.915684] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 183.921477] Lustre: 8426:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 184.004047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 187.518367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 187.535991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 187.678661] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 190.840509] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.142367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 207.407383] Lustre: lustre-OST0001: new disk, initializing [ 207.412285] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 207.421539] Lustre: 9497:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 207.521324] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 214.037534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 214.045632] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 214.114947] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 214.116669] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.539707] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 235.159440] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 242.786445] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing check_logdir /tmp/testlogs/ [ 248.773943] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing yml_node [ 254.934968] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 258.835891] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 261.653673] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 263.471112] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 09:31:42 EDT 2026 [ 282.744512] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 284.216703] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 301.027667] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.033963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.048348] Lustre: Skipped 3 previous similar messages [ 301.060410] Lustre: Skipped 3 previous similar messages [ 306.152971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.161431] Lustre: Skipped 3 previous similar messages [ 311.269195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 311.289249] Lustre: Skipped 3 previous similar messages [ 315.363347] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 315.936490] Lustre: server umount lustre-MDT0000 complete [ 320.395768] LustreError: 10316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442361 with bad export cookie 17617733783708877161 [ 320.399784] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 320.404579] LustreError: 10316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 321.507020] LustreError: 12141:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 321.510826] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.512391] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 321.537662] LustreError: 12141:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 321.578800] Lustre: Skipped 1 previous similar message [ 325.537257] LustreError: 6496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.539373] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 325.561806] LustreError: 6496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 325.578895] Lustre: Skipped 2 previous similar messages [ 326.942735] Lustre: server umount lustre-MDT0001 complete [ 332.910262] Lustre: server umount lustre-OST0000 complete [ 336.570851] Lustre: server umount lustre-OST0001 complete [ 343.244492] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 353.888304] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 394.641194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.954427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 395.002251] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 395.127416] Lustre: lustre-MDT0000: new disk, initializing [ 395.244735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 395.257773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 400.032844] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 411.553181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.650874] Lustre: 15057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 411.699480] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 411.702600] Lustre: Skipped 1 previous similar message [ 411.788549] Lustre: lustre-MDT0001: new disk, initializing [ 411.971230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 412.016940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 412.048139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 416.631908] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 426.801188] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 433.046971] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 441.128256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 441.343296] Lustre: lustre-OST0000: new disk, initializing [ 441.347317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 441.354356] Lustre: 17028:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 441.454611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 442.525664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 442.554678] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 442.811937] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 448.594351] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 458.635495] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 463.859113] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 464.089272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 468.978535] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 469.291382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 473.570538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 473.570983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 473.586687] LustreError: Skipped 1 previous similar message [ 473.589183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 473.603313] Lustre: Skipped 1 previous similar message [ 473.632510] Lustre: Skipped 1 previous similar message [ 477.580419] Lustre: server umount lustre-OST0000 complete [ 483.810256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 483.827772] Lustre: Skipped 1 previous similar message [ 488.136379] Lustre: server umount lustre-MDT0000 complete [ 488.931382] LustreError: 15064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.955826] LustreError: 15064:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 492.185250] LustreError: 15052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442532 with bad export cookie 17617733783708878701 [ 492.194980] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 492.210791] LustreError: 15052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 492.663425] Lustre: server umount lustre-MDT0001 complete [ 500.224771] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:35:38 (1788442538) [ 521.583632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 522.024615] LustreError: 19902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.108110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 527.232493] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 527.335652] LustreError: 19903:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.375766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 536.551390] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 536.557104] Lustre: Skipped 2 previous similar messages [ 541.693744] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 549.647966] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 556.299379] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 558.839275] Lustre: Failing over lustre-MDT0000 [ 559.143611] Lustre: server umount lustre-MDT0000 complete [ 561.642113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 561.660662] LustreError: 19902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 561.666601] Lustre: Skipped 1 previous similar message [ 561.685875] LustreError: 19902:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 578.017321] Lustre: 3623:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442602/real 1788442602] req@ffff900db7a94a80 x1875317459419520/t0(0) o400->MGC192.168.204.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788442618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 578.018387] LustreError: 19902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.057345] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 578.077808] LustreError: 19902:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 579.479845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 587.623900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 587.634659] Lustre: Skipped 1 previous similar message [ 592.093788] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 592.877401] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 592.881135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 592.901617] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 602.581566] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 606.904123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 606.910405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 606.916025] Lustre: Skipped 1 previous similar message [ 606.929737] Lustre: Skipped 1 previous similar message [ 613.347206] LustreError: 20905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 613.392861] LustreError: 20905:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 613.441557] Lustre: server umount lustre-MDT0000 complete [ 618.099717] LustreError: 19885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442658 with bad export cookie 17617733783708880115 [ 618.113858] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 618.125566] LustreError: 19885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 618.602339] Lustre: server umount lustre-MDT0001 complete [ 627.729470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.141974] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 641.455478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 646.257600] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 654.627521] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 661.287831] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 663.803620] Lustre: Failing over lustre-MDT0000 [ 664.050544] Lustre: server umount lustre-MDT0000 complete [ 666.916561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 677.346394] LustreError: 23240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 677.377717] LustreError: 23240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 682.466281] Lustre: 3621:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442707/real 1788442707] req@ffff900c81bb3100 x1875317459449856/t0(0) o400->MGC192.168.204.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788442723 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 682.524178] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 685.104659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 693.137470] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 698.297515] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 698.340207] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 698.346976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 698.359522] Lustre: Skipped 1 previous similar message [ 698.370399] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 708.714732] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 713.699239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 713.705887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 713.722574] Lustre: Skipped 1 previous similar message [ 713.728729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 713.733555] Lustre: Skipped 2 previous similar messages [ 719.335214] Lustre: server umount lustre-MDT0000 complete [ 724.089963] LustreError: 23220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442764 with bad export cookie 17617733783708881053 [ 724.096821] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 724.113403] LustreError: 23220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 724.683645] Lustre: server umount lustre-MDT0001 complete [ 735.513398] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:39:34 (1788442774) [ 757.050161] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 769.869417] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 816.321962] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 829.238935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 829.543122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 829.626222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 829.736400] Lustre: lustre-MDT0000: new disk, initializing [ 829.824273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 829.831576] Lustre: Skipped 3 previous similar messages [ 829.856393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 834.778619] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 846.924649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 847.094578] Lustre: 29619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 847.385449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 852.409157] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 863.868911] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 870.519502] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 877.397480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 877.802225] Lustre: 31586:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 879.686344] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 879.754816] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 879.845436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 879.871209] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 885.934944] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 898.065614] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 905.359269] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 931.299112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 931.307245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.308880] LustreError: Skipped 1 previous similar message [ 931.310139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 931.310147] Lustre: Skipped 2 previous similar messages [ 931.329309] Lustre: Skipped 2 previous similar messages [ 933.461182] Lustre: server umount lustre-OST0000 complete [ 939.489843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 944.499932] Lustre: server umount lustre-MDT0000 complete [ 946.660094] LustreError: 29630:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 946.677566] LustreError: 29630:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 948.696521] LustreError: 31585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442989 with bad export cookie 17617733783708881459 [ 948.706351] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 948.712842] LustreError: 31585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 949.511324] Lustre: server umount lustre-MDT0001 complete [ 962.871437] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 966.293400] Key type lgssc unregistered [ 966.565870] LNet: 33570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.581923] LNetError: 33570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.607615] LNet: Removed LNI 192.168.204.102@tcp [ 967.811165] Key type .llcrypt unregistered [ 967.814421] Key type ._llcrypt unregistered [ 996.692433] Key type ._llcrypt registered [ 996.699551] Key type .llcrypt registered [ 996.804380] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 998.363961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 998.423142] alg: No test for adler32 (adler32-zlib) [ 999.441666] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 999.671793] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1001.392164] Key type lgssc registered [ 1002.805494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1011.790135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1011.817550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1013.451722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1018.428489] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1023.970497] LustreError: 34989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1023.991834] LustreError: 34989:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1028.645121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1029.092133] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1029.222855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1034.538104] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1044.120567] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1051.429389] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1060.101956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1060.516227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1068.580564] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1073.896295] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1077.164275] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1084.146733] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1105.680819] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:45:44 (1788443144) [ 1108.080491] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1110.428762] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1118.551387] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:45:56 (1788443156) [ 1131.402622] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:46:09 (1788443169) [ 1132.695535] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1144.607494] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1144.613893] Lustre: Skipped 2 previous similar messages [ 1147.522684] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:46:25 (1788443185) [ 1150.033742] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1150.042042] Lustre: Skipped 2 previous similar messages [ 1158.116222] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1158.132896] Lustre: Skipped 1 previous similar message [ 1175.520890] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1175.530099] Lustre: Skipped 5 previous similar messages [ 1179.144652] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:46:57 (1788443217) [ 1182.517896] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1187.121154] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1197.548611] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:47:15 (1788443235) [ 1212.791564] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1212.794718] Lustre: Skipped 7 previous similar messages [ 1215.731728] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:47:33 (1788443253) [ 1237.989971] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1237.990119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1238.000152] LustreError: Skipped 1 previous similar message [ 1238.001417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1238.027787] Lustre: Skipped 1 previous similar message [ 1240.941674] Lustre: server umount lustre-OST0000 complete [ 1248.227158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1248.232423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1248.249847] Lustre: Skipped 1 previous similar message [ 1248.273019] Lustre: Skipped 2 previous similar messages [ 1251.283562] Lustre: server umount lustre-MDT0000 complete [ 1253.346176] LustreError: 34988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1253.366676] LustreError: 34988:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1255.536892] LustreError: 34968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443296 with bad export cookie 12379497112181783434 [ 1255.539609] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1255.554312] LustreError: 34968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1256.035269] Lustre: server umount lustre-MDT0001 complete [ 1270.931934] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1274.059959] Key type lgssc unregistered [ 1274.372966] LNet: 42875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1274.385519] LNetError: 42875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1274.405336] LNet: Removed LNI 192.168.204.102@tcp [ 1275.424342] Key type .llcrypt unregistered [ 1275.429909] Key type ._llcrypt unregistered [ 1299.318576] Key type ._llcrypt registered [ 1299.323681] Key type .llcrypt registered [ 1299.501189] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 1319.158066] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1319.889468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1320.015207] alg: No test for adler32 (adler32-zlib) [ 1321.102448] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1321.488817] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1323.216178] Key type lgssc registered [ 1324.499243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1366.850094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1366.907113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1368.223906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1368.263854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1368.393534] Lustre: lustre-MDT0000: new disk, initializing [ 1368.557208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1368.603079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1373.144628] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1384.744044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1384.960183] Lustre: 46032:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1385.062191] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1385.070731] Lustre: Skipped 1 previous similar message [ 1385.148382] Lustre: lustre-MDT0001: new disk, initializing [ 1385.328643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1385.381702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1385.397762] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1391.143436] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1404.161665] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1412.125078] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1421.125200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1421.517932] Lustre: lustre-OST0000: new disk, initializing [ 1421.527960] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1421.536827] Lustre: 48002:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1421.641388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1427.015924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1427.025615] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1427.179315] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1427.760724] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1439.014852] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1444.651491] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1444.904594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1450.550241] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1450.838292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1458.151349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1458.152845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1458.157968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1458.170613] LustreError: Skipped 1 previous similar message [ 1458.203767] Lustre: Skipped 1 previous similar message [ 1459.570279] Lustre: server umount lustre-OST0000 complete [ 1467.362061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1467.368132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1467.395106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1467.401810] Lustre: Skipped 1 previous similar message [ 1468.386192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1468.408322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1468.417292] Lustre: Skipped 1 previous similar message [ 1470.721772] Lustre: server umount lustre-MDT0000 complete [ 1475.069062] LustreError: 46026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443515 with bad export cookie 16917084578549915181 [ 1475.078170] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1475.081099] LustreError: 46026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1475.500896] Lustre: server umount lustre-MDT0001 complete [ 1486.333313] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:52:04 (1788443524) [ 1493.808929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1494.313752] LustreError: 50460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1494.347350] LustreError: 50460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1494.446320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1499.526556] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1499.623310] LustreError: 50461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1504.746482] LustreError: 50460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1509.020505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1509.330919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1514.502134] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1522.330843] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1529.314925] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1537.424579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1537.814772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1543.830873] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1552.170799] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1560.280812] LustreError: 52992:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1561.330967] Lustre: Modifying parameter general.timeout=1 in log params [ 1563.288516] Lustre: Modifying parameter general.timeout=3 in log params [ 1563.290698] Lustre: Skipped 1 previous similar message [ 1565.406531] Lustre: Modifying parameter general.timeout=5 in log params [ 1565.410463] Lustre: Skipped 1 previous similar message [ 1569.886768] Lustre: Modifying parameter general.timeout=10 in log params [ 1569.893760] Lustre: Skipped 4 previous similar messages [ 1578.230784] Lustre: Modifying parameter general.timeout=19 in log params [ 1578.235634] Lustre: Skipped 8 previous similar messages [ 1594.306118] Lustre: Modifying parameter general.timeout=35 in log params [ 1594.312542] Lustre: Skipped 15 previous similar messages [ 1626.439897] Lustre: Modifying parameter general.timeout=71 in log params [ 1626.454913] Lustre: Skipped 35 previous similar messages [ 1692.925579] Lustre: Disabling parameter general.timeout= in log params [ 1692.931276] Lustre: Skipped 58 previous similar messages [ 1697.345944] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:55:35 (1788443735) [ 1701.371994] Lustre: 59943:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1703.545141] Lustre: 60053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1707.165941] Lustre: 60249:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1707.177277] Lustre: 60249:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1711.457366] Lustre: 60465:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1711.477232] Lustre: 60465:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1739.932151] Lustre: 61828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1739.955690] LustreError: 61828:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1744.561217] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:56:23 (1788443783) [ 1758.693469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1758.696663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1758.723447] Lustre: Skipped 1 previous similar message [ 1758.741470] Lustre: Skipped 2 previous similar messages [ 1763.816335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1763.834333] Lustre: Skipped 2 previous similar messages [ 1764.750124] Lustre: server umount lustre-MDT0000 complete [ 1768.542637] LustreError: 58298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443809 with bad export cookie 16917084578549916063 [ 1768.544257] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1768.553856] LustreError: 58298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1768.971799] Lustre: server umount lustre-MDT0001 complete [ 1783.959922] Lustre: server umount lustre-OST0000 complete [ 1816.359742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1816.678674] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1817.069183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1823.024799] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1833.656606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1833.776648] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1833.841087] Lustre: 64356:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1839.212941] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1849.350802] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1857.268465] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1866.133167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1866.287871] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1866.604354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1866.615144] Lustre: Skipped 1 previous similar message [ 1874.637148] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1884.919661] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1887.193418] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1887.213661] Lustre: Skipped 23 previous similar messages [ 1887.291653] Lustre: 66928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1899.489402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1899.515814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1899.552241] Lustre: Skipped 1 previous similar message [ 1899.558322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1901.882583] Lustre: server umount lustre-OST0000 complete [ 1910.241533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1910.258839] Lustre: Skipped 2 previous similar messages [ 1913.086574] Lustre: server umount lustre-MDT0000 complete [ 1915.361296] LustreError: 64370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1915.380038] LustreError: 64370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1917.845264] LustreError: 64349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443958 with bad export cookie 16917084578549919171 [ 1917.849357] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1917.871671] LustreError: 64349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1918.469967] Lustre: server umount lustre-MDT0001 complete [ 1931.796408] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1935.200401] Key type lgssc unregistered [ 1935.612731] LNet: 68141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1935.630946] LNetError: 68141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1935.645965] LNet: Removed LNI 192.168.204.102@tcp [ 1937.003820] Key type .llcrypt unregistered [ 1937.006805] Key type ._llcrypt unregistered [ 1966.562524] Key type ._llcrypt registered [ 1966.564454] Key type .llcrypt registered [ 1966.683079] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1967.643553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1967.652759] alg: No test for adler32 (adler32-zlib) [ 1968.705763] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1969.039923] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1970.809230] Key type lgssc registered [ 1972.172649] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1980.960501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1981.000650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1982.531135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1987.081731] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1992.676165] LustreError: 69556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1992.714606] LustreError: 69556:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1996.276842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1996.959508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2002.265858] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2010.880178] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2016.994187] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2024.304859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2024.920127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2032.463486] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2044.370510] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2053.121780] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2061.794258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2061.797558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2061.806660] Lustre: Skipped 2 previous similar messages [ 2061.820133] Lustre: Skipped 2 previous similar messages [ 2066.915991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2066.931364] Lustre: Skipped 2 previous similar messages [ 2067.642287] Lustre: server umount lustre-MDT0000 complete [ 2072.038425] LustreError: 69556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2072.063881] LustreError: 69556:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2073.031435] LustreError: 69539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444113 with bad export cookie 1342534357729903351 [ 2073.048469] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2073.451275] Lustre: server umount lustre-MDT0001 complete [ 2088.891695] Lustre: server umount lustre-OST0000 complete [ 2117.019290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.399173] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2117.461488] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2117.774403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2122.963831] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2131.850127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2132.005276] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2132.061937] Lustre: 74442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2132.601047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2136.502315] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2144.423080] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2150.961980] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2160.047847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2160.410812] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2160.935271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2167.390882] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2174.896519] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2176.576561] Lustre: 77018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2178.284852] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2186.726337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2186.739670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2186.761713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2187.237450] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2189.536940] Lustre: server umount lustre-OST0000 complete [ 2193.892853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2193.902369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2193.912540] Lustre: Skipped 1 previous similar message [ 2193.921359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2193.926870] Lustre: Skipped 2 previous similar messages [ 2197.489652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.952814] Lustre: server umount lustre-MDT0000 complete [ 2202.594398] LustreError: 74453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2202.614520] LustreError: 74453:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2204.428181] LustreError: 74433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444245 with bad export cookie 1342534357729904198 [ 2204.434949] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2204.439775] LustreError: 74433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2204.733137] Lustre: server umount lustre-MDT0001 complete [ 2218.560996] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 2221.867992] Key type lgssc unregistered [ 2222.192992] LNet: 78233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2222.208856] LNetError: 78233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2222.229204] LNet: Removed LNI 192.168.204.102@tcp [ 2223.193507] Key type .llcrypt unregistered [ 2223.197672] Key type ._llcrypt unregistered [ 2251.967939] Key type ._llcrypt registered [ 2251.969681] Key type .llcrypt registered [ 2252.027347] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2252.770976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2252.803267] alg: No test for adler32 (adler32-zlib) [ 2253.846991] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2254.096321] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 2255.808220] Key type lgssc registered [ 2256.800913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2266.538594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2266.567228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2268.006472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2273.232484] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2277.346731] LustreError: 79647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2277.372350] LustreError: 79647:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2282.468836] LustreError: 79648:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2284.708804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2285.409797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2290.137315] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2299.723298] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2307.146554] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2317.191771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2317.707188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2326.306206] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2338.424962] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2345.580280] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2346.316097] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2347.477425] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2350.121219] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2350.128932] Lustre: Skipped 2 previous similar messages [ 2354.539410] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2354.545505] Lustre: Skipped 3 previous similar messages [ 2363.505776] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2363.519734] Lustre: Skipped 8 previous similar messages [ 2379.768177] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2379.776753] Lustre: Skipped 15 previous similar messages [ 2412.196936] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2412.202478] Lustre: Skipped 31 previous similar messages [ 2458.823223] Lustre: 87759:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2461.806560] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 10:08:20 (1788444500) [ 2464.233418] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2466.808099] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 10:08:25 (1788444505) [ 2471.394454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.406835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2471.411261] Lustre: Skipped 1 previous similar message [ 2471.432697] Lustre: Skipped 1 previous similar message [ 2476.517902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2476.533258] Lustre: Skipped 1 previous similar message [ 2477.343723] Lustre: server umount lustre-OST0000 complete [ 2484.708475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2484.717106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2484.737040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2486.756467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2486.757468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2486.785532] Lustre: Skipped 1 previous similar message [ 2488.504871] Lustre: server umount lustre-MDT0000 complete [ 2491.882496] LustreError: 80752:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2491.929707] LustreError: 80752:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2493.749305] LustreError: 79630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444534 with bad export cookie 15666027280280783051 [ 2493.763497] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2493.764620] LustreError: 79630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2494.312737] Lustre: server umount lustre-MDT0001 complete [ 2508.301662] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 2511.521070] Key type lgssc unregistered [ 2511.829173] LNet: 89010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2511.843194] LNetError: 89010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2511.867668] LNet: Removed LNI 192.168.204.102@tcp [ 2512.874110] Key type .llcrypt unregistered [ 2512.876835] Key type ._llcrypt unregistered [ 2518.112476] Key type ._llcrypt registered [ 2518.114154] Key type .llcrypt registered [ 2518.244593] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2523.555945] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules [ 2524.967273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2524.991102] alg: No test for adler32 (adler32-zlib) [ 2526.226775] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2526.567980] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 2528.354951] Key type lgssc registered [ 2530.111555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2531.542948] LustreError: 90224:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2551.879710] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2553.358371] LustreError: 90224:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2562.083125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2562.099604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2563.659051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2569.005486] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2573.792722] LustreError: 91239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.815883] LustreError: 91239:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2578.912983] LustreError: 91238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.521574] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 10:10:18 (1788444618) [ 2584.039402] LustreError: 91239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.011780] LustreError: 91270:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2585.044648] LustreError: 91270:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2585.072540] LustreError: 91270:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2585.640418] Lustre: server umount lustre-MDT0000 complete [ 2602.574164] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 2605.698722] Key type lgssc unregistered [ 2606.009307] LNet: 92767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2606.014776] LNetError: 92767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2606.042178] LNet: Removed LNI 192.168.204.102@tcp [ 2607.084568] Key type .llcrypt unregistered [ 2607.090519] Key type ._llcrypt unregistered [ 2630.026285] Key type ._llcrypt registered [ 2630.029290] Key type .llcrypt registered [ 2630.250283] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2633.293562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2633.331218] alg: No test for adler32 (adler32-zlib) [ 2634.669538] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2635.194261] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 2637.152268] Key type lgssc registered [ 2638.782430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2649.167111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2649.190218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2651.164921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2656.295987] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2660.332241] LustreError: 93987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2660.355134] LustreError: 93987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2665.442126] LustreError: 93988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2666.857468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2667.628931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2673.032856] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2681.296515] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2687.541922] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2694.862283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2695.711823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2702.488520] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2710.852174] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2767.141700] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 10:13:25 (1788444805) [ 2768.750228] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2770.801901] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 10:13:29 (1788444809) [ 2779.107530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2779.116615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2779.117889] Lustre: Skipped 2 previous similar messages [ 2779.141181] Lustre: Skipped 2 previous similar messages [ 2781.810180] Lustre: server umount lustre-MDT0000 complete [ 2784.230066] LustreError: 93984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2784.266572] LustreError: 93984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2785.815816] LustreError: 93967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444826 with bad export cookie 15800578903490523687 [ 2785.846384] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2786.349124] Lustre: server umount lustre-MDT0001 complete [ 2801.888826] Lustre: server umount lustre-OST0000 complete [ 2817.341341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2817.920963] LustreError: 98284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2818.239626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2823.653816] LustreError: 98285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2823.675333] LustreError: 98285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2823.913961] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2832.874638] LustreError: 98284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2832.883655] LustreError: 98284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2834.163524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2834.927978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2840.668524] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2849.231383] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2855.605982] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2873.294084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2873.528087] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2873.542786] LustreError: 98274:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2873.580963] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2873.589752] LustreError: 100368:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2873.596210] LustreError: 100368:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2873.621954] LustreError: 100368:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2873.710441] Lustre: server umount lustre-OST0000 complete [ 2873.713716] LustreError: 100368:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2882.922763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2883.174652] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2883.181712] LustreError: 98273:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2883.209269] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2883.217521] LustreError: 100837:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2883.223055] LustreError: 100837:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2883.231747] LustreError: 100837:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2883.356675] Lustre: server umount lustre-OST0000 complete [ 2883.359833] LustreError: 100837:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2894.852828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2895.541990] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2895.552163] Lustre: Found index 0 for lustre-OST0000, updating log [ 2895.561423] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2895.692276] Lustre: lustre-OST0000: new disk, initializing [ 2895.703648] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2895.715943] Lustre: 101373:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2896.013263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2904.251339] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2909.375859] Lustre: Failing over lustre-OST0000 [ 2909.685746] Lustre: server umount lustre-OST0000 complete [ 2916.335029] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2916.340764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2916.356400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2917.858097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2917.860042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2921.068035] Lustre: server umount lustre-MDT0000 complete [ 2922.980136] LustreError: 99233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2923.007724] LustreError: 99233:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2926.116274] Lustre: 93297:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444950/real 1788444950] req@00000000bdd7e288 x1875320163639424/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788444966 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2926.188413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2926.270710] LustreError: 100389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444967 with bad export cookie 15800578903490526431 [ 2926.271724] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2926.290725] LustreError: 100389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2926.756626] Lustre: server umount lustre-MDT0001 complete [ 2938.428735] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 10:16:16 (1788444976) [ 2958.591653] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2969.367956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2969.972324] LustreError: 103720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2969.991817] LustreError: 103720:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2970.111658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2975.178881] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2986.648659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2993.685668] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2997.347117] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3005.014230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3005.730416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3005.739812] Lustre: Skipped 1 previous similar message [ 3005.751089] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3011.875811] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3011.966616] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3011.966760] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3012.298536] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3023.368974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3023.525397] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3023.659673] Lustre: lustre-OST0001: new disk, initializing [ 3023.669153] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3023.686447] Lustre: 106039:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3031.149620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3031.155466] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3031.327202] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3032.335855] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3041.810867] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3052.926179] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3059.681328] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3059.687766] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3059.700632] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3059.707770] Lustre: Skipped 1 previous similar message [ 3064.248025] Lustre: server umount lustre-MDT0001 complete [ 3066.549667] LustreError: 106050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3066.573836] LustreError: 106050:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3076.033251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3076.165725] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3076.173276] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3076.254283] Lustre: 103708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3077.258317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3077.270745] Lustre: Skipped 1 previous similar message [ 3078.274403] LustreError: 107589:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3078.288347] LustreError: 107589:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3078.301489] LustreError: 103712:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3078.514674] LustreError: 107604:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.102@tcp: cfg command failed: rc = -17 [ 3078.540653] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3078.540653] [ 3082.753067] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3083.101284] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3095.256102] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 10:18:53 (1788445133) [ 3116.252439] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 3137.080389] Lustre: 109655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3161.580159] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3166.013489] Lustre: 110791:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3266.019970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3266.037605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3266.062850] Lustre: Skipped 2 previous similar messages [ 3266.075775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3266.087550] Lustre: Skipped 4 previous similar messages [ 3269.771199] Lustre: server umount lustre-MDT0000 complete [ 3272.174201] LustreError: 110586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3272.201791] LustreError: 110586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 3273.696396] LustreError: 103700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445314 with bad export cookie 15800578903490527425 [ 3273.698902] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3273.718856] LustreError: 103700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3274.136363] Lustre: server umount lustre-MDT0001 complete [ 3288.688880] Lustre: server umount lustre-OST0000 complete [ 3304.195795] Lustre: server umount lustre-OST0001 complete [ 3311.027213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3320.569331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3331.107980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3343.083988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3369.906794] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 3382.175854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3382.214935] Lustre: lustre-MDT0000: reset Object Index mappings [ 3382.991422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3387.604353] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3397.145744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3397.161062] Lustre: lustre-MDT0001: reset Object Index mappings [ 3403.221742] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3406.751756] Lustre: 115897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3415.298807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3422.953333] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 3424.497552] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3428.100866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3437.053295] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3442.685862] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3442.695181] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3444.800495] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3453.854270] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3457.970676] Lustre: 117773:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3517.612407] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 10:25:56 (1788445556) [ 3545.059800] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3545.073293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3545.096581] Lustre: Skipped 6 previous similar messages [ 3545.110727] Lustre: Skipped 7 previous similar messages [ 3557.344143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3557.587596] Lustre: server umount lustre-MDT0000 complete [ 3560.425263] LustreError: 117930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3560.445832] LustreError: 117930:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 3561.709475] LustreError: 114732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445602 with bad export cookie 15800578903490574318 [ 3561.713440] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3561.726892] LustreError: 114732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3565.549855] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3565.558757] Lustre: Skipped 9 previous similar messages [ 3576.291476] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3576.682762] Lustre: server umount lustre-MDT0001 complete [ 3581.308960] LustreError: 118541:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000400:0xb8:0x0].0x0 (ffff900c83387700) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3587.707646] Lustre: server umount lustre-OST0000 complete [ 3599.112529] Lustre: server umount lustre-OST0001 complete [ 3606.971511] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 3619.395571] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 3664.240387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3664.608698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3664.675154] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3664.809516] Lustre: lustre-MDT0000: new disk, initializing [ 3664.882455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3664.888034] Lustre: Skipped 3 previous similar messages [ 3664.906599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3670.191697] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3677.695711] Lustre: server umount lustre-MDT0000 complete [ 3688.570991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3688.914333] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3688.924719] Lustre: 122054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3688.934858] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3694.322399] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3700.587704] Lustre: server umount lustre-MDT0000 complete [ 3710.949067] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:29:09 (1788445749) [ 3712.920311] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3715.360479] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:29:14 (1788445754) [ 3736.532700] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 3748.168887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3753.915643] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3763.002913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3763.117563] Lustre: 124102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3763.166914] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3763.170925] Lustre: Skipped 1 previous similar message [ 3763.247273] Lustre: lustre-MDT0001: new disk, initializing [ 3763.407951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3763.425362] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3768.631203] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3774.102967] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3782.822740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3783.270677] Lustre: lustre-OST0000: new disk, initializing [ 3783.276892] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3783.285300] Lustre: 125680:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3784.889462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3784.903074] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3785.022620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3791.500150] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3804.630474] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3804.888211] Lustre: lustre-OST0001: new disk, initializing [ 3804.893143] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3804.902995] Lustre: 126552:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3806.734505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3806.758478] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3806.928134] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3813.279390] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3823.966411] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3828.278521] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4266.626338] Lustre: 132977:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4266.642882] Lustre: 132977:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4266.652390] Lustre: 132977:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4266.664221] Lustre: 132977:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4266.671248] Lustre: 132977:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4266.678966] Lustre: 132977:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4550.591748] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:43:08 (1788446588) [ 4564.962961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4564.964564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4564.983854] Lustre: Skipped 4 previous similar messages [ 4564.997984] Lustre: Skipped 7 previous similar messages [ 4570.084153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4570.091701] Lustre: Skipped 3 previous similar messages [ 4579.296210] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4579.553244] Lustre: server umount lustre-MDT0000 complete [ 4580.322628] LustreError: 124115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4580.352072] LustreError: 124115:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 4582.895613] LustreError: 133008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446623 with bad export cookie 15800578903490644577 [ 4582.902093] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4582.912281] LustreError: 133008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4585.443438] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4585.449657] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4585.460395] Lustre: Skipped 2 previous similar messages [ 4585.472083] Lustre: Skipped 5 previous similar messages [ 4589.315478] Lustre: server umount lustre-MDT0001 complete [ 4598.913194] Lustre: server umount lustre-OST0000 complete [ 4608.858357] Lustre: server umount lustre-OST0001 complete [ 4615.768771] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 4625.075978] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 4665.518505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4665.856211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4665.895658] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4666.045270] Lustre: lustre-MDT0000: new disk, initializing [ 4666.184473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4666.189649] Lustre: Skipped 5 previous similar messages [ 4666.204337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4671.240290] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4682.393585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4682.507647] Lustre: 136042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4682.560552] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4682.565609] Lustre: Skipped 1 previous similar message [ 4682.673830] Lustre: lustre-MDT0001: new disk, initializing [ 4682.822563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4682.832485] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4687.710984] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4697.377337] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4702.885972] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4710.779489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4711.106070] Lustre: lustre-OST0000: new disk, initializing [ 4711.111759] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4711.171132] Lustre: 138012:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4712.910507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4712.924565] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4713.038213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4718.013829] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 4726.735345] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4736.246992] Lustre: lustre-MDD0000: changelog on [ 4739.364087] Lustre: lustre-MDD0001: changelog on [ 6745.698752] Lustre: lustre-MDD0001: changelog off [ 6747.656956] Lustre: 136047:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6748.451318] Lustre: lustre-MDD0000: changelog off [ 6755.809341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6755.816084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6755.823187] LustreError: Skipped 1 previous similar message [ 6755.830417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6755.842087] Lustre: Skipped 1 previous similar message [ 6755.867916] Lustre: Skipped 3 previous similar messages [ 6760.942651] LustreError: 139632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6760.955784] LustreError: 139632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6761.028464] Lustre: server umount lustre-OST0000 complete [ 6766.057829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6766.062599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6766.073271] Lustre: Skipped 1 previous similar message [ 6766.087824] Lustre: Skipped 1 previous similar message [ 6771.172043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6771.192429] Lustre: Skipped 1 previous similar message [ 6773.796925] Lustre: server umount lustre-MDT0000 complete [ 6778.126247] LustreError: 138011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448818 with bad export cookie 15800578903491430103 [ 6778.136600] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6778.144987] LustreError: 138011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6778.555369] Lustre: server umount lustre-MDT0001 complete [ 6789.278795] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 6791.909486] Key type lgssc unregistered [ 6792.215217] LNet: 147180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6792.226345] LNetError: 147180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6792.242704] LNet: Removed LNI 192.168.204.102@tcp [ 6792.986333] Key type .llcrypt unregistered [ 6792.990802] Key type ._llcrypt unregistered [ 6813.085891] Key type ._llcrypt registered [ 6813.090989] Key type .llcrypt registered [ 6813.197706] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 6830.771358] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 6832.213825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6832.240201] alg: No test for adler32 (adler32-zlib) [ 6833.349629] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6833.760881] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 6835.528195] Key type lgssc registered [ 6836.995825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6882.950225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6883.018042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6884.378909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6884.404491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6884.473495] Lustre: lustre-MDT0000: new disk, initializing [ 6884.612155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6884.630535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6888.952808] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6898.467131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6898.585417] Lustre: 150382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6898.630526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6898.634036] Lustre: Skipped 1 previous similar message [ 6898.734436] Lustre: lustre-MDT0001: new disk, initializing [ 6898.861990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6898.944827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6898.953321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6904.191865] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6916.402277] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6922.324443] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6929.535942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6929.886555] Lustre: lustre-OST0000: new disk, initializing [ 6929.899862] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6929.912159] Lustre: 152350:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6930.050233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6936.216842] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 6940.294321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6940.303805] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6940.447678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6946.406207] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6952.002827] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_module obdecho/obdecho [ 6954.009852] LustreError: 153357:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6954.016641] LustreError: 153357:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6956.102756] LustreError: 153454:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6970.850201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6970.860502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6970.882713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6971.379667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6971.400213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6971.404649] Lustre: Skipped 1 previous similar message [ 6971.420516] Lustre: Skipped 2 previous similar messages [ 6974.029508] Lustre: server umount lustre-MDT0000 complete [ 6976.487210] LustreError: 150387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6976.510789] LustreError: 150387:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6977.792914] LustreError: 150374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449018 with bad export cookie 14837904334241605269 [ 6977.797662] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6977.813892] LustreError: 150374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6978.165799] Lustre: server umount lustre-MDT0001 complete [ 6992.314097] Lustre: server umount lustre-OST0000 complete [ 7016.366929] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 7027.754373] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 7068.749881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7069.095776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7069.131651] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7069.246458] Lustre: lustre-MDT0000: new disk, initializing [ 7069.416295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7069.450100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7074.334928] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7085.609780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7085.734405] Lustre: 157173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7085.784711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7085.787987] Lustre: Skipped 1 previous similar message [ 7085.899762] Lustre: lustre-MDT0001: new disk, initializing [ 7086.022834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7086.078205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7086.092815] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7090.176976] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7099.722667] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7106.523992] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7113.854025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7114.061638] Lustre: lustre-OST0000: new disk, initializing [ 7114.065937] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7114.073617] Lustre: 159141:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7114.165141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7119.490021] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7119.502217] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7119.641402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7121.339170] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7130.996844] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7135.924361] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7136.197755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7141.426744] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7141.722704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7145.442089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7145.444751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7145.455418] LustreError: Skipped 1 previous similar message [ 7145.462857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7145.484856] Lustre: Skipped 1 previous similar message [ 7145.504648] Lustre: Skipped 1 previous similar message [ 7149.971252] Lustre: server umount lustre-OST0000 complete [ 7155.681453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7155.699710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7155.703405] Lustre: Skipped 1 previous similar message [ 7160.318379] Lustre: server umount lustre-MDT0000 complete [ 7160.806365] LustreError: 158433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7160.834529] LustreError: 158433:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7164.749522] LustreError: 157166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449205 with bad export cookie 14837904334241606228 [ 7164.754153] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7164.770925] LustreError: 157166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7165.186811] Lustre: server umount lustre-MDT0001 complete [ 7169.726202] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 11:26:48 (1788449208) [ 7171.794614] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7173.740776] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 11:26:52 (1788449212) [ 7182.075241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7182.716742] LustreError: 161554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7182.798491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7187.331565] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7187.948598] LustreError: 161555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7193.061644] LustreError: 161554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7196.046414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7200.929514] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7207.987425] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7213.064836] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7220.064738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7220.640084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7220.645745] Lustre: Skipped 1 previous similar message [ 7227.266792] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7236.241692] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7243.234588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7243.236599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7243.258604] Lustre: Skipped 3 previous similar messages [ 7243.265166] Lustre: Skipped 1 previous similar message [ 7248.112508] Lustre: server umount lustre-MDT0000 complete [ 7248.357968] LustreError: 161554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7248.381238] LustreError: 161554:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7252.151442] LustreError: 161535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449292 with bad export cookie 14837904334241607110 [ 7252.157696] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7252.179041] LustreError: 161535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7253.480805] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7253.494426] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7253.506292] Lustre: Skipped 2 previous similar messages [ 7258.677806] Lustre: server umount lustre-MDT0001 complete [ 7266.770814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7280.529134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7280.788658] LustreError: 165399:0:(llog.c:1655:llog_backup()) MGC192.168.204.102@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7280.800926] Lustre: 165399:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.102@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7282.148189] LustreError: 165404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7282.184399] LustreError: 165404:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7316.974063] LustreError: 165405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7316.992754] LustreError: 165405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 25 previous similar messages [ 7342.052536] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7342.072932] Lustre: 165399:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.102@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7382.506456] LustreError: 165404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7382.528470] LustreError: 165404:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 42 previous similar messages [ 7383.024734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7387.152123] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7388.148635] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7395.533875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7397.358720] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcdead3609db346fc [ 7397.373111] Lustre: MGC192.168.204.102@tcp: Connection restored to 0@lo (at 0@lo) [ 7401.775412] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7402.994674] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7409.603241] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7411.045967] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7415.955209] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7417.386461] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7438.161204] Lustre: server umount lustre-MDT0000 complete [ 7438.818642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7438.831170] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7438.844055] Lustre: Skipped 2 previous similar messages [ 7441.878842] LustreError: 161534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449482 with bad export cookie 14837904334241613564 [ 7441.883082] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7441.889286] LustreError: 161534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7442.247145] Lustre: server umount lustre-MDT0001 complete [ 7456.611585] Lustre: server umount lustre-OST0000 complete [ 7481.162944] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 7492.691915] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 7530.264854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7530.535339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7530.560195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7530.682958] Lustre: lustre-MDT0000: new disk, initializing [ 7530.798895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7530.809380] Lustre: Skipped 1 previous similar message [ 7530.838156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7534.910857] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7546.006801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7546.124611] Lustre: 170502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7546.390809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7550.704548] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7559.879848] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7564.914781] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7571.600918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7571.898549] Lustre: lustre-OST0000: new disk, initializing [ 7571.902385] Lustre: Skipped 1 previous similar message [ 7571.910756] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7571.917488] Lustre: Skipped 2 previous similar messages [ 7571.925475] Lustre: 172474:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7577.945365] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7580.263996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7580.273323] Lustre: Skipped 1 previous similar message [ 7580.279306] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7580.365125] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7587.188631] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7592.707724] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7592.984056] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7597.651619] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7597.837983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7601.125636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7601.126554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7601.127703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7601.127712] Lustre: Skipped 2 previous similar messages [ 7601.140363] Lustre: Skipped 1 previous similar message [ 7606.088980] Lustre: server umount lustre-OST0000 complete [ 7616.787540] Lustre: server umount lustre-MDT0000 complete [ 7621.568247] LustreError: 171298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449662 with bad export cookie 14837904334241630399 [ 7621.572893] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7621.593782] LustreError: 171298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7622.302883] Lustre: server umount lustre-MDT0001 complete [ 7634.443120] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:34:33 (1788449673) [ 7646.616687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7647.490142] LustreError: 174888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7647.535318] LustreError: 174888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 7654.043763] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7664.560540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7669.701983] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7676.990833] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7683.314965] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7689.503579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7696.091032] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7702.167409] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7710.947489] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:35:50 (1788449750) [ 7718.885763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7718.890439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7718.892689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7718.892696] Lustre: Skipped 5 previous similar messages [ 7718.918362] Lustre: Skipped 3 previous similar messages [ 7720.335350] Lustre: server umount lustre-OST0000 complete [ 7731.034238] Lustre: server umount lustre-MDT0000 complete [ 7735.434658] LustreError: 174868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449776 with bad export cookie 14837904334241631281 [ 7735.452575] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7735.695916] Lustre: server umount lustre-MDT0001 complete [ 7747.215595] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 7750.311274] Key type lgssc unregistered [ 7750.665791] LNet: 178618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7750.677339] LNetError: 178618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7750.710032] LNet: Removed LNI 192.168.204.102@tcp [ 7751.838794] Key type .llcrypt unregistered [ 7751.843055] Key type ._llcrypt unregistered [ 7776.526312] Key type ._llcrypt registered [ 7776.530937] Key type .llcrypt registered [ 7776.631389] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 7777.818732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7777.840164] alg: No test for adler32 (adler32-zlib) [ 7778.998466] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7779.251543] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 7780.984203] Key type lgssc registered [ 7782.703743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7791.769725] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7791.786948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7807.392422] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7812.512595] LustreError: 179864:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.102@tcp: failed processing log, type 4: rc = -110 [ 7817.698306] LustreError: 179864:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7817.715704] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7817.723092] LustreError: MGC192.168.204.102@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7817.746845] LustreError: 179864:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7817.757228] LustreError: 179864:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7817.762938] LustreError: 179864:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7817.830383] Lustre: server umount lustre-OST0000 complete [ 7817.841905] LustreError: 179864:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7825.549943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7826.896454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7830.781764] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7837.155082] LustreError: 180363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7837.172908] LustreError: 180363:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7838.196239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7838.609318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7843.133488] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7849.965312] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7855.590049] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7861.791298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7862.001860] LustreError: 182180:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7862.020989] Lustre: 182180:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.102@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7862.222627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7868.143460] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7874.781349] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7879.654404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7879.663565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7879.674975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7881.697729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7881.699058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7881.715736] Lustre: Skipped 1 previous similar message [ 7881.723600] Lustre: Skipped 2 previous similar messages [ 7883.276796] Lustre: server umount lustre-MDT0000 complete [ 7886.822450] LustreError: 180364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7886.847698] LustreError: 180364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7886.879497] LustreError: 180344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449927 with bad export cookie 6054029983400785135 [ 7886.888690] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7886.898824] LustreError: 180344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7887.306108] Lustre: server umount lustre-MDT0001 complete [ 7901.917217] Lustre: server umount lustre-OST0000 complete [ 7910.903526] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 7920.382105] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 7953.525771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7953.731689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7953.755884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7953.843872] Lustre: lustre-MDT0000: new disk, initializing [ 7953.920158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7953.937213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7957.994251] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7967.544749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7967.713931] Lustre: 185702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7967.817292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7967.829070] Lustre: Skipped 1 previous similar message [ 7967.927717] Lustre: lustre-MDT0001: new disk, initializing [ 7968.069965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7968.139996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7968.151199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7972.898412] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 7984.339771] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7990.522928] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7997.403635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7997.790404] Lustre: lustre-OST0000: new disk, initializing [ 7997.797920] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7997.810429] Lustre: 187668:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7997.961169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8003.662533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8003.684996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8003.798433] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8004.215324] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8014.975032] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8020.211583] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8020.376909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8025.018426] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8025.336462] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8029.666280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8029.681062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8029.687863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8029.705772] Lustre: Skipped 1 previous similar message [ 8033.403756] Lustre: server umount lustre-OST0000 complete [ 8039.906264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8039.924061] Lustre: Skipped 1 previous similar message [ 8039.944146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8039.954227] Lustre: Skipped 1 previous similar message [ 8045.026982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8045.035223] Lustre: Skipped 1 previous similar message [ 8051.168153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8051.360859] Lustre: server umount lustre-MDT0000 complete [ 8054.828311] LustreError: 185695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450095 with bad export cookie 6054029983400785856 [ 8054.845335] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8055.179134] Lustre: server umount lustre-MDT0001 complete [ 8064.189421] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:41:43 (1788450103) [ 8082.760166] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 8085.887509] Key type lgssc unregistered [ 8086.192850] LNet: 190621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8086.200419] LNetError: 190621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8086.224931] LNet: Removed LNI 192.168.204.102@tcp [ 8087.084489] Key type .llcrypt unregistered [ 8087.086884] Key type ._llcrypt unregistered [ 8108.907797] Key type ._llcrypt registered [ 8108.910766] Key type .llcrypt registered [ 8109.031593] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 8110.180288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8110.305173] alg: No test for adler32 (adler32-zlib) [ 8111.485374] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8111.701939] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 8113.385815] Key type lgssc registered [ 8114.833726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8123.424804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8123.451662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8124.997186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8129.910594] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8135.142043] LustreError: 191839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8135.172267] LustreError: 191839:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8139.488032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8140.046575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8144.903736] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8152.710328] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8158.479670] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8167.564835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8167.739908] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8167.742515] LustreError: 193705:0:(llog.c:1696:llog_backup()) MGC192.168.204.102@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8167.748118] Lustre: 193705:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.102@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8167.845238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8174.343603] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8182.287558] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8186.338154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8186.352781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8186.381757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8189.414775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8189.416355] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8194.535134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8194.557302] Lustre: Skipped 4 previous similar messages [ 8199.136369] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8199.355449] Lustre: server umount lustre-MDT0000 complete [ 8199.650884] LustreError: 191836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8199.686657] LustreError: 191836:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8203.507771] LustreError: 191820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450244 with bad export cookie 8388474815848180685 [ 8203.524514] LustreError: 191820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8203.528821] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8203.853088] Lustre: server umount lustre-MDT0001 complete [ 8218.459200] Lustre: server umount lustre-OST0000 complete [ 8229.374501] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 8242.008536] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 8283.562590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8283.754936] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8283.789185] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8283.904324] Lustre: lustre-MDT0000: new disk, initializing [ 8283.988385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8284.002425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8288.688385] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8298.656378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8298.772455] Lustre: 197231:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8298.807511] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8298.811247] Lustre: Skipped 1 previous similar message [ 8298.814519] LustreError: 198014:0:(llog.c:1696:llog_backup()) MGC192.168.204.102@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8298.823247] LustreError: 198014:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8298.829113] Lustre: 198014:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.102@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8298.854356] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8298.861051] Lustre: Skipped 1 previous similar message [ 8298.961609] Lustre: lustre-MDT0001: new disk, initializing [ 8299.061313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8299.120078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8299.129920] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8303.456845] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8313.114934] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8319.108730] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8327.714445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8328.100477] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8328.109104] Lustre: Skipped 1 previous similar message [ 8328.111199] LustreError: 199179:0:(llog.c:1696:llog_backup()) MGC192.168.204.102@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8328.121631] LustreError: 199179:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8328.128863] Lustre: 199179:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.102@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8328.137923] Lustre: 199179:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8328.174907] Lustre: lustre-OST0000: new disk, initializing [ 8328.186427] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8328.193373] Lustre: 199202:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8328.358658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8334.931188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8334.944761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8335.072282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8335.588644] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8345.764232] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8351.691905] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8351.928624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8357.645870] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8357.865433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8360.931646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8360.934350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8360.963665] Lustre: Skipped 1 previous similar message [ 8360.972046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8360.978214] Lustre: Skipped 1 previous similar message [ 8366.051212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8366.070982] Lustre: Skipped 1 previous similar message [ 8374.240190] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8374.387242] Lustre: server umount lustre-OST0000 complete [ 8380.912451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8380.919970] LustreError: Skipped 1 previous similar message [ 8380.924403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8380.945694] Lustre: Skipped 1 previous similar message [ 8380.952397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8380.959561] Lustre: Skipped 2 previous similar messages [ 8384.273530] Lustre: server umount lustre-MDT0000 complete [ 8386.530728] LustreError: 197244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8386.550935] LustreError: 197244:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8387.979853] LustreError: 198038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450428 with bad export cookie 8388474815848181406 [ 8387.985584] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8387.990021] LustreError: 198038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8388.268510] Lustre: server umount lustre-MDT0001 complete [ 8397.677613] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:47:16 (1788450436) [ 8414.616218] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 8424.495524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8424.882778] LustreError: 202329:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8424.917942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8429.269928] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8430.062056] LustreError: 202330:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8438.443691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8445.078108] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8449.145287] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8459.055766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8459.337244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8459.343922] Lustre: Skipped 1 previous similar message [ 8466.165607] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8474.656828] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8474.795465] Lustre: lustre-OST0001: new disk, initializing [ 8474.799520] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8474.806135] Lustre: 204639:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8480.906340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8480.914898] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8481.040349] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8481.393049] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8492.138443] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8496.291438] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8498.154838] Lustre: DEBUG MARKER: ADD OST3 [ 8503.171125] loop: module loaded [ 8503.240548] loop0: detected capacity change from 0 to 204800000 [ 8503.429987] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8503.448777] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8503.463034] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8509.485247] loop0: detected capacity change from 0 to 204800000 [ 8510.314315] Lustre: DEBUG MARKER: STOP OST3 [ 8515.726765] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8516.005569] Lustre: lustre-OST0003: new disk, initializing [ 8516.010530] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8516.016129] Lustre: 206660:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8518.162411] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8518.366441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8518.394884] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8518.460378] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8519.802330] Lustre: Failing over lustre-OST0003 [ 8520.050771] Lustre: server umount lustre-OST0003 complete [ 8520.224087] LustreError: 206834:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8522.458945] LustreError: 204350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8522.471620] LustreError: 204350:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8522.897032] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8523.745040] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8523.758706] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8523.775547] Lustre: Skipped 2 previous similar messages [ 8531.868550] Lustre: DEBUG MARKER: START OST3 again [ 8539.110470] LustreError: 203816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8539.122348] LustreError: 203816:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 8539.353355] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8539.732794] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8539.745905] Lustre: Skipped 2 previous similar messages [ 8539.768704] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8541.455605] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8541.771968] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8541.772184] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8546.411157] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8569.028333] Lustre: Failing over lustre-OST0003 [ 8569.166508] Lustre: server umount lustre-OST0003 complete [ 8570.352222] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8570.371971] Lustre: Skipped 1 previous similar message [ 8575.478659] LustreError: 203816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8575.499664] LustreError: 203816:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8590.825518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8590.834680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8590.843924] Lustre: Skipped 3 previous similar messages [ 8590.859087] Lustre: Skipped 5 previous similar messages [ 8594.714294] Lustre: server umount lustre-MDT0000 complete [ 8598.254666] LustreError: 202312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450639 with bad export cookie 8388474815848182288 [ 8598.256512] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8598.266306] LustreError: 202312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8604.766855] Lustre: server umount lustre-MDT0001 complete [ 8608.937878] Lustre: server umount lustre-OST0000 complete [ 8612.706542] Lustre: server umount lustre-OST0001 complete [ 8620.270688] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 8630.407631] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 8673.530582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8673.868538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8673.941071] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8674.058873] Lustre: lustre-MDT0000: new disk, initializing [ 8674.150647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8674.206683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8679.891438] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8691.589836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8691.748226] Lustre: 212103:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8691.837251] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8691.842163] Lustre: Skipped 1 previous similar message [ 8691.924363] Lustre: lustre-MDT0001: new disk, initializing [ 8692.050782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8692.065921] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8697.035284] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8706.807994] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8712.941347] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8719.730887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8719.948627] Lustre: 214072:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8721.464073] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8721.595711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8726.214581] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8736.150375] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8740.951860] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8741.222486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8745.648254] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8745.873323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8752.610061] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8752.626556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8752.641150] Lustre: Skipped 2 previous similar messages [ 8752.647344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8752.662596] Lustre: Skipped 5 previous similar messages [ 8753.894989] Lustre: server umount lustre-OST0000 complete [ 8772.064192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8772.272869] Lustre: server umount lustre-MDT0000 complete [ 8773.091872] LustreError: 212112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8773.104185] LustreError: 212112:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 8775.408924] LustreError: 212096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450816 with bad export cookie 8388474815848184584 [ 8775.414883] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8775.422077] LustreError: 212096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8775.713279] Lustre: server umount lustre-MDT0001 complete [ 8778.353214] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:53:37 (1788450817) [ 8791.537868] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 8801.344729] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 8839.337630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8839.535940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8839.563985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8839.569165] Lustre: Skipped 1 previous similar message [ 8839.666239] Lustre: lustre-MDT0000: new disk, initializing [ 8839.669510] Lustre: Skipped 1 previous similar message [ 8839.789634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8839.802973] Lustre: Skipped 1 previous similar message [ 8843.653867] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8854.711068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8854.851364] Lustre: 218684:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8855.144796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8860.231898] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8870.023883] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8876.487468] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8883.899250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8884.219607] Lustre: 220653:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8886.210111] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8890.855315] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8899.844508] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8904.952467] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8905.224341] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8909.478873] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8909.769141] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8911.843622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8911.844954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8911.853970] LustreError: Skipped 2 previous similar messages [ 8911.868975] Lustre: Skipped 3 previous similar messages [ 8911.870558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8911.884391] Lustre: Skipped 7 previous similar messages [ 8917.648279] Lustre: server umount lustre-OST0000 complete [ 8927.594726] Lustre: server umount lustre-MDT0000 complete [ 8931.404615] LustreError: 218677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450972 with bad export cookie 8388474815848185466 [ 8931.413568] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8931.423902] LustreError: 218677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8931.793823] Lustre: server umount lustre-MDT0001 complete [ 8943.471586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8944.250150] LustreError: 222873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8944.381209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8944.389100] Lustre: Skipped 5 previous similar messages [ 8949.807908] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8960.228265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8967.347741] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 8976.943733] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8984.807279] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8993.821986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9001.133149] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9010.340724] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9027.046581] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9027.054970] LustreError: Skipped 1 previous similar message [ 9032.505809] Lustre: server umount lustre-OST0000 complete [ 9042.689682] Lustre: server umount lustre-MDT0000 complete [ 9046.920822] LustreError: 222855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451087 with bad export cookie 8388474815848186348 [ 9046.935167] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9046.937206] LustreError: 222855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9047.323291] Lustre: server umount lustre-MDT0001 complete [ 9150.511678] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 9153.481551] Key type lgssc unregistered [ 9153.779664] LNet: 226561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9153.791973] LNetError: 226561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9153.825102] LNet: Removed LNI 192.168.204.102@tcp [ 9155.028223] Key type .llcrypt unregistered [ 9155.031969] Key type ._llcrypt unregistered [ 9187.170322] Key type ._llcrypt registered [ 9187.173602] Key type .llcrypt registered [ 9187.280805] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 9203.342482] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 9204.651271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9204.700265] alg: No test for adler32 (adler32-zlib) [ 9205.802125] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9206.091120] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 9207.768170] Key type lgssc registered [ 9209.049630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9244.520201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9244.564064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9245.881834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9245.920121] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9246.031135] Lustre: lustre-MDT0000: new disk, initializing [ 9246.110386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9246.133929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9250.530723] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9262.142805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9262.284830] Lustre: 229928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9262.328275] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9262.334150] Lustre: Skipped 1 previous similar message [ 9262.407332] Lustre: lustre-MDT0001: new disk, initializing [ 9262.509864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9262.555527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9262.563517] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9267.442690] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9277.534363] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9283.900704] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9292.536866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9292.874154] Lustre: lustre-OST0000: new disk, initializing [ 9292.878701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9292.885650] Lustre: 231894:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9293.027478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9301.133568] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9302.680134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9302.692333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9302.906611] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9313.301685] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9318.972846] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9319.140747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9324.635523] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9324.890925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9328.611927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9328.614450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9328.621541] LustreError: Skipped 1 previous similar message [ 9328.629735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9328.634306] Lustre: Skipped 1 previous similar message [ 9328.661361] Lustre: Skipped 1 previous similar message [ 9333.426780] Lustre: server umount lustre-OST0000 complete [ 9338.850358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9338.859965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9338.860826] Lustre: Skipped 1 previous similar message [ 9338.875826] Lustre: Skipped 1 previous similar message [ 9343.549647] Lustre: server umount lustre-MDT0000 complete [ 9347.505406] LustreError: 230721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451388 with bad export cookie 10832514832192996927 [ 9347.509404] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9347.513849] LustreError: 230721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9347.849669] Lustre: server umount lustre-MDT0001 complete [ 9367.654071] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 9381.179230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9381.984087] LustreError: 234831:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9382.010606] LustreError: 234831:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9382.108408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9387.499217] LustreError: 234832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9387.539635] LustreError: 234832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9388.587953] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9391.594207] LustreError: 234831:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9396.712939] LustreError: 234832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9398.761126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9399.215338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9404.803794] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9407.535317] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9414.926204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9415.482742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9422.456565] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9432.603568] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9432.743862] Lustre: lustre-OST0001: new disk, initializing [ 9432.752343] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9432.762908] Lustre: 237140:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9432.942551] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9439.398450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9439.411607] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9439.557491] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9440.157947] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9451.907331] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9460.222202] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9474.216485] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 12:05:13 (1788451513) [ 9480.675960] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9480.678418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9480.712164] Lustre: Skipped 2 previous similar messages [ 9480.724903] Lustre: Skipped 1 previous similar message [ 9482.706317] Lustre: server umount lustre-MDT0000 complete [ 9485.802649] LustreError: 235897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9485.834141] LustreError: 235897:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9487.057810] LustreError: 234811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451527 with bad export cookie 10832514832192997809 [ 9487.066744] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9487.073327] LustreError: 234811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9487.533993] Lustre: server umount lustre-MDT0001 complete [ 9502.227879] Lustre: server umount lustre-OST0000 complete [ 9506.277432] Lustre: 228139:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788451531/real 1788451531] req@ffff900c8f1c1180 x1875327053842304/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788451547 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9506.298316] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9506.307900] Lustre: Skipped 1 previous similar message [ 9506.785350] Lustre: server umount lustre-OST0001 complete [ 9514.795799] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 9530.949988] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 9575.881679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9576.078087] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9576.118044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9576.187359] Lustre: lustre-MDT0000: new disk, initializing [ 9576.240402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9576.256356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9581.440450] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9593.269731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9593.396310] Lustre: 241109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9593.447591] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9593.451360] Lustre: Skipped 1 previous similar message [ 9593.513136] Lustre: lustre-MDT0001: new disk, initializing [ 9593.628122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9593.649617] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9598.896261] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9609.266586] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9615.335607] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9625.017151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9625.333642] Lustre: lustre-OST0000: new disk, initializing [ 9625.341937] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9625.348952] Lustre: 243078:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9627.295244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9627.312421] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9627.460605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9633.163554] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9644.240749] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9649.887529] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9650.079713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9656.076116] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9656.340529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9663.464036] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9663.474403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9663.484731] LustreError: Skipped 1 previous similar message [ 9663.488856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9663.516977] Lustre: Skipped 1 previous similar message [ 9665.002477] Lustre: server umount lustre-OST0000 complete [ 9670.627545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9670.651232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9673.700948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9673.726808] Lustre: Skipped 2 previous similar messages [ 9675.347535] Lustre: server umount lustre-MDT0000 complete [ 9678.828218] LustreError: 241120:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9678.862341] LustreError: 241120:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9679.833396] LustreError: 243077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451720 with bad export cookie 10832514832192999937 [ 9679.851064] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9679.854415] LustreError: 243077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9680.493315] Lustre: server umount lustre-MDT0001 complete [ 9689.522626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9690.104657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9690.108290] Lustre: Skipped 2 previous similar messages [ 9695.205436] LustreError: 245294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9695.226727] LustreError: 245294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9695.275378] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9705.673469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9712.941855] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9724.630612] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9731.786079] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9739.088648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9746.499784] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 9755.164411] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9900.513894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9900.525482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9900.550573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9900.580349] Lustre: Skipped 2 previous similar messages [ 9906.856656] Lustre: server umount lustre-OST0000 complete [ 9917.799637] Lustre: server umount lustre-MDT0000 complete [ 9918.946887] LustreError: 245290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9918.976433] LustreError: 245290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9922.317817] LustreError: 247140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451963 with bad export cookie 10832514832193000826 [ 9922.343333] LustreError: 247140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9922.348262] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9922.724534] Lustre: server umount lustre-MDT0001 complete [10023.249654] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [10026.487993] Key type lgssc unregistered [10026.779497] LNet: 248904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10026.798214] LNetError: 248904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10027.816188] LNet: Removed LNI 192.168.204.102@tcp [10028.859391] Key type .llcrypt unregistered [10028.869669] Key type ._llcrypt unregistered [10058.679421] Key type ._llcrypt registered [10058.683611] Key type .llcrypt registered [10058.861946] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [10075.618260] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [10076.522344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10076.614144] alg: No test for adler32 (adler32-zlib) [10077.628760] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10078.057498] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [10079.801830] Key type lgssc registered [10080.981687] Lustre: Echo OBD driver; http://www.lustre.org/ [10131.271759] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [10143.622715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10143.664701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10144.856621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10144.908856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10144.983930] Lustre: lustre-MDT0000: new disk, initializing [10145.118587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10145.135426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10151.991366] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10165.423032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10165.632305] Lustre: 252975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10165.712656] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10165.718029] Lustre: Skipped 1 previous similar message [10165.804606] Lustre: lustre-MDT0001: new disk, initializing [10165.945933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10165.986803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10166.014062] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10170.754437] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10175.862766] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10183.214152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10183.628604] Lustre: lustre-OST0000: new disk, initializing [10183.635511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10183.648343] Lustre: 254608:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10183.804575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10186.361189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10186.376605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10186.475903] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10189.715654] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10200.113355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10200.284932] Lustre: lustre-OST0001: new disk, initializing [10200.289716] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10200.297146] Lustre: 255480:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10200.394921] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10206.438420] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10208.941176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10208.958547] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10209.127218] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10218.241427] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10222.747052] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10234.857491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10234.864110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10234.888328] Lustre: Skipped 1 previous similar message [10234.902022] Lustre: Skipped 2 previous similar messages [10239.832177] Lustre: server umount lustre-MDT0000 complete [10243.215675] LustreError: 252969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452283 with bad export cookie 9088749017140148881 [10243.218088] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10243.223815] LustreError: 252969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10243.536281] Lustre: server umount lustre-MDT0001 complete [10257.719846] Lustre: server umount lustre-OST0000 complete [10260.451782] Lustre: 250482:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788452285/real 1788452285] req@ffff900dbd048e00 x1875327968161664/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788452301 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10260.495187] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10260.520019] Lustre: Skipped 2 previous similar messages [10262.185296] Lustre: server umount lustre-OST0001 complete [10281.383090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10291.706281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10301.371039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10311.728629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10334.963225] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [10346.155638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10346.193383] Lustre: lustre-MDT0000: reset Object Index mappings [10346.734954] LustreError: 260492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10346.760773] LustreError: 260492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10346.831818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10351.506871] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10352.098031] LustreError: 260493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10357.229201] LustreError: 260492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10359.946920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10359.959852] Lustre: lustre-MDT0001: reset Object Index mappings [10360.377065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10365.330478] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10368.102740] Lustre: 261638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10374.978064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10382.019913] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10385.641338] LustreError: 261993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10389.968073] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10390.230115] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10390.240486] Lustre: Skipped 1 previous similar message [10392.321493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10392.332170] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10396.360382] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10403.512504] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10407.650542] Lustre: 263510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10415.653988] Lustre: server umount lustre-MDT0000 complete [10416.613265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10416.624203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10416.643969] LustreError: 261226:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10420.638440] LustreError: 261238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452461 with bad export cookie 9088749017140155664 [10420.649181] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10420.667364] LustreError: 261238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10421.083926] Lustre: server umount lustre-MDT0001 complete [10435.953947] Lustre: server umount lustre-OST0000 complete [10450.269710] Lustre: server umount lustre-OST0001 complete [10470.577203] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 12:21:49 (1788452509) [10486.893440] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [10496.444401] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [10534.577640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10534.868508] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10534.916604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10535.015693] Lustre: lustre-MDT0000: new disk, initializing [10535.130822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10535.164511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10541.269532] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10553.206586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10553.289854] Lustre: 267764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10553.347662] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10553.354959] Lustre: Skipped 1 previous similar message [10553.407807] Lustre: lustre-MDT0001: new disk, initializing [10553.519124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10553.528523] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10558.720955] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10570.200480] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10577.576695] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10584.924239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10585.265858] Lustre: lustre-OST0000: new disk, initializing [10585.270236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10585.284164] Lustre: 269736:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10586.951252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10586.961088] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10587.058599] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10592.821224] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10604.025955] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10609.445372] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10609.737372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10615.541451] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10615.871904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10622.949254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10622.959548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10622.974696] Lustre: Skipped 3 previous similar messages [10622.983314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10622.999319] Lustre: Skipped 1 previous similar message [10624.963206] Lustre: server umount lustre-OST0000 complete [10630.626480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10630.646918] LustreError: Skipped 1 previous similar message [10630.656330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10630.681128] Lustre: Skipped 1 previous similar message [10630.702865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10633.186597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10636.576656] Lustre: server umount lustre-MDT0000 complete [10638.308749] LustreError: 267777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10638.319400] LustreError: 267777:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10641.917937] LustreError: 267757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452682 with bad export cookie 9088749017140157764 [10641.922417] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10641.947018] LustreError: 267757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10642.308975] Lustre: server umount lustre-MDT0001 complete [10666.088706] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [10677.932524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10678.444512] LustreError: 272673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10678.500877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10678.508193] Lustre: Skipped 2 previous similar messages [10683.391052] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10692.662167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10697.783842] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10701.136108] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10707.888605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10715.682462] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10724.956816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10725.174563] Lustre: lustre-OST0001: new disk, initializing [10725.179491] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10725.187075] Lustre: 274983:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10727.067707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10727.092066] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10727.253123] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10733.069391] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10743.026112] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10747.190614] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10754.362819] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10754.475864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10763.236754] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10763.265583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10763.269770] Lustre: Skipped 4 previous similar messages [10763.303201] Lustre: Skipped 4 previous similar messages [10768.494048] Lustre: server umount lustre-MDT0000 complete [10772.878089] LustreError: 272655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452813 with bad export cookie 9088749017140158646 [10772.891209] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10772.892136] LustreError: 272655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10773.290472] Lustre: server umount lustre-MDT0001 complete [10787.646320] Lustre: server umount lustre-OST0000 complete [10789.664134] Lustre: 250483:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788452814/real 1788452814] req@ffff900c83c25880 x1875327968326400/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788452830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10789.698932] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10792.750862] Lustre: server umount lustre-OST0001 complete [10812.308727] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [10823.018336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10823.444295] LustreError: 278041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10823.474905] LustreError: 278041:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10823.536799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10823.546094] Lustre: Skipped 3 previous similar messages [10828.211783] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10840.123415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10847.112305] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10851.018753] Lustre: 279183:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10860.507813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10868.484335] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10868.998085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10877.530878] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10883.073684] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10884.463744] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [10892.167465] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10895.841521] Lustre: 281059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10900.427489] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10900.505276] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10910.594686] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 12:29:09 (1788452949) [10928.097615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10928.110183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10928.120343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10928.127201] Lustre: Skipped 4 previous similar messages [10932.111782] Lustre: server umount lustre-MDT0000 complete [10934.258398] LustreError: 278041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10934.283060] LustreError: 278041:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [10935.665058] LustreError: 281062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452976 with bad export cookie 9088749017140165450 [10935.666952] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10935.678210] LustreError: 281062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10942.162946] Lustre: server umount lustre-MDT0001 complete [10952.420899] Lustre: server umount lustre-OST0000 complete [10970.080147] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10970.236217] Lustre: server umount lustre-OST0001 complete [10976.433376] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [10985.072266] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [11040.419128] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [11055.151996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11055.558794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11055.613866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11055.724700] Lustre: lustre-MDT0000: new disk, initializing [11055.878590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11061.925453] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11073.565088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11073.744064] Lustre: 285323:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11074.042063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11079.049550] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11091.163129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11091.442883] Lustre: 286956:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11091.539590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11091.549629] Lustre: Skipped 5 previous similar messages [11092.689607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11098.300173] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11109.198459] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11109.360989] Lustre: 287826:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11111.487611] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11111.507250] Lustre: Skipped 1 previous similar message [11111.681618] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11117.055668] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11127.648900] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11132.245373] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11132.260258] Lustre: Skipped 1 previous similar message [11153.341741] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 12:33:12 (1788453192) [11155.946432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11155.961284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11155.983434] Lustre: Skipped 5 previous similar messages [11155.991588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11156.003244] Lustre: Skipped 8 previous similar messages [11158.214923] Lustre: server umount lustre-MDT0000 complete [11162.003636] LustreError: 286118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453202 with bad export cookie 9088749017140172877 [11162.009823] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11162.012708] LustreError: 286118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11162.484301] Lustre: server umount lustre-MDT0001 complete [11176.849673] Lustre: server umount lustre-OST0000 complete [11178.786128] Lustre: 250483:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788453203/real 1788453203] req@ffff900c87638e00 x1875327968482816/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788453219 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11180.925933] Lustre: server umount lustre-OST0001 complete [11191.482210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11192.103962] LustreError: 290071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11192.128844] LustreError: 290071:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11198.213303] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11209.137135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11215.595437] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11225.506883] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11232.085367] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11240.260886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11247.487800] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11255.196264] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11284.645657] loop1: detected capacity change from 0 to 204800000 [11284.869827] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11284.891899] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11284.909650] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11290.516711] loop1: detected capacity change from 0 to 204800000 [11295.969956] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11296.037367] LustreError: 290059:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs21615: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11296.051391] LustreError: 290059:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs21615 [11296.055808] LustreError: 290059:0:(mgs_handler.c:517:mgs_target_reg()) fs21615-MDT0001: Error checking target index: rc = -13 [11296.069340] LustreError: fs21615-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11296.081708] LustreError: 293811:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11296.090308] LustreError: 293811:0:(tgt_mount.c:2129:server_put_super()) no obd fs21615-MDT0001 [11296.097118] LustreError: 293811:0:(tgt_mount.c:116:server_deregister_mount()) fs21615-MDT0001 not registered [11296.135919] Lustre: server umount fs21615-MDT0001 complete [11296.138409] LustreError: 293811:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11305.935282] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11306.119605] Lustre: Setting parameter fs21615-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs21615-MDT0001 [11306.226636] Lustre: srv-fs21615-MDT0001: No data found on store. Initialize space. [11306.234206] Lustre: Skipped 4 previous similar messages [11306.349565] Lustre: fs21615-MDT0001: new disk, initializing [11306.353795] Lustre: Skipped 3 previous similar messages [11306.473653] Lustre: srv-fs21615-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11306.482796] Lustre: 294277:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21615-MDT0001-mdtlov: can't create new llog: rc = -115 [11306.501771] LustreError: 294277:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21615-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11311.555462] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11319.274596] Lustre: server umount fs21615-MDT0001 complete [11333.104780] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11334.905604] loop1: detected capacity change from 0 to 204800000 [11341.984164] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11342.751078] Lustre: 295848:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21615-MDT0001-mdtlov: can't create new llog: rc = -115 [11342.769066] LustreError: 295848:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21615-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11349.209635] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11355.358368] Lustre: server umount fs21615-MDT0001 complete [11368.421255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11368.434527] LustreError: Skipped 1 previous similar message [11368.440102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11368.455232] Lustre: Skipped 4 previous similar messages [11368.462626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11368.472305] Lustre: Skipped 4 previous similar messages [11373.166861] Lustre: server umount lustre-MDT0000 complete [11383.302547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11383.384319] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11388.341800] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11388.916912] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11397.668617] loop1: detected capacity change from 0 to 204800000 [11404.939039] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11405.704512] Lustre: 298301:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs21615-MDT0001-mdtlov: can't create new llog: rc = -115 [11405.729163] LustreError: 298301:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs21615-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11411.076685] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11417.905831] Lustre: server umount fs21615-MDT0001 complete [11444.705977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11444.713539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11444.725592] Lustre: Skipped 3 previous similar messages [11447.035966] Lustre: server umount lustre-OST0000 complete [11458.024596] Lustre: server umount lustre-MDT0000 complete [11460.599900] LustreError: 290072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11460.637820] LustreError: 290072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 119 previous similar messages [11463.764687] LustreError: 290051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453504 with bad export cookie 9088749017140175558 [11463.781223] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11463.787223] LustreError: 290051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11464.736707] Lustre: server umount lustre-MDT0001 complete [11477.662902] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [11481.465320] Key type lgssc unregistered [11481.806683] LNet: 300649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11481.812915] LNetError: 300649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11481.834801] LNet: Removed LNI 192.168.204.102@tcp [11483.158283] Key type .llcrypt unregistered [11483.159974] Key type ._llcrypt unregistered [11508.371358] Key type ._llcrypt registered [11508.373825] Key type .llcrypt registered [11508.594867] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [11510.177981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11510.412136] alg: No test for adler32 (adler32-zlib) [11511.636382] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11511.984749] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [11513.721440] Key type lgssc registered [11514.799810] Lustre: Echo OBD driver; http://www.lustre.org/ [11523.514529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11523.529730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11525.023616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11529.436775] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11534.318265] LustreError: 301940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11534.346362] LustreError: 301940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11538.656892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11539.042350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11544.063860] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11552.477552] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11560.126900] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11571.046233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11572.265112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11582.330651] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11583.560390] LustreError: 303771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11588.600437] LustreError: 304164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11592.777099] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11593.702743] LustreError: 303772:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11599.077318] LustreError: 303771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11599.113761] LustreError: 303771:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11609.274325] LustreError: 303769:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11609.318574] LustreError: 303769:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11616.778668] loop1: detected capacity change from 0 to 204800000 [11616.975313] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11616.986644] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11617.002919] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11623.074819] loop1: detected capacity change from 0 to 204800000 [11626.475129] LustreError: 304164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11626.502600] LustreError: 304164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [11628.682714] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11628.964807] Lustre: fs975-OST0000: new disk, initializing [11628.973868] Lustre: srv-fs975-OST0000: No data found on store. Initialize space. [11628.986474] Lustre: 305675:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11629.109479] Lustre: fs975-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11635.814831] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11644.170552] Lustre: server umount fs975-OST0000 complete [11657.294048] loop1: detected capacity change from 0 to 204800000 [11660.468086] LustreError: 303771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11660.497452] LustreError: 303771:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [11662.337343] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11664.086933] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11664.404023] Lustre: fs975-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11671.137787] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11672.557253] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11679.989081] loop2: detected capacity change from 0 to 204800000 [11680.244615] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11680.262727] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11680.276339] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11685.311623] loop2: detected capacity change from 0 to 204800000 [11691.143959] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11691.222935] LustreError: fs975-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11691.235383] LustreError: fs975-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11691.243370] LustreError: 308717:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11691.249529] LustreError: 308717:0:(tgt_mount.c:2129:server_put_super()) no obd fs975-OST0001 [11691.254952] LustreError: 308717:0:(tgt_mount.c:116:server_deregister_mount()) fs975-OST0001 not registered [11691.287571] Lustre: server umount fs975-OST0001 complete [11691.297282] LustreError: 308717:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [11701.602686] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11701.742868] Lustre: fs975-OST0001: new disk, initializing [11701.748654] Lustre: srv-fs975-OST0001: No data found on store. Initialize space. [11701.759668] Lustre: 309268:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11701.853946] Lustre: fs975-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11708.298813] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11716.519454] Lustre: server umount fs975-OST0000 complete [11725.036560] Lustre: server umount fs975-OST0001 complete [11727.031957] LustreError: 303772:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11727.063207] LustreError: 303772:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [11739.257529] loop1: detected capacity change from 0 to 204800000 [11744.341871] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11744.580740] Lustre: fs975-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11750.426258] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11757.183137] loop2: detected capacity change from 0 to 204800000 [11762.428935] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11768.371284] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11777.701572] Lustre: server umount fs975-OST0000 complete [11786.372767] Lustre: server umount fs975-OST0001 complete [11798.500277] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11798.516194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11798.535217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11800.545748] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11800.552153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11800.553648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11805.667370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11805.674208] Lustre: Skipped 1 previous similar message [11810.789133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11810.801627] Lustre: Skipped 2 previous similar messages [11812.320205] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11812.558603] Lustre: server umount lustre-OST0000 complete [11820.516389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11820.526737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11820.546923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11823.315198] Lustre: server umount lustre-MDT0000 complete [11827.693942] LustreError: 301922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453868 with bad export cookie 8013172053318891687 [11827.702693] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11827.708988] LustreError: 301922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11828.158975] Lustre: server umount lustre-MDT0001 complete [11839.427676] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [11842.036165] Key type lgssc unregistered [11842.308535] LNet: 315193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11842.317759] LNetError: 315193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11842.358145] LNet: Removed LNI 192.168.204.102@tcp [11843.356755] Key type .llcrypt unregistered [11843.358675] Key type ._llcrypt unregistered [11879.284585] Key type ._llcrypt registered [11879.291815] Key type .llcrypt registered [11879.461646] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [11897.296804] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [11898.793480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11898.854678] alg: No test for adler32 (adler32-zlib) [11900.089035] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11900.334568] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [11902.000982] Key type lgssc registered [11903.192757] Lustre: Echo OBD driver; http://www.lustre.org/ [11943.021947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11943.046102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11944.245482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11944.269897] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11944.384560] Lustre: lustre-MDT0000: new disk, initializing [11944.471170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11944.490608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11950.260860] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11965.720732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11965.981508] Lustre: 319056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11966.103761] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11966.109866] Lustre: Skipped 1 previous similar message [11966.316489] Lustre: lustre-MDT0001: new disk, initializing [11966.499355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11966.603050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11966.620462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11971.525276] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [11982.768908] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11989.215982] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11997.246517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11997.572665] Lustre: lustre-OST0000: new disk, initializing [11997.584639] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11997.597559] Lustre: 321027:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11997.707675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12005.853845] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12007.544802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12007.557504] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12007.718726] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12018.581540] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12021.452297] Lustre: Mounted lustre-client - version 2.17.56_51_g2aa7cea [12023.113926] Lustre: Unmounted lustre-client [12038.625400] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12038.631460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12038.653910] LustreError: Skipped 1 previous similar message [12038.661332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12038.674250] Lustre: Skipped 1 previous similar message [12042.232606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12042.250830] Lustre: Skipped 1 previous similar message [12043.760533] Lustre: server umount lustre-OST0000 complete [12048.354526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12048.363829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12048.395779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12048.412509] Lustre: Skipped 1 previous similar message [12052.450572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12052.455184] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12054.817218] Lustre: server umount lustre-MDT0000 complete [12059.233847] LustreError: 319049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454100 with bad export cookie 15837395907324342071 [12059.239587] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12059.250976] LustreError: 319049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12059.681795] Lustre: server umount lustre-MDT0001 complete [12071.964247] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [12074.939401] Key type lgssc unregistered [12082.656271] LNet: 1 local NIs in recovery (showing 1): 192.168.204.102@tcp [12083.489566] LNet: 323193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12083.500471] LNetError: 323193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12150.061881] LNet: Removed LNI 192.168.204.102@tcp [12150.986457] Key type .llcrypt unregistered [12150.988827] Key type ._llcrypt unregistered [12172.555140] Key type ._llcrypt registered [12172.561042] Key type .llcrypt registered [12172.740050] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [12191.121948] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [12192.687420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12192.756124] alg: No test for adler32 (adler32-zlib) [12194.079441] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12194.350533] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [12196.024772] Key type lgssc registered [12198.096619] Lustre: Echo OBD driver; http://www.lustre.org/ [12234.256414] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:51:12 (1788454272) [12243.509057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12243.528929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12244.900692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12244.928425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12245.106459] Lustre: lustre-MDT0000: new disk, initializing [12245.257734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12245.291091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12250.631943] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12263.196810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12263.349101] Lustre: 326421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12263.409321] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12263.413193] Lustre: Skipped 1 previous similar message [12263.504603] Lustre: lustre-MDT0001: new disk, initializing [12263.639626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12263.685627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12263.692717] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12268.534917] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12279.188496] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12284.913447] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12292.135606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12292.434788] Lustre: lustre-OST0000: new disk, initializing [12292.439511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12292.447831] Lustre: 328388:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12292.541127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12298.431043] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12299.361615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12299.380407] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12299.577548] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12307.687969] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12325.351807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12325.352839] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12325.356897] LustreError: Skipped 1 previous similar message [12325.359662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12325.390764] Lustre: Skipped 1 previous similar message [12329.405054] Lustre: server umount lustre-OST0000 complete [12335.586207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12335.592529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12335.617785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12335.630754] Lustre: Skipped 2 previous similar messages [12340.705869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12340.713716] Lustre: Skipped 1 previous similar message [12345.829194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12345.842673] Lustre: Skipped 1 previous similar message [12348.384149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12348.669197] Lustre: server umount lustre-MDT0000 complete [12352.611902] LustreError: 326412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454393 with bad export cookie 17820677194039278888 [12352.618646] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12352.630111] LustreError: 326412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12353.062778] Lustre: server umount lustre-MDT0001 complete [12364.017726] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [12367.522790] Key type lgssc unregistered [12367.849037] LNet: 330302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12367.855295] LNetError: 330302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12367.875153] LNet: Removed LNI 192.168.204.102@tcp [12368.816541] Key type .llcrypt unregistered [12368.819943] Key type ._llcrypt unregistered [12391.121650] Key type ._llcrypt registered [12391.125581] Key type .llcrypt registered [12391.256710] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [12392.481877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12392.689884] alg: No test for adler32 (adler32-zlib) [12393.805546] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12393.998314] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [12395.673893] Key type lgssc registered [12396.955961] Lustre: Echo OBD driver; http://www.lustre.org/ [12404.943456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12404.974440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12406.462457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12411.396276] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12417.000989] LustreError: 331575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12417.019120] LustreError: 331575:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12420.176846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12420.550764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12424.615079] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12432.557527] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12438.642881] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12447.845746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12448.327631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12456.293385] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12464.433586] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12474.570967] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12476.524714] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12481.009142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12481.020942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12481.045047] Lustre: Skipped 2 previous similar messages [12481.085947] Lustre: Skipped 2 previous similar messages [12481.778714] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.2@tcp (stopping) [12486.117848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12486.137652] Lustre: Skipped 2 previous similar messages [12487.440439] Lustre: server umount lustre-MDT0000 complete [12491.242793] LustreError: 331572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12491.259280] LustreError: 331572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12492.113288] LustreError: 331556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454532 with bad export cookie 3609423664133472886 [12492.119924] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12492.124548] LustreError: 331556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12496.355199] LustreError: 331571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12496.356347] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12496.386563] LustreError: 331571:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12496.415993] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12496.430780] Lustre: Skipped 1 previous similar message [12498.492095] Lustre: server umount lustre-MDT0001 complete [12502.706634] Lustre: server umount lustre-OST0000 complete [12511.129438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12515.648184] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12518.942900] Lustre: 335756:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.102@tcp [12521.995347] Lustre: server umount MGS complete [12531.420561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12531.740693] LustreError: 336329:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12531.758756] LustreError: 336329:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12531.804284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12536.807769] LustreError: 336330:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12537.987473] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12547.055729] LustreError: 336329:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12547.085024] LustreError: 336329:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12549.381771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12549.720503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12554.551840] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12564.627492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12565.054742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12573.012591] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12581.998794] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12583.825986] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12590.084390] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12591.816446] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12597.217220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12597.222704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12597.236208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12597.245219] Lustre: Skipped 1 previous similar message [12597.255515] Lustre: Skipped 3 previous similar messages [12603.487922] Lustre: server umount lustre-OST0000 complete [12611.044885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12611.050898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12611.064528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12611.071889] Lustre: Skipped 2 previous similar messages [12622.304295] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12622.826443] LustreError: 336329:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12622.850309] LustreError: 336329:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12623.125648] Lustre: server umount lustre-MDT0000 complete [12628.616154] LustreError: 337787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454669 with bad export cookie 3609423664133473726 [12628.624887] LustreError: 337787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12628.627880] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12628.864991] Lustre: server umount lustre-MDT0001 complete [12642.034415] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [12645.204834] Key type lgssc unregistered [12645.534818] LNet: 339774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12645.537954] LNetError: 339774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12645.561263] LNet: Removed LNI 192.168.204.102@tcp [12646.738599] Key type .llcrypt unregistered [12646.740600] Key type ._llcrypt unregistered [12668.066934] Key type ._llcrypt registered [12668.069705] Key type .llcrypt registered [12668.162853] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [12689.894055] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [12692.216708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12692.294550] alg: No test for adler32 (adler32-zlib) [12693.281690] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12693.548886] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [12695.240356] Key type lgssc registered [12696.418931] Lustre: Echo OBD driver; http://www.lustre.org/ [12736.093915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12736.127134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12737.365166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12737.427755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12737.593739] Lustre: lustre-MDT0000: new disk, initializing [12737.692893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12737.716716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12742.701484] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12754.681260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12754.941270] Lustre: 342942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12755.063662] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12755.072957] Lustre: Skipped 1 previous similar message [12755.214280] Lustre: lustre-MDT0001: new disk, initializing [12755.376746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12755.459531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12755.485896] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12760.808391] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12772.393083] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12780.096154] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12789.000083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12789.495185] Lustre: lustre-OST0000: new disk, initializing [12789.501962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12789.515995] Lustre: 344911:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12789.723854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12798.563329] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12799.176428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12799.193625] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12799.358953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12811.962680] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12817.875438] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12818.174525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12824.372375] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12824.650728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12830.180045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12830.182374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12830.202303] LustreError: Skipped 1 previous similar message [12830.208110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12830.244709] Lustre: Skipped 1 previous similar message [12833.625706] Lustre: server umount lustre-OST0000 complete [12840.424721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12840.438755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12840.445872] Lustre: Skipped 1 previous similar message [12840.479919] Lustre: Skipped 2 previous similar messages [12845.544564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12845.559759] Lustre: Skipped 1 previous similar message [12850.660311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12850.689389] Lustre: Skipped 1 previous similar message [12854.240177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12854.544787] Lustre: server umount lustre-MDT0000 complete [12859.139655] LustreError: 344910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454899 with bad export cookie 4868959126070736392 [12859.150603] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12859.155096] LustreError: 344910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12859.936279] Lustre: server umount lustre-MDT0001 complete [12872.119845] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 13:01:50 (1788454910) [12891.269581] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [12902.350445] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [12939.941345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12940.188966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12940.231946] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12940.288242] Lustre: lustre-MDT0000: new disk, initializing [12940.381726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12940.419992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12945.085131] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12957.088576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12957.249627] Lustre: 349686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12957.307799] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12957.311971] Lustre: Skipped 1 previous similar message [12957.405493] Lustre: lustre-MDT0001: new disk, initializing [12957.564493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12957.603159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12957.628117] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12964.154708] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12975.727553] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12982.191841] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12989.209131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12989.489694] Lustre: lustre-OST0000: new disk, initializing [12989.493818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12989.503413] Lustre: 351657:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12989.608561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12996.150742] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [12996.667452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12996.682585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12996.845961] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13005.264187] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13048.299203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13048.307901] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13048.312547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13048.319516] LustreError: Skipped 1 previous similar message [13048.327213] Lustre: Skipped 1 previous similar message [13052.322703] Lustre: server umount lustre-OST0000 complete [13058.533252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13058.544795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13058.560618] Lustre: Skipped 2 previous similar messages [13058.572428] Lustre: Skipped 1 previous similar message [13062.242283] Lustre: server umount lustre-MDT0000 complete [13063.651608] LustreError: 349696:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13063.672231] LustreError: 349696:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13066.442734] LustreError: 349676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455107 with bad export cookie 4868959126070737274 [13066.450617] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13066.457408] LustreError: 349676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13066.953316] Lustre: server umount lustre-MDT0001 complete [13077.731200] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13080.760723] Key type lgssc unregistered [13081.030176] LNet: 353835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13081.047935] LNetError: 353835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13081.068336] LNet: Removed LNI 192.168.204.102@tcp [13082.004382] Key type .llcrypt unregistered [13082.007665] Key type ._llcrypt unregistered [13102.748041] Key type ._llcrypt registered [13102.750398] Key type .llcrypt registered [13102.834726] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [13104.044816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13104.093930] alg: No test for adler32 (adler32-zlib) [13105.262246] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13105.602631] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [13107.322583] Key type lgssc registered [13108.517436] Lustre: Echo OBD driver; http://www.lustre.org/ [13115.938783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13115.985622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13117.548413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13121.969738] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [13128.164164] LustreError: 355107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13128.180679] LustreError: 355107:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13132.084030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13132.523857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13137.112329] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [13145.586671] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13152.381610] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13160.405060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13161.145097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13167.999752] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [13176.295841] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13188.064962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13188.070161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13188.080911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13188.104050] Lustre: Skipped 1 previous similar message [13188.110519] Lustre: Skipped 1 previous similar message [13193.194320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13193.224657] Lustre: Skipped 1 previous similar message [13194.092837] Lustre: server umount lustre-OST0000 complete [13203.443959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13203.474488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13203.500666] Lustre: Skipped 1 previous similar message [13208.556363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13209.042182] Lustre: server umount lustre-MDT0000 complete [13213.680225] LustreError: 355102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13213.722179] LustreError: 355102:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13217.416149] LustreError: 355089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455258 with bad export cookie 1626111006707158447 [13217.446070] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13218.624053] Lustre: server umount lustre-MDT0001 complete [13236.872910] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13240.998402] Key type lgssc unregistered [13241.422545] LNet: 360438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13241.439966] LNetError: 360438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13241.466173] LNet: Removed LNI 192.168.204.102@tcp [13242.620033] Key type .llcrypt unregistered [13242.629311] Key type ._llcrypt unregistered [13261.774512] Key type ._llcrypt registered [13261.777104] Key type .llcrypt registered [13261.901855] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [13263.075021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13263.154620] alg: No test for adler32 (adler32-zlib) [13264.339855] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13264.692179] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [13266.360134] Key type lgssc registered [13267.560744] Lustre: Echo OBD driver; http://www.lustre.org/ [13276.582796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13276.621577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13278.210782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13282.524620] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [13287.396310] LustreError: 364135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13287.415842] LustreError: 364135:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13291.763612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13292.223512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13297.361536] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [13304.980308] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13310.957750] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13318.972169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13319.344754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13326.067172] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [13334.934135] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13347.300663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13347.319870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13347.335042] Lustre: Skipped 1 previous similar message [13347.364178] Lustre: Skipped 1 previous similar message [13350.944068] Lustre: server umount lustre-OST0000 complete [13357.543893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13357.568904] Lustre: Skipped 1 previous similar message [13357.569840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13357.589847] Lustre: Skipped 1 previous similar message [13362.013869] Lustre: server umount lustre-MDT0000 complete [13362.657515] LustreError: 364127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13362.686438] LustreError: 364127:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13366.866028] LustreError: 364065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455407 with bad export cookie 6074963380999390623 [13366.875233] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13366.879233] LustreError: 364065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13367.514318] Lustre: server umount lustre-MDT0001 complete [13381.248276] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13385.291088] Key type lgssc unregistered [13385.870545] LNet: 371490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13385.890813] LNetError: 371490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13385.911779] LNet: Removed LNI 192.168.204.102@tcp [13387.359124] Key type .llcrypt unregistered [13387.361523] Key type ._llcrypt unregistered [13411.698307] Key type ._llcrypt registered [13411.700553] Key type .llcrypt registered [13411.817824] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [13413.048815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13413.071937] alg: No test for adler32 (adler32-zlib) [13414.889672] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13415.463530] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [13417.305092] Key type lgssc registered [13419.514499] Lustre: Echo OBD driver; http://www.lustre.org/ [13430.859800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13430.898769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13432.699910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13438.321355] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [13443.043489] LustreError: 375021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13443.066707] LustreError: 375021:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13448.173725] LustreError: 375022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13449.823246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13450.302921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13456.079443] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [13464.613713] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13471.452505] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13479.691651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13480.155178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13488.649451] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [13499.127860] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13518.305434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13518.323122] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13518.339722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13518.359537] Lustre: Skipped 1 previous similar message [13523.180840] Lustre: server umount lustre-OST0000 complete [13528.547236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13528.582653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13528.597082] Lustre: Skipped 2 previous similar messages [13533.666025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13533.685458] Lustre: Skipped 1 previous similar message [13534.241523] Lustre: server umount lustre-MDT0000 complete [13538.788153] LustreError: 375021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13538.814739] LustreError: 375021:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13539.596629] LustreError: 374961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455580 with bad export cookie 9214886374569178214 [13539.623026] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13540.369759] Lustre: server umount lustre-MDT0001 complete [13552.188367] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13554.949446] Key type lgssc unregistered [13555.247881] LNet: 380849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13555.258687] LNetError: 380849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13556.325873] LNet: Removed LNI 192.168.204.102@tcp [13557.650354] Key type .llcrypt unregistered [13557.653280] Key type ._llcrypt unregistered [13681.457384] Key type ._llcrypt registered [13681.459498] Key type .llcrypt registered [13681.548291] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13681.815286] Key type .llcrypt unregistered [13681.821854] Key type ._llcrypt unregistered [13758.472591] Key type ._llcrypt registered [13758.475051] Key type .llcrypt registered [13758.593659] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13758.998316] Key type .llcrypt unregistered [13759.001552] Key type ._llcrypt unregistered [13786.723846] Key type ._llcrypt registered [13786.726551] Key type .llcrypt registered [13786.870665] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13787.302529] Key type .llcrypt unregistered [13787.307794] Key type ._llcrypt unregistered [13824.138172] Key type ._llcrypt registered [13824.141432] Key type .llcrypt registered [13824.282668] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13824.625655] Key type .llcrypt unregistered [13824.629128] Key type ._llcrypt unregistered [13898.363189] Key type ._llcrypt registered [13898.364598] Key type .llcrypt registered [13898.561122] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13898.891308] Key type .llcrypt unregistered [13898.893787] Key type ._llcrypt unregistered [13931.098195] Key type ._llcrypt registered [13931.103841] Key type .llcrypt registered [13931.257155] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13931.886744] Key type .llcrypt unregistered [13931.891759] Key type ._llcrypt unregistered [13971.283953] Key type ._llcrypt registered [13971.286610] Key type .llcrypt registered [13971.398068] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [13972.002577] Key type .llcrypt unregistered [13972.007549] Key type ._llcrypt unregistered [14007.226944] Key type ._llcrypt registered [14007.228716] Key type .llcrypt registered [14007.295268] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [14007.753294] Key type .llcrypt unregistered [14007.756407] Key type ._llcrypt unregistered [14039.783880] Key type ._llcrypt registered [14039.785173] Key type .llcrypt registered [14039.877500] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [14040.209386] Key type .llcrypt unregistered [14040.213355] Key type ._llcrypt unregistered [14082.826176] Key type ._llcrypt registered [14082.827932] Key type .llcrypt registered [14083.127497] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [14083.594369] Key type .llcrypt unregistered [14083.599367] Key type ._llcrypt unregistered [14152.201334] Key type ._llcrypt registered [14152.203920] Key type .llcrypt registered [14152.369560] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [14152.753322] Key type .llcrypt unregistered [14152.755210] Key type ._llcrypt unregistered [14185.251673] Key type ._llcrypt registered [14185.258631] Key type .llcrypt registered [14185.348893] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [14185.597328] Key type .llcrypt unregistered [14185.606754] Key type ._llcrypt unregistered [14220.007839] Key type ._llcrypt registered [14220.011456] Key type .llcrypt registered [14220.155443] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [14220.490492] Key type .llcrypt unregistered [14220.495377] Key type ._llcrypt unregistered [14282.676719] Key type ._llcrypt registered [14282.680096] Key type .llcrypt registered [14282.790052] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [14304.380744] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [14306.726497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14306.805316] alg: No test for adler32 (adler32-zlib) [14307.996972] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [14308.187525] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [14309.819690] Key type lgssc registered [14311.429126] Lustre: Echo OBD driver; http://www.lustre.org/ [14355.725790] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14355.767086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14357.069343] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14357.112332] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14357.186330] Lustre: tst250fs-MDT0000: new disk, initializing [14357.256509] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14357.281982] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14362.407868] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [14374.344337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14374.568856] Lustre: 401474:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14374.641368] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14374.652351] Lustre: Skipped 1 previous similar message [14374.782805] Lustre: tst250fs-MDT0001: new disk, initializing [14374.952570] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14375.013686] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14375.030979] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14380.499515] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [14386.546722] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14394.372722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14394.784098] Lustre: tst250fs-OST0000: new disk, initializing [14394.792476] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14394.803030] Lustre: 403108:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14394.972692] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14401.549390] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [14404.189552] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14404.205532] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14404.365154] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14413.416384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14413.704923] Lustre: tst250fs-OST0001: new disk, initializing [14413.710556] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14413.725854] Lustre: 403979:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14413.863961] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14421.132257] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [14422.202297] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14422.221581] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14422.336981] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14428.039314] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14430.320553] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14430.336470] Lustre: Skipped 1 previous similar message [14456.801860] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14456.820554] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14456.851148] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14458.339938] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14458.359731] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14462.610907] Lustre: server umount tst250fs-MDT0000 complete [14467.887832] LustreError: 401467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456508 with bad export cookie 1275782782554835236 [14467.897373] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14467.932929] LustreError: 401467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14468.577293] LustreError: 401480:0:(ldlm_lib.c:1192:target_handle_connect()) tst250fs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14468.588370] LustreError: 401480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [14468.588851] Lustre: tst250fs-MDT0001-lwp-OST0000: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14468.588858] Lustre: Skipped 2 previous similar messages [14468.713748] Lustre: server umount tst250fs-MDT0001 complete [14475.032118] Lustre: server umount tst250fs-OST0000 complete [14480.542810] Lustre: server umount tst250fs-OST0001 complete [14507.256087] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [14520.322168] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [14560.582884] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 13:29:58 (1788456598) [14562.242348] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14588.278789] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [14605.224083] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [14649.967276] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14383 sec ========= 13:31:27 (1788456687) [14652.780899] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:31:30 (1788456690) === [14658.289060] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:31:35 (1788456695) === [14712.782906] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [14717.184883] Key type lgssc unregistered [14717.558039] LNet: 413150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14717.569579] LNetError: 413150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14717.601439] LNet: Removed LNI 192.168.204.102@tcp [14718.948260] Key type .llcrypt unregistered [14718.953266] Key type ._llcrypt unregistered