[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 493271498 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003075] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005009] kvm-guest: setup PV IPIs [ 0.008582] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011123] LSM: Security Framework initializing [ 0.012059] Yama: becoming mindful. [ 0.013029] SELinux: Initializing. [ 0.015018] *** VALIDATE selinux *** [ 0.023039] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028887] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029224] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031103] *** VALIDATE tmpfs *** [ 0.032459] *** VALIDATE proc *** [ 0.034301] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036290] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038070] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040030] Spectre V2 : User space: Vulnerable [ 0.041010] Speculative Store Bypass: Vulnerable [ 0.043994] debug: unmapping init [mem 0xffffffffb0a59000-0xffffffffb0a60fff] [ 0.046143] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047704] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048026] ... version: 2 [ 0.049015] ... bit width: 48 [ 0.050012] ... generic registers: 4 [ 0.051016] ... value mask: 0000ffffffffffff [ 0.052016] ... max period: 00007fffffffffff [ 0.053018] ... fixed-purpose events: 3 [ 0.054012] ... event mask: 000000070000000f [ 0.055291] rcu: Hierarchical SRCU implementation. [ 0.057444] smp: Bringing up secondary CPUs ... [ 0.058603] x86: Booting SMP configuration: [ 0.059031] .... node #0, CPUs: #1 #2 #3 [ 0.062716] smp: Brought up 1 node, 4 CPUs [ 0.064014] smpboot: Max logical packages: 1 [ 0.065019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.266073] node 0 deferred pages initialised in 199ms [ 0.271087] devtmpfs: initialized [ 0.272363] x86/mm: Memory block size: 128MB [ 0.275831] gcov: version magic: 0x41383552 [ 0.279383] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.283152] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.286536] pinctrl core: initialized pinctrl subsystem [ 0.289292] [ 0.289953] ************************************************************* [ 0.292020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.295018] ** ** [ 0.298022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.301019] ** ** [ 0.304019] ** This means that this kernel is built to expose internal ** [ 0.306017] ** IOMMU data structures, which may compromise security on ** [ 0.309021] ** your system. ** [ 0.312016] ** ** [ 0.314015] ** If you see this message and you are not debugging the ** [ 0.316021] ** kernel, report this immediately to your vendor! ** [ 0.319018] ** ** [ 0.321018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.324019] ************************************************************* [ 0.326780] NET: Registered protocol family 16 [ 0.329383] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.332099] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.335102] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.340040] cpuidle: using governor menu [ 0.342645] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.345698] PCI: Using configuration type 1 for base access [ 0.348128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.357190] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.358024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.361067] cryptd: max_cpu_qlen set to 1000 [ 0.363342] ACPI: Added _OSI(Module Device) [ 0.365026] ACPI: Added _OSI(Processor Device) [ 0.367025] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.369024] ACPI: Added _OSI(Processor Aggregator Device) [ 0.373879] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.381553] ACPI: Interpreter enabled [ 0.383077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.385026] ACPI: Using IOAPIC for interrupt routing [ 0.387122] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.390428] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.400541] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.403123] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.407036] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.411152] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.417292] acpiphp: Slot [2] registered [ 0.418089] acpiphp: Slot [5] registered [ 0.419133] acpiphp: Slot [6] registered [ 0.421165] acpiphp: Slot [7] registered [ 0.423219] acpiphp: Slot [8] registered [ 0.425106] acpiphp: Slot [9] registered [ 0.426212] acpiphp: Slot [10] registered [ 0.428154] acpiphp: Slot [3] registered [ 0.430120] acpiphp: Slot [4] registered [ 0.431113] acpiphp: Slot [11] registered [ 0.433117] acpiphp: Slot [12] registered [ 0.435132] acpiphp: Slot [13] registered [ 0.437122] acpiphp: Slot [14] registered [ 0.438104] acpiphp: Slot [15] registered [ 0.441214] acpiphp: Slot [16] registered [ 0.443146] acpiphp: Slot [17] registered [ 0.444154] acpiphp: Slot [18] registered [ 0.446235] acpiphp: Slot [19] registered [ 0.448149] acpiphp: Slot [20] registered [ 0.449143] acpiphp: Slot [21] registered [ 0.451165] acpiphp: Slot [22] registered [ 0.453155] acpiphp: Slot [23] registered [ 0.455150] acpiphp: Slot [24] registered [ 0.456111] acpiphp: Slot [25] registered [ 0.458172] acpiphp: Slot [26] registered [ 0.460194] acpiphp: Slot [27] registered [ 0.462157] acpiphp: Slot [28] registered [ 0.464181] acpiphp: Slot [29] registered [ 0.466184] acpiphp: Slot [30] registered [ 0.467149] acpiphp: Slot [31] registered [ 0.469095] PCI host bridge to bus 0000:00 [ 0.472039] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.475037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.478037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.481037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.485032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.487044] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.490122] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.495235] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.499631] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.509770] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.516447] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.519027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.523034] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.527029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.529643] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.532948] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.537062] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.542041] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.548023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.562023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.567023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.574305] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.581133] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.588023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.607021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.618723] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.626021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.633022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.653027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.664595] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.672024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.679020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.696018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.707497] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.717022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.726016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.744017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.754872] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.766017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.773016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.798028] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.809360] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.821015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.832017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.855019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.868581] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.870284] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.873432] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.875265] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.878256] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.883645] iommu: Default domain type: Passthrough [ 0.885460] SCSI subsystem initialized [ 0.887109] ACPI: bus type USB registered [ 0.888062] usbcore: registered new interface driver usbfs [ 0.889060] usbcore: registered new interface driver hub [ 0.891072] usbcore: registered new device driver usb [ 0.892187] pps_core: LinuxPPS API ver. 1 registered [ 0.894013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.897067] PTP clock support registered [ 0.899080] EDAC MC: Ver: 3.0.0 [ 0.901204] PCI: Using ACPI for IRQ routing [ 0.902655] NetLabel: Initializing [ 0.904009] NetLabel: domain hash size = 128 [ 0.906013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.908165] NetLabel: unlabeled traffic allowed by default [ 0.911293] vgaarb: loaded [ 0.913310] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.916030] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.924698] clocksource: Switched to clocksource kvm-clock [ 1.029835] VFS: Disk quotas dquot_6.6.0 [ 1.031903] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.035214] *** VALIDATE ramfs *** [ 1.036536] *** VALIDATE hugetlbfs *** [ 1.038772] pnp: PnP ACPI init [ 1.041399] pnp: PnP ACPI: found 6 devices [ 1.059566] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.063852] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.066359] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.069051] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.072058] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.075149] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.078339] NET: Registered protocol family 2 [ 1.081417] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.087366] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.091208] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.096603] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.100087] TCP: Hash tables configured (established 65536 bind 65536) [ 1.102812] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.106074] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.108933] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.112274] NET: Registered protocol family 1 [ 1.115273] RPC: Registered named UNIX socket transport module. [ 1.118034] RPC: Registered udp transport module. [ 1.120539] RPC: Registered tcp transport module. [ 1.122644] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.125882] NET: Registered protocol family 44 [ 1.127912] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.130624] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.132850] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.135758] PCI: CLS 0 bytes, default 64 [ 1.137539] Unpacking initramfs... [ 2.699063] debug: unmapping init [mem 0xffff96dffcc54000-0xffff96dffffbffff] [ 2.703765] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.706608] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.710266] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.199763] Initialise system trusted keyrings [ 5.204529] Key type blacklist registered [ 5.208365] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.230228] zbud: loaded [ 5.235399] *** VALIDATE nfs *** [ 5.238891] *** VALIDATE nfs4 *** [ 5.243186] pstore: using deflate compression [ 5.256732] Platform Keyring initialized [ 5.502499] NET: Registered protocol family 38 [ 5.504375] Key type asymmetric registered [ 5.509786] Asymmetric key parser 'x509' registered [ 5.511614] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.518495] io scheduler mq-deadline registered [ 5.520612] io scheduler kyber registered [ 5.527860] io scheduler bfq registered [ 5.530225] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.535599] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.540430] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.545279] ACPI: Power Button [PWRF] [ 5.556782] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.567320] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.597604] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.614838] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.650366] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.684490] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.728652] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.737758] Non-volatile memory driver v1.3 [ 5.742233] Linux agpgart interface v0.103 [ 5.832549] virtio_blk virtio1: [vda] 149784 512-byte logical blocks (76.7 MB/73.1 MiB) [ 5.843782] vda: detected capacity change from 0 to 76689408 [ 5.895352] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.906402] vdb: detected capacity change from 0 to 1073741824 [ 5.949157] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.954079] vdc: detected capacity change from 0 to 2621440000 [ 5.981817] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.988200] vdd: detected capacity change from 0 to 2621440000 [ 6.022275] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.029538] vde: detected capacity change from 0 to 4294967296 [ 6.059311] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.063606] vdf: detected capacity change from 0 to 4294967296 [ 6.075957] libphy: Fixed MDIO Bus: probed [ 6.085309] usbcore: registered new interface driver usbserial_generic [ 6.088983] usbserial: USB Serial support registered for generic [ 6.092370] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.098848] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.101674] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.105110] mousedev: PS/2 mouse device common for all mice [ 6.109688] rtc_cmos 00:05: RTC can wake from S4 [ 6.112245] rtc_cmos 00:05: registered as rtc0 [ 6.118456] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.122353] intel_pstate: CPU model not supported [ 6.126283] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.131760] hid: raw HID events driver (C) Jiri Kosina [ 6.136310] usbcore: registered new interface driver usbhid [ 6.137723] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.145822] usbhid: USB HID core driver [ 6.146057] drop_monitor: Initializing network drop monitor service [ 6.146187] Initializing XFRM netlink socket [ 6.146547] NET: Registered protocol family 10 [ 6.147887] Segment Routing with IPv6 [ 6.147921] NET: Registered protocol family 17 [ 6.148073] mpls_gso: MPLS GSO support [ 6.170365] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.179840] RAS: Correctable Errors collector initialized. [ 6.182952] AVX version of gcm_enc/dec engaged. [ 6.185143] AES CTR mode by8 optimization enabled [ 6.319355] sched_clock: Marking stable (6319327465, 0)->(7250728994, -931401529) [ 6.325328] registered taskstats version 1 [ 6.328664] Loading compiled-in X.509 certificates [ 6.331964] zswap: loaded using pool lzo/zbud [ 6.372812] Key type big_key registered [ 6.402382] Key type encrypted registered [ 6.407160] ima: No TPM chip found, activating TPM-bypass! [ 6.412164] ima: Allocated hash algorithm: sha1 [ 6.416413] ima: No architecture policies found [ 6.421397] evm: Initialising EVM extended attributes: [ 6.426368] evm: security.selinux [ 6.429846] evm: security.ima [ 6.432854] evm: security.capability [ 6.436438] evm: HMAC attrs: 0x1 [ 6.443134] rtc_cmos 00:05: setting system clock to 2026-09-06 03:28:33 UTC (1788665313) [ 6.456505] debug: unmapping init [mem 0xffffffffb1a03000-0xffffffffb1bfffff] [ 6.463309] debug: unmapping init [mem 0xffffffffb0782000-0xffffffffb0a58fff] [ 6.479110] Write protecting the kernel read-only data: 28672k [ 6.485349] debug: unmapping init [mem 0xffffffffaee03000-0xffffffffaeffffff] [ 6.491147] debug: unmapping init [mem 0xffffffffaf714000-0xffffffffaf7fffff] [ 6.597297] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.629733] systemd[1]: Detected virtualization kvm. [ 6.633274] systemd[1]: Detected architecture x86-64. [ 6.635274] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.712266] systemd[1]: No hostname configured. [ 6.716384] systemd[1]: Set hostname to . [ 6.720391] random: systemd: uninitialized urandom read (16 bytes read) [ 6.724224] systemd[1]: Initializing machine ID from random generator. [ 6.819462] random: ln: uninitialized urandom read (6 bytes read) [ 6.997399] random: systemd: uninitialized urandom read (16 bytes read) [ 7.002246] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 7.017102] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 7.030379] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.424798] device-mapper: uevent: version 1.0.3 [ 8.438478] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.224777] random: fast init done [ 10.237977] virtio_net virtio0 ens2: renamed from eth0 [ 10.401714] scsi host0: ata_piix [ 10.513875] scsi host1: ata_piix [ 10.516376] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.526247] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.975671] random: crng init done [ 14.977738] random: 7 urandom warning(s) missed due to ratelimiting [ 17.146950] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.426242] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.314355] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.322503] SELinux: Disabled at runtime. [ 23.472394] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.491985] systemd[1]: Detected virtualization kvm. [ 23.495164] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.750749] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.763658] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.780537] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.793077] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.805793] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.825114] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.839897] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ 26.422243] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 27.296298] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.310725] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.382882] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.881561] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.099406] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 34.494432] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 35.278815] NFS: Registering the id_resolver key type [ 35.281237] Key type id_resolver registered [ 35.283140] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg346-server login: [ 52.826224] hrtimer: interrupt took 2360411 ns [ 103.278355] libcfs: loading out-of-tree module taints kernel. [ 103.322310] Key type ._llcrypt registered [ 103.324825] Key type .llcrypt registered [ 103.426538] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [ 124.038851] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 125.613274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 125.626307] alg: No test for adler32 (adler32-zlib) [ 127.016812] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 127.891911] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 129.647172] Key type lgssc registered [ 131.768387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 152.155642] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 193.755517] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 208.843305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 208.911502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 210.244690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 210.293447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 210.442974] Lustre: lustre-MDT0000: new disk, initializing [ 210.574218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 210.617288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 216.055460] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.243733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 230.400164] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 230.506706] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 230.510855] Lustre: Skipped 1 previous similar message [ 230.592621] Lustre: lustre-MDT0001: new disk, initializing [ 230.672583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 230.714319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 230.726109] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 235.274477] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.720962] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 250.004577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 250.408779] Lustre: lustre-OST0000: new disk, initializing [ 250.417165] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 250.430406] Lustre: 8452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 250.520883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 255.540181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 255.558311] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 255.684662] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 257.166661] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 273.718921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 273.909395] Lustre: lustre-OST0001: new disk, initializing [ 273.916480] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 273.922897] Lustre: 9525:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 273.983321] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 280.600061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 280.608427] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 280.702373] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 281.031726] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 295.547827] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 307.402459] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 313.708847] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing check_logdir /tmp/testlogs/ [ 318.544313] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing yml_node [ 322.648672] Lustre: DEBUG MARKER: Client: 2.17.58.3 [ 325.056644] Lustre: DEBUG MARKER: MDS: 2.17.58.3 [ 327.306915] Lustre: DEBUG MARKER: OSS: 2.17.58.3 [ 328.654496] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 23:33:54 EDT 2026 [ 344.340359] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 345.748334] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 362.464025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.478319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 362.487243] Lustre: Skipped 1 previous similar message [ 367.585614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 367.593478] Lustre: Skipped 6 previous similar messages [ 372.709146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 372.719729] Lustre: Skipped 3 previous similar messages [ 375.263225] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 375.417184] Lustre: server umount lustre-MDT0000 complete [ 378.751256] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665685 with bad export cookie 18334108443488997718 [ 378.753276] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 378.757785] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 382.944627] LustreError: 9396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 382.946644] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 382.947614] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 382.976791] LustreError: 9396:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 383.020712] Lustre: Skipped 3 previous similar messages [ 383.968312] LustreError: 9395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 383.994877] LustreError: 9395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 388.064125] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 388.065287] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 388.090401] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 388.116845] Lustre: Skipped 4 previous similar messages [ 393.183129] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 393.184235] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 393.241746] LustreError: 6521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 393.554691] Lustre: server umount lustre-MDT0001 complete [ 397.840531] Lustre: server umount lustre-OST0000 complete [ 401.585671] Lustre: server umount lustre-OST0001 complete [ 407.718443] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [ 416.285907] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 449.135754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 449.388829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 449.417328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 449.491380] Lustre: lustre-MDT0000: new disk, initializing [ 449.570081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.591203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 453.795321] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 463.030933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 463.105767] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 463.183647] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 463.191847] Lustre: Skipped 1 previous similar message [ 463.265815] Lustre: lustre-MDT0001: new disk, initializing [ 463.365372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 463.403595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 463.413571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 467.321323] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 476.172866] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 481.735525] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 488.243654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 488.531242] Lustre: lustre-OST0000: new disk, initializing [ 488.534955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 488.542232] Lustre: 17050:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 488.629527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 490.045788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 490.055995] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 490.172852] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 495.151958] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 504.063798] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 508.746833] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 508.928437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 513.900895] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 514.078511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 516.067091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.084554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 516.087240] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 516.101405] Lustre: Skipped 3 previous similar messages [ 530.399126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 530.527654] Lustre: server umount lustre-OST0000 complete [ 535.008688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 535.024678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 535.043859] Lustre: Skipped 1 previous similar message [ 535.059608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 535.074762] Lustre: Skipped 4 previous similar messages [ 548.320487] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 548.629331] Lustre: server umount lustre-MDT0000 complete [ 551.904900] LustreError: 15089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.922688] LustreError: 15089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 552.611792] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665859 with bad export cookie 18334108443488999251 [ 552.613157] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 552.632312] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 553.103969] Lustre: server umount lustre-MDT0001 complete [ 557.274231] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 23:37:43 (1788665863) [ 559.698347] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 561.463392] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 23:37:47 (1788665867) [ 567.727184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.322182] LustreError: 19462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.379988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 572.846555] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 581.416970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 586.814244] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 594.418973] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 599.732707] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 606.093125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 612.776702] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 619.924389] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 628.195617] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 628.207754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.231605] Lustre: Skipped 1 previous similar message [ 628.252541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 628.258334] Lustre: Skipped 6 previous similar messages [ 633.866952] Lustre: server umount lustre-OST0000 complete [ 638.432140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 638.450049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.467458] Lustre: Skipped 1 previous similar message [ 643.828458] Lustre: server umount lustre-MDT0000 complete [ 644.577459] LustreError: 19463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 644.598605] LustreError: 19463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 647.570796] LustreError: 19443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665954 with bad export cookie 18334108443489000133 [ 647.582752] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 647.584274] LustreError: 19443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 648.042935] Lustre: server umount lustre-MDT0001 complete [ 658.633935] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 661.418280] Key type lgssc unregistered [ 661.778786] LNet: 22992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 661.788206] LNetError: 22992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 661.817271] LNet: Removed LNI 192.168.203.146@tcp [ 662.853266] Key type .llcrypt unregistered [ 662.856826] Key type ._llcrypt unregistered [ 684.723644] Key type ._llcrypt registered [ 684.726213] Key type .llcrypt registered [ 684.874789] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 685.870599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 685.958751] alg: No test for adler32 (adler32-zlib) [ 687.019711] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 687.274626] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 688.983175] Key type lgssc registered [ 689.989314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 697.244467] loop: module loaded [ 697.342402] loop0: detected capacity change from 0 to 204800000 [ 697.562872] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 697.595620] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 697.979025] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.446181] loop0: detected capacity change from 0 to 204800000 [ 703.663877] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.682896] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.694230] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 708.402794] loop0: detected capacity change from 0 to 204800000 [ 713.466091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 713.493250] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 714.828296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 714.877796] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 715.005592] Lustre: lustre-MDT0000: new disk, initializing [ 715.123891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 715.149268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 719.765453] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 727.212935] loop1: detected capacity change from 0 to 204800000 [ 732.035913] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 732.288610] Lustre: lustre-OST07c6: new disk, initializing [ 732.293127] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 732.298393] Lustre: Skipped 1 previous similar message [ 732.302626] Lustre: 26177:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 732.375646] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 733.287571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 733.298574] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 733.334049] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 737.769864] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 742.865288] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 742.880067] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 742.883978] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 742.889163] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 742.900402] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 742.910988] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 762.540540] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 762.553418] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 762.567280] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 762.578671] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 762.591257] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 762.602218] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 767.245261] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 767.261618] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 767.276189] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 767.286868] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 767.299646] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 767.321432] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 767.335670] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 767.345194] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 767.353203] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 767.361758] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 767.369351] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 767.377676] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.584174] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 770.596281] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 770.605503] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 770.616995] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.629102] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 770.635098] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.644323] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 770.651153] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.655878] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 770.660526] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.665995] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 770.670691] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.702882] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 775.717017] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 775.726273] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 775.732896] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.747552] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 775.757460] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.763329] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 775.770143] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.775546] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 775.781541] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 775.787511] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 775.792622] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.743281] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 786.754582] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 786.773330] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 786.785373] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 786.798077] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 786.803791] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 786.811153] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 786.827508] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 786.842378] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 786.850184] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 786.860174] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 786.868351] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 835.558468] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 835.574172] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 835.594645] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 840.674790] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 845.793417] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 847.327168] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 847.497081] Lustre: server umount lustre-OST07c6 complete [ 856.594738] Lustre: server umount lustre-MDT0000 complete [ 871.742400] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 874.673575] Key type lgssc unregistered [ 875.091598] LNet: 28734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.100692] LNetError: 28734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.122383] LNet: Removed LNI 192.168.203.146@tcp [ 876.087741] Key type .llcrypt unregistered [ 876.094838] Key type ._llcrypt unregistered [ 897.995347] Key type ._llcrypt registered [ 897.997585] Key type .llcrypt registered [ 898.176080] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 899.576438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 899.609431] alg: No test for adler32 (adler32-zlib) [ 900.792688] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 901.224274] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 902.975472] Key type lgssc registered [ 904.224668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 912.446876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 912.475342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 914.085398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 918.518372] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 924.644414] LustreError: 30004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 924.702632] LustreError: 30004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 927.584727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 927.978246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 932.625504] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 941.189220] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 946.802712] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 953.691551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 953.970809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 959.106623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 960.281875] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 967.843561] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 988.132950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 988.138536] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 988.138764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 988.148144] Lustre: Skipped 1 previous similar message [ 989.602479] Lustre: server umount lustre-OST0000 complete [ 993.249833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.265253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 993.271462] Lustre: Skipped 1 previous similar message [ 993.288848] Lustre: Skipped 2 previous similar messages [ 998.370069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 998.374689] Lustre: Skipped 1 previous similar message [ 999.277304] Lustre: server umount lustre-MDT0000 complete [ 1002.916376] LustreError: 29983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666309 with bad export cookie 17885609323035419786 [ 1002.924885] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1002.936445] LustreError: 29983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1003.367991] Lustre: server umount lustre-MDT0001 complete [ 1013.193595] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 1015.771518] Key type lgssc unregistered [ 1016.073270] LNet: 33738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.082380] LNetError: 33738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1016.098993] LNet: Removed LNI 192.168.203.146@tcp [ 1016.859627] Key type .llcrypt unregistered [ 1016.863779] Key type ._llcrypt unregistered [ 1034.357342] Key type ._llcrypt registered [ 1034.359233] Key type .llcrypt registered [ 1034.437437] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 1035.409797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1035.444082] alg: No test for adler32 (adler32-zlib) [ 1036.506948] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 1036.716448] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 1038.367182] Key type lgssc registered [ 1039.236755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1046.429720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1046.465510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1047.992709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1051.982886] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 1058.281271] LustreError: 35006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1058.308495] LustreError: 35006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1060.900529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1061.189429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1065.936544] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 1074.963241] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1081.140674] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1087.912275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1088.230569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1093.368198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1093.494903] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 1100.668665] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1114.085771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1114.096755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1114.118629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1117.156970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1117.162842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1117.166876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1117.188543] Lustre: Skipped 1 previous similar message [ 1120.175404] Lustre: server umount lustre-OST0000 complete [ 1127.392796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1127.415956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1129.780811] Lustre: server umount lustre-MDT0000 complete [ 1132.512312] LustreError: 35006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1132.531468] LustreError: 35006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1133.286663] LustreError: 34987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666440 with bad export cookie 12991793468628352747 [ 1133.289343] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1133.299858] LustreError: 34987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1133.635883] Lustre: server umount lustre-MDT0001 complete [ 1142.746309] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 1145.392292] Key type lgssc unregistered [ 1145.654848] LNet: 38557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1145.669616] LNetError: 38557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1145.712827] LNet: Removed LNI 192.168.203.146@tcp [ 1146.366232] Key type .llcrypt unregistered [ 1146.373253] Key type ._llcrypt unregistered [ 1168.338977] Key type ._llcrypt registered [ 1168.341919] Key type .llcrypt registered [ 1168.480697] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 1170.176179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1170.277285] alg: No test for adler32 (adler32-zlib) [ 1171.344991] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 1171.618569] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 1173.327427] Key type lgssc registered [ 1175.136337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1185.497226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1185.550418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1187.289469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1193.323423] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 1197.543383] LustreError: 39969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1197.564138] LustreError: 39969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1202.660913] LustreError: 39970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1202.936621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1203.299613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1207.561907] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 1216.072176] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1221.010644] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1227.253291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1227.618832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1232.769445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1234.602512] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 1242.064272] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1249.249669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1249.264078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1249.288319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1249.804216] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.46@tcp (stopping) [ 1253.345205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1253.345543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1253.362715] Lustre: Skipped 1 previous similar message [ 1253.382284] Lustre: Skipped 2 previous similar messages [ 1255.242804] Lustre: server umount lustre-MDT0000 complete [ 1258.473741] LustreError: 39969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1258.509980] LustreError: 39969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1259.639239] LustreError: 41813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666566 with bad export cookie 17655670099910792315 [ 1259.640680] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1259.652250] LustreError: 41813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1260.045518] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.46@tcp (stopping) [ 1260.056388] Lustre: Skipped 1 previous similar message [ 1263.585557] LustreError: 39965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1263.586030] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1263.608891] LustreError: 39965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1264.612951] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1264.621602] Lustre: Skipped 1 previous similar message [ 1266.147403] Lustre: server umount lustre-MDT0001 complete [ 1287.688067] Lustre: server umount lustre-OST0000 complete [ 1305.273568] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 1308.179666] Key type lgssc unregistered [ 1308.420682] LNet: 43828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1308.434385] LNetError: 43828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1308.448933] LNet: Removed LNI 192.168.203.146@tcp [ 1309.250203] Key type .llcrypt unregistered [ 1309.255653] Key type ._llcrypt unregistered [ 1333.333608] Key type ._llcrypt registered [ 1333.338420] Key type .llcrypt registered [ 1333.477739] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 1334.345334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1334.428727] alg: No test for adler32 (adler32-zlib) [ 1335.681431] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 1335.942146] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 1337.671662] Key type lgssc registered [ 1338.767342] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1346.290912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1346.309972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1347.789397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1352.504267] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 1357.288634] LustreError: 45241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1357.313610] LustreError: 45241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1361.215644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1361.639972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1366.399833] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 1373.331479] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1379.174318] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1386.107363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1386.498629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1392.625787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1392.851517] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 1399.786835] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1406.948221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1406.958577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1406.973193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1407.520379] Lustre: lustre-OST0000: Not available for connect from 192.168.203.46@tcp (stopping) [ 1407.977689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1408.002133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1412.600981] Lustre: lustre-OST0000: Not available for connect from 192.168.203.46@tcp (stopping) [ 1412.611283] Lustre: Skipped 2 previous similar messages [ 1417.718884] Lustre: lustre-OST0000: Not available for connect from 192.168.203.46@tcp (stopping) [ 1417.728055] Lustre: Skipped 2 previous similar messages [ 1421.279305] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1421.531237] Lustre: server umount lustre-OST0000 complete [ 1443.808830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1443.809562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1443.817130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1443.847881] Lustre: Skipped 1 previous similar message [ 1443.864553] Lustre: Skipped 3 previous similar messages [ 1447.283148] Lustre: server umount lustre-MDT0000 complete [ 1447.969158] LustreError: 47396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1447.983850] LustreError: 47396:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1451.210829] LustreError: 45222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666758 with bad export cookie 11821284670134562763 [ 1451.214787] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1451.231797] LustreError: 45222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1457.654128] Lustre: server umount lustre-MDT0001 complete [ 1468.521893] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 1471.405553] Key type lgssc unregistered [ 1471.679450] LNet: 48950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1471.686470] LNetError: 48950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1471.705349] LNet: Removed LNI 192.168.203.146@tcp [ 1472.512990] Key type .llcrypt unregistered [ 1472.522525] Key type ._llcrypt unregistered [ 1495.318954] Key type ._llcrypt registered [ 1495.322753] Key type .llcrypt registered [ 1495.453924] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 1496.846386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1496.883827] alg: No test for adler32 (adler32-zlib) [ 1497.989770] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 1498.318695] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 1499.991370] Key type lgssc registered [ 1501.090894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1508.535848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1508.551309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1510.038817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1513.684952] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 1520.627680] LustreError: 50382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1520.647289] LustreError: 50382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1521.323961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1521.658521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1526.086198] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 1533.240198] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1538.501257] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1545.227301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1545.616185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1551.264401] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 1555.902070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1558.522354] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1564.770933] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1568.432342] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1580.573034] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0523h54m45s [ 1582.154445] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1586.662959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1586.668708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1586.685816] Lustre: Skipped 2 previous similar messages [ 1586.692388] Lustre: Skipped 1 previous similar message [ 1587.755179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1587.767802] Lustre: Skipped 5 previous similar messages [ 1590.483508] Lustre: server umount lustre-MDT0000 complete [ 1592.826169] LustreError: 50379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1593.643521] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1593.891234] LustreError: 50377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1593.910874] LustreError: 50377:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1598.969077] LustreError: 50378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1600.215180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1600.314232] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1600.543411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1600.625205] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1601.615753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1604.794408] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 1608.822248] Lustre: DEBUG MARKER: Wait for df (36687) ... [ 1610.968034] Lustre: DEBUG MARKER: done [ 1616.866930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1616.895297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1616.909486] Lustre: Skipped 1 previous similar message [ 1621.987615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1622.006720] Lustre: Skipped 2 previous similar messages [ 1622.229370] Lustre: server umount lustre-OST0000 complete [ 1627.104656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1627.138483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1627.158257] Lustre: Skipped 1 previous similar message [ 1632.427139] Lustre: server umount lustre-MDT0000 complete [ 1633.256444] LustreError: 50383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1633.285532] LustreError: 50383:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1636.077820] LustreError: 50362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666943 with bad export cookie 12705656550170744886 [ 1636.081313] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1636.097650] LustreError: 50362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1636.756227] Lustre: server umount lustre-MDT0001 complete [ 1647.278085] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 1649.913058] Key type lgssc unregistered [ 1650.209746] LNet: 55305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1650.220701] LNetError: 55305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1650.236792] LNet: Removed LNI 192.168.203.146@tcp [ 1651.025113] Key type .llcrypt unregistered [ 1651.030690] Key type ._llcrypt unregistered [ 1686.975655] Key type ._llcrypt registered [ 1686.979977] Key type .llcrypt registered [ 1687.078492] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 1688.446261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1688.463495] alg: No test for adler32 (adler32-zlib) [ 1689.471985] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 1689.718845] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 1691.439697] Key type lgssc registered [ 1692.620753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1700.932120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1700.963192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1702.321278] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1702.343596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1702.477016] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1706.736625] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 1716.108109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1716.177274] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1716.204872] Lustre: 57506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1716.523021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1721.312567] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 1728.544291] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1733.785803] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1740.159567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1740.435176] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1740.783241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1746.623786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1747.268699] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 1754.592923] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1761.304888] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0523h57m47s [ 1762.834305] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1769.815546] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1772.667895] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1772.717950] LustreError: 57515:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1793.079804] Lustre: lustre-MDT0000: Client d3193f34-9f49-49f1-bce6-f66f0a6c9181 (at 192.168.203.46@tcp) reconnecting [ 1812.807317] LustreError: 57515:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1815.438901] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1818.476335] Lustre: DEBUG MARKER: done [ 1828.328784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1828.348242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1828.376177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1828.842641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1831.230605] Lustre: server umount lustre-OST0000 complete [ 1839.075459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.084674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1839.105929] Lustre: Skipped 2 previous similar messages [ 1839.120959] Lustre: Skipped 2 previous similar messages [ 1840.171314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1840.176829] Lustre: Skipped 1 previous similar message [ 1846.249643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1849.311185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1849.563967] Lustre: server umount lustre-MDT0000 complete [ 1850.401451] LustreError: 57519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1850.423709] LustreError: 57519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1853.406159] LustreError: 57500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667160 with bad export cookie 2594195812600816150 [ 1853.412338] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1853.419246] LustreError: 57500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1853.786741] Lustre: server umount lustre-MDT0001 complete [ 1864.635493] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 1867.456605] Key type lgssc unregistered [ 1867.746436] LNet: 61835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1867.754958] LNetError: 61835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1867.778515] LNet: Removed LNI 192.168.203.146@tcp [ 1868.722199] Key type .llcrypt unregistered [ 1868.726110] Key type ._llcrypt unregistered [ 1909.551960] Key type ._llcrypt registered [ 1909.553962] Key type .llcrypt registered [ 1909.640606] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 1910.718116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1910.885415] alg: No test for adler32 (adler32-zlib) [ 1911.910570] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 1912.108750] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 1913.759176] Key type lgssc registered [ 1915.042700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1918.642360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1918.672086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1920.035659] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1920.095146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1920.314038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1921.464907] Lustre: Failing over lustre-MDT0000 [ 1921.936514] Lustre: server umount lustre-MDT0000 complete [ 1928.258119] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 00:00:34 (1788667234) [ 1934.779140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.296466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1939.985830] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 1949.773620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1949.940791] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1949.977836] Lustre: 64633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1950.236294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1955.046535] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 1962.332763] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1968.397905] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1976.337057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1976.644044] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1976.947791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1984.396632] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 1985.754328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1992.913532] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1999.111165] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2011.618054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2011.632391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.653759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2011.658722] Lustre: Skipped 1 previous similar message [ 2013.251469] Lustre: server umount lustre-MDT0000 complete [ 2016.996172] LustreError: 64624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667324 with bad export cookie 8197387244193434319 [ 2017.006792] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2017.012957] LustreError: 64624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2017.440889] Lustre: server umount lustre-MDT0001 complete [ 2020.599642] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2025.123924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2033.998980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2036.959156] Lustre: 63555:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788667328/real 1788667328] req@00000000587223f8 x1875553517872384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788667344 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2036.998360] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.025023] Lustre: Skipped 2 previous similar messages [ 2038.239157] Lustre: 63552:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788667329/real 1788667329] req@00000000fd9f7862 x1875553517872512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788667345 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2042.381535] LustreError: 66493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2042.398487] LustreError: 66493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2042.404570] Lustre: 63552:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788667333/real 1788667333] req@000000008adec31b x1875553517872640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788667349 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2042.482526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2042.598316] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2046.500060] LustreError: 66494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2047.525486] LustreError: 68413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2047.559180] LustreError: 68413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2047.646487] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 2048.543440] Lustre: 63554:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788667339/real 1788667339] req@0000000091d6efbb x1875553517872896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788667355 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2052.577056] LustreError: 66493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2052.594816] LustreError: 66493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2056.916111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2057.478983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2062.831403] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2063.035164] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 2065.960654] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2072.233158] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2078.781673] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2086.369507] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2086.371474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2086.381927] Lustre: Skipped 1 previous similar message [ 2086.389096] Lustre: Skipped 3 previous similar messages [ 2091.489970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2091.509695] Lustre: Skipped 2 previous similar messages [ 2092.276514] Lustre: server umount lustre-MDT0000 complete [ 2096.614714] LustreError: 69150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2096.638504] LustreError: 69150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2096.730267] LustreError: 64624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667403 with bad export cookie 8197387244193436594 [ 2096.747398] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2096.755113] LustreError: 64624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2097.145752] Lustre: server umount lustre-MDT0001 complete [ 2104.459174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2114.613605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.087185] Lustre: 63552:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788667408/real 1788667408] req@000000002b4f5809 x1875553517901312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788667424 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2117.116588] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2117.132245] Lustre: Skipped 1 previous similar message [ 2121.248109] LustreError: 63551:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007f237466 x1875553517902080/t0(0) o250->MGC192.168.203.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2121.586524] LustreError: 66493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2121.678820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2121.738028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2125.950443] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 2126.777180] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2128.869326] Lustre: 63554:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788667419/real 1788667419] req@000000001031dfb6 x1875553517901696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788667435 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2128.923685] Lustre: 63554:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2134.691578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2139.706874] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 2140.151301] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2146.696625] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2152.863933] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2159.509590] Lustre: server umount lustre-MDT0000 complete [ 2160.610678] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2160.617542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2160.622386] LustreError: 71098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2160.622401] LustreError: 71098:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2160.627895] Lustre: Skipped 1 previous similar message [ 2163.298689] LustreError: 64623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667470 with bad export cookie 8197387244193437973 [ 2163.315058] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2163.777985] Lustre: server umount lustre-MDT0001 complete [ 2170.925376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2175.230256] Lustre: DEBUG MARKER: files compared the same [ 2188.155397] Lustre: server umount lustre-OST0000 complete [ 2204.142393] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 2207.156835] Key type lgssc unregistered [ 2207.469699] LNet: 74375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2207.477434] LNetError: 74375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2207.503893] LNet: Removed LNI 192.168.203.146@tcp [ 2208.423103] Key type .llcrypt unregistered [ 2208.428663] Key type ._llcrypt unregistered [ 2230.407797] Key type ._llcrypt registered [ 2230.410837] Key type .llcrypt registered [ 2230.503227] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 2231.286221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2231.400519] alg: No test for adler32 (adler32-zlib) [ 2232.567317] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 2232.794794] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 2234.455240] Key type lgssc registered [ 2235.474944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2242.527432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2242.556082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2243.929744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2247.670301] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug +malloc all [ 2253.279926] LustreError: 75790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2253.306827] LustreError: 75790:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2256.273744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.585034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2261.235104] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug +malloc all [ 2268.847568] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2274.027819] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2280.412756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2280.735185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2286.862853] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug +malloc all [ 2288.877259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2293.979220] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2302.432064] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2302.444076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2302.464561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2304.481151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2304.481206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2307.922661] Lustre: server umount lustre-OST0000 complete [ 2313.194297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2313.208219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2313.230864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2313.241373] Lustre: Skipped 1 previous similar message [ 2318.204621] Lustre: server umount lustre-MDT0000 complete [ 2319.840366] LustreError: 75789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2319.864122] LustreError: 75789:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2322.649034] LustreError: 75769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667629 with bad export cookie 2868490232707609551 [ 2322.657764] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2323.119864] Lustre: server umount lustre-MDT0001 complete [ 2335.566656] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 2338.748865] Key type lgssc unregistered [ 2339.083033] LNet: 79320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2339.105320] LNetError: 79320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2339.132797] LNet: Removed LNI 192.168.203.146@tcp [ 2340.367367] Key type .llcrypt unregistered [ 2340.370904] Key type ._llcrypt unregistered [ 2365.563989] Key type ._llcrypt registered [ 2365.566395] Key type .llcrypt registered [ 2365.654237] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 2366.812281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2366.940811] alg: No test for adler32 (adler32-zlib) [ 2367.991743] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 2368.308330] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 2369.983170] Key type lgssc registered [ 2371.457760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2380.987114] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2381.022712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2396.640679] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2401.759297] LustreError: 80712:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.146@tcp: failed processing log, type 4: rc = -110 [ 2428.447254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2435.018875] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 2442.533926] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2450.036597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2450.169953] Lustre: *** cfs_fail_loc=706, val=0*** [ 2450.571970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2450.661051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2455.145228] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 2456.037362] LustreError: 80738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2456.068712] LustreError: 80738:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2461.155289] LustreError: 80737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2463.218519] LustreError: 81797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2463.246630] LustreError: 81797:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2463.390828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2463.738097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2467.922166] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 2474.558847] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2480.254981] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2484.195648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2484.196594] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2484.219411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2484.235307] Lustre: Skipped 1 previous similar message [ 2488.422854] Lustre: server umount lustre-OST0000 complete [ 2494.434726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2494.438551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2494.454743] Lustre: Skipped 1 previous similar message [ 2494.460305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2494.478405] Lustre: Skipped 1 previous similar message [ 2498.804811] Lustre: server umount lustre-MDT0000 complete [ 2499.555717] LustreError: 83577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2499.585277] LustreError: 83577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2502.982175] LustreError: 82537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667810 with bad export cookie 3911876197342600910 [ 2503.000566] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2503.300709] Lustre: server umount lustre-MDT0001 complete [ 2514.915271] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 2517.727892] Key type lgssc unregistered [ 2518.142586] LNet: 84327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2518.157540] LNetError: 84327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2519.205970] LNet: Removed LNI 192.168.203.146@tcp [ 2520.253448] Key type .llcrypt unregistered [ 2520.256313] Key type ._llcrypt unregistered [ 2545.614288] Key type ._llcrypt registered [ 2545.620731] Key type .llcrypt registered [ 2545.725223] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 2546.703151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2546.743174] alg: No test for adler32 (adler32-zlib) [ 2547.886501] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 2548.137035] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 2549.799243] Key type lgssc registered [ 2550.931844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2557.832147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2557.874532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2563.441237] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 2570.890093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2571.372052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2575.490711] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 2576.866149] LustreError: 86427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2576.887705] LustreError: 86427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2580.981716] LustreError: 86431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2581.003543] LustreError: 86431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2583.638389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2584.042096] LustreError: 86430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.063490] LustreError: 86430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2584.073957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2589.161914] LustreError: 86431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.183919] LustreError: 86431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2589.640363] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 2594.279900] LustreError: 86773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2594.299829] LustreError: 86773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2598.803890] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2598.831948] Lustre: MGS: Client 5cf7080d-fe82-4f94-934b-d43ed69ccef3 (at 0@lo) reconnecting [ 2598.876647] LustreError: 86441:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2599.188428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2599.255375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2604.728970] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 2619.886757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2619.902833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2619.910612] LustreError: Skipped 1 previous similar message [ 2619.918491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2619.973917] Lustre: Skipped 1 previous similar message [ 2624.992408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2625.004657] Lustre: Skipped 1 previous similar message [ 2625.792586] Lustre: server umount lustre-OST0000 complete [ 2635.240895] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2635.241748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2635.242868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2635.254912] Lustre: Skipped 1 previous similar message [ 2635.277403] Lustre: Skipped 2 previous similar messages [ 2636.972707] Lustre: server umount lustre-MDT0000 complete [ 2640.356533] LustreError: 86430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.400576] LustreError: 86430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2642.132996] Lustre: server umount lustre-MDT0001 complete [ 2646.302465] Lustre: server umount MGS complete [ 2658.532954] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 00:12:44 (1788667964) [ 2687.633781] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [ 2696.504458] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 2730.776369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2735.512950] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 2744.739030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2744.993577] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2745.042813] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2745.111717] Lustre: lustre-MDT0001: new disk, initializing [ 2745.193494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2745.221429] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2745.231839] Lustre: 93615:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2745.240899] LustreError: 93615:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2750.442581] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 2765.175856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2765.450834] Lustre: lustre-OST0000: new disk, initializing [ 2765.461376] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2765.470350] Lustre: 94381:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2765.591691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2772.733290] Lustre: 94560:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000006508c1d1 x1875554184766208/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788668095 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2775.279442] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 2786.663041] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2793.439885] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2793.535344] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2793.542516] Lustre: MGS: Client fb5ff107-18fe-4090-b452-992a9b80cacb (at 0@lo) reconnecting [ 2793.584241] Lustre: 92883:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2793.599700] LustreError: 93601:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2793.616288] LustreError: 93601:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2793.634309] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2793.726540] Lustre: lustre-MDT0000: new disk, initializing [ 2793.927939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2793.952408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2794.976996] Lustre: 95451:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000008eba1e9f x1875554184773632/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788668118 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2798.571461] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 2800.224306] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2804.138939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2804.152668] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2804.299546] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2829.282426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2829.296229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2829.311757] Lustre: Skipped 1 previous similar message [ 2829.331881] Lustre: Skipped 1 previous similar message [ 2833.968163] Lustre: server umount lustre-OST0000 complete [ 2839.552485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2839.559169] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2839.571961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2839.594704] Lustre: Skipped 1 previous similar message [ 2839.611258] Lustre: Skipped 1 previous similar message [ 2844.857354] Lustre: server umount lustre-MDT0000 complete [ 2854.736490] Lustre: server umount lustre-MDT0001 complete [ 2858.033650] Lustre: server umount MGS complete [ 2869.674410] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 00:16:14 (1788668174) [ 2890.431352] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 2893.579361] Key type lgssc unregistered [ 2894.028319] LNet: 97949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2894.042713] LNetError: 97949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2894.072913] LNet: Removed LNI 192.168.203.146@tcp [ 2895.091220] Key type .llcrypt unregistered [ 2895.094672] Key type ._llcrypt unregistered [ 2914.021307] Key type ._llcrypt registered [ 2914.022940] Key type .llcrypt registered [ 2914.158375] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 2916.021660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2916.087533] alg: No test for adler32 (adler32-zlib) [ 2917.245663] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 2917.531724] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 2919.279184] Key type lgssc registered [ 2920.622551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2927.846407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2927.876504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2929.368609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2933.629713] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 2939.875235] LustreError: 99167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2939.902936] LustreError: 99167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2941.666449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2942.032701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2946.399183] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 2954.598492] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2960.254370] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2967.197737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2967.550866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2974.520851] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 2974.705441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2983.686947] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2999.219966] Lustre: server umount lustre-OST0000 complete [ 3006.444712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3006.446586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3006.482516] Lustre: Skipped 1 previous similar message [ 3006.504273] Lustre: Skipped 1 previous similar message [ 3011.553871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3011.572888] Lustre: Skipped 1 previous similar message [ 3015.811815] Lustre: server umount lustre-MDT0000 complete [ 3016.672942] LustreError: 99167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3016.690209] LustreError: 99167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3017.572779] Lustre: 98478:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788668308/real 1788668308] req@00000000147c9a3f x1875554571679872/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788668324 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3017.612190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3018.656028] Lustre: 98479:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788668308/real 1788668308] req@00000000caa7eeed x1875554571680256/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788668324 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3020.248193] LustreError: 99147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668327 with bad export cookie 9167323569863630989 [ 3020.254107] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3020.268809] LustreError: 99147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3020.685044] Lustre: server umount lustre-MDT0001 complete [ 3033.437523] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 3036.356575] Key type lgssc unregistered [ 3036.666797] LNet: 102718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3036.677331] LNetError: 102718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3036.703791] LNet: Removed LNI 192.168.203.146@tcp [ 3037.577553] Key type .llcrypt unregistered [ 3037.581392] Key type ._llcrypt unregistered [ 3057.206538] Key type ._llcrypt registered [ 3057.209485] Key type .llcrypt registered [ 3057.339102] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 3058.605412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3058.646256] alg: No test for adler32 (adler32-zlib) [ 3059.810841] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 3060.238119] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 3061.887227] Key type lgssc registered [ 3063.160576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3069.420670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3069.420744] LustreError: 103800:0:(tgt_mount.c:2575:server_fill_super()) cfs_race id 716 sleeping [ 3069.437140] Lustre: Skipped 1 previous similar message [ 3069.450191] LustreError: 103801:0:(tgt_mount.c:2575:server_fill_super()) cfs_fail_race id 716 waking [ 3069.456665] LustreError: 103800:0:(tgt_mount.c:2575:server_fill_super()) cfs_fail_race id 716 awake: rc=4988 [ 3069.467689] LustreError: 103800:0:(tgt_mount.c:2587:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3069.479738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3069.484281] LustreError: 103800:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3071.063495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3078.942573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3079.209715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3083.131838] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 3088.772921] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing lsmod [ 3091.545365] LustreError: 105000:0:(tgt_mount.c:2575:server_fill_super()) cfs_race id 716 sleeping [ 3091.577803] LustreError: 105007:0:(tgt_mount.c:2575:server_fill_super()) cfs_fail_race id 716 waking [ 3091.587474] LustreError: 105000:0:(tgt_mount.c:2575:server_fill_super()) cfs_fail_race id 716 awake: rc=4968 [ 3091.594916] LustreError: 105000:0:(tgt_mount.c:2587:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3091.608129] LustreError: 105000:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3091.624708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3092.207065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3097.570421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3097.581051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3097.587738] Lustre: Skipped 1 previous similar message [ 3097.600674] Lustre: Skipped 2 previous similar messages [ 3102.688471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3102.699556] Lustre: Skipped 2 previous similar messages [ 3104.739356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3109.343966] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3109.598303] Lustre: server umount lustre-MDT0000 complete [ 3109.865069] LustreError: 105194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3109.896566] LustreError: 105194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3113.854578] LustreError: 103817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668420 with bad export cookie 234352151470241088 [ 3113.877984] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3113.882763] LustreError: 103817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3114.995542] LustreError: 103835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3115.006569] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3115.060771] Lustre: Skipped 1 previous similar message [ 3115.061230] LustreError: 104269:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a95cc7d4 x1875554721617664/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3115.064815] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3115.064822] Lustre: Skipped 2 previous similar messages [ 3115.275850] Lustre: server umount lustre-MDT0001 complete [ 3122.121751] Lustre: server umount lustre-OST0000 complete [ 3132.378373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3132.927337] LustreError: 106073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3133.018190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3137.611387] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 3138.019966] LustreError: 106074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3143.157129] LustreError: 106073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3146.729309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3147.181484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3151.736916] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 3159.514594] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3165.831815] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3173.681850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3174.083879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3178.172285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3180.695270] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 3188.874285] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3202.032648] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3202.052534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3202.074125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3203.053628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3208.253536] Lustre: server umount lustre-OST0000 complete [ 3213.282723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3213.285362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3213.308299] Lustre: Skipped 2 previous similar messages [ 3213.318347] Lustre: Skipped 2 previous similar messages [ 3218.850461] Lustre: server umount lustre-MDT0000 complete [ 3222.568547] LustreError: 106055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668529 with bad export cookie 234352151470241802 [ 3222.587875] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3223.040465] Lustre: server umount lustre-MDT0001 complete [ 3235.943530] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 3239.301487] Key type lgssc unregistered [ 3239.638220] LNet: 109627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3239.643269] LNetError: 109627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3239.665730] LNet: Removed LNI 192.168.203.146@tcp [ 3240.778499] Key type .llcrypt unregistered [ 3240.784903] Key type ._llcrypt unregistered [ 3268.175237] Key type ._llcrypt registered [ 3268.182431] Key type .llcrypt registered [ 3268.358443] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 3269.803560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3269.820249] alg: No test for adler32 (adler32-zlib) [ 3270.935293] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 3271.206421] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 3272.879203] Key type lgssc registered [ 3274.349837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3282.449731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3282.470139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3283.872807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3288.758365] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 3293.161614] LustreError: 111057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3293.188989] LustreError: 111057:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3298.279773] LustreError: 111058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3298.835971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3299.284510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3304.103392] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 3311.736715] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3317.099376] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3324.223710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3324.669787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3328.760881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3329.819775] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 3337.164241] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3343.019305] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3347.049996] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3354.592754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3354.598444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3354.631267] Lustre: Skipped 1 previous similar message [ 3359.177943] Lustre: server umount lustre-OST0000 complete [ 3364.839982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3364.874761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3364.877853] Lustre: Skipped 1 previous similar message [ 3369.953200] LustreError: 111058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3370.113150] Lustre: server umount lustre-MDT0000 complete [ 3374.049575] LustreError: 111052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3374.085073] LustreError: 111052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3374.179150] LustreError: 113590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668681 with bad export cookie 12625978880163219634 [ 3374.180039] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3374.189623] LustreError: 113590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3374.641958] Lustre: server umount lustre-MDT0001 complete [ 3388.040760] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 3390.528905] Key type lgssc unregistered [ 3390.804681] LNet: 114701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3390.814671] LNetError: 114701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3390.836122] LNet: Removed LNI 192.168.203.146@tcp [ 3391.677974] Key type .llcrypt unregistered [ 3391.682747] Key type ._llcrypt unregistered [ 3409.364805] Key type ._llcrypt registered [ 3409.368637] Key type .llcrypt registered [ 3409.470438] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 3410.853776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3410.873748] alg: No test for adler32 (adler32-zlib) [ 3412.075434] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 3412.486671] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 3414.175156] Key type lgssc registered [ 3415.379452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3424.090277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3424.115828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3425.754130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3430.708371] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 3434.979296] LustreError: 115928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3435.003198] LustreError: 115928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3440.099247] LustreError: 115927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3441.221726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3441.494748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3447.065042] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 3455.499088] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3461.712684] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3469.681237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3470.105768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3471.159880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3476.538803] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 3484.740913] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3496.928482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3496.933302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3496.944441] LustreError: Skipped 1 previous similar message [ 3496.949772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3496.979488] Lustre: Skipped 1 previous similar message [ 3500.354972] Lustre: server umount lustre-OST0000 complete [ 3507.169683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3507.177752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3507.187449] Lustre: Skipped 1 previous similar message [ 3507.198219] Lustre: Skipped 2 previous similar messages [ 3511.052469] Lustre: server umount lustre-MDT0000 complete [ 3512.289587] LustreError: 115922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3512.329660] LustreError: 115922:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3514.440898] LustreError: 115909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668821 with bad export cookie 3329540709114166759 [ 3514.445245] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3514.454334] LustreError: 115909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3514.674690] Lustre: server umount lustre-MDT0001 complete [ 3525.933447] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 3528.867610] Key type lgssc unregistered [ 3529.169828] LNet: 119459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3529.179797] LNetError: 119459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3530.219760] LNet: Removed LNI 192.168.203.146@tcp [ 3531.114928] Key type .llcrypt unregistered [ 3531.116943] Key type ._llcrypt unregistered [ 3554.210877] Key type ._llcrypt registered [ 3554.213447] Key type .llcrypt registered [ 3554.312275] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 3555.545767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3555.574064] alg: No test for adler32 (adler32-zlib) [ 3556.724859] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 3557.071202] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 3558.735942] Key type lgssc registered [ 3559.981031] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3567.003151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3567.023904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3568.460101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3572.873249] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 3578.849683] LustreError: 120873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3578.876896] LustreError: 120873:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3582.791151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3583.189252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3587.660342] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 3595.087745] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3601.484105] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3608.255080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3608.678690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3613.815071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3614.978785] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 3621.807285] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3631.383167] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3635.245590] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3635.252772] Lustre: Skipped 2 previous similar messages [ 3639.374156] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3639.396473] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3639.603848] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3639.610519] Lustre: Skipped 2 previous similar messages [ 3646.008195] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3646.014565] Lustre: Skipped 1 previous similar message [ 3650.152945] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.46@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3650.173077] Lustre: Skipped 2 previous similar messages [ 3654.267923] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.46@tcp 192.168.0.[2,10]@tcp [ 3658.427187] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.46/24@tcp in log lustre-MDT0000 [ 3658.435853] Lustre: Skipped 2 previous similar messages [ 3663.453416] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.46/24@tcp [ 3663.464029] Lustre: Skipped 1 previous similar message [ 3674.593115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3674.597552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3674.618202] Lustre: Skipped 1 previous similar message [ 3674.618713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3674.625680] Lustre: Skipped 1 previous similar message [ 3677.368340] Lustre: server umount lustre-OST0000 complete [ 3684.833136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3684.836791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3684.849967] Lustre: Skipped 1 previous similar message [ 3684.864714] Lustre: Skipped 1 previous similar message [ 3687.936512] Lustre: server umount lustre-MDT0000 complete [ 3689.954992] LustreError: 121612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3689.980855] LustreError: 121612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3691.651893] LustreError: 120855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668998 with bad export cookie 15019583564062268268 [ 3691.666220] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3691.673198] LustreError: 120855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3691.909512] Lustre: server umount lustre-MDT0001 complete [ 3702.510650] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 3705.153558] Key type lgssc unregistered [ 3705.383429] LNet: 125816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3705.391815] LNetError: 125816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3705.422211] LNet: Removed LNI 192.168.203.146@tcp [ 3706.245874] Key type .llcrypt unregistered [ 3706.248522] Key type ._llcrypt unregistered [ 3733.889737] Key type ._llcrypt registered [ 3733.892657] Key type .llcrypt registered [ 3733.985646] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 3735.079240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3735.107984] alg: No test for adler32 (adler32-zlib) [ 3736.303451] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 3736.561515] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 3738.288611] Key type lgssc registered [ 3739.777120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3747.866669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3747.908484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3749.502905] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3749.534394] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.46/24@tcp [ 3749.540842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3753.588396] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 3759.077304] LustreError: 127376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3759.096190] LustreError: 127376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3761.847678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3762.155759] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3762.185952] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.46/24@tcp [ 3762.283534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3766.805046] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 3773.488081] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3779.247287] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3785.614501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3786.005950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3791.369387] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 3794.271079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3798.275131] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3808.740464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3808.751950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3808.784826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3809.773826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3809.774507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3809.812503] Lustre: Skipped 1 previous similar message [ 3814.382643] Lustre: server umount lustre-OST0000 complete [ 3818.469854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3818.478650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3818.496965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3823.011448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3823.016044] Lustre: Skipped 3 previous similar messages [ 3823.655578] Lustre: server umount lustre-MDT0000 complete [ 3826.873435] LustreError: 127358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669133 with bad export cookie 2404763825483428825 [ 3826.876410] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3826.883895] LustreError: 127358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3827.211744] Lustre: server umount lustre-MDT0001 complete [ 3836.870245] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 3839.718912] Key type lgssc unregistered [ 3840.138285] LNet: 130962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3840.145567] LNetError: 130962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3840.176234] LNet: Removed LNI 192.168.203.146@tcp [ 3840.959653] Key type .llcrypt unregistered [ 3840.964208] Key type ._llcrypt unregistered [ 3870.655642] Key type ._llcrypt registered [ 3870.659728] Key type .llcrypt registered [ 3870.767215] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [ 3885.049692] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 3885.925624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3885.972062] alg: No test for adler32 (adler32-zlib) [ 3887.152552] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 3887.414307] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 3889.095340] Key type lgssc registered [ 3890.700302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3928.980917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3929.029222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3930.311262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3930.340358] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3930.431171] Lustre: lustre-MDT0000: new disk, initializing [ 3930.490788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3930.513289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3934.838502] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 3944.362322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3944.483773] Lustre: 134357:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3944.527440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3944.530354] Lustre: Skipped 1 previous similar message [ 3944.601453] Lustre: lustre-MDT0001: new disk, initializing [ 3944.725659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3944.776589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3944.800609] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3948.846810] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 3956.812590] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3962.622938] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3969.146860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3969.458831] Lustre: lustre-OST0000: new disk, initializing [ 3969.462759] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3969.468635] Lustre: 136325:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3969.539175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3975.674645] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 3978.849110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3978.868288] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3978.992414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3985.850668] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3990.649321] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3990.825922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3995.477806] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3995.678043] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3999.712058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3999.725234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3999.730145] LustreError: Skipped 1 previous similar message [ 3999.731102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3999.769514] Lustre: Skipped 1 previous similar message [ 4003.803342] Lustre: server umount lustre-OST0000 complete [ 4009.970351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4009.979584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4009.982263] Lustre: Skipped 1 previous similar message [ 4010.001116] Lustre: Skipped 2 previous similar messages [ 4015.075045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4015.090930] Lustre: Skipped 1 previous similar message [ 4019.172048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4019.176560] Lustre: Skipped 1 previous similar message [ 4022.239157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4022.453352] Lustre: server umount lustre-MDT0000 complete [ 4026.497846] LustreError: 134350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669333 with bad export cookie 1466640176075141593 [ 4026.498561] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4026.530870] LustreError: 134350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4026.902252] Lustre: server umount lustre-MDT0001 complete [ 4034.962763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4035.350092] LustreError: 138546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4035.367180] LustreError: 138546:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4035.406672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4039.467952] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 4040.679040] LustreError: 138547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4045.797089] LustreError: 138546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4048.117335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4048.476151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4052.549390] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 4060.091961] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4065.557034] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4072.787103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4073.069602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4079.028985] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 4086.569294] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4090.918805] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4091.154804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4096.211846] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4096.570866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4107.373921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4107.610506] Lustre: lustre-OST0001: new disk, initializing [ 4107.615517] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4107.626146] Lustre: 141916:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4107.812548] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4114.486222] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 4115.591683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4115.602235] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4115.726609] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4122.577786] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4122.781970] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4127.339618] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4127.505456] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4133.646806] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8eb688c52800.ost_server_uuid 50 [ 4135.157528] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8eb688c52800.ost_server_uuid in IDLE state after 0 sec [ 4145.659694] Lustre: server umount lustre-OST0001 complete [ 4146.670612] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4146.677463] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4146.691127] LustreError: 140800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4146.715765] LustreError: 140800:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4151.266886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4151.276406] LustreError: Skipped 1 previous similar message [ 4151.279860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4151.292835] Lustre: Skipped 1 previous similar message [ 4151.299909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4151.786422] LustreError: 142597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4151.803026] LustreError: 142597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4155.304674] Lustre: server umount lustre-OST0000 complete [ 4160.992598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4160.999570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4161.013942] Lustre: Skipped 1 previous similar message [ 4161.020038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4161.028968] Lustre: Skipped 1 previous similar message [ 4173.279136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4173.513760] Lustre: server umount lustre-MDT0000 complete [ 4176.361249] LustreError: 139787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4176.393027] LustreError: 139787:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4177.299445] LustreError: 141926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669484 with bad export cookie 1466640176075142475 [ 4177.305702] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4177.314779] LustreError: 141926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4177.646498] Lustre: server umount lustre-MDT0001 complete [ 4197.862537] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 4201.129059] Key type lgssc unregistered [ 4201.432168] LNet: 144918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4201.457930] LNetError: 144918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4201.473940] LNet: Removed LNI 192.168.203.146@tcp [ 4202.590635] Key type .llcrypt unregistered [ 4202.595321] Key type ._llcrypt unregistered [ 4244.488897] Key type ._llcrypt registered [ 4244.492896] Key type .llcrypt registered [ 4244.620406] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [ 4259.586656] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 4260.910514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4260.928105] alg: No test for adler32 (adler32-zlib) [ 4262.156454] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 4262.558496] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 4264.265872] Key type lgssc registered [ 4265.290780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4299.300757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4299.315339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4300.646740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4300.671492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4300.737673] Lustre: lustre-MDT0000: new disk, initializing [ 4300.809940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4300.834694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4304.563880] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 4316.001173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4316.089851] Lustre: 149069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4316.136896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4316.139960] Lustre: Skipped 1 previous similar message [ 4316.192298] Lustre: lustre-MDT0001: new disk, initializing [ 4316.284510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4316.319342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4316.335093] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4321.278770] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 4331.204801] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4338.747720] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4346.421935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4346.754576] Lustre: lustre-OST0000: new disk, initializing [ 4346.767507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4346.786333] Lustre: 151037:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4346.907680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4354.174657] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 4355.618579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4355.629720] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4355.759184] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4363.411181] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4368.117724] Lustre: Failing over lustre-OST0000 [ 4368.262465] Lustre: server umount lustre-OST0000 complete [ 4385.737557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4386.021611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4386.039838] Lustre: Skipped 1 previous similar message [ 4386.058872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4386.074430] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4386.257443] Lustre: 147295:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788669677/real 1788669677] req@000000006fb32810 x1875555982017920/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788669693 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4386.895787] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4387.042958] Lustre: 147294:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788669677/real 1788669677] req@00000000d8b57f15 x1875555982018048/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788669693 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4387.061983] Lustre: 147294:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4391.448146] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4391.452276] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4392.178275] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 4392.415542] Lustre: 147295:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788669683/real 1788669683] req@00000000765dd9c1 x1875555982019456/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788669699 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4392.444444] Lustre: 147295:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4395.808336] Lustre: Failing over lustre-MDT0000 [ 4396.042422] Lustre: server umount lustre-MDT0000 complete [ 4396.524076] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4396.531770] LustreError: 149077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4396.541843] Lustre: Skipped 1 previous similar message [ 4396.563609] LustreError: 149077:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4397.088320] LustreError: 149076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4397.607976] Lustre: 147295:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788669688/real 1788669688] req@00000000b1b0df96 x1875555982020480/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788669704 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4397.644473] Lustre: 147295:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4401.635760] LustreError: 149080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4401.659767] LustreError: 149080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4406.760071] LustreError: 149076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4406.782750] LustreError: 149076:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4411.873133] LustreError: 151044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4411.903676] LustreError: 151044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4412.895166] Lustre: 147295:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788669703/real 1788669703] req@000000000241003b x1875555982029184/t0(0) o400->MGC192.168.203.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788669719 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4412.933886] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4414.970807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4422.643037] LustreError: 151044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4422.660397] LustreError: 151044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 4423.445471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4423.495486] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4427.769762] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4428.131720] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 4428.778072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4428.794725] Lustre: Skipped 1 previous similar message [ 4428.831543] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4428.900412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4437.473293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4437.483059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4437.506337] Lustre: Skipped 1 previous similar message [ 4437.515468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4439.014645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4442.782520] Lustre: server umount lustre-OST0000 complete [ 4449.253179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.260916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4449.264344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4449.280134] Lustre: Skipped 2 previous similar messages [ 4449.315878] Lustre: Skipped 2 previous similar messages [ 4453.485915] Lustre: server umount lustre-MDT0000 complete [ 4454.368367] LustreError: 149081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4454.402178] LustreError: 149081:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4457.116268] LustreError: 149062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669764 with bad export cookie 3698343591762400901 [ 4457.129952] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4457.137424] LustreError: 149062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4457.561548] Lustre: server umount lustre-MDT0001 complete [ 4468.092171] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 4470.799579] Key type lgssc unregistered [ 4471.058977] LNet: 154695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4471.073850] LNetError: 154695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4471.092093] LNet: Removed LNI 192.168.203.146@tcp [ 4472.040718] Key type .llcrypt unregistered [ 4472.042576] Key type ._llcrypt unregistered [ 4508.393255] Key type ._llcrypt registered [ 4508.395689] Key type .llcrypt registered [ 4508.511643] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 4509.813143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4509.846138] alg: No test for adler32 (adler32-zlib) [ 4511.017630] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 4511.313356] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 4513.039189] Key type lgssc registered [ 4514.263212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4522.255463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4522.274337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4523.795448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4528.970539] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 4533.226561] LustreError: 156214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4533.240583] LustreError: 156214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4538.343250] LustreError: 156215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4540.002758] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4543.483038] LustreError: 156214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4544.656753] LustreError: 156243:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4544.680827] LustreError: 156243:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4544.693133] LustreError: 156243:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4545.228509] Lustre: server umount lustre-MDT0000 complete [ 4561.315238] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [ 4572.123673] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 4610.392757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4610.650100] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4610.683755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4610.763535] Lustre: lustre-MDT0000: new disk, initializing [ 4610.833305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4610.850431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4616.594874] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 4628.249485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4628.476795] Lustre: 159556:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4628.558384] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4628.562929] Lustre: Skipped 1 previous similar message [ 4628.715494] Lustre: lustre-MDT0001: new disk, initializing [ 4628.853717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4628.902431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4628.918521] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4633.922785] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 4643.233894] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4648.949523] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4656.161491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4656.568906] Lustre: lustre-OST0000: new disk, initializing [ 4656.577235] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4656.595556] Lustre: 161521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4656.688923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4663.064088] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 4663.883991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4663.904085] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4664.068137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4674.263598] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4679.263716] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4681.060762] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 00:46:26 (1788669986) [ 4925.919764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4925.923496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4925.951734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4930.531115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4930.532701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4930.547500] Lustre: Skipped 1 previous similar message [ 4930.565979] Lustre: Skipped 2 previous similar messages [ 4932.161542] Lustre: server umount lustre-MDT0000 complete [ 4935.648927] LustreError: 159566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4935.682051] LustreError: 159566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4935.703914] LustreError: 159547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670242 with bad export cookie 10733418990436128571 [ 4935.704792] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4935.717570] LustreError: 159547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4940.768686] LustreError: 159563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4940.784549] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4940.809481] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4942.140985] Lustre: server umount lustre-MDT0001 complete [ 4952.512509] Lustre: server umount lustre-OST0000 complete [ 4963.786777] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [ 4974.617725] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 5011.694971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5011.892594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5011.920379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5011.996973] Lustre: lustre-MDT0000: new disk, initializing [ 5012.083918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5012.101747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5016.948976] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 5027.404681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5027.548570] Lustre: 166185:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5027.621859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5027.630157] Lustre: Skipped 1 previous similar message [ 5027.772359] Lustre: lustre-MDT0001: new disk, initializing [ 5027.934732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5027.993054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5028.014261] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5033.869677] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 5044.815676] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5050.966457] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5058.760466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5059.111718] Lustre: lustre-OST0000: new disk, initializing [ 5059.115606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5059.126661] Lustre: 168156:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5059.256652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5066.219272] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 5066.863818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5066.878704] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5066.975926] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5077.584771] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5083.438792] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5083.788052] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5088.666197] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5089.091294] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5092.834149] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5092.836702] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5092.843400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5092.861052] LustreError: Skipped 1 previous similar message [ 5092.882302] Lustre: Skipped 1 previous similar message [ 5098.045410] Lustre: server umount lustre-OST0000 complete [ 5104.609612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5104.633136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5104.661330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5104.675697] Lustre: Skipped 1 previous similar message [ 5110.590975] Lustre: server umount lustre-MDT0000 complete [ 5113.326258] LustreError: 166198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5113.362532] LustreError: 166198:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5115.991512] LustreError: 166180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670423 with bad export cookie 10733418990436287401 [ 5115.999992] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5116.007163] LustreError: 166180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5116.703163] Lustre: server umount lustre-MDT0001 complete [ 5126.462401] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 00:53:52 (1788670432) [ 5144.645458] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [ 5155.742648] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 5197.457240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5197.719476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5197.755784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5197.882420] Lustre: lustre-MDT0000: new disk, initializing [ 5198.059952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5198.077712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5203.063721] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 5214.155672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5214.405311] Lustre: 172914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5214.731640] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5220.274190] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 5231.911994] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5240.189138] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5249.495945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5249.791847] Lustre: lustre-OST0000: new disk, initializing [ 5249.795462] Lustre: Skipped 1 previous similar message [ 5249.806433] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5249.814834] Lustre: Skipped 2 previous similar messages [ 5249.818491] Lustre: 174892:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5255.238507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5255.247915] Lustre: Skipped 1 previous similar message [ 5255.268303] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5255.387761] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5256.909371] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 5267.121491] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5276.129881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5276.141625] Lustre: Skipped 1 previous similar message [ 5276.148557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5276.156403] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5276.162074] Lustre: Skipped 2 previous similar messages [ 5276.187824] LustreError: Skipped 1 previous similar message [ 5281.253737] LustreError: 174870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5281.283112] LustreError: 174870:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5281.526706] Lustre: server umount lustre-OST0000 complete [ 5286.368294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5291.883961] Lustre: server umount lustre-MDT0000 complete [ 5295.447582] LustreError: 172905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670602 with bad export cookie 10733418990436288290 [ 5295.450120] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5295.465179] LustreError: 172905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5295.967443] Lustre: server umount lustre-MDT0001 complete [ 5304.723743] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 00:56:50 (1788670610) [ 5323.642099] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [ 5335.838355] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 5380.009844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5380.406992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5380.473207] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5380.654394] Lustre: lustre-MDT0000: new disk, initializing [ 5380.747306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5380.753843] Lustre: Skipped 2 previous similar messages [ 5380.786113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5385.453965] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 5395.661743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5395.758688] Lustre: 179217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5396.000154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5400.585891] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 5409.852465] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5415.620242] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5422.992846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5423.328195] Lustre: 181195:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5429.343027] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5429.447355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5430.178809] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 5439.377493] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5450.212102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5450.214377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5450.231249] Lustre: Skipped 4 previous similar messages [ 5450.232989] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5450.251054] Lustre: Skipped 6 previous similar messages [ 5452.059887] Lustre: server umount lustre-OST0000 complete [ 5461.548430] Lustre: server umount lustre-MDT0000 complete [ 5465.204751] LustreError: 179209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670772 with bad export cookie 10733418990436289368 [ 5465.214927] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5465.224150] LustreError: 179209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5465.737373] Lustre: server umount lustre-MDT0001 complete [ 5477.579687] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 5480.777748] Key type lgssc unregistered [ 5481.241939] LNet: 183048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5481.246358] LNetError: 183048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5481.275917] LNet: Removed LNI 192.168.203.146@tcp [ 5482.370569] Key type .llcrypt unregistered [ 5482.378249] Key type ._llcrypt unregistered [ 5508.959880] Key type ._llcrypt registered [ 5508.961651] Key type .llcrypt registered [ 5509.093900] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 5510.341163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5510.643283] alg: No test for adler32 (adler32-zlib) [ 5511.614447] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 5511.886356] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 5513.535736] Key type lgssc registered [ 5514.906876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5523.684195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5523.700413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5525.294896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5529.670093] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 5535.714355] LustreError: 184461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5535.729180] LustreError: 184461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5538.301150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5538.681537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5543.048342] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 5550.185542] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5557.270644] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5564.626123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5564.948225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5570.090312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5571.082561] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 5578.846234] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5600.738551] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5600.745296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5600.758562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5603.814850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5603.817747] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5603.818491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5603.831661] Lustre: Skipped 1 previous similar message [ 5606.510902] Lustre: server umount lustre-OST0000 complete [ 5610.466640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5610.483859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5610.499194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5614.049219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5614.050163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5616.645635] Lustre: server umount lustre-MDT0000 complete [ 5619.171130] LustreError: 184457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5619.198415] LustreError: 184457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5620.442618] LustreError: 184441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670927 with bad export cookie 1972776136732708407 [ 5620.449487] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5620.454527] LustreError: 184441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5620.805678] Lustre: server umount lustre-MDT0001 complete [ 5632.275331] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 5635.448869] Key type lgssc unregistered [ 5635.863659] LNet: 188032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5635.873841] LNetError: 188032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5636.903707] LNet: Removed LNI 192.168.203.146@tcp [ 5637.913317] Key type .llcrypt unregistered [ 5637.916182] Key type ._llcrypt unregistered [ 5664.311538] Key type ._llcrypt registered [ 5664.315182] Key type .llcrypt registered [ 5664.445276] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 5665.321972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5665.397131] alg: No test for adler32 (adler32-zlib) [ 5666.538298] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 5666.720971] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 5668.351365] Key type lgssc registered [ 5669.354918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5677.018169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5677.032406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5678.459745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5683.028641] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 5688.800306] LustreError: 189445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5688.829717] LustreError: 189445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5691.378872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5691.763916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5697.606704] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 5704.375821] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5709.604648] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5716.103542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5716.382802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5722.367625] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 5723.663650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5729.438977] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5737.482208] Lustre: lustre-OST0000: Not available for connect from 192.168.203.46@tcp (stopping) [ 5737.956730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5737.971289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5737.986219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5738.983597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5739.000487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5739.025084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5739.035374] Lustre: Skipped 1 previous similar message [ 5742.622991] Lustre: server umount lustre-OST0000 complete [ 5750.904180] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8eb686c79800.ost_server_uuid 50 [ 5752.605782] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5768.672724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5768.678403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5768.692445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5774.097703] Lustre: server umount lustre-MDT0000 complete [ 5774.823038] LustreError: 190174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5774.849728] LustreError: 190174:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5778.538951] LustreError: 189427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671085 with bad export cookie 4025890836822950655 [ 5778.542392] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5778.548343] LustreError: 189427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5778.934581] Lustre: server umount lustre-MDT0001 complete [ 5789.047858] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 01:04:54 (1788671094) [ 5796.512713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5797.172119] LustreError: 193365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5797.264307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5802.441474] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 5802.467235] LustreError: 193366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5807.595691] LustreError: 193365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5811.309384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5811.618446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5816.130337] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 5823.962558] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5829.458318] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5835.934367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5836.200137] LustreError: 195185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5836.363605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5842.211388] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 5843.448065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5849.425201] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5856.534838] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5856.840880] Lustre: lustre-OST0001: new disk, initializing [ 5856.847995] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5856.858814] Lustre: 196209:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5857.053318] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5863.503527] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 5867.113338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5867.132410] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5867.327869] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5873.798825] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5877.223964] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5877.241825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5877.274239] Lustre: Skipped 1 previous similar message [ 5877.281338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5877.294719] Lustre: Skipped 2 previous similar messages [ 5883.100505] Lustre: server umount lustre-OST0000 complete [ 5884.933291] LustreError: 196881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5891.023057] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5891.269685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5896.372410] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5896.642869] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5913.569281] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5913.603978] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5913.618619] Lustre: Skipped 1 previous similar message [ 5913.626196] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5913.637251] Lustre: Skipped 5 previous similar messages [ 5917.480751] Lustre: server umount lustre-OST0001 complete [ 5923.810964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5923.840237] Lustre: Skipped 1 previous similar message [ 5927.947657] Lustre: server umount lustre-MDT0000 complete [ 5928.930561] LustreError: 193360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5928.960316] LustreError: 193360:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 5931.832507] LustreError: 193346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671238 with bad export cookie 4025890836822951733 [ 5931.842435] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5931.849361] LustreError: 193346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5932.420331] Lustre: server umount lustre-MDT0001 complete [ 5958.220040] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 01:07:43 (1788671263) [ 5965.578741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5965.974943] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5966.000530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5966.274851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5971.104195] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 5982.532225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5982.645888] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5982.728520] Lustre: 199649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5989.020236] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 5997.287610] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6004.137941] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6011.834746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6012.136200] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6019.144779] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 6021.231759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6026.288609] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6033.291613] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6033.362844] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6033.677991] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6033.686889] Lustre: Skipped 2 previous similar messages [ 6039.250935] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6040.924216] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 6050.373750] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6054.887783] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6054.915045] Lustre: Skipped 1 previous similar message [ 6054.935307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6054.945130] Lustre: Skipped 3 previous similar messages [ 6059.551127] Lustre: server umount lustre-OST0000 complete [ 6060.013795] LustreError: 201502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6060.033962] LustreError: 201502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6080.494762] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6080.508291] LustreError: Skipped 2 previous similar messages [ 6082.419359] Lustre: server umount lustre-OST0001 complete [ 6090.722648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6090.737825] Lustre: Skipped 3 previous similar messages [ 6090.743907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6090.766048] Lustre: Skipped 4 previous similar messages [ 6093.218809] Lustre: server umount lustre-MDT0000 complete [ 6098.116308] LustreError: 199639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671405 with bad export cookie 4025890836822953133 [ 6098.116946] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6098.141189] LustreError: 199639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6098.701269] Lustre: server umount lustre-MDT0001 complete [ 6127.076591] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 01:10:32 (1788671432) [ 6144.322628] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [ 6156.292792] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 6195.284148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6195.524174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6195.561880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6195.704256] Lustre: lustre-MDT0000: new disk, initializing [ 6195.768098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6195.780212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6200.587598] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 6211.957970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6212.081198] Lustre: 207747:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6212.137525] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6212.142720] Lustre: Skipped 1 previous similar message [ 6212.252864] Lustre: lustre-MDT0001: new disk, initializing [ 6212.360444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6212.380232] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6217.167632] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 6226.829865] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6232.445660] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6239.792785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6240.005417] Lustre: lustre-OST0000: new disk, initializing [ 6240.008869] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6240.021218] Lustre: 209711:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6246.001886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6246.011932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6246.175096] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6246.951668] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 6257.331140] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6261.454481] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6261.683638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6266.394179] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6266.595742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6271.973610] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6271.981678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6271.989215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6271.996078] Lustre: Skipped 2 previous similar messages [ 6274.843489] Lustre: server umount lustre-OST0000 complete [ 6284.944794] Lustre: server umount lustre-MDT0000 complete [ 6287.330718] LustreError: 210240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6287.348101] LustreError: 210240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 6288.522992] LustreError: 207740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671595 with bad export cookie 4025890836822954925 [ 6288.529031] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6288.545762] LustreError: 207740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6288.851580] Lustre: server umount lustre-MDT0001 complete [ 6298.240866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6304.486528] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 6313.693816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6318.516330] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 6326.299945] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6332.674405] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6340.135027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6347.494412] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 6355.425029] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6361.023019] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6361.253865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6367.031784] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6367.273592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6376.220236] Lustre: server umount lustre-OST0000 complete [ 6383.028663] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6387.656340] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6416.675519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6423.721418] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 6431.343394] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6435.331688] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6439.400193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6439.421879] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6439.426429] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6439.462966] Lustre: Skipped 1 previous similar message [ 6444.514767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6444.515029] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6444.526913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6444.526922] Lustre: Skipped 6 previous similar messages [ 6444.536589] Lustre: Skipped 5 previous similar messages [ 6444.544504] LustreError: Skipped 3 previous similar messages [ 6445.901164] Lustre: server umount lustre-OST0000 complete [ 6455.539963] Lustre: server umount lustre-MDT0000 complete [ 6458.919177] LustreError: 211914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671765 with bad export cookie 4025890836822955807 [ 6458.923926] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6458.931319] LustreError: 211914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6459.271704] Lustre: server umount lustre-MDT0001 complete [ 6467.707391] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 01:16:13 (1788671773) [ 6473.947226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6474.530447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6474.545498] Lustre: Skipped 6 previous similar messages [ 6479.671729] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 6488.192043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6493.265045] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 6502.791548] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6509.018866] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6515.838653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6522.505285] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 6529.981034] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6534.190834] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6534.380141] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6538.820916] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6539.042872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6545.983430] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6546.163317] Lustre: lustre-OST0001: new disk, initializing [ 6546.167359] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6546.174804] Lustre: 221272:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6552.353804] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 6555.749128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6555.768605] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6555.898799] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6561.988475] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6567.511946] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6567.760206] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6572.407688] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6572.673869] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6576.619945] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6576.638279] LustreError: Skipped 1 previous similar message [ 6580.722834] Lustre: server umount lustre-OST0001 complete [ 6581.736721] LustreError: 219722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6581.761608] LustreError: 219722:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6587.450469] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6587.675844] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6592.670859] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6592.852334] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6622.341386] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6627.838896] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6627.880630] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6627.881444] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6627.917869] Lustre: Skipped 1 previous similar message [ 6629.330866] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 6637.393953] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6639.413884] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6655.967240] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6656.088109] Lustre: server umount lustre-OST0001 complete [ 6679.796437] Lustre: server umount lustre-OST0000 complete [ 6689.973374] Lustre: server umount lustre-MDT0000 complete [ 6693.600423] LustreError: 217881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672000 with bad export cookie 4025890836822956773 [ 6693.601472] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6693.618190] LustreError: 217881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6700.041760] Lustre: server umount lustre-MDT0001 complete [ 6723.836882] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 01:20:29 (1788672029) [ 6730.777170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6731.266550] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6731.283109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6736.447359] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 6745.113310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6745.231197] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6745.278864] Lustre: 226518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6750.488787] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 6757.868216] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6763.464020] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6772.101368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6772.284141] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6778.919967] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 6786.522748] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6796.184097] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6796.290780] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6803.180584] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 6810.423877] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6812.015718] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6816.353186] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6816.590173] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6821.104917] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6821.352669] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6827.635953] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8eb6980de000.ost_server_uuid 50 [ 6829.151180] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8eb6980de000.ost_server_uuid in IDLE state after 0 sec [ 6830.387540] Lustre: Permanently deactivating lustre-OST0001 [ 6830.417600] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6833.853978] Lustre: Permanently reactivating lustre-OST0001 [ 6839.780805] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6839.792277] LustreError: Skipped 3 previous similar messages [ 6839.802586] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6839.818454] Lustre: Skipped 10 previous similar messages [ 6839.831465] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6839.839632] Lustre: Skipped 21 previous similar messages [ 6844.389583] Lustre: server umount lustre-OST0001 complete [ 6853.922631] Lustre: server umount lustre-OST0000 complete [ 6863.868072] Lustre: server umount lustre-MDT0000 complete [ 6867.623024] LustreError: 226508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672174 with bad export cookie 4025890836822958061 [ 6867.629765] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6867.640462] LustreError: 226508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6868.166929] Lustre: server umount lustre-MDT0001 complete [ 6892.197128] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 01:23:18 (1788672198) [ 6899.340123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6899.695829] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6899.734151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6899.745411] Lustre: Skipped 5 previous similar messages [ 6904.918121] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 6915.441694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6915.554975] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6915.595046] Lustre: 233211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6921.842367] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 6929.096339] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6936.100744] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6942.974538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6943.206619] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6943.232847] Lustre: Permanently deactivating lustre-OST0000 [ 6943.242903] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6949.433201] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6949.586326] LustreError: 235388:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 [ 6950.575676] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 6959.138949] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6965.840349] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6965.913654] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6972.372798] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 6973.635411] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6979.574502] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6981.281934] Lustre: Permanently reactivating lustre-OST0000 [ 6981.296024] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6981.301072] Lustre: Skipped 2 previous similar messages [ 6981.564328] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6981.613750] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 32 seconds. I think it's dead, and I am evicting it. exp 00000000a372de3a, cur 1788672288 deadline 1788672286 last 1788672256 [ 6981.682255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7005.931350] Lustre: server umount lustre-OST0001 complete [ 7017.871074] Lustre: server umount lustre-OST0000 complete [ 7036.383225] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7036.809066] Lustre: server umount lustre-MDT0000 complete [ 7041.178294] LustreError: 233206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672348 with bad export cookie 4025890836822959741 [ 7041.200185] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7041.525777] Lustre: server umount lustre-MDT0001 complete [ 7054.569628] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 7058.256764] Key type lgssc unregistered [ 7058.669570] LNet: 238098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7058.692901] LNetError: 238098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7059.755703] LNet: Removed LNI 192.168.203.146@tcp [ 7060.747054] Key type .llcrypt unregistered [ 7060.754437] Key type ._llcrypt unregistered [ 7085.638220] Key type ._llcrypt registered [ 7085.642253] Key type .llcrypt registered [ 7085.784306] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 7086.937144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7086.967407] alg: No test for adler32 (adler32-zlib) [ 7088.194522] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 7088.458350] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 7090.127178] Key type lgssc registered [ 7091.326364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7101.624436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7101.674233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7103.293512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7108.118879] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 7113.699255] LustreError: 239559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7113.718715] LustreError: 239559:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7118.004160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7118.203674] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7118.266498] Lustre: 239549:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7118.291867] Lustre: Permanently deactivating lustre-MDT0001 [ 7118.323298] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7118.480768] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7118.493929] LustreError: 239590:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7118.519042] LustreError: 239590:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7118.536201] LustreError: 239590:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 7118.798876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7124.186136] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 7132.168443] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7137.561066] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7143.713571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7144.063211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7148.135960] LustreError: 241400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7148.162681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7149.582491] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 7153.636577] LustreError: 241401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7155.998183] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7158.771063] LustreError: 241400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7163.892557] LustreError: 241400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7163.911160] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7163.923368] LustreError: 241400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7164.528558] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7164.885344] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7170.074083] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7170.084985] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7173.450692] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 7182.479243] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7186.517953] Lustre: Permanently reactivating lustre-MDT0001 [ 7186.531252] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7186.534586] Lustre: Skipped 2 previous similar messages [ 7190.291450] Lustre: lustre-MDT0001: Client 58f70bce-c4dd-412b-9552-189a18d6281e (at 192.168.203.46@tcp) reconnecting [ 7197.521961] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7199.143689] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7204.858272] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7205.128233] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7210.205429] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7210.425582] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7213.306316] Lustre: Permanently deactivating lustre-MDT0001 [ 7213.323576] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7213.329245] Lustre: Skipped 2 previous similar messages [ 7219.305855] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7228.392616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7228.396088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7228.419141] Lustre: Skipped 2 previous similar messages [ 7228.431418] Lustre: Skipped 3 previous similar messages [ 7230.108957] Lustre: server umount lustre-MDT0000 complete [ 7233.505865] LustreError: 244055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7233.534636] LustreError: 244055:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7234.463368] LustreError: 239541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672541 with bad export cookie 4438370563465165514 [ 7234.470546] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7234.471593] LustreError: 239541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7235.020334] Lustre: server umount lustre-MDT0001 complete [ 7241.101548] Lustre: server umount lustre-OST0000 complete [ 7253.865855] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 01:29:19 (1788672559) [ 7265.755783] Lustre: server umount lustre-OST0001 complete [ 7272.850470] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [ 7282.831574] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 7317.540182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7317.746917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7317.753510] Lustre: Skipped 2 previous similar messages [ 7317.783923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7317.876278] Lustre: lustre-MDT0000: new disk, initializing [ 7317.974933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7317.995497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7322.635075] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 7332.984246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7333.131846] Lustre: 247819:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7333.188270] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7333.192319] Lustre: Skipped 1 previous similar message [ 7333.309656] Lustre: lustre-MDT0001: new disk, initializing [ 7333.462636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7333.520949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7333.542432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7339.179402] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 7350.970789] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7358.621306] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7366.947533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7367.220386] Lustre: lustre-OST0000: new disk, initializing [ 7367.226045] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7367.245941] Lustre: 249788:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7367.371316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7374.005112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7374.046884] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7374.235297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7375.145218] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 7385.400214] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7387.120629] LustreError: 247825:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7396.142161] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7396.328367] Lustre: lustre-OST0001: new disk, initializing [ 7396.335673] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7396.346763] Lustre: 250903:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7397.207466] LustreError: 247825:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7397.241652] LustreError: 249796:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7403.723814] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 7404.109537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7404.123389] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7404.271859] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7407.271127] LustreError: 249796:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7414.179397] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7416.237279] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7419.877324] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7419.883816] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7419.897963] Lustre: Skipped 1 previous similar message [ 7419.918523] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7419.926580] Lustre: Skipped 1 previous similar message [ 7421.472074] Lustre: lustre-OST0001: Not available for connect from 192.168.203.46@tcp (stopping) [ 7425.002490] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7425.009951] Lustre: Skipped 1 previous similar message [ 7430.122870] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7430.128770] Lustre: Skipped 2 previous similar messages [ 7433.183273] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7433.309086] Lustre: server umount lustre-OST0001 complete [ 7435.246319] LustreError: 249773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7435.263486] LustreError: 249773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7450.610464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7450.636071] Lustre: Skipped 1 previous similar message [ 7450.643639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7450.652426] Lustre: Skipped 2 previous similar messages [ 7456.446138] Lustre: server umount lustre-OST0000 complete [ 7460.850495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7460.878244] Lustre: Skipped 2 previous similar messages [ 7466.494964] Lustre: server umount lustre-MDT0000 complete [ 7470.160349] LustreError: 247812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672777 with bad export cookie 4438370563465168328 [ 7470.174225] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7470.180435] LustreError: 247812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7476.541263] Lustre: server umount lustre-MDT0001 complete [ 7485.834676] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 7488.165931] Key type lgssc unregistered [ 7488.457128] LNet: 252935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7488.476243] LNetError: 252935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7488.510183] LNet: Removed LNI 192.168.203.146@tcp [ 7489.392898] Key type .llcrypt unregistered [ 7489.395139] Key type ._llcrypt unregistered [ 7525.070528] Key type ._llcrypt registered [ 7525.075853] Key type .llcrypt registered [ 7525.182954] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 7526.349959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7526.370242] alg: No test for adler32 (adler32-zlib) [ 7527.494760] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 7527.713673] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 7529.359193] Key type lgssc registered [ 7530.479456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7538.644072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7538.662455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7539.976446] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7540.009202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7540.159039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7545.265832] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 7554.355388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7554.526617] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7554.572536] Lustre: 255138:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7554.870516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7559.072678] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 7566.787828] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7571.905559] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7578.294976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7578.552960] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7578.745907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7584.927934] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 7588.060485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7592.833800] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7644.642206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7644.644814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7644.648255] LustreError: Skipped 1 previous similar message [ 7644.652773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7644.675816] Lustre: Skipped 1 previous similar message [ 7649.779597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7649.785926] Lustre: Skipped 2 previous similar messages [ 7650.244602] Lustre: server umount lustre-OST0000 complete [ 7654.474848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7667.944099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7668.474164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7674.425296] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 7674.607237] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7674.615649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7674.624188] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7681.445950] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7724.581858] Lustre: server umount lustre-OST0000 complete [ 7731.189186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7731.231952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7731.244945] Lustre: Skipped 1 previous similar message [ 7736.292902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7736.298196] Lustre: Skipped 1 previous similar message [ 7739.455943] Lustre: server umount lustre-MDT0000 complete [ 7742.431844] Lustre: 254461:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788673033/real 1788673033] req@00000000882d249f x1875559405654528/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788673049 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7742.439302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7742.460846] Lustre: 254461:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7742.490331] Lustre: Skipped 1 previous similar message [ 7743.282291] LustreError: 255130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673050 with bad export cookie 14292020611257785608 [ 7743.283973] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7743.299931] LustreError: 255130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7743.670109] Lustre: server umount lustre-MDT0001 complete [ 7754.463829] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 7757.352479] Key type lgssc unregistered [ 7757.628436] LNet: 260565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7757.649498] LNetError: 260565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7757.676455] LNet: Removed LNI 192.168.203.146@tcp [ 7758.626386] Key type .llcrypt unregistered [ 7758.627853] Key type ._llcrypt unregistered [ 7781.855107] Key type ._llcrypt registered [ 7781.858036] Key type .llcrypt registered [ 7781.951410] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 7783.243893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7783.337664] alg: No test for adler32 (adler32-zlib) [ 7784.468442] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 7784.741323] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 7786.431536] Key type lgssc registered [ 7787.461169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7796.382684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7796.416460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7797.946486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7802.683433] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 7807.460902] LustreError: 261982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7807.482867] LustreError: 261982:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7812.152133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7812.589107] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7812.795274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7817.423694] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 7824.457899] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7831.100805] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7838.453883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7838.963883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7842.051179] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7846.154737] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 7854.735295] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7883.232762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7883.241334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7883.249082] LustreError: Skipped 1 previous similar message [ 7883.274674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7883.298867] Lustre: Skipped 1 previous similar message [ 7887.668736] Lustre: server umount lustre-OST0000 complete [ 7893.473970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7893.482767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7893.491159] Lustre: Skipped 2 previous similar messages [ 7893.503143] Lustre: Skipped 1 previous similar message [ 7898.594891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7898.605670] Lustre: Skipped 1 previous similar message [ 7903.714080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7903.722243] Lustre: Skipped 1 previous similar message [ 7905.247390] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7905.411671] Lustre: server umount lustre-MDT0000 complete [ 7908.832559] LustreError: 261977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7908.843657] LustreError: 261977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7909.051982] LustreError: 261962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673216 with bad export cookie 16457954715852761975 [ 7909.060064] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7909.066360] LustreError: 261962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7909.387828] Lustre: server umount lustre-MDT0001 complete [ 7919.943542] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 7922.878986] Key type lgssc unregistered [ 7923.158865] LNet: 266285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7923.163963] LNetError: 266285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7924.201847] LNet: Removed LNI 192.168.203.146@tcp [ 7924.992777] Key type .llcrypt unregistered [ 7924.994958] Key type ._llcrypt unregistered [ 7941.657703] Key type ._llcrypt registered [ 7941.665234] Key type .llcrypt registered [ 7941.770594] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 7942.800520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7942.841467] alg: No test for adler32 (adler32-zlib) [ 7944.005165] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 7944.268818] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 7945.944416] Key type lgssc registered [ 7947.041799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7954.885568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7954.903809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7956.296843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7960.147487] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 7966.693892] LustreError: 267505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7966.715304] LustreError: 267505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7967.756242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7968.239234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7973.060387] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 7980.417412] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7986.323725] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7992.087651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7992.563260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7999.736844] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 8001.776042] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8007.487233] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8019.426827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8019.437298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8019.452580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8022.496929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8022.514371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8022.516794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8022.558549] Lustre: Skipped 1 previous similar message [ 8023.915620] Lustre: server umount lustre-OST0000 complete [ 8029.665183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8029.668909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8029.688418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8032.738692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8032.740464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8032.782663] Lustre: Skipped 1 previous similar message [ 8034.173075] Lustre: server umount lustre-MDT0000 complete [ 8037.876254] LustreError: 267501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8037.916786] LustreError: 267501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8038.265341] LustreError: 267484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673345 with bad export cookie 15795662598157125478 [ 8038.271399] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8038.281222] LustreError: 267484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8039.215851] Lustre: server umount lustre-MDT0001 complete [ 8050.290766] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 8052.931971] Key type lgssc unregistered [ 8053.200158] LNet: 271181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8053.210403] LNetError: 271181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8053.228809] LNet: Removed LNI 192.168.203.146@tcp [ 8054.343275] Key type .llcrypt unregistered [ 8054.346381] Key type ._llcrypt unregistered [ 8078.121574] Key type ._llcrypt registered [ 8078.124932] Key type .llcrypt registered [ 8078.270793] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 8079.268767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8079.305371] alg: No test for adler32 (adler32-zlib) [ 8080.498916] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 8080.721856] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 8082.367278] Key type lgssc registered [ 8083.758607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8092.259191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8092.272405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8093.850332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8099.670146] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 8103.393263] LustreError: 272595:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8103.407497] LustreError: 272595:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8108.523985] LustreError: 272596:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8108.642573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8109.085620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8113.789745] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 8121.062244] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8126.724878] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8133.973365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8134.339204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8138.483637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8140.692983] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 8148.639591] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8174.561888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8174.569103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8174.580105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8179.683923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8179.688929] Lustre: Skipped 1 previous similar message [ 8180.692405] Lustre: server umount lustre-OST0000 complete [ 8184.800562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8184.813137] Lustre: Skipped 1 previous similar message [ 8184.819533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8184.824399] Lustre: Skipped 2 previous similar messages [ 8189.922819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8189.926099] Lustre: Skipped 1 previous similar message [ 8190.194757] Lustre: server umount lustre-MDT0000 complete [ 8193.935120] LustreError: 274438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673500 with bad export cookie 14105383459438509757 [ 8193.938307] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8193.949969] LustreError: 274438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8194.286335] Lustre: server umount lustre-MDT0001 complete [ 8204.003049] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 8206.440770] Key type lgssc unregistered [ 8206.692575] LNet: 276900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8206.710514] LNetError: 276900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8206.745675] LNet: Removed LNI 192.168.203.146@tcp [ 8207.698523] Key type .llcrypt unregistered [ 8207.702757] Key type ._llcrypt unregistered [ 8225.633332] Key type ._llcrypt registered [ 8225.636708] Key type .llcrypt registered [ 8225.745700] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 8226.784692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8226.890521] alg: No test for adler32 (adler32-zlib) [ 8227.935858] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 8228.231178] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 8229.951177] Key type lgssc registered [ 8231.066898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8238.567970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8238.599936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8240.288709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8244.646549] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 8250.849487] LustreError: 278132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8250.874394] LustreError: 278132:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8253.132408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8253.460606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8257.875538] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 8265.756979] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8272.779647] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8280.397918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8280.900729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8284.989924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8287.929925] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 8295.938619] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8314.849796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8314.865722] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8314.866050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8314.871870] Lustre: Skipped 1 previous similar message [ 8319.851484] Lustre: server umount lustre-OST0000 complete [ 8325.088800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8325.089184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8325.110538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8325.110547] Lustre: Skipped 1 previous similar message [ 8330.213761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8330.224975] Lustre: Skipped 2 previous similar messages [ 8330.636470] Lustre: server umount lustre-MDT0000 complete [ 8334.603490] LustreError: 278098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673641 with bad export cookie 15136461673720338447 [ 8334.611024] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8334.633768] LustreError: 278098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8335.002979] Lustre: server umount lustre-MDT0001 complete [ 8348.404183] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 8351.714928] Key type lgssc unregistered [ 8352.042197] LNet: 281833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8352.049530] LNetError: 281833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8352.075049] LNet: Removed LNI 192.168.203.146@tcp [ 8353.274529] Key type .llcrypt unregistered [ 8353.278447] Key type ._llcrypt unregistered [ 8366.817810] Key type ._llcrypt registered [ 8366.822779] Key type .llcrypt registered [ 8366.959364] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8388.095905] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [ 8406.793346] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 8407.664383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8407.741854] alg: No test for adler32 (adler32-zlib) [ 8408.872985] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 8409.245519] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 8410.975276] Key type lgssc registered [ 8411.952643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8449.047751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8449.097538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8450.398878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8450.425832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8450.524193] Lustre: lustre-MDT0000: new disk, initializing [ 8450.617353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8450.645740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8455.535264] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 8466.324742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8466.439159] Lustre: 285385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8466.537992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8466.546606] Lustre: Skipped 1 previous similar message [ 8466.647270] Lustre: lustre-MDT0001: new disk, initializing [ 8466.786583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8466.845786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8466.864242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8471.522368] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 8482.013486] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8489.038503] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8497.128258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8497.436590] Lustre: lustre-OST0000: new disk, initializing [ 8497.447071] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8497.458041] Lustre: 287351:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8497.634489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8504.629600] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 8506.968029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8506.991078] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8507.145639] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8516.216879] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8521.710775] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8521.925089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8527.286947] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8527.538820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8532.965086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8532.987900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8532.996573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8533.011058] Lustre: Skipped 1 previous similar message [ 8535.573863] Lustre: server umount lustre-OST0000 complete [ 8543.204721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8543.212314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8543.221346] Lustre: Skipped 2 previous similar messages [ 8546.057187] Lustre: server umount lustre-MDT0000 complete [ 8550.339647] LustreError: 285378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673857 with bad export cookie 8349353819876675492 [ 8550.343316] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8550.355230] LustreError: 285378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8550.920037] Lustre: server umount lustre-MDT0001 complete [ 8561.486204] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 01:51:06 (1788673866) [ 8569.388470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8570.065717] LustreError: 289766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8570.093273] LustreError: 289766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8570.182739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8575.457478] LustreError: 289767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8576.262071] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 8580.611352] LustreError: 289766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8585.715974] LustreError: 289767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8587.219975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8587.718229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8592.231201] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 8600.767815] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8607.239476] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8614.917425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8615.306844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8622.387467] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 8630.241977] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8642.018147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8642.029719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8642.056293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8642.065264] Lustre: Skipped 1 previous similar message [ 8643.040501] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8646.998285] Lustre: server umount lustre-OST0000 complete [ 8653.281224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8653.283185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8653.304227] Lustre: Skipped 2 previous similar messages [ 8653.324203] Lustre: Skipped 3 previous similar messages [ 8658.189780] Lustre: server umount lustre-MDT0000 complete [ 8658.402719] LustreError: 290496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8658.432762] LustreError: 290496:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8661.885239] LustreError: 289748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788673968 with bad export cookie 8349353819876676374 [ 8661.911250] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8662.330157] Lustre: server umount lustre-MDT0001 complete [ 8675.157698] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 8678.235630] Key type lgssc unregistered [ 8678.608604] LNet: 293302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8678.621932] LNetError: 293302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8678.650432] LNet: Removed LNI 192.168.203.146@tcp [ 8679.815237] Key type .llcrypt unregistered [ 8679.819109] Key type ._llcrypt unregistered [ 8707.564164] Key type ._llcrypt registered [ 8707.566814] Key type .llcrypt registered [ 8707.701612] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [ 8722.639766] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 8723.499297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8723.586341] alg: No test for adler32 (adler32-zlib) [ 8724.725189] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 8724.928278] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 8726.615175] Key type lgssc registered [ 8727.568512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8771.949647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8771.970959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8773.302132] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8773.336344] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8773.433837] Lustre: lustre-MDT0000: new disk, initializing [ 8773.532029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8773.553453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8778.817791] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 8788.556747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8788.650615] Lustre: 297125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8788.700224] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8788.703623] Lustre: Skipped 1 previous similar message [ 8788.760978] Lustre: lustre-MDT0001: new disk, initializing [ 8788.833049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8788.870139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8788.881524] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8793.239459] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 8802.570124] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8807.267485] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8813.710596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8813.947608] Lustre: lustre-OST2710: new disk, initializing [ 8813.955404] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8813.968352] Lustre: 299092:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8814.064061] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8820.330398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8820.349855] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8820.455926] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8820.559893] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 8829.735452] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8837.333254] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8837.492592] Lustre: lustre-OST03e8: new disk, initializing [ 8837.497146] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8837.504512] Lustre: 300158:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8837.652594] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8844.308864] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 8845.945961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8845.961376] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8846.064851] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8852.562207] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8859.532501] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8859.783765] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8864.086740] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8864.269467] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8868.283899] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8868.512162] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8872.772693] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8872.967344] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8877.025663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8877.047640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8877.059302] Lustre: Skipped 3 previous similar messages [ 8882.154661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8882.184124] Lustre: Skipped 3 previous similar messages [ 8883.044252] Lustre: server umount lustre-MDT0000 complete [ 8886.526508] LustreError: 297116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674193 with bad export cookie 17156013396343279423 [ 8886.537601] LustreError: 297116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8886.539592] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8886.899391] Lustre: server umount lustre-MDT0001 complete [ 8900.644498] Lustre: server umount lustre-OST2710 complete [ 8903.455233] Lustre: 294922:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788674194/real 1788674194] req@00000000ec778ead x1875560660796160/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788674210 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8903.491316] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8903.499257] Lustre: Skipped 3 previous similar messages [ 8904.538671] Lustre: server umount lustre-OST03e8 complete [ 8923.324959] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [ 8932.587743] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 8964.976773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8965.258829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8965.293768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8965.392918] Lustre: lustre-MDT0000: new disk, initializing [ 8965.500106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8965.515181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8969.931348] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 8981.591523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8981.826838] Lustre: 305407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8981.880906] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8981.883941] Lustre: Skipped 1 previous similar message [ 8981.950536] Lustre: lustre-MDT0001: new disk, initializing [ 8982.068304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8982.116665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8982.144249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8986.990696] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 8998.323554] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9005.030925] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9012.001655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9012.314781] Lustre: lustre-OST0000: new disk, initializing [ 9012.321168] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9012.333593] Lustre: 307375:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9012.429593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9019.260924] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 9021.028437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9021.048911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9021.119963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9030.185610] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9035.785642] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9035.976337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9042.242509] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9042.395694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9047.010679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9047.012789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9047.024859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9047.053292] Lustre: Skipped 1 previous similar message [ 9050.695440] Lustre: server umount lustre-OST0000 complete [ 9057.259132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9057.264408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9057.287144] Lustre: Skipped 1 previous similar message [ 9057.311898] Lustre: Skipped 1 previous similar message [ 9061.394332] Lustre: server umount lustre-MDT0000 complete [ 9066.213359] LustreError: 306211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674373 with bad export cookie 17156013396343281110 [ 9066.220254] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9066.225623] LustreError: 306211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9066.782033] Lustre: server umount lustre-MDT0001 complete [ 9076.544305] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 01:59:42 (1788674382) [ 9078.072534] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9079.812243] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 01:59:45 (1788674385) [ 9083.822025] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 9110.610346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9110.790708] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9110.802591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9110.920597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9115.685112] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 9124.411296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9124.568256] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9124.607481] Lustre: 311234:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9129.774900] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 9152.459895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9152.729726] Lustre: Denying initial registration attempt from nid 192.168.203.146@tcp, specified as failover [ 9152.739980] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9152.753385] LustreError: 312689:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -99 [ 9152.757846] LustreError: 312689:0:(tgt_mount.c:2134:server_put_super()) no obd lustre-OST0000 [ 9152.766642] LustreError: 312689:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9152.869437] Lustre: server umount lustre-OST0000 complete [ 9152.872977] LustreError: 312689:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9160.672499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9160.683074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9160.703585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9160.727740] Lustre: Skipped 1 previous similar message [ 9166.440224] Lustre: server umount lustre-MDT0000 complete [ 9167.332166] LustreError: 311245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9167.361763] LustreError: 311245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9170.660212] LustreError: 311227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674477 with bad export cookie 17156013396343281992 [ 9170.667589] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9170.670995] LustreError: 311227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9170.916789] Lustre: server umount lustre-MDT0001 complete [ 9183.196891] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 9186.630435] Key type lgssc unregistered [ 9187.043390] LNet: 313822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9187.050984] LNetError: 313822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9187.081893] LNet: Removed LNI 192.168.203.146@tcp [ 9188.167322] Key type .llcrypt unregistered [ 9188.170476] Key type ._llcrypt unregistered [ 9199.086307] Key type ._llcrypt registered [ 9199.088241] Key type .llcrypt registered [ 9199.219576] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 9200.413414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9200.444219] alg: No test for adler32 (adler32-zlib) [ 9201.491354] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 9201.781949] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 9203.465777] Key type lgssc registered [ 9204.517523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9251.833169] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 9261.332679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9261.350338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9262.608932] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9262.644207] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9262.814920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9267.485990] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 9275.452726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9275.595710] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9275.650336] Lustre: 316582:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9275.850801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9280.286883] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 9288.670514] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9295.439923] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9315.369285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9315.601165] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9315.829443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9321.474370] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 9328.281493] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9334.240413] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9334.256704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9334.270974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9334.757190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9334.772844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9334.790230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9336.459632] Lustre: server umount lustre-OST0000 complete [ 9342.441357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9342.453120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9342.468404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9344.994649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9345.005827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9346.529269] Lustre: server umount lustre-MDT0000 complete [ 9350.702148] LustreError: 318448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674657 with bad export cookie 12311161610609679552 [ 9350.704206] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9350.717849] LustreError: 318448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9351.075644] Lustre: server umount lustre-MDT0001 complete [ 9361.261215] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 9364.354717] Key type lgssc unregistered [ 9364.661751] LNet: 320157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9364.680531] LNetError: 320157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9364.696344] LNet: Removed LNI 192.168.203.146@tcp [ 9365.594779] Key type .llcrypt unregistered [ 9365.596259] Key type ._llcrypt unregistered [ 9388.740919] Key type ._llcrypt registered [ 9388.742788] Key type .llcrypt registered [ 9388.853528] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 9389.884663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9390.023303] alg: No test for adler32 (adler32-zlib) [ 9391.146644] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 9391.412895] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 9393.095162] Key type lgssc registered [ 9394.232928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9402.413955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9402.450162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9403.983486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9408.628943] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 9414.118109] LustreError: 321572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9414.143045] LustreError: 321572:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9417.754492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9418.144711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9422.304589] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 9429.475110] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9434.138112] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9441.960804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9442.352941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9447.301281] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 9453.348770] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9464.288609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9464.300550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9464.324975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9464.810345] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9464.832505] Lustre: Skipped 1 previous similar message [ 9464.837454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9464.846495] Lustre: Skipped 1 previous similar message [ 9466.382591] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.46@tcp (stopping) [ 9468.267470] Lustre: server umount lustre-MDT0000 complete [ 9469.923534] LustreError: 321571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9469.940019] LustreError: 321571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9471.475083] LustreError: 323056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9472.720655] LustreError: 321553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674779 with bad export cookie 2621350867282277552 [ 9472.737307] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9472.748455] LustreError: 321553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9475.040267] LustreError: 321567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9475.043649] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9475.053838] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9481.712661] LustreError: 321567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9481.713874] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.46@tcp (stopping) [ 9481.742052] LustreError: 321567:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9481.805959] Lustre: Skipped 2 previous similar messages [ 9487.327399] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9487.678123] Lustre: server umount lustre-MDT0001 complete [ 9495.081434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9505.178495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9517.392325] LustreError: 323403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9517.427975] LustreError: 323403:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9517.502829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9517.591751] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9522.974992] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 9532.884584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9533.511282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9538.494735] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 9542.667646] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9547.345596] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9549.303730] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9555.165128] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9556.897662] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9563.114356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9563.138749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9563.152386] Lustre: Skipped 4 previous similar messages [ 9568.499595] Lustre: server umount lustre-OST0000 complete [ 9573.344520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9573.358366] Lustre: Skipped 2 previous similar messages [ 9583.584808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9583.590365] Lustre: Skipped 7 previous similar messages [ 9587.167152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9587.394149] Lustre: server umount lustre-MDT0000 complete [ 9588.707367] LustreError: 325168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9588.735418] LustreError: 325168:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9590.762421] LustreError: 321553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788674897 with bad export cookie 2621350867282284699 [ 9590.763179] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9590.770182] LustreError: 321553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9591.132804] Lustre: server umount lustre-MDT0001 complete [ 9601.864457] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 9604.884795] Key type lgssc unregistered [ 9605.228794] LNet: 327837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9605.238062] LNetError: 327837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9605.264023] LNet: Removed LNI 192.168.203.146@tcp [ 9606.276185] Key type .llcrypt unregistered [ 9606.283101] Key type ._llcrypt unregistered [ 9632.918916] Key type ._llcrypt registered [ 9632.927626] Key type .llcrypt registered [ 9633.068682] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 9634.188083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9634.248652] alg: No test for adler32 (adler32-zlib) [ 9635.460865] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 9635.844958] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 9637.585708] Key type lgssc registered [ 9638.907041] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9648.192062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9648.214330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9649.661211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9653.883257] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 9659.878058] LustreError: 329250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9659.905863] LustreError: 329250:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9662.016690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9662.408266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9667.105725] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 9675.090891] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9681.226844] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9683.803120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9683.805215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9683.817781] Lustre: Skipped 1 previous similar message [ 9687.907795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9687.920589] Lustre: Skipped 2 previous similar messages [ 9693.155879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9693.164225] Lustre: Skipped 1 previous similar message [ 9698.271129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9698.276132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9698.295924] Lustre: Skipped 1 previous similar message [ 9698.450556] Lustre: server umount lustre-MDT0000 complete [ 9702.563520] LustreError: 330867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675009 with bad export cookie 5360603343098881688 [ 9702.582872] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9702.584185] LustreError: 330867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9703.041780] Lustre: server umount lustre-MDT0001 complete [ 9718.542595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9718.868877] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9718.920772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9719.084800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9723.525784] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 9732.561599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9732.681498] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9732.754430] Lustre: 331842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9733.012964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9737.047802] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 9743.931612] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9749.301881] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9758.188191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9758.424702] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9758.437591] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9758.447508] LustreError: 333724:0:(tgt_mount.c:2619:server_fill_super()) Unable to start targets: -2 [ 9758.455551] LustreError: 333724:0:(tgt_mount.c:2134:server_put_super()) no obd lustre-OST0000 [ 9758.459949] LustreError: 333724:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9758.519426] Lustre: server umount lustre-OST0000 complete [ 9758.525283] LustreError: 333724:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9769.001473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9769.280345] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9769.472099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9775.693103] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 9777.796418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9782.569527] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9790.611787] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9790.739828] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9790.810374] Lustre: lustre-OST0001: new disk, initializing [ 9790.814141] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9790.818970] Lustre: 335263:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9790.904490] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9797.092896] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [ 9799.244353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9799.252417] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9799.348071] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9804.167581] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9809.902443] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9809.908302] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9809.914676] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9809.938282] Lustre: Skipped 1 previous similar message [ 9820.135143] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9820.148019] Lustre: Skipped 4 previous similar messages [ 9821.152688] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9821.250162] Lustre: server umount lustre-OST0001 complete [ 9825.271305] LustreError: 335945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9825.294236] LustreError: 335945:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9828.834240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9828.847115] LustreError: Skipped 1 previous similar message [ 9828.850397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9830.375838] LustreError: 335739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9830.392784] LustreError: 335739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9835.498586] LustreError: 334192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9835.526561] LustreError: 334192:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9840.097059] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9840.421245] Lustre: server umount lustre-OST0000 complete [ 9845.733690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9845.736826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9845.739278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9845.739288] Lustre: Skipped 6 previous similar messages [ 9845.755562] Lustre: Skipped 2 previous similar messages [ 9845.816263] LustreError: Skipped 1 previous similar message [ 9851.366982] Lustre: server umount lustre-MDT0000 complete [ 9855.839747] LustreError: 331834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675162 with bad export cookie 5360603343098882241 [ 9855.841556] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9855.859102] LustreError: 331834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9856.189627] Lustre: server umount lustre-MDT0001 complete [ 9866.844917] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [ 9869.780366] Key type lgssc unregistered [ 9870.083515] LNet: 337225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9870.098557] LNetError: 337225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9870.132584] LNet: Removed LNI 192.168.203.146@tcp [ 9871.187412] Key type .llcrypt unregistered [ 9871.198174] Key type ._llcrypt unregistered [ 9944.204395] Key type ._llcrypt registered [ 9944.213814] Key type .llcrypt registered [ 9944.409459] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [ 9960.117979] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [ 9961.704504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9961.721027] alg: No test for adler32 (adler32-zlib) [ 9962.768742] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [ 9963.074216] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [ 9964.767284] Key type lgssc registered [ 9966.338661] Lustre: Echo OBD driver; http://www.lustre.org/ [10007.381418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10007.420190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10008.768657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10008.850740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10009.053589] Lustre: lustre-MDT0000: new disk, initializing [10009.192247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10009.215249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10014.868099] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [10027.223534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10027.359547] Lustre: 342163:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10027.412612] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10027.416694] Lustre: Skipped 1 previous similar message [10027.476816] Lustre: lustre-MDT0001: new disk, initializing [10027.585616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10027.632056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10027.654877] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10032.431355] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [10042.533211] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10049.377144] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10057.404860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10057.742749] Lustre: lustre-OST0000: new disk, initializing [10057.747650] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10057.755547] Lustre: 344133:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10057.906544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10063.962381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10063.973118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10064.108564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10064.729521] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [10074.969322] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10079.887023] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10080.120601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10084.797765] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10085.091347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10089.960427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10089.969747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10089.972113] LustreError: Skipped 1 previous similar message [10090.005440] Lustre: Skipped 1 previous similar message [10090.005775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10093.589301] Lustre: server umount lustre-OST0000 complete [10099.168672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10099.176833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10099.189860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10099.195763] Lustre: Skipped 1 previous similar message [10100.192788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10100.195242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10100.236880] Lustre: Skipped 1 previous similar message [10104.299725] Lustre: server umount lustre-MDT0000 complete [10108.183916] LustreError: 342157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675415 with bad export cookie 12024027191057288143 [10108.190392] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10108.209335] LustreError: 342157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10108.746835] Lustre: server umount lustre-MDT0001 complete [10117.678375] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 02:17:03 (1788675423) [10126.497596] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 02:17:12 (1788675432) [10137.095676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10137.757082] LustreError: 346886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10137.772558] LustreError: 346886:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10137.903731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10143.204919] LustreError: 346887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10143.315559] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [10147.305469] LustreError: 346886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10152.426664] LustreError: 346887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10155.250185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10155.752381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10160.714577] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [10168.724308] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10175.441450] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10182.128152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10182.471018] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10189.484224] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [10197.598707] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10210.394356] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10212.310388] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10213.887721] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10215.759465] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10217.810660] Lustre: Failing over lustre-MDT0000 [10218.188120] Lustre: server umount lustre-MDT0000 complete [10221.090571] LustreError: 349375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10221.536845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10221.575483] Lustre: Skipped 2 previous similar messages [10231.288134] LustreError: 346883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10231.307913] LustreError: 346883:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10237.921039] Lustre: 340388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788675528/real 1788675528] req@0000000021096f26 x1875561959991040/t0(0) o400->MGC192.168.203.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788675544 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10237.961625] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10238.101476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10248.165567] LustreError: 349375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10248.185287] LustreError: 349375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [10248.550749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10248.633429] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10251.772225] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10253.381911] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [10253.807614] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10253.844051] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10253.876954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10263.310715] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10265.538586] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10270.104432] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10272.490157] Lustre: Failing over lustre-MDT0000 [10272.785890] Lustre: server umount lustre-MDT0000 complete [10274.273100] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10274.278155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10274.295458] Lustre: Skipped 1 previous similar message [10282.480857] LustreError: 346881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10282.506901] LustreError: 346881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10290.657468] Lustre: 340388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788675581/real 1788675581] req@000000002fabb7b7 x1875561960012672/t0(0) o400->MGC192.168.203.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788675597 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10290.685634] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10293.048309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10300.910502] LustreError: 340385:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f51ba268 x1875561960018176/t0(0) o250->MGC192.168.203.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10301.252947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10301.332129] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10303.987550] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10305.790239] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [10306.538332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10306.544561] Lustre: Skipped 2 previous similar messages [10306.571565] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10306.593180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10315.222381] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10316.646172] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10321.893669] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10321.903781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10321.903945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10321.910290] Lustre: Skipped 2 previous similar messages [10327.009227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10327.020732] Lustre: Skipped 4 previous similar messages [10328.054428] Lustre: server umount lustre-MDT0000 complete [10331.332959] LustreError: 346867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675638 with bad export cookie 12024027191057290866 [10331.336587] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10331.345105] LustreError: 346867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10331.561586] Lustre: server umount lustre-MDT0001 complete [10345.568988] Lustre: server umount lustre-OST0000 complete [10364.283282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10364.907988] LustreError: 353259:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10364.934173] LustreError: 353259:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [10370.815310] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [10380.166839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10380.682824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10380.699787] Lustre: Skipped 1 previous similar message [10385.416335] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [10392.899515] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10399.297939] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10406.347468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10409.970755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10413.328375] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [10421.335685] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10435.557085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10435.559489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10435.561818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10435.564326] LustreError: Skipped 1 previous similar message [10435.572701] Lustre: Skipped 2 previous similar messages [10447.327219] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10447.504334] Lustre: server umount lustre-OST0000 complete [10452.449752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10452.457456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10452.465259] Lustre: Skipped 5 previous similar messages [10457.651298] Lustre: server umount lustre-MDT0000 complete [10461.329449] LustreError: 353240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675768 with bad export cookie 12024027191057291601 [10461.338870] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10461.589145] Lustre: server umount lustre-MDT0001 complete [10472.008955] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [10475.381904] Key type lgssc unregistered [10475.615705] LNet: 356792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10475.631426] LNetError: 356792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10475.668110] LNet: Removed LNI 192.168.203.146@tcp [10476.557629] Key type .llcrypt unregistered [10476.561254] Key type ._llcrypt unregistered [10502.101095] Key type ._llcrypt registered [10502.105427] Key type .llcrypt registered [10502.253940] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [10503.487413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10503.630262] alg: No test for adler32 (adler32-zlib) [10504.743040] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [10505.049760] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [10506.855183] Key type lgssc registered [10508.495769] Lustre: Echo OBD driver; http://www.lustre.org/ [10518.284375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10518.317092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10519.860531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10524.800254] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [10529.249064] LustreError: 358311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10529.277134] LustreError: 358311:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10533.570471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10533.941357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10537.874641] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [10544.667691] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10549.548462] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10555.364075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10555.694392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10561.410566] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [10564.911328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10569.971577] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10583.120332] Lustre: server umount lustre-MDT0000 complete [10585.060436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10585.076134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10585.103865] LustreError: 358311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10586.584250] LustreError: 358290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675893 with bad export cookie 7974300083564152828 [10586.596300] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10586.609413] LustreError: 358290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10587.036612] Lustre: server umount lustre-MDT0001 complete [10600.442377] Lustre: server umount lustre-OST0000 complete [10616.392183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10616.944116] LustreError: 362025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10616.961856] LustreError: 362025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10617.049036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10621.609121] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [10622.435640] LustreError: 362026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10622.454827] LustreError: 362026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10627.557709] LustreError: 362025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10629.628438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10629.964349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10634.413923] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [10641.220590] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10646.311737] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10653.249958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10653.617217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10660.470653] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [10662.855125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10668.185547] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10681.313888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10681.320143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10681.345089] Lustre: Skipped 2 previous similar messages [10681.356446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10683.365923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10683.367441] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10683.393810] Lustre: Skipped 1 previous similar message [10686.243624] Lustre: server umount lustre-MDT0000 complete [10688.495496] LustreError: 364162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10688.540841] LustreError: 364162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10690.741047] LustreError: 362005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788675997 with bad export cookie 7974300083564153948 [10690.742484] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10690.758446] LustreError: 362005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10691.235959] Lustre: server umount lustre-MDT0001 complete [10707.529493] Lustre: server umount lustre-OST0000 complete [10724.170365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10724.608892] LustreError: 365937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10724.710036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10728.892933] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [10737.288524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10741.825398] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [10749.364940] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10756.987812] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10764.042870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10764.490824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10764.501762] Lustre: Skipped 1 previous similar message [10769.593334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10771.010680] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [10777.877198] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10783.718786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10783.723149] Lustre: Skipped 2 previous similar messages [10783.726138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10784.905469] Lustre: server umount lustre-OST0000 complete [10788.834106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10788.849408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10788.869855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10788.875972] Lustre: Skipped 2 previous similar messages [10793.952409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10793.965040] Lustre: Skipped 1 previous similar message [10794.689869] Lustre: server umount lustre-MDT0000 complete [10798.022487] LustreError: 365918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676105 with bad export cookie 7974300083564154970 [10798.024402] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10798.028761] LustreError: 365918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10798.294386] Lustre: server umount lustre-MDT0001 complete [10808.372995] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [10811.104457] Key type lgssc unregistered [10811.446279] LNet: 369469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10811.464203] LNetError: 369469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10811.480783] LNet: Removed LNI 192.168.203.146@tcp [10812.329752] Key type .llcrypt unregistered [10812.332515] Key type ._llcrypt unregistered [10837.079241] Key type ._llcrypt registered [10837.086416] Key type .llcrypt registered [10837.184285] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [10838.530418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10838.579738] alg: No test for adler32 (adler32-zlib) [10839.621110] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [10839.832080] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [10841.480907] Key type lgssc registered [10842.478529] Lustre: Echo OBD driver; http://www.lustre.org/ [10850.400892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10850.433774] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10850.442240] LustreError: 370906:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10850.463959] LustreError: 370906:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10850.468913] LustreError: 370906:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10850.480408] LustreError: 370906:0:(tgt_mount.c:2587:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10850.486287] LustreError: 370906:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10857.515141] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10857.526501] LustreError: 371303:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10857.560569] LustreError: 371303:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10857.571826] LustreError: 371303:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10857.576717] LustreError: 371303:0:(tgt_mount.c:2587:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10857.583841] LustreError: 371303:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10877.735938] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [10880.361375] Key type lgssc unregistered [10880.582512] LNet: 372307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10880.587748] LNetError: 372307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10880.608112] LNet: Removed LNI 192.168.203.146@tcp [10881.541324] Key type .llcrypt unregistered [10881.545765] Key type ._llcrypt unregistered [10900.597394] Key type ._llcrypt registered [10900.600099] Key type .llcrypt registered [10900.697052] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [10914.940931] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [10915.957499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10915.998409] alg: No test for adler32 (adler32-zlib) [10917.060791] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [10917.314928] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [10918.959168] Key type lgssc registered [10920.053576] Lustre: Echo OBD driver; http://www.lustre.org/ [10952.163614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10952.202677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10953.490142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10953.522584] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10953.627207] Lustre: lustre-MDT0000: new disk, initializing [10953.717761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10953.750642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10957.806304] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [10966.769483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10966.878828] Lustre: 375460:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10966.922103] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10966.924380] Lustre: Skipped 1 previous similar message [10966.978297] Lustre: lustre-MDT0001: new disk, initializing [10967.058250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10967.099671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10967.109815] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10971.263840] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [10979.790456] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10985.109441] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10991.176779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10991.418200] Lustre: lustre-OST0000: new disk, initializing [10991.423990] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10991.434394] Lustre: 377424:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10991.526413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10996.676730] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11000.387502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11000.397870] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11000.498918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11005.216892] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11009.490083] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11009.670848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11014.498961] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11014.718070] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11021.287521] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11021.289050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11021.326413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11021.345873] Lustre: Skipped 1 previous similar message [11022.776835] Lustre: server umount lustre-OST0000 complete [11028.449278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11028.460856] LustreError: Skipped 1 previous similar message [11028.469092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11028.488418] Lustre: Skipped 1 previous similar message [11028.492866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11031.524603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11031.526057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11031.538144] Lustre: Skipped 1 previous similar message [11033.749052] Lustre: server umount lustre-MDT0000 complete [11037.845164] LustreError: 375452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676344 with bad export cookie 595652965816567164 [11037.853210] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11037.861342] LustreError: 375452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11038.062882] Lustre: server umount lustre-MDT0001 complete [11047.157504] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11049.029346] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 02:32:34 (1788676354) [11056.369402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11056.874420] LustreError: 379884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11056.885069] LustreError: 379884:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11056.911844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11061.452470] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11062.240306] LustreError: 379885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11066.345207] LustreError: 379884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11069.499057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11069.820101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11073.850903] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11080.892373] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11085.853259] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11091.333997] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11091.554305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11097.235804] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11103.436904] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11109.968125] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11110.148936] Lustre: lustre-OST0001: new disk, initializing [11110.153675] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11110.161272] Lustre: 382724:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11110.272622] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11115.630770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11115.643478] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11115.715957] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11116.123668] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11124.902627] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11131.366520] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11131.367448] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11131.369309] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11131.378517] LustreError: Skipped 1 previous similar message [11131.400019] Lustre: Skipped 1 previous similar message [11133.929171] Lustre: server umount lustre-OST0001 complete [11136.490706] LustreError: 381705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11136.509789] LustreError: 381705:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11141.607776] LustreError: 381707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11141.629185] LustreError: 381707:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11151.328909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11151.348259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11151.378522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11151.390244] Lustre: Skipped 2 previous similar messages [11151.849076] LustreError: 383047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11151.868757] LustreError: 383047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11155.597800] Lustre: server umount lustre-OST0000 complete [11162.080161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11162.085215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11162.086310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11162.099989] Lustre: Skipped 2 previous similar messages [11162.116518] Lustre: Skipped 2 previous similar messages [11165.014843] Lustre: server umount lustre-MDT0000 complete [11168.358230] LustreError: 379864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676475 with bad export cookie 595652965816568046 [11168.368243] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11168.387440] LustreError: 379864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11174.765360] Lustre: server umount lustre-MDT0001 complete [11184.186349] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [11186.384664] Key type lgssc unregistered [11186.727129] LNet: 384710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11186.737766] LNetError: 384710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11186.757795] LNet: Removed LNI 192.168.203.146@tcp [11187.479674] Key type .llcrypt unregistered [11187.481727] Key type ._llcrypt unregistered [11234.857684] Key type ._llcrypt registered [11234.859462] Key type .llcrypt registered [11234.953522] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [11235.692363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11235.777080] alg: No test for adler32 (adler32-zlib) [11236.890401] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [11237.085488] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [11238.735162] Key type lgssc registered [11240.185162] Lustre: Echo OBD driver; http://www.lustre.org/ [11248.556590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11248.600119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11249.861460] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11249.880853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11250.002401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11254.152667] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11261.882374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11261.975404] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11262.019402] Lustre: 387506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11262.236868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11266.076691] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11273.495867] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11279.036967] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11282.912550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11282.924846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11282.936544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11284.961030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11284.962962] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11287.501793] Lustre: server umount lustre-MDT0000 complete [11291.104864] LustreError: 387499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676598 with bad export cookie 4683360250019840591 [11291.111239] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11291.129466] LustreError: 387499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11291.415350] Lustre: server umount lustre-MDT0001 complete [11301.287284] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 02:36:46 (1788676606) [11308.358340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11308.837896] LustreError: 390017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11308.858977] LustreError: 390017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11308.879860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11313.534501] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11314.145957] LustreError: 390018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11319.268611] LustreError: 390017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11324.004362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11324.320872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11329.257796] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11337.669317] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11343.532700] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11350.757373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11351.008088] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11351.248357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11357.053884] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11363.401202] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11370.899976] Lustre: Permanently deactivating lustre-OST0000 [11370.935216] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11377.691863] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11379.869713] LustreError: 393004:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11379.881178] LustreError: 393004:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11379.888889] LustreError: 393004:0:(mgs_handler.c:1247:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11382.805349] LustreError: 393053:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11382.815039] LustreError: 393053:0:(mgs_handler.c:1247:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11390.787708] Lustre: server umount lustre-OST0000 complete [11393.246819] LustreError: 393303:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11393.255478] LustreError: 393303:0:(mgs_handler.c:1247:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11396.064951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11396.074697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11396.091304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11396.098658] Lustre: Skipped 1 previous similar message [11398.113032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11398.113386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11403.232549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11403.241292] Lustre: Skipped 2 previous similar messages [11405.334733] Lustre: server umount lustre-MDT0000 complete [11408.353699] LustreError: 391009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11408.371493] LustreError: 391009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11408.691307] LustreError: 389998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676715 with bad export cookie 4683360250019841165 [11408.691877] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11408.702359] LustreError: 389998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11418.257424] Lustre: server umount lustre-MDT0001 complete [11425.457224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11430.059201] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11432.326104] LustreError: 394463:0:(mgs_handler.c:1247:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11437.673660] Lustre: 394802:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.146@tcp [11439.435464] Lustre: server umount MGS complete [11448.968773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11449.346780] LustreError: 395525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11449.361432] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11449.376631] Lustre: Skipped 1 previous similar message [11449.422399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11453.608978] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11459.561526] LustreError: 395526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11459.582379] LustreError: 395526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11461.326469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11461.535083] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11466.120969] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11473.589947] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11478.883409] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11480.610181] Lustre: Permanently reactivating lustre-OST0000 [11480.632357] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11480.640660] Lustre: Skipped 2 previous similar messages [11507.679907] LustreError: 397308:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11514.387517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11514.694985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11514.699069] Lustre: Skipped 1 previous similar message [11520.329061] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11527.163992] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11538.400075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11538.404518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11538.411037] LustreError: Skipped 1 previous similar message [11538.414382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11538.428359] Lustre: Skipped 1 previous similar message [11538.449306] Lustre: Skipped 1 previous similar message [11539.619442] Lustre: server umount lustre-OST0000 complete [11543.522556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11557.343113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11557.583617] Lustre: server umount lustre-MDT0000 complete [11558.882102] LustreError: 395520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11558.898211] LustreError: 395520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11561.047302] LustreError: 395506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788676868 with bad export cookie 4683360250019842222 [11561.051653] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11561.060498] LustreError: 395506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11561.376451] Lustre: server umount lustre-MDT0001 complete [11571.103874] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [11574.152778] Key type lgssc unregistered [11574.446083] LNet: 399940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11574.454330] LNetError: 399940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11574.466656] LNet: Removed LNI 192.168.203.146@tcp [11575.307340] Key type .llcrypt unregistered [11575.309553] Key type ._llcrypt unregistered [11592.795457] Key type ._llcrypt registered [11592.797871] Key type .llcrypt registered [11592.879564] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [11607.693581] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [11608.923094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11608.963766] alg: No test for adler32 (adler32-zlib) [11610.132827] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [11610.414648] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [11612.103154] Key type lgssc registered [11613.324249] Lustre: Echo OBD driver; http://www.lustre.org/ [11648.667229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11648.703139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11650.010553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11650.058692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11650.176767] Lustre: lustre-MDT0000: new disk, initializing [11650.294695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11650.316645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11655.952905] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11666.392046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11666.493539] Lustre: 403097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11666.555511] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11666.563045] Lustre: Skipped 1 previous similar message [11666.674574] Lustre: lustre-MDT0001: new disk, initializing [11666.773492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11666.807198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11666.822719] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11671.017438] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11680.400972] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11685.829794] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11691.871796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11692.137264] Lustre: lustre-OST0000: new disk, initializing [11692.140985] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11692.148757] Lustre: 405063:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11692.243932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11697.756202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11697.768079] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11697.873712] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11697.954912] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11707.940631] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11712.641965] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11712.848627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11718.015576] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11718.252569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11723.750820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11723.753199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11723.760445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11723.793327] Lustre: Skipped 1 previous similar message [11726.375420] Lustre: server umount lustre-OST0000 complete [11733.478867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11733.495901] LustreError: Skipped 1 previous similar message [11733.501078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11733.516076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11733.522512] Lustre: Skipped 1 previous similar message [11735.986885] Lustre: server umount lustre-MDT0000 complete [11739.558474] LustreError: 403089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788677046 with bad export cookie 9080375092705929425 [11739.564602] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11739.575544] LustreError: 403089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11739.928901] Lustre: server umount lustre-MDT0001 complete [11747.485487] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11748.982902] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 02:44:15 (1788677055) [11755.436934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11755.869154] LustreError: 407526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11755.897898] LustreError: 407526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11755.947523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11760.193980] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11761.126784] LustreError: 407527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11765.219828] LustreError: 407526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11768.059632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11768.369288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11772.624026] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11779.612448] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11785.366679] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11791.324492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11791.645653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11797.053732] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11804.086740] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11806.101743] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11808.322748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11814.381530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11814.397682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11814.413419] Lustre: Skipped 1 previous similar message [11814.419666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11814.430355] Lustre: Skipped 1 previous similar message [11816.421136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11816.435228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11816.445218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11818.521242] Lustre: server umount lustre-OST0000 complete [11824.613510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11824.621042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11824.635714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11824.642300] Lustre: Skipped 1 previous similar message [11836.895969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11836.910375] Lustre: Skipped 5 previous similar messages [11837.407448] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11837.639334] Lustre: server umount lustre-MDT0000 complete [11840.998077] LustreError: 407527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11841.035310] LustreError: 407527:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11841.390419] LustreError: 407508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788677148 with bad export cookie 9080375092705930307 [11841.406803] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11841.412043] LustreError: 407508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11841.643333] Lustre: server umount lustre-MDT0001 complete [11854.218786] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [11857.134276] Key type lgssc unregistered [11857.545774] LNet: 411251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11857.557786] LNetError: 411251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11857.581681] LNet: Removed LNI 192.168.203.146@tcp [11858.302449] Key type .llcrypt unregistered [11858.306137] Key type ._llcrypt unregistered [11880.705541] Key type ._llcrypt registered [11880.710424] Key type .llcrypt registered [11880.839193] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [11881.139284] Key type .llcrypt unregistered [11881.141214] Key type ._llcrypt unregistered [11902.210879] Key type ._llcrypt registered [11902.213291] Key type .llcrypt registered [11902.309666] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [11903.673863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11903.761345] alg: No test for adler32 (adler32-zlib) [11904.789689] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [11905.225139] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [11906.911220] Key type lgssc registered [11908.249494] Lustre: Echo OBD driver; http://www.lustre.org/ [11915.940681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11915.976368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11917.394395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11921.827312] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11927.523173] LustreError: 413458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11927.542445] LustreError: 413458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11929.461817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11929.779271] LustreError: 413457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11929.793343] LustreError: 413457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11929.804539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11935.206791] LustreError: 413452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11935.235751] LustreError: 413452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11935.361435] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11940.323792] LustreError: 413458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11940.354869] LustreError: 413458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11942.412031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11942.950688] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11947.495073] LustreError: 413454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11947.512641] LustreError: 413454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11949.689125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11950.088651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11954.470202] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [11962.851474] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11962.859087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11962.882142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11964.671609] Lustre: server umount lustre-OST0000 complete [11970.021799] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11970.032923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11970.062478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11970.548778] LustreError: 413452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11970.750435] Lustre: server umount lustre-MDT0000 complete [11974.327649] LustreError: 413438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788677281 with bad export cookie 11024241003144625595 [11974.330410] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11974.348280] LustreError: 413438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11985.035763] Lustre: server umount lustre-MDT0001 complete [11995.338790] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [11997.763860] Key type lgssc unregistered [11998.057448] LNet: 416603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11998.064256] LNetError: 416603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11998.084048] LNet: Removed LNI 192.168.203.146@tcp [11998.800225] Key type .llcrypt unregistered [11998.802268] Key type ._llcrypt unregistered [12023.026207] Key type ._llcrypt registered [12023.027796] Key type .llcrypt registered [12023.116735] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [12024.113265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12024.176878] alg: No test for adler32 (adler32-zlib) [12025.588493] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [12025.927716] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [12027.607650] Key type lgssc registered [12028.905463] Lustre: Echo OBD driver; http://www.lustre.org/ [12036.861781] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12036.884304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12052.512127] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12057.631334] LustreError: 417991:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.146@tcp: failed processing log, type 4: rc = -110 [12084.448181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12091.563928] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [12098.976260] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12106.502323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12107.224891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12107.320563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12111.245671] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [12112.357513] LustreError: 418017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12112.383498] LustreError: 418017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12117.477391] LustreError: 418016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12119.535436] LustreError: 419028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12119.569149] LustreError: 419028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12119.607970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12120.018771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12124.710247] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [12132.316705] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12138.408503] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12145.637142] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12145.649692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12145.664771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12145.668835] Lustre: Skipped 1 previous similar message [12145.680333] Lustre: Skipped 1 previous similar message [12146.383573] Lustre: server umount lustre-OST0000 complete [12150.755721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12150.757288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12150.764421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12150.777890] Lustre: Skipped 1 previous similar message [12152.432544] Lustre: server umount lustre-MDT0000 complete [12155.885532] LustreError: 419032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12155.911136] LustreError: 419032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12156.030414] LustreError: 419768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788677463 with bad export cookie 2708005268475025379 [12156.033326] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12156.040644] LustreError: 419768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12156.565951] Lustre: server umount lustre-MDT0001 complete [12168.765857] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [12171.929404] Key type lgssc unregistered [12172.215757] LNet: 421559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12172.222790] LNetError: 421559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12172.243229] LNet: Removed LNI 192.168.203.146@tcp [12173.055240] Key type .llcrypt unregistered [12173.056693] Key type ._llcrypt unregistered [12198.529315] Key type ._llcrypt registered [12198.534219] Key type .llcrypt registered [12198.707848] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [12199.877845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12199.946321] alg: No test for adler32 (adler32-zlib) [12200.999376] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [12201.259338] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [12202.911724] Key type lgssc registered [12204.071226] Lustre: Echo OBD driver; http://www.lustre.org/ [12211.773893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12211.795843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12213.315098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12217.731204] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [12223.456252] LustreError: 422975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12223.482931] LustreError: 422975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12225.806284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12226.116178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12230.242258] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [12238.600477] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12244.207072] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12250.958055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12251.228588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12257.647767] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [12258.430468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12264.560625] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12269.025640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12269.043402] Lustre: Skipped 2 previous similar messages [12269.044420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12269.062386] Lustre: Skipped 2 previous similar messages [12271.109498] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.46@tcp (stopping) [12274.159828] LustreError: 422969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12274.168946] LustreError: 422969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12274.216453] Lustre: server umount lustre-MDT0000 complete [12276.369608] LustreError: 422955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.46@tcp arrived at 1788677583 with bad export cookie 11753716652140216663 [12276.385698] LustreError: 422955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12278.274335] LustreError: 422954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788677585 with bad export cookie 11753716652140215767 [12278.286737] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12279.265878] LustreError: 422970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12279.267621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12279.285861] LustreError: 422970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12279.314818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12283.299124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12283.299259] LustreError: 424000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12283.325365] LustreError: 424000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12284.512255] Lustre: server umount lustre-OST0000 complete [12288.420425] LustreError: 422975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12288.442214] LustreError: 422975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12291.395873] Lustre: server umount lustre-MDT0001 complete [12302.540646] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [12305.079092] Key type lgssc unregistered [12305.376747] LNet: 426660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12305.382937] LNetError: 426660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12305.418682] LNet: Removed LNI 192.168.203.146@tcp [12306.351599] Key type .llcrypt unregistered [12306.356203] Key type ._llcrypt unregistered [12332.419376] Key type ._llcrypt registered [12332.421505] Key type .llcrypt registered [12332.572108] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [12333.882903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12334.162981] alg: No test for adler32 (adler32-zlib) [12335.234431] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [12335.482558] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [12337.159937] Key type lgssc registered [12338.454651] Lustre: Echo OBD driver; http://www.lustre.org/ [12346.555433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12346.568031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12347.994604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12352.685521] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [12358.122067] LustreError: 428074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12358.140837] LustreError: 428074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12361.995272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12362.486147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12368.126062] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [12376.955492] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12383.349279] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12390.749500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12391.055641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12393.148934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12396.965920] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [12404.874768] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12408.799920] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12408.816380] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12408.847914] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12411.894286] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.46@tcp (stopping) [12413.928109] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12413.939474] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12415.313952] Lustre: server umount lustre-MDT0001 complete [12417.009572] LustreError: 428071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12419.045560] LustreError: 428803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12419.064526] LustreError: 428803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12424.161091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12424.170678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12424.173214] LustreError: 428074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12424.193078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12424.193086] Lustre: Skipped 1 previous similar message [12427.622613] Lustre: server umount lustre-OST0000 complete [12429.280134] LustreError: 428803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12431.647740] Lustre: server umount lustre-MDT0000 complete [12446.102371] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [12449.407978] Key type lgssc unregistered [12449.745157] LNet: 431758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12449.754032] LNetError: 431758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12449.781921] LNet: Removed LNI 192.168.203.146@tcp [12450.953481] Key type .llcrypt unregistered [12450.956664] Key type ._llcrypt unregistered [12474.282462] Key type ._llcrypt registered [12474.284844] Key type .llcrypt registered [12474.371423] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [12493.295636] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [12494.845787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12494.929129] alg: No test for adler32 (adler32-zlib) [12495.943350] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [12496.201245] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [12497.847291] Key type lgssc registered [12498.803644] Lustre: Echo OBD driver; http://www.lustre.org/ [12532.931075] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [12545.905303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12545.929127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12547.166189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12547.200202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12547.257534] Lustre: lustre-MDT0000: new disk, initializing [12547.347320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12547.362233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12551.436562] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [12565.616645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12565.849595] Lustre: lustre-OST0000: new disk, initializing [12565.856764] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12565.862255] Lustre: Skipped 1 previous similar message [12565.869173] Lustre: 436788:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12565.938447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12570.960979] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [12575.345086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12575.358303] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12575.398186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12578.782720] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12593.957055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12594.062698] Lustre: 435768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12594.122556] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12594.191299] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.46@tcp (not set up) [12594.202740] Lustre: lustre-MDT0001: new disk, initializing [12594.318508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12594.377048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12594.394802] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12598.746355] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [12608.534516] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12608.837652] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12614.067676] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12614.355530] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12619.797211] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.46@tcp (stopping) [12623.341371] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12623.342285] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12623.361601] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12623.377777] Lustre: Skipped 1 previous similar message [12625.893023] Lustre: server umount lustre-MDT0001 complete [12633.574291] LustreError: 435778:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12633.597437] LustreError: 435778:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12638.694844] LustreError: 435778:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12638.734912] LustreError: 435778:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12643.808625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12643.810589] LustreError: 435781:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12643.823198] Lustre: Skipped 1 previous similar message [12643.827413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12643.859776] Lustre: Skipped 1 previous similar message [12648.819400] Lustre: server umount lustre-OST0000 complete [12648.935490] LustreError: 435782:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12659.057950] Lustre: server umount lustre-MDT0000 complete [12671.232893] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [12673.449868] Key type lgssc unregistered [12673.765141] LNet: 440535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12673.774306] LNetError: 440535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12673.790710] LNet: Removed LNI 192.168.203.146@tcp [12674.483057] Key type .llcrypt unregistered [12674.488165] Key type ._llcrypt unregistered [12711.769652] Key type ._llcrypt registered [12711.772458] Key type .llcrypt registered [12711.867951] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [12712.144443] Key type .llcrypt unregistered [12712.148772] Key type ._llcrypt unregistered [12732.329194] Key type ._llcrypt registered [12732.331693] Key type .llcrypt registered [12732.517368] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [12733.939517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12733.955570] alg: No test for adler32 (adler32-zlib) [12735.015508] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [12735.303854] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [12737.055170] Key type lgssc registered [12738.385713] Lustre: Echo OBD driver; http://www.lustre.org/ [12764.071973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12764.090049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12765.370420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12765.405422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12765.498359] Lustre: lustre-MDT0000: new disk, initializing [12765.566389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12765.590232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12769.543488] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [12779.144251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12779.258349] Lustre: 443912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12779.313140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12779.317354] Lustre: Skipped 1 previous similar message [12779.379362] Lustre: lustre-MDT0001: new disk, initializing [12779.480350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12779.515631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12779.533631] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12783.710338] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [12792.055612] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12797.107482] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12803.081147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12803.314065] Lustre: lustre-OST0000: new disk, initializing [12803.317330] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12803.323247] Lustre: 445881:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12803.402439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12808.959758] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [12813.402698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12813.416392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12813.538304] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12818.412773] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12825.570895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12825.580119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12825.595906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12829.163413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12829.167807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12829.175609] Lustre: Skipped 1 previous similar message [12829.189544] Lustre: Skipped 1 previous similar message [12831.578461] Lustre: server umount lustre-MDT0000 complete [12835.274044] LustreError: 443904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788678142 with bad export cookie 7863215775554924114 [12835.275036] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12835.290374] LustreError: 443904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12835.665222] Lustre: server umount lustre-MDT0001 complete [12849.636603] Lustre: server umount lustre-OST0000 complete [12859.606990] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 03:02:45 (1788678165) [12867.780321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12868.259334] LustreError: 447910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12868.281621] LustreError: 447910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12868.347246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12872.471612] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [12873.703441] LustreError: 447911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12878.826934] LustreError: 447910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12881.332899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12881.718336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12885.980548] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [12893.207611] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12898.554776] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12919.363548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12919.692822] Lustre: Found index 0 for lustre-OST0000, updating log [12920.055943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12927.021655] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [12933.801221] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12939.524850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12944.871718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12944.874455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12944.884546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12944.901440] Lustre: Skipped 2 previous similar messages [12946.345786] Lustre: server umount lustre-OST0000 complete [12963.816240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12963.823488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12963.834668] Lustre: Skipped 1 previous similar message [12963.846025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12963.855121] Lustre: Skipped 2 previous similar messages [12972.516819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12972.537139] Lustre: Skipped 3 previous similar messages [12977.119163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12977.410233] Lustre: server umount lustre-MDT0000 complete [12978.656937] LustreError: 448951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12978.680727] LustreError: 448951:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12981.303571] LustreError: 447891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788678288 with bad export cookie 7863215775554925500 [12981.317770] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12981.330411] LustreError: 447891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12981.978536] Lustre: server umount lustre-MDT0001 complete [12985.192962] LNet: 1 local NIs in recovery (showing 1): 192.168.203.146@tcp [12993.398809] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 03:04:59 (1788678299) [13614.089172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13614.481207] LustreError: 454328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13614.549598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13618.384796] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [13619.686530] LustreError: 454329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13624.804509] LustreError: 454328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13626.583289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13626.996019] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13631.687237] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [13639.180269] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13644.880895] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13664.881270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13665.229398] Lustre: Found index 0 for lustre-OST0000, updating log [13665.524670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13671.870755] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [13676.455129] LNet: 1 local NIs in recovery (showing 1): 192.168.203.146@tcp [13678.924711] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13681.300410] Lustre: server umount lustre-OST0000 complete [13686.144894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13686.160069] Lustre: Skipped 1 previous similar message [13686.164721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13686.179729] Lustre: Skipped 2 previous similar messages [13687.201138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13687.213755] Lustre: Skipped 1 previous similar message [13689.250077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13689.258452] Lustre: Skipped 3 previous similar messages [13691.333207] LustreError: 454323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13691.357298] LustreError: 454323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13691.705865] Lustre: server umount lustre-MDT0000 complete [13695.467730] LustreError: 455570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13695.486596] LustreError: 455570:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [13695.781880] LustreError: 454308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788679002 with bad export cookie 7863215775554926592 [13695.800776] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13696.205915] Lustre: server umount lustre-MDT0001 complete [13727.999192] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [13738.992199] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [13784.621598] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 03:18:10 (1788679090) [13804.244769] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [13807.510964] Key type lgssc unregistered [13807.802209] LNet: 460935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13807.816173] LNetError: 460935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13927.653554] LNet: Removed LNI 192.168.203.146@tcp [13928.615299] Key type .llcrypt unregistered [13928.617534] Key type ._llcrypt unregistered [13945.853763] Key type ._llcrypt registered [13945.864642] Key type .llcrypt registered [13945.965185] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [13946.939523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13947.056514] alg: No test for adler32 (adler32-zlib) [13948.279718] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [13948.549956] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [13950.239224] Key type lgssc registered [13951.789771] Lustre: Echo OBD driver; http://www.lustre.org/ [13956.071059] LNet: Added LNI 192.168.203.146@tcp100 [8/256/0/180] [13956.960355] LNet: Added LNI 192.168.203.146@tcp101 [8/256/0/180] [13957.812780] LNet: Added LNI 192.168.203.146@tcp102 [8/256/0/180] [13958.548078] LNet: Added LNI 192.168.203.146@tcp103 [8/256/0/180] [13959.450828] LNet: Added LNI 192.168.203.146@tcp104 [8/256/0/180] [13960.324208] LNet: Added LNI 192.168.203.146@tcp105 [8/256/0/180] [13961.082319] LNet: Added LNI 192.168.203.146@tcp106 [8/256/0/180] [13962.039574] LNet: Added LNI 192.168.203.146@tcp107 [8/256/0/180] [13962.813876] LNet: Added LNI 192.168.203.146@tcp108 [8/256/0/180] [13963.572290] LNet: Added LNI 192.168.203.146@tcp109 [8/256/0/180] [13964.421745] LNet: Added LNI 192.168.203.146@tcp110 [8/256/0/180] [13965.256600] LNet: Added LNI 192.168.203.146@tcp111 [8/256/0/180] [13966.144435] LNet: Added LNI 192.168.203.146@tcp112 [8/256/0/180] [13966.973843] LNet: Added LNI 192.168.203.146@tcp113 [8/256/0/180] [13967.727629] LNet: Added LNI 192.168.203.146@tcp114 [8/256/0/180] [13968.798499] LNet: Added LNI 192.168.203.146@tcp115 [8/256/0/180] [13969.737854] LNet: Added LNI 192.168.203.146@tcp116 [8/256/0/180] [13970.678756] LNet: Added LNI 192.168.203.146@tcp117 [8/256/0/180] [13971.796273] LNet: Added LNI 192.168.203.146@tcp118 [8/256/0/180] [13972.711811] LNet: Added LNI 192.168.203.146@tcp119 [8/256/0/180] [13973.774995] LNet: Added LNI 192.168.203.146@tcp120 [8/256/0/180] [13974.802392] LNet: Added LNI 192.168.203.146@tcp121 [8/256/0/180] [13975.755790] LNet: Added LNI 192.168.203.146@tcp122 [8/256/0/180] [13976.660065] LNet: Added LNI 192.168.203.146@tcp123 [8/256/0/180] [13977.518751] LNet: Added LNI 192.168.203.146@tcp124 [8/256/0/180] [13978.484442] LNet: Added LNI 192.168.203.146@tcp125 [8/256/0/180] [13979.429182] LNet: Added LNI 192.168.203.146@tcp126 [8/256/0/180] [13980.402693] LNet: Added LNI 192.168.203.146@tcp127 [8/256/0/180] [13981.257932] LNet: Added LNI 192.168.203.146@tcp128 [8/256/0/180] [13982.229223] LNet: Added LNI 192.168.203.146@tcp129 [8/256/0/180] [13983.173865] LNet: Added LNI 192.168.203.146@tcp130 [8/256/0/180] [13984.311464] LNet: Added LNI 192.168.203.146@tcp131 [8/256/0/180] [13985.481040] LNet: Added LNI 192.168.203.146@tcp132 [8/256/0/180] [13986.474836] LNet: Added LNI 192.168.203.146@tcp133 [8/256/0/180] [13987.580742] LNet: Added LNI 192.168.203.146@tcp134 [8/256/0/180] [13988.564032] LNet: Added LNI 192.168.203.146@tcp135 [8/256/0/180] [13996.179756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13996.241480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13997.361871] Lustre: 463963:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.146@tcp: can't add NID 192.168.203.146@tcp130: rc = -75 [13997.450830] LNetError: 463991:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.146 [13997.468218] LNetError: Protocol error connecting to 192.168.203.146@tcp116 at host 192.168.203.146:988: is it running a compatible version of Lustre? [13997.481671] LNetError: Skipped 9 previous similar messages [13997.561644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13997.583963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13997.658541] Lustre: lustre-MDT0000: new disk, initializing [13997.730115] LNet: 5 local NIs in recovery (showing 5): 192.168.203.146@tcp127, 192.168.203.146@tcp129, 192.168.203.146@tcp124, 192.168.203.146@tcp121, 192.168.203.146@tcp128 [13997.734654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13997.755433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13998.496840] LNetError: 463987:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.146 [13998.510624] LNetError: 463987:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 12 previous similar messages [13998.519636] LNetError: Protocol error connecting to 192.168.203.146@tcp119 at host 192.168.203.146:988: is it running a compatible version of Lustre? [13998.532407] LNetError: Skipped 1 previous similar message [14000.801447] LNetError: 461418:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.146 [14000.825643] LNetError: 461418:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [14000.841844] LNetError: Protocol error connecting to 192.168.203.146@tcp127 at host 192.168.203.146:988: is it running a compatible version of Lustre? [14000.860161] LNetError: Skipped 5 previous similar messages [14003.078924] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [14004.896975] LNetError: 463986:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.146 [14004.926397] LNetError: Protocol error connecting to 192.168.203.146@tcp127 at host 192.168.203.146:988: is it running a compatible version of Lustre? [14012.074044] LNetError: 463992:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.146 [14012.095045] LNetError: Protocol error connecting to 192.168.203.146@tcp127 at host 192.168.203.146:988: is it running a compatible version of Lustre? [14016.002254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14016.109796] Lustre: 463980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14016.176624] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14016.180168] Lustre: Skipped 1 previous similar message [14016.242115] Lustre: lustre-MDT0001: new disk, initializing [14016.302803] Lustre: 464786:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.146@tcp: can't add NID 192.168.203.146@tcp130: rc = -75 [14016.326934] Lustre: 464786:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14016.524753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14016.599820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14016.618677] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14021.492842] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [14024.768944] Lustre: 465149:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.146@tcp: can't add NID 192.168.203.146@tcp130: rc = -75 [14024.780090] Lustre: 465149:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14028.455439] LNetError: 463996:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.146 [14028.478550] LNetError: Protocol error connecting to 192.168.203.146@tcp122 at host 192.168.203.146:988: is it running a compatible version of Lustre? [14032.985071] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14040.431189] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14044.834198] LNetError: 461421:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.146 [14044.847961] LNetError: 461421:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [14044.860588] LNetError: Protocol error connecting to 192.168.203.146@tcp120 at host 192.168.203.146:988: is it running a compatible version of Lustre? [14044.888716] LNetError: Skipped 10 previous similar messages [14049.227262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14049.814728] Lustre: lustre-OST0000: new disk, initializing [14049.839242] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14049.853021] Lustre: 465960:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14050.031541] Lustre: 465965:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.146@tcp: can't add NID 192.168.203.146@tcp130: rc = -75 [14050.040149] Lustre: 465965:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14050.122225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14057.644534] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [14059.098751] Lustre: 466384:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.146@tcp: can't add NID 192.168.203.146@tcp130: rc = -75 [14059.107598] Lustre: 466384:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14059.192720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14059.208970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14059.366783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14067.599974] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14079.970962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14079.973712] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14079.976864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14079.981382] LustreError: Skipped 1 previous similar message [14080.015445] Lustre: Skipped 1 previous similar message [14081.943941] Lustre: server umount lustre-OST0000 complete [14088.165200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14088.175160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14088.191506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14088.209401] Lustre: Skipped 1 previous similar message [14090.213377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14090.221710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14092.607391] Lustre: server umount lustre-MDT0000 complete [14092.962886] LNetError: 463996:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.146 [14092.983046] LNetError: 463996:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 21 previous similar messages [14092.989079] LNetError: Protocol error connecting to 192.168.203.146@tcp122 at host 192.168.203.146:988: is it running a compatible version of Lustre? [14092.998963] LNetError: Skipped 21 previous similar messages [14096.149800] LustreError: 463972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788679403 with bad export cookie 6591420144361723845 [14096.155213] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14096.159644] LustreError: 463972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14096.628727] Lustre: server umount lustre-MDT0001 complete [14107.206754] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [14110.040900] Key type lgssc unregistered [14110.463564] LNet: 467874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14110.470198] LNetError: 467874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14110.479621] LNet: Removed LNI 192.168.203.146@tcp [14111.519640] LNet: Removed LNI 192.168.203.146@tcp100 [14111.526078] LNetError: 467874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14111.535228] LNetError: 467874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14111.544992] LNet: Removed LNI 192.168.203.146@tcp101 [14112.609883] LNet: Removed LNI 192.168.203.146@tcp102 [14112.614483] LNetError: 467874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14112.622386] LNetError: 467874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14112.636872] LNet: Removed LNI 192.168.203.146@tcp103 [14113.695408] LNet: Removed LNI 192.168.203.146@tcp104 [14114.719580] LNet: Removed LNI 192.168.203.146@tcp105 [14114.722204] LNetError: 467874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14114.731674] LNetError: 467874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14114.737960] LNet: Removed LNI 192.168.203.146@tcp106 [14115.743216] LNet: Removed LNI 192.168.203.146@tcp107 [14116.767173] LNet: Removed LNI 192.168.203.146@tcp108 [14117.791237] LNet: Removed LNI 192.168.203.146@tcp109 [14118.815396] LNet: Removed LNI 192.168.203.146@tcp110 [14118.822343] LNetError: 467874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14118.831199] LNetError: 467874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14118.854504] LNet: Removed LNI 192.168.203.146@tcp111 [14119.904044] LNet: Removed LNI 192.168.203.146@tcp112 [14120.927653] LNet: Removed LNI 192.168.203.146@tcp113 [14121.963414] LNet: Removed LNI 192.168.203.146@tcp114 [14122.977466] LNet: Removed LNI 192.168.203.146@tcp115 [14123.999165] LNet: Removed LNI 192.168.203.146@tcp116 [14125.023195] LNet: Removed LNI 192.168.203.146@tcp117 [14126.047228] LNet: Removed LNI 192.168.203.146@tcp118 [14127.071205] LNet: Removed LNI 192.168.203.146@tcp119 [14127.078054] LNetError: 467874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14127.084971] LNetError: 467874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14127.096055] LNet: Removed LNI 192.168.203.146@tcp120 [14128.159419] LNet: Removed LNI 192.168.203.146@tcp121 [14129.184218] LNet: Removed LNI 192.168.203.146@tcp122 [14130.207392] LNet: Removed LNI 192.168.203.146@tcp123 [14131.231216] LNet: Removed LNI 192.168.203.146@tcp124 [14132.255796] LNet: Removed LNI 192.168.203.146@tcp125 [14133.279209] LNet: Removed LNI 192.168.203.146@tcp126 [14134.303193] LNet: Removed LNI 192.168.203.146@tcp127 [14135.327345] LNet: Removed LNI 192.168.203.146@tcp128 [14136.351248] LNet: Removed LNI 192.168.203.146@tcp129 [14136.362327] LNet: Removed LNI 192.168.203.146@tcp130 [14136.370477] LNet: Removed LNI 192.168.203.146@tcp131 [14136.378628] LNet: Removed LNI 192.168.203.146@tcp132 [14136.387426] LNet: Removed LNI 192.168.203.146@tcp133 [14136.395892] LNet: Removed LNI 192.168.203.146@tcp134 [14136.425799] LNet: Removed LNI 192.168.203.146@tcp135 [14137.383135] Key type .llcrypt unregistered [14137.386112] Key type ._llcrypt unregistered [14161.489143] Key type ._llcrypt registered [14161.491346] Key type .llcrypt registered [14161.621455] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [14180.951817] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [14182.988679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14183.065161] alg: No test for adler32 (adler32-zlib) [14184.220794] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [14184.537750] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [14186.240268] Key type lgssc registered [14187.750624] Lustre: Echo OBD driver; http://www.lustre.org/ [14227.204837] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 03:25:32 (1788679532) [14248.055780] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [14251.439703] Key type lgssc unregistered [14251.629704] LNet: 471958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14251.639280] LNetError: 471958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14251.661199] LNet: Removed LNI 192.168.203.146@tcp [14252.699075] Key type .llcrypt unregistered [14252.706477] Key type ._llcrypt unregistered [14267.812891] Key type ._llcrypt registered [14267.815755] Key type .llcrypt registered [14267.919827] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [14268.868488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14269.055055] alg: No test for adler32 (adler32-zlib) [14270.148157] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [14270.389678] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [14272.071135] Key type lgssc registered [14273.191861] Lustre: Echo OBD driver; http://www.lustre.org/ [14282.457882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14282.483532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14283.799364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14283.864099] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14283.967293] Lustre: lustre-MDT0000: new disk, initializing [14284.096781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14284.126903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14288.426548] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [14299.597440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14299.840456] Lustre: 473167:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14299.909573] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14299.914814] Lustre: Skipped 1 previous similar message [14300.007671] Lustre: lustre-MDT0001: new disk, initializing [14300.147699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14300.200596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14300.218728] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14304.815798] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [14314.323872] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14320.591713] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14327.684057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14327.934392] Lustre: lustre-OST0000: new disk, initializing [14327.938688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14327.944854] Lustre: 475138:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14328.036456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14334.064554] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [14335.107677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14335.121372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14335.260797] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14342.983533] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14347.398318] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing lnet_if_list [14352.387544] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14352.422284] LNet: Added LNI 192.168.203.146@tcp10 [8/256/0/180] [14357.764685] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14357.796187] LNet: Added LNI 192.168.203.146@tcp11 [8/256/0/180] [14362.315353] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14362.341820] LNet: Added LNI 192.168.203.146@tcp12 [8/256/0/180] [14366.936444] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14366.967584] LNet: Added LNI 192.168.203.146@tcp13 [8/256/0/180] [14370.914514] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14370.998111] LNet: Added LNI 192.168.203.146@tcp14 [8/256/0/180] [14375.298610] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14375.319099] LNet: Added LNI 192.168.203.146@tcp15 [8/256/0/180] [14379.658900] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14379.683171] LNet: Added LNI 192.168.203.146@tcp16 [8/256/0/180] [14384.481189] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14384.502863] LNet: Added LNI 192.168.203.146@tcp17 [8/256/0/180] [14390.651373] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14390.685964] LNet: Added LNI 192.168.203.146@tcp18 [8/256/0/180] [14395.736448] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14395.767027] LNet: Added LNI 192.168.203.146@tcp19 [8/256/0/180] [14401.011679] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14401.046031] LNet: Added LNI 192.168.203.146@tcp20 [8/256/0/180] [14406.501297] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14406.553574] LNet: Added LNI 192.168.203.146@tcp21 [8/256/0/180] [14412.264283] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14412.308743] LNet: Added LNI 192.168.203.146@tcp22 [8/256/0/180] [14419.437403] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14419.466718] LNet: Added LNI 192.168.203.146@tcp23 [8/256/0/180] [14427.193544] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14427.231058] LNet: Added LNI 192.168.203.146@tcp24 [8/256/0/180] [14434.855347] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14434.901521] LNet: Added LNI 192.168.203.146@tcp25 [8/256/0/180] [14442.119970] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14442.148615] LNet: Added LNI 192.168.203.146@tcp26 [8/256/0/180] [14448.503803] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14448.530580] LNet: Added LNI 192.168.203.146@tcp27 [8/256/0/180] [14453.849726] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14453.877388] LNet: Added LNI 192.168.203.146@tcp28 [8/256/0/180] [14459.669659] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14459.715581] LNet: Added LNI 192.168.203.146@tcp29 [8/256/0/180] [14466.541272] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14466.576828] LNet: Added LNI 192.168.203.146@tcp30 [8/256/0/180] [14472.410564] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14472.431457] LNet: Added LNI 192.168.203.146@tcp31 [8/256/0/180] [14478.353678] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14478.390942] LNet: Added LNI 192.168.203.146@tcp32 [8/256/0/180] [14484.405731] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14484.456422] LNet: Added LNI 192.168.203.146@tcp33 [8/256/0/180] [14489.575282] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14489.601792] LNet: Added LNI 192.168.203.146@tcp34 [8/256/0/180] [14495.242766] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14495.271298] LNet: Added LNI 192.168.203.146@tcp35 [8/256/0/180] [14501.291083] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14501.338902] LNet: Added LNI 192.168.203.146@tcp36 [8/256/0/180] [14507.911789] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14507.944836] LNet: Added LNI 192.168.203.146@tcp37 [8/256/0/180] [14513.946476] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14513.979736] LNet: Added LNI 192.168.203.146@tcp38 [8/256/0/180] [14519.684240] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14519.731403] LNet: Added LNI 192.168.203.146@tcp39 [8/256/0/180] [14525.988192] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14526.013793] LNet: Added LNI 192.168.203.146@tcp40 [8/256/0/180] [14533.771375] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14533.807763] LNet: Added LNI 192.168.203.146@tcp41 [8/256/0/180] [14541.654892] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14541.694531] LNet: Added LNI 192.168.203.146@tcp42 [8/256/0/180] [14550.232156] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14550.271142] LNet: Added LNI 192.168.203.146@tcp43 [8/256/0/180] [14559.179733] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14559.221733] LNet: Added LNI 192.168.203.146@tcp44 [8/256/0/180] [14567.037908] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14567.077099] LNet: Added LNI 192.168.203.146@tcp45 [8/256/0/180] [14575.020605] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14575.050280] LNet: Added LNI 192.168.203.146@tcp46 [8/256/0/180] [14583.411786] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14583.432631] LNet: Added LNI 192.168.203.146@tcp47 [8/256/0/180] [14592.730029] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14592.766295] LNet: Added LNI 192.168.203.146@tcp48 [8/256/0/180] [14604.550946] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14604.602828] LNet: Added LNI 192.168.203.146@tcp49 [8/256/0/180] [14614.816229] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14614.861169] LNet: Added LNI 192.168.203.146@tcp50 [8/256/0/180] [14632.928264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14632.931695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14632.958826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14637.537379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14637.544340] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14638.985496] Lustre: server umount lustre-MDT0000 complete [14642.609338] LustreError: 473159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788679949 with bad export cookie 16345385827350176934 [14642.621225] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14642.644743] LustreError: 473159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14642.665520] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14642.669426] Lustre: Skipped 1 previous similar message [14642.675960] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14642.683815] Lustre: Skipped 2 previous similar messages [14642.874393] Lustre: server umount lustre-MDT0001 complete [14663.369282] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [14673.153826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14687.713445] LustreError: 472438:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a260efc6 x1875566476300928/t0(0) o250->MGC192.168.203.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14688.115379] LustreError: 475131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14688.139773] LustreError: 475131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14688.161725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14692.258866] LustreError: 475132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14692.272752] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14692.309856] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [14697.448434] LustreError: 475131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14700.987978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14701.480689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14705.812531] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [14707.698545] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14712.172406] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14718.197904] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14738.414189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14738.429331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14738.439107] Lustre: Skipped 1 previous similar message [14738.458400] Lustre: Skipped 1 previous similar message [14743.522086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14743.529913] Lustre: Skipped 1 previous similar message [14743.831717] Lustre: server umount lustre-OST0000 complete [14748.643042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14748.652373] Lustre: Skipped 1 previous similar message [14753.766293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14753.781620] Lustre: Skipped 3 previous similar messages [14754.109826] Lustre: server umount lustre-MDT0000 complete [14757.824967] LustreError: 473159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788680064 with bad export cookie 16345385827350179524 [14757.828421] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14757.846039] LustreError: 473159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14758.250802] Lustre: server umount lustre-MDT0001 complete [14770.435970] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [14773.071903] Key type lgssc unregistered [14773.511334] LNet: 502177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14773.517453] LNetError: 502177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14773.523556] LNet: Removed LNI 192.168.203.146@tcp [14773.527832] LNet: Removed LNI 192.168.203.146@tcp10 [14773.531536] LNet: Removed LNI 192.168.203.146@tcp11 [14773.537089] LNet: Removed LNI 192.168.203.146@tcp12 [14773.539146] LNet: Removed LNI 192.168.203.146@tcp13 [14773.542125] LNet: Removed LNI 192.168.203.146@tcp14 [14773.545188] LNet: Removed LNI 192.168.203.146@tcp15 [14773.547212] LNet: Removed LNI 192.168.203.146@tcp16 [14773.550488] LNet: Removed LNI 192.168.203.146@tcp17 [14773.552973] LNet: Removed LNI 192.168.203.146@tcp18 [14773.559055] LNet: Removed LNI 192.168.203.146@tcp19 [14773.563455] LNet: Removed LNI 192.168.203.146@tcp20 [14773.567981] LNet: Removed LNI 192.168.203.146@tcp21 [14773.571895] LNet: Removed LNI 192.168.203.146@tcp22 [14773.581921] LNet: Removed LNI 192.168.203.146@tcp23 [14773.586820] LNet: Removed LNI 192.168.203.146@tcp24 [14773.589669] LNet: Removed LNI 192.168.203.146@tcp25 [14773.592250] LNet: Removed LNI 192.168.203.146@tcp26 [14773.594531] LNet: Removed LNI 192.168.203.146@tcp27 [14773.597698] LNet: Removed LNI 192.168.203.146@tcp28 [14773.604465] LNet: Removed LNI 192.168.203.146@tcp29 [14773.614869] LNet: Removed LNI 192.168.203.146@tcp30 [14773.616984] LNet: Removed LNI 192.168.203.146@tcp31 [14773.619234] LNet: Removed LNI 192.168.203.146@tcp32 [14773.622520] LNet: Removed LNI 192.168.203.146@tcp33 [14773.627601] LNet: Removed LNI 192.168.203.146@tcp34 [14773.630284] LNet: Removed LNI 192.168.203.146@tcp35 [14773.633328] LNet: Removed LNI 192.168.203.146@tcp36 [14773.635263] LNet: Removed LNI 192.168.203.146@tcp37 [14773.640603] LNet: Removed LNI 192.168.203.146@tcp38 [14773.644156] LNet: Removed LNI 192.168.203.146@tcp39 [14773.647601] LNet: Removed LNI 192.168.203.146@tcp40 [14773.653516] LNet: Removed LNI 192.168.203.146@tcp41 [14773.660539] LNet: Removed LNI 192.168.203.146@tcp42 [14773.663972] LNet: Removed LNI 192.168.203.146@tcp43 [14773.665956] LNet: Removed LNI 192.168.203.146@tcp44 [14773.668627] LNet: Removed LNI 192.168.203.146@tcp45 [14773.676131] LNet: Removed LNI 192.168.203.146@tcp46 [14773.678821] LNet: Removed LNI 192.168.203.146@tcp47 [14773.683544] LNet: Removed LNI 192.168.203.146@tcp48 [14773.687941] LNet: Removed LNI 192.168.203.146@tcp49 [14773.698929] LNet: Removed LNI 192.168.203.146@tcp50 [14774.601641] Key type .llcrypt unregistered [14774.603435] Key type ._llcrypt unregistered [14796.149080] Key type ._llcrypt registered [14796.151237] Key type .llcrypt registered [14796.259444] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [14811.482032] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [14812.462278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14812.479779] alg: No test for adler32 (adler32-zlib) [14813.739185] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [14813.931634] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [14815.624741] Key type lgssc registered [14816.924098] Lustre: Echo OBD driver; http://www.lustre.org/ [14856.274106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14856.306619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14857.727994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14857.763625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14857.900776] Lustre: lustre-MDT0000: new disk, initializing [14858.112695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14858.145834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14863.108946] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [14874.017064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14874.139891] Lustre: 505369:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14874.216312] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14874.223264] Lustre: Skipped 1 previous similar message [14874.322382] Lustre: lustre-MDT0001: new disk, initializing [14874.449383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14874.507547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14874.517072] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14879.279511] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [14888.216335] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14893.320088] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14901.988290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14902.322531] Lustre: *** cfs_fail_loc=90f, val=0*** [14902.395480] Lustre: lustre-OST0000: new disk, initializing [14902.398661] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14902.404777] Lustre: 507388:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14902.492897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14909.078192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14909.104523] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14909.228701] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14909.711321] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [14920.449462] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14929.889389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14929.892629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14929.913055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14929.933379] Lustre: Skipped 1 previous similar message [14935.017930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14935.021688] Lustre: Skipped 1 previous similar message [14940.137605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14940.144860] Lustre: Skipped 1 previous similar message [14943.199318] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14943.330865] Lustre: server umount lustre-OST0000 complete [14950.375063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14950.389727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14950.405056] Lustre: Skipped 2 previous similar messages [14953.880790] Lustre: server umount lustre-MDT0000 complete [14958.122520] LustreError: 507385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788680265 with bad export cookie 10893686064150616663 [14958.123555] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14958.140911] LustreError: 507385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14958.533145] Lustre: server umount lustre-MDT0001 complete [14969.422225] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [14972.595385] Key type lgssc unregistered [14972.986891] LNet: 509296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14972.992511] LNetError: 509296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14973.023575] LNet: Removed LNI 192.168.203.146@tcp [14973.830800] Key type .llcrypt unregistered [14973.833762] Key type ._llcrypt unregistered [14992.944708] Key type ._llcrypt registered [14992.948665] Key type .llcrypt registered [14993.060022] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [14994.073732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14994.264356] alg: No test for adler32 (adler32-zlib) [14995.411340] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [14995.748116] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [14997.439179] Key type lgssc registered [14998.490105] Lustre: Echo OBD driver; http://www.lustre.org/ [15004.894631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15004.922514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15006.305160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15010.500244] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [15016.425388] LustreError: 510562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15016.444338] LustreError: 510562:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15019.048775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15019.538835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15023.214703] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [15029.725527] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15035.244653] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15041.366893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15041.759102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15047.445597] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [15054.201559] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15065.297622] loop0: detected capacity change from 0 to 204800000 [15065.548580] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15065.585995] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15065.622807] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15072.187068] loop0: detected capacity change from 0 to 204800000 [15072.342397] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15072.365711] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15072.377246] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15077.857160] loop0: detected capacity change from 0 to 204800000 [15083.710409] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15083.950791] Lustre: fs15246-OST0000: new disk, initializing [15083.958419] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15083.970607] Lustre: 514155:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15084.060252] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15091.292076] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [15101.343133] loop1: detected capacity change from 0 to 204800000 [15106.937165] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15107.017219] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15107.071795] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15107.141959] Lustre: fs15246-MDT0000: new disk, initializing [15107.250985] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15107.285939] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15107.392128] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15108.449930] Lustre: 515276:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000007d738ad1 x1875567236395392/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788680431 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15111.466900] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [15117.607288] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15117.624129] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15117.667496] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15131.621384] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15131.638962] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15135.764901] Lustre: server umount fs15246-MDT0000 complete [15136.739832] LustreError: 510557:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15141.863342] LustreError: 510558:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15143.372831] Lustre: server umount fs15246-OST0000 complete [15157.241892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15157.245691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15157.255245] Lustre: Skipped 1 previous similar message [15157.273281] Lustre: Skipped 1 previous similar message [15160.750728] Lustre: server umount lustre-OST0000 complete [15167.456894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15167.459539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15167.470978] Lustre: Skipped 1 previous similar message [15170.350988] Lustre: server umount lustre-MDT0000 complete [15172.576260] LustreError: 513335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15172.618813] LustreError: 513335:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15174.096170] LustreError: 516324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788680481 with bad export cookie 6869288939198632241 [15174.102453] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15174.105642] LustreError: 516324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15174.483714] Lustre: server umount lustre-MDT0001 complete [15185.256394] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [15188.151459] Key type lgssc unregistered [15188.463032] LNet: 518372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15188.474883] LNetError: 518372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15188.502181] LNet: Removed LNI 192.168.203.146@tcp [15189.418740] Key type .llcrypt unregistered [15189.421961] Key type ._llcrypt unregistered [15207.460801] Key type ._llcrypt registered [15207.469483] Key type .llcrypt registered [15207.653238] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [15209.203249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15209.279525] alg: No test for adler32 (adler32-zlib) [15210.435740] Lustre: Lustre: Build Version: 2.17.58_3_gb7bcc87 [15210.748852] LNet: Added LNI 192.168.203.146@tcp [8/256/0/180] [15212.457971] Key type lgssc registered [15213.699249] Lustre: Echo OBD driver; http://www.lustre.org/ [15252.576444] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [15261.511579] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [15294.172808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15294.210791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15295.462884] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15295.489708] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15295.546403] Lustre: lustre-MDT0000: new disk, initializing [15295.595276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15295.609353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15299.687081] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [15309.872815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15310.004418] Lustre: 522836:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15310.079476] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15310.086483] Lustre: Skipped 1 previous similar message [15310.142857] Lustre: lustre-MDT0001: new disk, initializing [15310.233517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15310.268720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15310.284467] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15315.163842] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [15324.744704] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15330.788085] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15337.885271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15338.199226] Lustre: lustre-OST0000: new disk, initializing [15338.212328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15338.225039] Lustre: 524802:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15338.327817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15344.307120] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_default_debug -1 all [15346.758990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15346.773348] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15346.848161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15353.649389] Lustre: DEBUG MARKER: oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15358.531294] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15358.867997] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15364.174518] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15364.378587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15367.650381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15367.650770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15367.662411] LustreError: Skipped 1 previous similar message [15367.687241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15367.703366] Lustre: Skipped 1 previous similar message [15372.801603] Lustre: server umount lustre-OST0000 complete [15376.864603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15376.884782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15376.907240] Lustre: Skipped 1 previous similar message [15376.917700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15377.890160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15382.893927] Lustre: server umount lustre-MDT0000 complete [15386.950481] LustreError: 523630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788680694 with bad export cookie 6867217857079370825 [15386.953105] LustreError: MGC192.168.203.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15386.960640] LustreError: 523630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15387.372115] Lustre: server umount lustre-MDT0001 complete [15411.151612] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing set_hostid [15420.636399] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing load_modules_local [15448.180643] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15118 sec ========= 03:45:54 (1788680754) [15449.479985] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:45:55 (1788680755) === [15452.082852] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:45:58 (1788680758) === [15488.638183] Lustre: DEBUG MARKER: oleg346-server.virtnet: executing unload_modules_local [15491.313576] Key type lgssc unregistered [15491.548098] LNet: 531442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15491.553977] LNetError: 531442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15491.567080] LNet: Removed LNI 192.168.203.146@tcp [15492.285472] Key type .llcrypt unregistered [15492.288518] Key type ._llcrypt unregistered