[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-22) (GCC)) #7 SMP Sat Jan 18 21:01:29 EST 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-1.fc39 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 561884298 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcbe3000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2448MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 513 pages/cpu s2064384 r8192 d28672 u4194304 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2818960K/4306352K available (20483K kernel code, 12066K rwdata, 7356K rodata, 4680K init, 23504K bss, 542476K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41388 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x616/0x99a with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar [ 0.001000] ... MAX_LOCKDEP_SUBCLASSES: 8 [ 0.001000] ... MAX_LOCK_DEPTH: 48 [ 0.001000] ... MAX_LOCKDEP_KEYS: 8192 [ 0.001000] ... CLASSHASH_SIZE: 4096 [ 0.001000] ... MAX_LOCKDEP_ENTRIES: 32768 [ 0.001000] ... MAX_LOCKDEP_CHAINS: 65536 [ 0.001000] ... CHAINHASH_SIZE: 32768 [ 0.001000] memory used by lock dependency info: 4149 kB [ 0.001000] per task-struct memory footprint: 2688 bytes [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003220] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011016] pid_max: default: 32768 minimum: 301 [ 0.013406] LSM: Security Framework initializing [ 0.014143] Yama: becoming mindful. [ 0.015141] SELinux: Initializing. [ 0.016298] *** VALIDATE selinux *** [ 0.028362] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.033352] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.034305] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035187] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.036340] *** VALIDATE tmpfs *** [ 0.038668] *** VALIDATE proc *** [ 0.040386] *** VALIDATE cgroup *** [ 0.041019] *** VALIDATE cgroup2 *** [ 0.042432] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.043260] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.044019] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.045047] Spectre V2 : User space: Vulnerable [ 0.046020] Speculative Store Bypass: Vulnerable [ 0.049321] debug: unmapping init [mem 0xffffffff88b03000-0xffffffff88b0afff] [ 0.051000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.053327] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.054063] ... version: 2 [ 0.055022] ... bit width: 48 [ 0.056027] ... generic registers: 4 [ 0.057025] ... value mask: 0000ffffffffffff [ 0.058028] ... max period: 00007fffffffffff [ 0.059027] ... fixed-purpose events: 3 [ 0.060022] ... event mask: 000000070000000f [ 0.061695] rcu: Hierarchical SRCU implementation. [ 0.066496] smp: Bringing up secondary CPUs ... [ 0.068937] x86: Booting SMP configuration: [ 0.069035] .... node #0, CPUs: #1 [ 0.075996] #2 [ 0.083169] #3 [ 0.085307] smp: Brought up 1 node, 4 CPUs [ 0.086052] smpboot: Max logical packages: 1 [ 0.087028] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.191044] node 0 deferred pages initialised in 99ms [ 0.192508] pgdatinit0 (35) used greatest stack depth: 14312 bytes left [ 0.199392] devtmpfs: initialized [ 0.201158] x86/mm: Memory block size: 128MB [ 0.211726] gcov: version magic: 0x41383552 [ 0.217231] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.218177] futex hash table entries: 1024 (order: 5, 131072 bytes, vmalloc) [ 0.219848] pinctrl core: initialized pinctrl subsystem [ 0.220832] [ 0.221021] ************************************************************* [ 0.222022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.223022] ** ** [ 0.224021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.225024] ** ** [ 0.226024] ** This means that this kernel is built to expose internal ** [ 0.227022] ** IOMMU data structures, which may compromise security on ** [ 0.228026] ** your system. ** [ 0.229022] ** ** [ 0.230025] ** If you see this message and you are not debugging the ** [ 0.231021] ** kernel, report this immediately to your vendor! ** [ 0.232022] ** ** [ 0.233024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.234025] ************************************************************* [ 0.237460] NET: Registered protocol family 16 [ 0.240158] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.241131] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.242144] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.246145] cpuidle: using governor menu [ 0.248578] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.253597] PCI: Using configuration type 1 for base access [ 0.257296] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.306710] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.309424] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.332055] cryptd: max_cpu_qlen set to 1000 [ 0.337697] ACPI: Added _OSI(Module Device) [ 0.339030] ACPI: Added _OSI(Processor Device) [ 0.340024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.342028] ACPI: Added _OSI(Processor Aggregator Device) [ 0.383098] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.396478] ACPI: Interpreter enabled [ 0.398260] ACPI: PM: (supports S0 S3 S4 S5) [ 0.400025] ACPI: Using IOAPIC for interrupt routing [ 0.401348] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.407000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.496393] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.499109] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.502034] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.506491] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.521063] acpiphp: Slot [2] registered [ 0.522431] acpiphp: Slot [5] registered [ 0.523471] acpiphp: Slot [6] registered [ 0.524364] acpiphp: Slot [7] registered [ 0.525464] acpiphp: Slot [8] registered [ 0.526366] acpiphp: Slot [9] registered [ 0.527372] acpiphp: Slot [10] registered [ 0.528448] acpiphp: Slot [3] registered [ 0.529368] acpiphp: Slot [4] registered [ 0.530370] acpiphp: Slot [11] registered [ 0.531406] acpiphp: Slot [12] registered [ 0.532415] acpiphp: Slot [13] registered [ 0.533347] acpiphp: Slot [14] registered [ 0.534355] acpiphp: Slot [15] registered [ 0.535320] acpiphp: Slot [16] registered [ 0.536327] acpiphp: Slot [17] registered [ 0.537326] acpiphp: Slot [18] registered [ 0.538389] acpiphp: Slot [19] registered [ 0.539341] acpiphp: Slot [20] registered [ 0.540307] acpiphp: Slot [21] registered [ 0.541432] acpiphp: Slot [22] registered [ 0.542374] acpiphp: Slot [23] registered [ 0.543383] acpiphp: Slot [24] registered [ 0.544372] acpiphp: Slot [25] registered [ 0.545382] acpiphp: Slot [26] registered [ 0.546324] acpiphp: Slot [27] registered [ 0.547372] acpiphp: Slot [28] registered [ 0.548330] acpiphp: Slot [29] registered [ 0.549332] acpiphp: Slot [30] registered [ 0.550406] acpiphp: Slot [31] registered [ 0.551207] PCI host bridge to bus 0000:00 [ 0.552057] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.553058] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.554058] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.555064] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.556093] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.557067] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.558469] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.561580] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.565461] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.571640] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.574986] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.575034] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.576029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.577031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.580040] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.581883] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.582063] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.585175] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.587845] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.593916] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.595028] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.614387] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.617027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.620028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.627028] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.648698] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.651022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.654023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.662029] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.690182] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.699034] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.709025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.727026] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.756907] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.759028] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.762024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.770019] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.794335] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.804047] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.813030] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.836020] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.869365] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.878028] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.887025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.911026] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.947554] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.951603] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.954284] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.957081] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.958920] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.968410] iommu: Default domain type: Passthrough [ 0.969449] SCSI subsystem initialized [ 0.970700] ACPI: bus type USB registered [ 0.972576] usbcore: registered new interface driver usbfs [ 0.975365] usbcore: registered new interface driver hub [ 0.977246] usbcore: registered new device driver usb [ 0.979905] pps_core: LinuxPPS API ver. 1 registered [ 0.982023] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.984171] PTP clock support registered [ 0.988089] EDAC MC: Ver: 3.0.0 [ 0.990984] PCI: Using ACPI for IRQ routing [ 0.993069] NetLabel: Initializing [ 0.994020] NetLabel: domain hash size = 128 [ 0.996022] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.998443] NetLabel: unlabeled traffic allowed by default [ 1.001832] vgaarb: loaded [ 1.003550] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.006022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.018779] clocksource: Switched to clocksource kvm-clock [ 1.503801] VFS: Disk quotas dquot_6.6.0 [ 1.505523] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.508464] *** VALIDATE ramfs *** [ 1.509870] *** VALIDATE hugetlbfs *** [ 1.512269] pnp: PnP ACPI init [ 1.522224] pnp: PnP ACPI: found 6 devices [ 1.563189] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.568664] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.571222] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.573758] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.580472] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.583189] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 1.587114] NET: Registered protocol family 2 [ 1.592410] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.601311] tcp_listen_portaddr_hash hash table entries: 4096 (order: 6, 360448 bytes, vmalloc) [ 1.605434] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.612866] TCP bind hash table entries: 65536 (order: 10, 5242880 bytes, vmalloc) [ 1.620254] TCP: Hash tables configured (established 65536 bind 65536) [ 1.624622] MPTCP token hash table entries: 8192 (order: 7, 786432 bytes, vmalloc) [ 1.629680] UDP hash table entries: 4096 (order: 7, 786432 bytes, vmalloc) [ 1.636698] UDP-Lite hash table entries: 4096 (order: 7, 786432 bytes, vmalloc) [ 1.641813] NET: Registered protocol family 1 [ 1.645994] RPC: Registered named UNIX socket transport module. [ 1.648730] RPC: Registered udp transport module. [ 1.650824] RPC: Registered tcp transport module. [ 1.653127] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.655527] NET: Registered protocol family 44 [ 1.657720] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.660221] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.662634] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.665855] PCI: CLS 0 bytes, default 64 [ 1.667915] Unpacking initramfs... [ 5.215962] debug: unmapping init [mem 0xffff9a50bcbe3000-0xffff9a50bffbffff] [ 5.227355] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.234757] software IO TLB: mapped [mem 0x00000000b8be3000-0x00000000bcbe3000] (64MB) [ 5.239863] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.280916] cryptomgr_test (66) used greatest stack depth: 14248 bytes left [ 9.273912] Initialise system trusted keyrings [ 9.279586] Key type blacklist registered [ 9.294677] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 9.582624] zbud: loaded [ 9.615936] *** VALIDATE nfs *** [ 9.621438] *** VALIDATE nfs4 *** [ 9.631443] pstore: using deflate compression [ 9.651283] Platform Keyring initialized [ 9.669916] cryptomgr_test (74) used greatest stack depth: 14024 bytes left [ 9.688048] cryptomgr_test (81) used greatest stack depth: 14008 bytes left [ 9.743952] cryptomgr_test (87) used greatest stack depth: 13800 bytes left [ 9.831373] cryptomgr_test (95) used greatest stack depth: 13640 bytes left [ 10.170531] cryptomgr_test (120) used greatest stack depth: 13592 bytes left [ 10.185549] NET: Registered protocol family 38 [ 10.188514] Key type asymmetric registered [ 10.191248] Asymmetric key parser 'x509' registered [ 10.195135] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 10.201580] io scheduler mq-deadline registered [ 10.204187] io scheduler kyber registered [ 10.207372] io scheduler bfq registered [ 10.210862] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 10.220168] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 10.224786] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 10.228682] ACPI: Power Button [PWRF] [ 13.414682] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 17.126058] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 22.103704] hrtimer: interrupt took 8299125 ns [ 25.249443] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 29.005131] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 37.655261] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 37.726984] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 37.795800] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 37.826931] Non-volatile memory driver v1.3 [ 37.831958] Linux agpgart interface v0.103 [ 38.243598] virtio_blk virtio1: [vda] 131896 512-byte logical blocks (67.5 MB/64.4 MiB) [ 38.259578] vda: detected capacity change from 0 to 67530752 [ 38.376126] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 38.380431] vdb: detected capacity change from 0 to 1073741824 [ 38.432678] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 38.438667] vdc: detected capacity change from 0 to 2621440000 [ 38.496885] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 38.504109] vdd: detected capacity change from 0 to 2621440000 [ 38.536301] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 38.539430] vde: detected capacity change from 0 to 4294967296 [ 38.579247] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 38.582282] vdf: detected capacity change from 0 to 4294967296 [ 38.625954] libphy: Fixed MDIO Bus: probed [ 38.649572] usbcore: registered new interface driver usbserial_generic [ 38.655160] usbserial: USB Serial support registered for generic [ 38.658795] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 38.666959] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 38.670662] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 38.681932] mousedev: PS/2 mouse device common for all mice [ 38.691822] rtc_cmos 00:05: RTC can wake from S4 [ 38.699854] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 38.710271] rtc_cmos 00:05: registered as rtc0 [ 38.712531] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 38.715612] intel_pstate: CPU model not supported [ 38.723079] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 38.736211] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 38.739708] hid: raw HID events driver (C) Jiri Kosina [ 38.749603] usbcore: registered new interface driver usbhid [ 38.751872] usbhid: USB HID core driver [ 38.754836] drop_monitor: Initializing network drop monitor service [ 38.757940] Initializing XFRM netlink socket [ 38.772271] NET: Registered protocol family 10 [ 38.788366] Segment Routing with IPv6 [ 38.790088] NET: Registered protocol family 17 [ 38.794786] mpls_gso: MPLS GSO support [ 38.813243] RAS: Correctable Errors collector initialized. [ 38.815192] AVX version of gcm_enc/dec engaged. [ 38.824971] AES CTR mode by8 optimization enabled [ 39.297734] sched_clock: Marking stable (39297694358, 0)->(40543038511, -1245344153) [ 39.311330] registered taskstats version 1 [ 39.319535] Loading compiled-in X.509 certificates [ 39.330871] zswap: loaded using pool lzo/zbud [ 39.511244] Key type big_key registered [ 39.583527] Key type encrypted registered [ 39.600959] ima: No TPM chip found, activating TPM-bypass! [ 39.604055] ima: Allocated hash algorithm: sha1 [ 39.605949] ima: No architecture policies found [ 39.609275] evm: Initialising EVM extended attributes: [ 39.611742] evm: security.selinux [ 39.612954] evm: security.ima [ 39.614331] evm: security.capability [ 39.618749] evm: HMAC attrs: 0x1 [ 39.646485] rtc_cmos 00:05: setting system clock to 2025-04-01 08:38:27 UTC (1743496707) [ 39.743187] debug: unmapping init [mem 0xffffffff8a003000-0xffffffff8a1fffff] [ 39.773175] debug: unmapping init [mem 0xffffffff88671000-0xffffffff88b02fff] [ 39.804386] Write protecting the kernel read-only data: 30720k [ 39.808435] debug: unmapping init [mem 0xffffffff86c03000-0xffffffff86dfffff] [ 39.811782] debug: unmapping init [mem 0xffffffff8752f000-0xffffffff875fffff] [ 39.956675] systemd[1]: systemd 239 (239-82.el8_10.3) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 39.979263] systemd[1]: Detected virtualization kvm. [ 39.985585] systemd[1]: Detected architecture x86-64. [ 39.989525] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 40.067753] systemd[1]: No hostname configured. [ 40.075750] systemd[1]: Set hostname to . [ 40.083984] random: systemd: uninitialized urandom read (16 bytes read) [ 40.095463] systemd[1]: Initializing machine ID from random generator. [ 40.870319] random: systemd: uninitialized urandom read (16 bytes read) [ 40.874059] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 40.896382] random: systemd: uninitialized urandom read (16 bytes read) [ 40.910450] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 40.987541] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ 41.123429] urandom_read: 4 callbacks suppressed [ 41.123437] random: systemd: uninitialized urandom read (16 bytes read) [ OK ] Reached target Local File Systems. [ 41.146546] random: systemd: uninitialized urandom read (16 bytes read) [ OK ] Reached target Paths. [ 41.168674] random: systemd: uninitialized urandom read (16 bytes read) [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ 41.892385] mount (249) used greatest stack depth: 13528 bytes left [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service.[ 42.731980] random: fast init done [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 44.171315] device-mapper: uevent: version 1.0.3 [ 44.177359] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 47.766891] udevadm (425) used greatest stack depth: 13344 bytes left [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 48.858840] virtio_net virtio0 ens2: renamed from eth0 [ 51.100238] scsi host0: ata_piix [ 51.133219] scsi host1: ata_piix [ 51.136166] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 51.138959] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 52.599164] systemd-udevd (447) used greatest stack depth: 12664 bytes left [ 53.287516] random: crng init done [ 54.012161] ip (529) used greatest stack depth: 11496 bytes left [ 58.819930] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 62.411909] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 68.760113] printk: systemd: 26 output lines suppressed due to ratelimiting [ 70.172333] SELinux: Disabled at runtime. [ 70.363638] systemd[1]: systemd 239 (239-82.el8_10.3) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 70.382509] systemd[1]: Detected virtualization kvm. [ 70.392880] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 73.267677] systemd[1]: initrd-switch-root.service: Succeeded. [ 73.293441] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 73.320913] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 73.337521] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 73.351451] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 73.416989] systemd[1]: Starting Journal Service... Starting Journal Service... [ 73.456411] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ 74.322535] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 77.284706] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 80.584151] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 80.664340] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ OK ] Started udev Coldplug all Devices. [ 82.471448] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 82.695465] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (13s / no limit) [** ] A start job is running for Configur…only root support (13s / no limit) [*** ] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit) [ ***] A start job is running for Configur…only root support (15s / no limit) [ **] A start job is running for Configur…only root support (16s / no limit) [ *] A start job is running for Configur…only root support (17s / no limit) [ **] A start job is running for Configur…only root support (17s / no limit) [ ***] A start job is running for Configur…only root support (18s / no limit) [ *** ] A start job is running for Configur…only root support (18s / no limit) [ *** ] A start job is running for Configur…only root support (19s / no limit) [*** ] A start job is running for Configur…only root support (19s / no limit) [** ] A start job is running for Configur…only root support (20s / no limit) [* ] A start job is running for Configur…only root support (21s / no limit)[ 94.735315] Key type dns_resolver registered [** ] A start job is running for Configur…only root support (21s / no limit) [*** ] A start job is running for Configur…only root support (22s / no limit) [ *** ] A start job is running for Configur…only root support (22s / no limit)[ 95.888635] NFS: Registering the id_resolver key type [ 95.897819] Key type id_resolver registered [ 95.899463] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (23s / no limit)[ 96.990230] mount.nfs (977) used greatest stack depth: 10456 bytes left [ ***] A start job is running for Configur…only root support (24s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg617-server login: [ 206.013287] libcfs: loading out-of-tree module taints kernel. [ 206.135567] Key type ._llcrypt registered [ 206.145872] Key type .llcrypt registered [ 206.493098] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 244.298330] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 249.783431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 249.850810] alg: No test for adler32 (adler32-zlib) [ 252.301985] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 253.761397] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 253.769635] LNet: Accept secure, port 988 [ 255.889456] Key type lgssc registered [ 259.726424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 290.032818] ZFS: Loaded module v2.3.0-1, ZFS pool version 5000, ZFS filesystem version 5 [ 290.044861] modprobe (4346) used greatest stack depth: 5584 bytes left [ 296.052291] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 318.172602] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 332.718615] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 347.728545] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 382.497351] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 412.745434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 412.938242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 412.987730] ------------[ cut here ]------------ [ 412.993636] DEBUG_LOCKS_WARN_ON(!lockdep_enabled()) [ 412.993668] WARNING: CPU: 3 PID: 6530 at kernel/locking/lockdep.c:4713 lockdep_init_map_type+0x29d/0x410 [ 413.006219] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 413.043805] CPU: 3 PID: 6530 Comm: mount.lustre Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #7 [ 413.051316] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-1.fc39 04/01/2014 [ 413.055310] RIP: 0010:lockdep_init_map_type+0x29d/0x410 [ 413.064379] Code: c0 0f 85 db fe ff ff 48 c7 c6 66 64 39 87 48 c7 c7 a7 4e 37 87 48 83 05 f0 df 25 03 01 e8 27 14 f5 ff 48 83 05 eb df 25 03 01 <0f> 0b 48 83 05 e9 df 25 03 01 48 83 05 e9 df 25 03 01 e9 a1 fe ff [ 413.103234] RSP: 0018:ffffa5d302573748 EFLAGS: 00010202 [ 413.117105] RAX: 0000000000000000 RBX: ffff9a513e59c4e8 RCX: 0000000000000001 [ 413.120655] RDX: 0000000000000001 RSI: 00000000ffff7fff RDI: ffff9a5141fde800 [ 413.127206] RBP: ffffffffc15637c0 R08: 0000000000000000 R09: c0000000ffff7fff [ 413.139134] R10: 0000000000000001 R11: ffffa5d302573538 R12: 0000000000000002 [ 413.144630] R13: ffff9a51387bc000 R14: 0000000000000000 R15: 0000000000000001 [ 413.160390] FS: 00007f31b5dbbb40(0000) GS:ffff9a5141e00000(0000) knlGS:0000000000000000 [ 413.164710] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 413.170203] CR2: 00007ffd6f8a7000 CR3: 0000000110010006 CR4: 0000000000170ee0 [ 413.180040] Call Trace: [ 413.185812] ? show_regs.cold.9+0x22/0x2f [ 413.187333] ? __warn+0xc8/0x150 [ 413.191360] ? lockdep_init_map_type+0x29d/0x410 [ 413.198786] ? report_bug+0x113/0x140 [ 413.206910] ? do_error_trap+0xb6/0x130 [ 413.212416] ? do_invalid_op+0x46/0x60 [ 413.222123] ? lockdep_init_map_type+0x29d/0x410 [ 413.226019] ? invalid_op+0x14/0x20 [ 413.236547] ? lockdep_init_map_type+0x29d/0x410 [ 413.238279] ? lockdep_init_map_type+0x295/0x410 [ 413.252930] ldiskfs_enable_quotas+0x1b9/0x4a0 [ldiskfs] [ 413.262475] ldiskfs_fill_super+0x3a56/0x43c0 [ldiskfs] [ 413.270115] ? ldiskfs_calculate_overhead+0x670/0x670 [ldiskfs] [ 413.280898] ? mount_bdev+0x226/0x270 [ 413.289685] mount_bdev+0x226/0x270 [ 413.295237] ldiskfs_mount+0x19/0x30 [ldiskfs] [ 413.297063] legacy_get_tree+0x35/0x90 [ 413.307517] vfs_get_tree+0x2a/0x140 [ 413.314045] fc_mount+0x16/0x60 [ 413.315757] vfs_kern_mount+0x91/0x100 [ 413.317322] osd_mount+0x5c4/0x1080 [osd_ldiskfs] [ 413.321844] osd_device_init0+0x2e1/0xc20 [osd_ldiskfs] [ 413.326257] osd_device_alloc+0x22a/0x290 [osd_ldiskfs] [ 413.336995] obd_setup+0x196/0x430 [obdclass] [ 413.346812] class_setup+0x6f5/0x9f0 [obdclass] [ 413.359994] class_process_config+0x1658/0x2b60 [obdclass] [ 413.374109] do_lcfg+0x376/0x740 [obdclass] [ 413.383599] lustre_start_simple+0x8f/0x220 [obdclass] [ 413.395638] osd_start+0x6aa/0xb60 [ptlrpc] [ 413.411075] ? server_name2index+0x79/0xe0 [obdclass] [ 413.413954] ? lsi_prepare+0x2e7/0x690 [ptlrpc] [ 413.434584] server_fill_super+0x99/0x1190 [ptlrpc] [ 413.437990] ? obd_zombie_barrier+0x63/0x120 [obdclass] [ 413.442575] ? debug_mutex_init+0x43/0x60 [ 413.448600] lustre_fill_super+0x4a6/0x5e0 [lustre] [ 413.455512] ? lustre_mount+0x30/0x30 [lustre] [ 413.460171] mount_nodev+0x56/0xf0 [ 413.462945] lustre_mount+0x1c/0x30 [lustre] [ 413.470809] legacy_get_tree+0x35/0x90 [ 413.481243] vfs_get_tree+0x2a/0x140 [ 413.486691] do_mount+0xd84/0x1190 [ 413.512371] ksys_mount+0x11d/0x150 [ 413.513809] __x64_sys_mount+0x29/0x40 [ 413.515240] do_syscall_64+0xc1/0x450 [ 413.517712] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 413.521849] RIP: 0033:0x7f31b1eefdbe [ 413.525396] Code: 48 8b 0d cd 60 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 9a 60 39 00 f7 d8 64 89 01 48 [ 413.542789] RSP: 002b:00007ffd6f8a5728 EFLAGS: 00000286 ORIG_RAX: 00000000000000a5 [ 413.550391] RAX: ffffffffffffffda RBX: 0000000000430cf6 RCX: 00007f31b1eefdbe [ 413.555882] RDX: 0000000000430cf6 RSI: 00007ffd6f8abdd0 RDI: 0000000000f1e940 [ 413.565835] RBP: 00007ffd6f8abdd0 R08: 0000000000f1e960 R09: 0000000000f1e010 [ 413.571988] R10: 0000000001000000 R11: 0000000000000286 R12: 0000000000000000 [ 413.581989] R13: 0000000000654920 R14: 00000000fffffff5 R15: 00000000fffffff5 [ 413.594858] ---[ end trace b1742f8e452b0fca ]--- [ 413.634362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.096430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 415.167801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 415.293973] Lustre: lustre-MDT0000: new disk, initializing [ 415.486945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.539875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 421.048909] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 442.019424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 442.159683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 442.403325] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 442.427078] Lustre: 7463:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 442.466087] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 442.468074] Lustre: Skipped 1 previous similar message [ 442.538889] Lustre: lustre-MDT0001: new disk, initializing [ 442.681797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 442.734638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 442.761216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 448.644032] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 456.061240] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 470.146063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 470.303287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 470.733190] Lustre: 8420:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 470.763307] Lustre: lustre-OST0000: new disk, initializing [ 470.783343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 470.890701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 479.548086] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 479.860282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 479.879055] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 480.144144] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 499.912383] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 500.024395] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 500.275261] Lustre: 9436:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 500.326113] Lustre: lustre-OST0001: new disk, initializing [ 500.344544] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 500.524867] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 508.543942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 508.554844] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 508.721586] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 509.337970] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 525.954798] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 535.138974] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 549.009750] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing check_logdir /tmp/testlogs/ [ 557.382875] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing yml_node [ 569.233618] Lustre: DEBUG MARKER: Client: 2.16.52.73 [ 574.191212] Lustre: DEBUG MARKER: MDS: 2.16.52.73 [ 578.447878] Lustre: DEBUG MARKER: OSS: 2.16.52.73 [ 581.917345] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Apr 1 04:47:26 EDT 2025 [ 612.232333] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 615.253195] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 636.901330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 636.903443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 636.910519] Lustre: Skipped 1 previous similar message [ 640.379623] Lustre: server umount lustre-MDT0000 complete [ 646.288120] LustreError: 6538:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743497314 with bad export cookie 9413308935058509515 [ 646.290617] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 646.304385] LustreError: 6538:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 647.136409] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.139777] LustreError: 9956:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 647.140044] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 647.140049] Lustre: Skipped 3 previous similar messages [ 647.149331] Lustre: Skipped 3 previous similar messages [ 647.201862] LustreError: 9956:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 6 previous similar messages [ 651.169136] LustreError: 9956:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 651.176391] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 651.189302] LustreError: 9956:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 651.204825] Lustre: Skipped 2 previous similar messages [ 652.854948] Lustre: server umount lustre-MDT0001 complete [ 658.903714] Lustre: server umount lustre-OST0000 complete [ 665.076932] Lustre: server umount lustre-OST0001 complete [ 675.562993] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 695.983207] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 713.749041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 725.901060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 735.214302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 745.736490] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 755.960822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 756.141827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 756.524637] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 756.614963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 756.767757] Lustre: lustre-MDT0000: new disk, initializing [ 756.919870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 756.964982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 762.560740] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 777.450896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 777.590666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 777.727128] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 777.758860] Lustre: 15694:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 777.802040] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 777.805377] Lustre: Skipped 1 previous similar message [ 777.871352] Lustre: lustre-MDT0001: new disk, initializing [ 778.069505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 778.188300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 778.203459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 783.697708] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 795.082994] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 801.466334] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 812.063943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 812.221512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 812.684102] Lustre: 16645:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 812.750693] Lustre: lustre-OST0000: new disk, initializing [ 812.755711] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 812.907401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 813.932989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 813.943357] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 814.216091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 821.009479] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 832.925626] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 838.694588] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 839.051496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 844.345341] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 844.851879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 849.892704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 849.894501] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 849.902651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 849.909652] Lustre: Skipped 1 previous similar message [ 853.543424] Lustre: server umount lustre-OST0000 complete [ 860.130635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 860.133974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 860.151303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 860.154109] Lustre: Skipped 2 previous similar messages [ 865.256831] LustreError: 16561:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 865.358341] Lustre: server umount lustre-MDT0000 complete [ 869.345976] LustreError: 14943:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 869.364974] LustreError: 14943:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 870.416775] LustreError: 14925:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743497538 with bad export cookie 9413308935058511048 [ 870.418772] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 870.425646] LustreError: 14925:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 870.912294] Lustre: server umount lustre-MDT0001 complete [ 877.138616] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 04:52:21 (1743497541) [ 881.853880] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 885.678827] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 04:52:29 (1743497549) [ 895.032949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 895.739382] LustreError: 18918:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 895.870389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 902.011795] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 905.199067] LustreError: 18918:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 905.206352] LustreError: 18918:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 913.508856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 913.662380] Lustre: 19618:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 919.011376] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 927.496690] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 934.124318] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 943.084565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 943.290136] Lustre: 20500:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 950.635562] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 958.679439] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 981.478622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 981.484059] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 981.499648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 981.501876] Lustre: Skipped 2 previous similar messages [ 986.232727] Lustre: server umount lustre-OST0000 complete [ 991.713358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 991.720118] Lustre: Skipped 1 previous similar message [ 998.113790] Lustre: server umount lustre-MDT0000 complete [ 1001.954928] LustreError: 18918:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1001.991562] LustreError: 18918:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 1003.372732] LustreError: 18899:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743497671 with bad export cookie 9413308935058511930 [ 1003.380334] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1003.387507] LustreError: 18899:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1003.916389] Lustre: server umount lustre-MDT0001 complete [ 1026.329434] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 1031.241364] Key type lgssc unregistered [ 1031.828845] LNet: 22125:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1032.873356] LNet: Removed LNI 192.168.206.117@tcp [ 1034.593859] Key type .llcrypt unregistered [ 1034.595488] Key type ._llcrypt unregistered [ 1073.298302] Key type ._llcrypt registered [ 1073.299890] Key type .llcrypt registered [ 1073.489337] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 1075.448336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1075.618161] alg: No test for adler32 (adler32-zlib) [ 1076.926353] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 1077.271434] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 1077.275763] LNet: Accept secure, port 988 [ 1078.999215] Key type lgssc registered [ 1080.268525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1091.168341] loop: module loaded [ 1091.266201] loop0: detected capacity change from 0 to 204800000 [ 1091.557307] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1091.568553] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1091.784512] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1091.793060] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1091.857742] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1091.913439] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1091.949973] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1092.016215] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1099.767407] loop0: detected capacity change from 0 to 204800000 [ 1100.092169] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1100.104586] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1100.124665] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1100.307479] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1107.016807] loop0: detected capacity change from 0 to 204800000 [ 1114.126384] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1114.192761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1114.212089] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1115.623879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 1115.712425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1115.785793] Lustre: lustre-MDT0000: new disk, initializing [ 1115.909142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1115.941910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1120.694779] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1131.618440] loop1: detected capacity change from 0 to 204800000 [ 1138.581543] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1138.674045] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1138.959066] Lustre: 25245:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1138.996862] Lustre: lustre-OST07c6: new disk, initializing [ 1139.008236] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 1139.017952] Lustre: Skipped 1 previous similar message [ 1139.149739] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 1145.982591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 1145.990764] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 1146.027606] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 1146.905423] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1248.225479] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1248.233631] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1248.257537] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 1251.790173] Lustre: server umount lustre-OST07c6 complete [ 1264.365845] Lustre: server umount lustre-MDT0000 complete [ 1294.064783] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 1299.354506] Key type lgssc unregistered [ 1299.936125] LNet: 27753:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1300.965961] LNet: Removed LNI 192.168.206.117@tcp [ 1302.571212] Key type .llcrypt unregistered [ 1302.572937] Key type ._llcrypt unregistered [ 1332.985536] Key type ._llcrypt registered [ 1332.987423] Key type .llcrypt registered [ 1333.188873] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 1335.352500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1335.393412] alg: No test for adler32 (adler32-zlib) [ 1336.641023] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 1337.041700] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 1337.045405] LNet: Accept secure, port 988 [ 1338.767693] Key type lgssc registered [ 1340.243458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1350.245874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1350.286859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1351.831896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1356.184483] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1361.377657] LustreError: 28995:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1361.397958] LustreError: 28995:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 1366.500856] LustreError: 28995:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1367.352834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1367.664184] Lustre: 29692:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1368.061138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1372.973864] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1380.673576] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1386.850709] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1395.452932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1395.655690] Lustre: 30589:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1395.844955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1396.867231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1402.524989] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1409.741529] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1418.531253] LustreError: 31187:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 1433.057958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1433.062334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1433.064559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1433.067958] Lustre: Skipped 1 previous similar message [ 1438.190128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1438.195308] Lustre: Skipped 2 previous similar messages [ 1443.300909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1445.343282] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1445.647327] Lustre: server umount lustre-OST0000 complete [ 1453.536828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1453.552822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1453.556210] Lustre: Skipped 1 previous similar message [ 1456.475726] Lustre: server umount lustre-MDT0000 complete [ 1458.658648] LustreError: 28995:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1458.670793] LustreError: 28995:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 1460.920303] LustreError: 28977:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743498128 with bad export cookie 17610378407887366112 [ 1460.940115] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1461.458872] Lustre: server umount lustre-MDT0001 complete [ 1483.071843] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 1488.136195] Key type lgssc unregistered [ 1488.656391] LNet: 32397:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1489.701572] LNet: Removed LNI 192.168.206.117@tcp [ 1491.279771] Key type .llcrypt unregistered [ 1491.281580] Key type ._llcrypt unregistered [ 1520.010604] Key type ._llcrypt registered [ 1520.012136] Key type .llcrypt registered [ 1520.160895] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 1522.717038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1522.759113] alg: No test for adler32 (adler32-zlib) [ 1523.978153] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 1524.388382] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 1524.391826] LNet: Accept secure, port 988 [ 1526.159430] Key type lgssc registered [ 1527.544781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1537.205150] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1537.247501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1538.842910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1543.163297] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1548.258238] LustreError: 33640:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1548.274820] LustreError: 33640:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 1553.386521] LustreError: 33640:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1553.503474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1553.623772] Lustre: 34337:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1553.901533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1558.583484] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1565.892373] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1571.305667] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1579.408823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1579.675113] Lustre: 35220:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1579.863554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1584.003990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1586.767664] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1594.314519] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1609.700200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1609.708380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1609.717258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1611.131267] Lustre: server umount lustre-OST0000 complete [ 1618.914226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1618.918382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1618.934861] Lustre: Skipped 1 previous similar message [ 1618.945233] Lustre: Skipped 1 previous similar message [ 1621.793061] Lustre: server umount lustre-MDT0000 complete [ 1624.043386] LustreError: 34598:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1626.775486] LustreError: 33620:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743498294 with bad export cookie 17852484793223900519 [ 1626.781557] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1626.793847] LustreError: 33620:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1636.554929] Lustre: server umount lustre-MDT0001 complete [ 1656.252354] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 1661.096780] Key type lgssc unregistered [ 1661.595026] LNet: 36840:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1662.639193] LNet: Removed LNI 192.168.206.117@tcp [ 1663.908508] Key type .llcrypt unregistered [ 1663.910918] Key type ._llcrypt unregistered [ 1702.041622] Key type ._llcrypt registered [ 1702.043391] Key type .llcrypt registered [ 1702.182616] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 1704.584815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1704.670923] alg: No test for adler32 (adler32-zlib) [ 1706.104566] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 1706.379184] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 1706.382021] LNet: Accept secure, port 988 [ 1708.143219] Key type lgssc registered [ 1709.643674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1719.338139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1719.370432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1720.983838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1725.378280] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1730.531029] LustreError: 38231:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1730.550156] LustreError: 38231:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 1735.581421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1735.650163] LustreError: 38231:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1735.740514] Lustre: 38928:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1736.028792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1740.680962] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1747.495649] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1753.080383] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1761.192101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1761.391465] Lustre: 39810:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1761.566561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1765.698857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1767.772957] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1775.098622] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1786.341791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.354813] Lustre: Skipped 2 previous similar messages [ 1786.354929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1786.360987] Lustre: Skipped 1 previous similar message [ 1787.573892] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.17@tcp (stopping) [ 1787.580984] Lustre: Skipped 1 previous similar message [ 1790.899482] Lustre: server umount lustre-MDT0000 complete [ 1791.459677] LustreError: 38231:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1791.481431] LustreError: 38231:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 1795.142511] LustreError: 38213:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743498462 with bad export cookie 4440811934967718178 [ 1795.158708] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1796.576474] LustreError: 40372:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1796.577662] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1796.598723] LustreError: 40372:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 1796.619722] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1800.609813] LustreError: 38228:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1800.611239] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1800.631898] LustreError: 38228:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 1800.640047] Lustre: Skipped 1 previous similar message [ 1801.531732] Lustre: server umount lustre-MDT0001 complete [ 1817.181753] Lustre: server umount lustre-OST0000 complete [ 1843.905427] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 1848.484324] Key type lgssc unregistered [ 1848.898737] LNet: 41739:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1849.961769] LNet: Removed LNI 192.168.206.117@tcp [ 1851.247898] Key type .llcrypt unregistered [ 1851.249645] Key type ._llcrypt unregistered [ 1888.630950] Key type ._llcrypt registered [ 1888.632781] Key type .llcrypt registered [ 1888.776943] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 1891.350258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1891.432362] alg: No test for adler32 (adler32-zlib) [ 1892.753566] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 1893.029089] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 1893.032683] LNet: Accept secure, port 988 [ 1894.735214] Key type lgssc registered [ 1896.451789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1905.976793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1906.006502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1907.578115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1912.375818] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1917.922502] LustreError: 43126:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1917.936164] LustreError: 43126:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 1922.899151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1923.014192] Lustre: 43823:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1923.053205] LustreError: 43125:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1923.316653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1928.202909] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1936.562902] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1942.837948] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1950.672137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1950.870431] Lustre: 44706:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 1951.067444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1953.140317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1957.277533] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1964.567354] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1977.010266] Lustre: lustre-OST0000: Not available for connect from 192.168.206.17@tcp (stopping) [ 1978.853306] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1978.857241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1978.860817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1978.865657] Lustre: Skipped 1 previous similar message [ 1978.872232] Lustre: Skipped 1 previous similar message [ 1980.498357] Lustre: server umount lustre-OST0000 complete [ 2000.352672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2000.356055] LustreError: Skipped 1 previous similar message [ 2000.358100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2000.385622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2003.425464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2003.427589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.913231] Lustre: server umount lustre-MDT0000 complete [ 2008.549806] LustreError: 43122:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2008.559157] LustreError: 43122:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 2010.434738] LustreError: 43105:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743498678 with bad export cookie 3111386206414213353 [ 2010.441529] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2016.893379] Lustre: server umount lustre-MDT0001 complete [ 2037.682994] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 2042.381843] Key type lgssc unregistered [ 2042.881554] LNet: 46481:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2043.943099] LNet: Removed LNI 192.168.206.117@tcp [ 2045.358755] Key type .llcrypt unregistered [ 2045.361618] Key type ._llcrypt unregistered [ 2079.627571] Key type ._llcrypt registered [ 2079.630322] Key type .llcrypt registered [ 2079.747245] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 2082.142484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2082.177509] alg: No test for adler32 (adler32-zlib) [ 2083.502705] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 2083.907239] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 2083.911097] LNet: Accept secure, port 988 [ 2085.631223] Key type lgssc registered [ 2086.970193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2095.303722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2095.325559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2096.828385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2101.086850] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2106.339029] LustreError: 47870:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2106.357694] LustreError: 47870:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 2110.708124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2110.838262] Lustre: 48567:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2111.108979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2115.053631] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2121.835368] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2126.635273] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2134.627553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2134.804513] Lustre: 49448:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2134.946088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2140.355808] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2142.134430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2147.341109] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2154.495905] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2159.180422] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 2164.725217] LNetError: Refusing connection from 192.168.206.117 for 127.0.0.2@tcp: No matching NI [ 2164.739548] LNetError: 47173:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 2164.743250] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 2172.239423] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-04-01 5h13m56s [ 2174.364677] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 2177.509027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2177.516243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.528386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2177.540438] LNetError: Refusing connection from 192.168.206.117 for 127.0.0.2@tcp: No matching NI [ 2177.547899] LNetError: 47176:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 2177.552169] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 2178.599243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2178.601968] Lustre: Skipped 2 previous similar messages [ 2180.254983] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.17@tcp (stopping) [ 2180.258278] Lustre: Skipped 2 previous similar messages [ 2182.821768] Lustre: server umount lustre-MDT0000 complete [ 2183.650941] LNetError: Refusing connection from 192.168.206.117 for 127.0.0.2@tcp: No matching NI [ 2183.659253] LNetError: 47174:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 2183.667401] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 2184.678610] LustreError: 47870:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2186.978567] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 2187.409665] LustreError: 49532:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2187.426295] LustreError: 49532:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 2189.799617] LNetError: Refusing connection from 192.168.206.117 for 127.0.0.2@tcp: No matching NI [ 2189.806893] LNetError: 47175:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 2189.813692] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 2190.823804] LustreError: 47870:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2190.838971] LustreError: 47870:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 2194.434387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2194.595941] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2194.837051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2194.904792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2195.883451] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.117@tcp (at 0@lo) [ 2198.813558] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2202.707907] Lustre: DEBUG MARKER: Wait for df (34741) ... [ 2205.351049] Lustre: DEBUG MARKER: done [ 2210.275884] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2210.284099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2210.291675] Lustre: Skipped 2 previous similar messages [ 2210.298298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2210.300957] Lustre: Skipped 1 previous similar message [ 2211.302203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2216.421091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2216.425466] Lustre: Skipped 3 previous similar messages [ 2224.095140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2224.319158] Lustre: server umount lustre-OST0000 complete [ 2231.777154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2231.780679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2231.780886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2231.789945] Lustre: Skipped 1 previous similar message [ 2231.792255] Lustre: Skipped 2 previous similar messages [ 2231.795691] LNetError: Refusing connection from 192.168.206.117 for 127.0.0.2@tcp: No matching NI [ 2231.817518] LNetError: 47173:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 2231.828509] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 2234.691983] Lustre: server umount lustre-MDT0000 complete [ 2238.458298] LustreError: 47850:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743498906 with bad export cookie 8776568340356720304 [ 2238.463160] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2238.465167] LustreError: 47850:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2238.751726] Lustre: server umount lustre-MDT0001 complete [ 2255.665129] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 2259.698925] Key type lgssc unregistered [ 2260.173777] LNet: 52417:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2261.222157] LNet: Removed LNI 192.168.206.117@tcp [ 2262.359500] Key type .llcrypt unregistered [ 2262.361569] Key type ._llcrypt unregistered [ 2296.896685] Key type ._llcrypt registered [ 2296.899804] Key type .llcrypt registered [ 2297.017676] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 2318.083297] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 2320.487104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2320.510940] alg: No test for adler32 (adler32-zlib) [ 2321.743396] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 2322.044294] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 2322.047023] LNet: Accept secure, port 988 [ 2323.767319] Key type lgssc registered [ 2325.082031] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2335.972096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2344.027559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2351.210669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2359.914042] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2367.682257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2367.774270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2367.793346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2369.104172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2369.151116] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2369.211628] Lustre: lustre-MDT0000: new disk, initializing [ 2369.329705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2369.365705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2373.533143] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2384.258506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2384.339284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2384.479806] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2384.504687] Lustre: 57116:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2384.548642] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2384.552138] Lustre: Skipped 1 previous similar message [ 2384.606116] Lustre: lustre-MDT0001: new disk, initializing [ 2384.703622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2384.749630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2384.754570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2388.796757] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2398.130052] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2403.399670] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2411.375211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2411.486122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2411.745590] Lustre: 58066:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2411.778568] Lustre: lustre-OST0000: new disk, initializing [ 2411.789509] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2411.913419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2418.259135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2418.268813] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2418.360917] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2418.562647] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2427.577566] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2432.249250] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2432.544988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2436.814790] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2437.134268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2444.264436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2444.268804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2444.271570] LustreError: Skipped 1 previous similar message [ 2444.272589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2444.292547] Lustre: Skipped 1 previous similar message [ 2445.628773] Lustre: server umount lustre-OST0000 complete [ 2451.425462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2451.429397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2451.438145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2451.441592] Lustre: Skipped 1 previous similar message [ 2454.496941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2454.501489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2455.857273] Lustre: server umount lustre-MDT0000 complete [ 2459.820063] LustreError: 56347:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743499127 with bad export cookie 783385043630737419 [ 2459.822673] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2459.834983] LustreError: 56347:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2460.271031] Lustre: server umount lustre-MDT0001 complete [ 2473.880113] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 05:18:58 (1743499138) [ 2482.105226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2482.581118] LustreError: 60342:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.602875] LustreError: 60342:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 2482.703594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2486.735872] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2487.786563] LustreError: 60342:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.908365] LustreError: 60342:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2495.394732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2495.501748] Lustre: 61037:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2495.752517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2499.715242] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2506.598049] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2511.675507] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2518.511466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2518.681421] Lustre: 61921:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2518.818567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.277032] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2530.045081] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2537.194913] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-04-01 5h20m02s [ 2539.547393] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2546.667794] LNetError: Refusing connection from 192.168.206.117 for 127.0.0.2@tcp: No matching NI [ 2546.672604] LNetError: 54556:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 2546.676369] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 2547.689790] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2550.810788] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2551.122053] LustreError: 61851:0:(mdt_handler.c:2326:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2571.430396] Lustre: lustre-MDT0000: Client 174b6625-b71e-4d71-8808-e88e954964e8 (at 192.168.206.17@tcp) reconnecting [ 2591.135810] LustreError: 61851:0:(mdt_handler.c:2326:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2593.696805] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2596.515799] Lustre: DEBUG MARKER: done [ 2602.978314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2602.982789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2602.995406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2602.998699] Lustre: Skipped 1 previous similar message [ 2604.454680] Lustre: server umount lustre-OST0000 complete [ 2608.612112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2608.616049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2608.624476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2608.629405] LNetError: Refusing connection from 192.168.206.117 for 127.0.0.2@tcp: No matching NI [ 2608.635756] LNetError: 54559:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 2608.639514] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 2610.721304] LNetError: Refusing connection from 192.168.206.117 for 127.0.0.2@tcp: No matching NI [ 2610.740440] LNetError: 54557:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 2610.750200] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 2616.805195] LNetError: Refusing connection from 192.168.206.117 for 127.0.0.2@tcp: No matching NI [ 2616.809058] LNetError: 54558:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 2616.815265] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 2617.827090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2617.830191] Lustre: Skipped 6 previous similar messages [ 2617.913854] Lustre: server umount lustre-MDT0000 complete [ 2620.901486] LustreError: 60337:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2620.911763] LustreError: 60337:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 2621.322726] LustreError: 60321:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743499289 with bad export cookie 783385043630738301 [ 2621.323854] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2621.329564] LustreError: 60321:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2621.583149] Lustre: server umount lustre-MDT0001 complete [ 2638.084531] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 2641.866678] Key type lgssc unregistered [ 2642.256258] LNet: 64278:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2643.302292] LNet: Removed LNI 192.168.206.117@tcp [ 2644.343911] Key type .llcrypt unregistered [ 2644.345699] Key type ._llcrypt unregistered [ 2677.436298] Key type ._llcrypt registered [ 2677.439564] Key type .llcrypt registered [ 2677.556943] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 2697.010892] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 2698.915631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2698.950351] alg: No test for adler32 (adler32-zlib) [ 2700.237892] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 2700.513367] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 2700.516325] LNet: Accept secure, port 988 [ 2702.191206] Key type lgssc registered [ 2703.227202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2711.995034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2719.125359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2725.368611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2732.133640] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2738.114502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2738.161743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2738.176491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2739.423068] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 2739.462430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2739.511736] Lustre: lustre-MDT0000: new disk, initializing [ 2739.603897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2739.623633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2742.702873] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2752.054201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2752.125043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2752.230889] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 2752.253863] Lustre: 68975:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2752.285206] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2752.287950] Lustre: Skipped 1 previous similar message [ 2752.334882] Lustre: lustre-MDT0001: new disk, initializing [ 2752.424282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2752.457262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2752.462188] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2755.572397] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2763.002089] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2767.397473] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2774.303461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2774.394573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2774.611468] Lustre: 69927:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2774.645611] Lustre: lustre-OST0000: new disk, initializing [ 2774.654823] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2774.735255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2779.777521] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2782.780694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2782.786784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2782.951711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2787.531995] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2791.296088] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2791.555776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2794.974878] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2795.312243] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2798.563873] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2798.567381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2798.574376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2803.684765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2803.687158] Lustre: Skipped 2 previous similar messages [ 2808.805675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2808.816836] Lustre: Skipped 1 previous similar message [ 2811.359595] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2811.506914] Lustre: server umount lustre-OST0000 complete [ 2819.042411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2819.042461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2819.044155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2819.048135] Lustre: Skipped 2 previous similar messages [ 2819.052762] LustreError: Skipped 1 previous similar message [ 2824.161554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2824.168395] Lustre: Skipped 2 previous similar messages [ 2829.279124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2829.427555] Lustre: server umount lustre-MDT0000 complete [ 2832.730961] LustreError: 68205:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743499500 with bad export cookie 439745984810933493 [ 2832.734949] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2832.737592] LustreError: 68205:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2832.986130] Lustre: server umount lustre-MDT0001 complete [ 2843.192535] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 05:25:08 (1743499508) [ 2845.493504] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 2847.741790] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 05:25:13 (1743499513) [ 2862.770556] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 2868.604948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2869.006459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2869.876216] Lustre: Failing over lustre-MDT0000 [ 2869.885531] LustreError: 72613:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2869.890412] LustreError: 72613:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2869.895171] LustreError: 72613:0:(lod_dev.c:506:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 2870.157753] Lustre: server umount lustre-MDT0000 complete [ 2878.779187] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 05:25:44 (1743499544) [ 2885.019393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2885.371358] LustreError: 73231:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2885.381464] LustreError: 73231:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 2885.429171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2888.707379] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2890.722371] LustreError: 73231:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2895.840904] LustreError: 73231:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2895.940625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2896.019910] Lustre: 73926:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2896.216222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2899.712956] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2905.392736] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2909.649199] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2915.565597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2915.710485] Lustre: 74812:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2915.856270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2921.020947] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2926.602403] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2937.826325] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2943.945445] Lustre: server umount lustre-MDT0000 complete [ 2945.504767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2945.509976] LustreError: 75444:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2945.515056] Lustre: Skipped 2 previous similar messages [ 2945.529233] LustreError: 75444:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 2947.285668] LustreError: 73211:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743499615 with bad export cookie 439745984810934648 [ 2947.292091] LustreError: 73211:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2947.293160] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2947.540162] Lustre: server umount lustre-MDT0001 complete [ 2951.193230] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2955.606796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2964.346307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2967.007227] Lustre: 66432:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743499618/real 1743499618] req@0000000037c3aa82 x1828191593373312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1743499634 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 2967.020927] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2969.055146] Lustre: 66429:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743499620/real 1743499620] req@000000004edd530d x1828191593373440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1743499636 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 2971.807722] Lustre: 66430:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743499623/real 1743499623] req@0000000013c92d14 x1828191593373568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1743499639 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 2973.154980] LustreError: 66428:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cd230e4e x1828191593374464/t0(0) o250->MGC192.168.206.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 2973.460671] LustreError: 74828:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2973.539616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2973.668244] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2977.333748] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2978.592276] Lustre: 66432:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743499630/real 1743499630] req@00000000182028fe x1828191593373952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1743499646 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 2978.608685] Lustre: 66432:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2981.869433] LustreError: 74827:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2981.875356] LustreError: 74827:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 2985.601782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2985.724791] Lustre: 77340:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 2987.057538] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.206.117@tcp (at 0@lo) [ 2989.908542] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2995.584354] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3000.256206] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3006.932027] Lustre: server umount lustre-MDT0000 complete [ 3007.462368] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.472867] LustreError: 76632:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3007.482252] LustreError: 76632:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 3010.316333] LustreError: 73210:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743499678 with bad export cookie 439745984810936552 [ 3010.321122] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3010.325453] LustreError: 73210:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3012.605391] Lustre: server umount lustre-MDT0001 complete [ 3018.273110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3026.659725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3028.895318] Lustre: 66432:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743499680/real 1743499680] req@000000001cb7340f x1828191593399424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1743499696 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 3028.906865] Lustre: 66432:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3028.910185] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3028.916394] Lustre: Skipped 2 previous similar messages [ 3036.127831] LustreError: 66428:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e7499261 x1828191593400576/t0(0) o250->MGC192.168.206.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 3036.495508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 3037.152136] Lustre: 66430:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743499688/real 1743499688] req@00000000c9f9081f x1828191593399936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1743499704 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 3037.168060] Lustre: 66430:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 3039.730776] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3043.809785] LustreError: 79080:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3043.816617] LustreError: 79080:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 4 previous similar messages [ 3047.441152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3047.533062] Lustre: 79783:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3047.740796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3047.743916] Lustre: Skipped 2 previous similar messages [ 3048.743106] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.206.117@tcp (at 0@lo) [ 3048.747386] Lustre: Skipped 1 previous similar message [ 3051.237175] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3056.533929] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3060.751519] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3068.387312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3068.395473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3068.404213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3068.408248] Lustre: Skipped 2 previous similar messages [ 3069.048390] Lustre: server umount lustre-MDT0000 complete [ 3072.722955] LustreError: 73942:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743499740 with bad export cookie 439745984810937924 [ 3072.730145] LustreError: 73942:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3072.733709] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3072.988796] Lustre: server umount lustre-MDT0001 complete [ 3078.818208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3082.767549] Lustre: DEBUG MARKER: files compared the same [ 3095.057358] Lustre: server umount lustre-OST0000 complete [ 3116.701527] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 3120.054400] Key type lgssc unregistered [ 3120.436676] LNet: 82115:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3121.510122] LNet: Removed LNI 192.168.206.117@tcp [ 3122.459784] Key type .llcrypt unregistered [ 3122.461686] Key type ._llcrypt unregistered [ 3152.001889] Key type ._llcrypt registered [ 3152.003508] Key type .llcrypt registered [ 3152.149050] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 3153.966421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3154.001380] alg: No test for adler32 (adler32-zlib) [ 3155.156875] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 3155.452179] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 3155.457211] LNet: Accept secure, port 988 [ 3157.143195] Key type lgssc registered [ 3158.212466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3165.905486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3165.929225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3167.381461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3171.081737] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug +malloc all [ 3177.956474] LustreError: 83503:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3177.964195] LustreError: 83503:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 3178.931809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3179.034575] Lustre: 84200:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3179.257924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3182.912420] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug +malloc all [ 3188.875186] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3193.343566] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3199.896361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3200.108752] Lustre: 85082:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3200.261055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3205.155983] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug +malloc all [ 3210.740402] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3212.585179] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 3219.937285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3219.941224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3219.950686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3223.012126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3223.016529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3223.028455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3223.045846] Lustre: Skipped 1 previous similar message [ 3228.128605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3228.131597] Lustre: Skipped 1 previous similar message [ 3233.247143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3233.252148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3233.261894] Lustre: Skipped 1 previous similar message [ 3233.378093] Lustre: server umount lustre-OST0000 complete [ 3238.371317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3238.375471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3238.384569] Lustre: Skipped 1 previous similar message [ 3243.455073] Lustre: server umount lustre-MDT0000 complete [ 3243.491053] LustreError: 83503:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3243.499930] LustreError: 83503:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 3246.901572] LustreError: 83485:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743499914 with bad export cookie 16952608312484640946 [ 3246.907875] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3247.194211] Lustre: server umount lustre-MDT0001 complete [ 3263.822683] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 3267.844605] Key type lgssc unregistered [ 3268.327509] LNet: 86680:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3269.349608] LNet: Removed LNI 192.168.206.117@tcp [ 3270.534486] Key type .llcrypt unregistered [ 3270.535878] Key type ._llcrypt unregistered [ 3302.700918] Key type ._llcrypt registered [ 3302.702448] Key type .llcrypt registered [ 3302.841461] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 3304.672595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3304.693881] alg: No test for adler32 (adler32-zlib) [ 3305.878027] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 3306.170347] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 3306.173272] LNet: Accept secure, port 988 [ 3307.855231] Key type lgssc registered [ 3308.881580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3315.447538] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3315.464103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3333.024613] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3339.167338] LustreError: 88042:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.117@tcp: failed processing log, type 4: rc = -110 [ 3369.951296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3375.161329] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3380.721074] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3387.761161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3387.871701] Lustre: *** cfs_fail_loc=706, val=0*** [ 3388.180746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3388.247567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 3391.296896] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3393.507565] LustreError: 88067:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3393.515131] LustreError: 88067:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 3398.628167] LustreError: 88067:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3399.013573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3399.140731] Lustre: 89711:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3399.394358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3402.637587] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3408.354555] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3413.174933] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3419.113848] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3419.116885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3419.132708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3419.139012] Lustre: Skipped 1 previous similar message [ 3421.779426] Lustre: server umount lustre-OST0000 complete [ 3429.345845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3429.350448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3429.360576] Lustre: Skipped 2 previous similar messages [ 3429.366766] Lustre: Skipped 1 previous similar message [ 3431.540137] Lustre: server umount lustre-MDT0000 complete [ 3434.465600] LustreError: 89011:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3434.474564] LustreError: 89011:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 3435.434869] LustreError: 89727:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743500103 with bad export cookie 4964999337187183135 [ 3435.438641] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3435.446492] LustreError: 89727:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3435.789301] Lustre: server umount lustre-MDT0001 complete [ 3451.515539] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 3455.058612] Key type lgssc unregistered [ 3455.421864] LNet: 91301:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3456.485496] LNet: Removed LNI 192.168.206.117@tcp [ 3457.424821] Key type .llcrypt unregistered [ 3457.426617] Key type ._llcrypt unregistered [ 3487.167352] Key type ._llcrypt registered [ 3487.169035] Key type .llcrypt registered [ 3487.283671] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 3488.844347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3489.012510] alg: No test for adler32 (adler32-zlib) [ 3490.252776] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 3490.566105] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 3490.569364] LNet: Accept secure, port 988 [ 3492.247189] Key type lgssc registered [ 3493.495629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3501.330924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3501.350572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3506.270578] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3513.211896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3513.407209] Lustre: 93343:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3513.607706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3516.787590] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3518.950184] LustreError: 93350:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3518.961989] LustreError: 93350:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 3524.067183] LustreError: 93354:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3524.834353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3525.001038] Lustre: 94050:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3525.118769] LustreError: 93349:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3525.126271] LustreError: 93349:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 3525.145719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3529.920390] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3530.210697] LustreError: 93355:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3530.223290] LustreError: 93355:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 3535.333682] LustreError: 93349:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3535.349539] LustreError: 93349:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 3537.873615] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3537.882428] Lustre: MGS: Client 833b07d3-0d41-4c95-bb2d-d86fec556867 (at 0@lo) reconnecting [ 3538.157075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3538.239149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 3541.821499] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3553.765830] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3553.774714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3553.784234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3554.792569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3554.794768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3554.800393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3559.877983] Lustre: server umount lustre-OST0000 complete [ 3565.026178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3565.028347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3565.032912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3565.044820] Lustre: Skipped 2 previous similar messages [ 3569.328276] Lustre: server umount lustre-MDT0000 complete [ 3570.146252] LustreError: 94851:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3570.160329] LustreError: 94851:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 3573.088109] Lustre: server umount lustre-MDT0001 complete [ 3576.721703] Lustre: server umount MGS complete [ 3589.932583] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 05:37:35 (1743500255) [ 3616.030725] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 3628.011659] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 3638.337398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3645.844266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3652.883563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3659.646949] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3666.336483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3666.433620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3669.868179] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3678.952292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3679.062522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3679.287647] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 3679.315095] Lustre: 100358:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3679.356169] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3679.409825] Lustre: lustre-MDT0001: new disk, initializing [ 3679.522815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3679.559756] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3679.569697] Lustre: 100396:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3679.576545] LustreError: 100396:0:(lod_dev.c:506:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3683.099420] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3692.146851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3692.233757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3692.483275] Lustre: 101117:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3692.519204] Lustre: lustre-OST0000: new disk, initializing [ 3692.523895] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3692.575995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3698.459535] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3699.297121] Lustre: 101452:0:(client.c:1515:after_reply()) @@@ resending request on EINPROGRESS req@00000000a1443955 x1828192421714176/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1743500383 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 [ 3706.108290] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3712.632852] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3712.642823] Lustre: MGS: Client 6e069d35-a490-4595-acd6-b3584127f5e5 (at 0@lo) reconnecting [ 3712.697820] Lustre: Modifying parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 3712.716777] LustreError: 99687:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 3712.751394] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3712.814681] Lustre: lustre-MDT0000: new disk, initializing [ 3712.981114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3713.011096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3713.166494] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3714.080137] Lustre: 102095:0:(client.c:1515:after_reply()) @@@ resending request on EINPROGRESS req@00000000d0221df7 x1828192421720576/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1743500397 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 [ 3716.690846] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3719.648534] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3723.180379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3723.188374] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3723.336297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3746.276585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3746.279576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3746.285154] Lustre: Skipped 2 previous similar messages [ 3751.395022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3751.398258] Lustre: Skipped 2 previous similar messages [ 3751.644962] Lustre: server umount lustre-OST0000 complete [ 3756.513151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3756.513565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3756.522111] Lustre: Skipped 1 previous similar message [ 3761.634328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3761.637389] Lustre: Skipped 3 previous similar messages [ 3769.311190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3769.419599] Lustre: server umount lustre-MDT0000 complete [ 3771.873196] LustreError: 100367:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3771.887382] LustreError: 100367:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 3778.809636] Lustre: server umount lustre-MDT0001 complete [ 3781.776559] Lustre: server umount MGS complete [ 3794.450873] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 05:40:59 (1743500459) [ 3816.434150] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 3819.721776] Key type lgssc unregistered [ 3820.074483] LNet: 104570:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3821.093966] LNet: Removed LNI 192.168.206.117@tcp [ 3822.087823] Key type .llcrypt unregistered [ 3822.091700] Key type ._llcrypt unregistered [ 3842.833755] Key type ._llcrypt registered [ 3842.835546] Key type .llcrypt registered [ 3842.951523] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 3844.815887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3844.842093] alg: No test for adler32 (adler32-zlib) [ 3846.059342] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 3846.373340] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 3846.378160] LNet: Accept secure, port 988 [ 3848.047144] Key type lgssc registered [ 3849.286454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3855.900323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3855.917265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3857.267427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3860.217056] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3866.828452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3866.914843] Lustre: 106458:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3867.107029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3869.862441] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3874.912463] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3879.027287] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3884.923293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3885.063830] Lustre: 107342:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3885.227001] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3889.607691] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3895.117890] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3902.636187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3912.162105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3912.165899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3912.173406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3913.192104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3913.192216] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3913.194918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3918.170501] Lustre: server umount lustre-OST0000 complete [ 3923.424789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3923.425710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3923.433299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3923.433493] Lustre: Skipped 1 previous similar message [ 3923.436122] Lustre: Skipped 1 previous similar message [ 3927.373366] Lustre: server umount lustre-MDT0000 complete [ 3928.555397] LustreError: 105760:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3928.563779] LustreError: 105760:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 3930.464073] LustreError: 105742:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743500598 with bad export cookie 6029972806667636376 [ 3930.468142] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3930.470733] LustreError: 105742:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3930.727770] Lustre: server umount lustre-MDT0001 complete [ 3947.061890] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 3950.678709] Key type lgssc unregistered [ 3951.093111] LNet: 108970:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3952.170778] LNet: Removed LNI 192.168.206.117@tcp [ 3953.222051] Key type .llcrypt unregistered [ 3953.223749] Key type ._llcrypt unregistered [ 3972.304807] Key type ._llcrypt registered [ 3972.306548] Key type .llcrypt registered [ 3972.420623] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 3974.079077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3974.098818] alg: No test for adler32 (adler32-zlib) [ 3975.295317] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 3975.599788] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 3975.607058] LNet: Accept secure, port 988 [ 3977.280510] Key type lgssc registered [ 3978.389942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3984.031223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3984.035135] LustreError: 110021:0:(tgt_mount.c:2222:server_fill_super()) cfs_race id 716 sleeping [ 3984.045735] LustreError: 110023:0:(tgt_mount.c:2222:server_fill_super()) cfs_fail_race id 716 waking [ 3984.048649] LustreError: 110021:0:(tgt_mount.c:2222:server_fill_super()) cfs_fail_race id 716 awake: rc=4990 [ 3984.052986] LustreError: 110021:0:(tgt_mount.c:2234:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3984.059465] LustreError: 110021:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3984.063045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3985.501281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3992.013812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3992.094112] Lustre: 110485:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 3992.274816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3995.083820] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3998.788876] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing lsmod [ 4001.406505] LustreError: 111183:0:(tgt_mount.c:2222:server_fill_super()) cfs_race id 716 sleeping [ 4001.414608] LustreError: 111187:0:(tgt_mount.c:2222:server_fill_super()) cfs_fail_race id 716 waking [ 4001.418542] LustreError: 111183:0:(tgt_mount.c:2222:server_fill_super()) cfs_fail_race id 716 awake: rc=4992 [ 4001.422185] LustreError: 111183:0:(tgt_mount.c:2234:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 4001.428274] LustreError: 111183:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 4001.439495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4001.650350] Lustre: 111187:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4001.799866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4007.914487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4007.915415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4007.919037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4007.921508] Lustre: Skipped 1 previous similar message [ 4010.445894] Lustre: server umount lustre-MDT0000 complete [ 4013.026539] LustreError: 110061:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4013.034276] LustreError: 110061:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 4 previous similar messages [ 4013.241769] LustreError: 110042:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743500681 with bad export cookie 18002626182472021990 [ 4013.244602] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4013.247025] LustreError: 110042:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4013.253958] Lustre: 110053:0:(mgc_request.c:1858:mgc_process_log()) MGC192.168.206.117@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4013.485813] Lustre: server umount lustre-MDT0001 complete [ 4026.496450] Lustre: server umount lustre-OST0000 complete [ 4033.143428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4033.528731] LustreError: 112252:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4033.603847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4036.651096] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4038.627188] LustreError: 112252:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4043.748042] LustreError: 112252:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4043.794599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4043.886221] Lustre: 112948:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4044.096067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4047.536863] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4053.292852] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4057.532134] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4063.176462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4063.316676] Lustre: 113830:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4063.422411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4067.861690] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4073.581512] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4078.778712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 4084.201063] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4084.201669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4084.205198] LustreError: Skipped 1 previous similar message [ 4084.211081] Lustre: Skipped 1 previous similar message [ 4084.213118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4084.218825] Lustre: Skipped 3 previous similar messages [ 4085.440585] Lustre: server umount lustre-OST0000 complete [ 4089.315027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4089.319335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4089.325842] Lustre: Skipped 2 previous similar messages [ 4089.330875] Lustre: Skipped 1 previous similar message [ 4094.434231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4094.437300] Lustre: Skipped 1 previous similar message [ 4094.541645] Lustre: server umount lustre-MDT0000 complete [ 4097.740568] LustreError: 113856:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743500765 with bad export cookie 18002626182472022669 [ 4097.741382] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4097.745375] LustreError: 113856:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4097.992874] Lustre: server umount lustre-MDT0001 complete [ 4113.574350] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 4116.728952] Key type lgssc unregistered [ 4117.100952] LNet: 115427:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4118.117585] LNet: Removed LNI 192.168.206.117@tcp [ 4119.090459] Key type .llcrypt unregistered [ 4119.091871] Key type ._llcrypt unregistered [ 4149.136785] Key type ._llcrypt registered [ 4149.138585] Key type .llcrypt registered [ 4149.237227] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 4151.139965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4151.183039] alg: No test for adler32 (adler32-zlib) [ 4152.433556] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 4152.702063] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 4152.704688] LNet: Accept secure, port 988 [ 4154.375181] Key type lgssc registered [ 4155.367492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4161.867134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4161.885785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4163.267710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4166.525547] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4173.795121] LustreError: 116812:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4173.801751] LustreError: 116812:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 4174.097707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4174.196617] Lustre: 117508:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4174.460028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4177.642440] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4183.257901] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4187.690509] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4194.126346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4194.309742] Lustre: 118389:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4194.461091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4199.311785] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4204.975127] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4209.845419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 4214.839663] Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client [ 4218.119524] Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client [ 4225.508043] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4225.508087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4225.510726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4225.511375] LustreError: Skipped 1 previous similar message [ 4225.521323] Lustre: Skipped 1 previous similar message [ 4227.744992] Lustre: server umount lustre-OST0000 complete [ 4235.749700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4235.750201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4235.752118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4235.759161] Lustre: Skipped 1 previous similar message [ 4235.761252] Lustre: Skipped 1 previous similar message [ 4237.275433] Lustre: server umount lustre-MDT0000 complete [ 4240.449977] LustreError: 118416:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743500908 with bad export cookie 5212119227574424950 [ 4240.455788] LustreError: 118416:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4240.457549] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4240.720125] Lustre: server umount lustre-MDT0001 complete [ 4257.641280] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 4261.042857] Key type lgssc unregistered [ 4261.417619] LNet: 120092:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4262.438283] LNet: Removed LNI 192.168.206.117@tcp [ 4263.486722] Key type .llcrypt unregistered [ 4263.488557] Key type ._llcrypt unregistered [ 4282.696773] Key type ._llcrypt registered [ 4282.698398] Key type .llcrypt registered [ 4282.790227] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 4284.523164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4284.574140] alg: No test for adler32 (adler32-zlib) [ 4285.733432] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 4285.942992] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 4285.945814] LNet: Accept secure, port 988 [ 4287.599197] Key type lgssc registered [ 4288.719772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4295.966238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4295.982918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4297.468659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4301.231132] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4307.938025] LustreError: 121284:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4307.947635] LustreError: 121284:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 4309.371788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4309.452170] Lustre: 121982:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4309.696804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4313.453734] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4319.195890] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4323.800842] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4330.497965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4330.685440] Lustre: 122864:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4330.849535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4335.963508] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4342.087825] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4343.157962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 4358.629029] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4358.630765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4358.634670] Lustre: Skipped 1 previous similar message [ 4358.635297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4358.643122] Lustre: Skipped 1 previous similar message [ 4361.869507] Lustre: server umount lustre-OST0000 complete [ 4366.305915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4366.310633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4366.318685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4368.865821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4368.867952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4371.417731] Lustre: server umount lustre-MDT0000 complete [ 4373.986375] LustreError: 121279:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4373.997928] LustreError: 121279:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 4374.802821] LustreError: 121266:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743501042 with bad export cookie 8345846622249935241 [ 4374.806753] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4374.811229] LustreError: 121266:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4375.080318] Lustre: server umount lustre-MDT0001 complete [ 4392.414566] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 4395.788418] Key type lgssc unregistered [ 4396.132287] LNet: 124482:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4397.158628] LNet: Removed LNI 192.168.206.117@tcp [ 4398.043975] Key type .llcrypt unregistered [ 4398.045621] Key type ._llcrypt unregistered [ 4427.372063] Key type ._llcrypt registered [ 4427.373387] Key type .llcrypt registered [ 4427.486256] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 4429.276566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4429.305283] alg: No test for adler32 (adler32-zlib) [ 4430.656217] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 4430.918444] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 4430.923425] LNet: Accept secure, port 988 [ 4432.639172] Key type lgssc registered [ 4433.685861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4440.861907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4440.888107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4442.339178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4445.930147] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4452.835131] LustreError: 125869:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4452.847843] LustreError: 125869:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 4454.156824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4454.242758] Lustre: 126567:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4454.540280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4457.783802] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4463.665774] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4467.884412] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4473.998794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4474.169581] Lustre: 127448:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4474.311565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4479.004868] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4484.513920] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4487.664766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4495.744577] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 4499.034176] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 4499.037678] Lustre: Skipped 2 previous similar messages [ 4502.081643] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4502.115491] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4502.300763] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash in log lustre-MDT0000 [ 4502.304012] Lustre: Skipped 2 previous similar messages [ 4508.694986] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4508.697502] Lustre: Skipped 1 previous similar message [ 4512.958351] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 4512.961558] Lustre: Skipped 2 previous similar messages [ 4516.128364] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.17@tcp 192.168.0.[2,10]@tcp [ 4523.160946] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids in log lustre-MDT0000 [ 4523.163960] Lustre: Skipped 2 previous similar messages [ 4525.740481] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.17/24@tcp [ 4525.742970] Lustre: Skipped 1 previous similar message [ 4533.734737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4533.739654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4533.749270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4533.752261] Lustre: Skipped 1 previous similar message [ 4538.343599] Lustre: server umount lustre-OST0000 complete [ 4541.408960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4541.412214] LustreError: Skipped 1 previous similar message [ 4541.414135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4541.419472] Lustre: Skipped 1 previous similar message [ 4541.422250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4543.969350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4543.970272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4547.638780] Lustre: server umount lustre-MDT0000 complete [ 4549.089493] LustreError: 125869:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4549.099361] LustreError: 125869:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 4551.065975] LustreError: 126582:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743501218 with bad export cookie 13641164944905731305 [ 4551.070449] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4551.071895] LustreError: 126582:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4551.353217] Lustre: server umount lustre-MDT0001 complete [ 4566.528660] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 4569.845924] Key type lgssc unregistered [ 4570.197829] LNet: 130460:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4571.237288] LNet: Removed LNI 192.168.206.117@tcp [ 4572.119434] Key type .llcrypt unregistered [ 4572.120964] Key type ._llcrypt unregistered [ 4603.909342] Key type ._llcrypt registered [ 4603.910897] Key type .llcrypt registered [ 4604.004293] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 4605.613182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4605.646031] alg: No test for adler32 (adler32-zlib) [ 4606.836095] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 4607.157438] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 4607.162788] LNet: Accept secure, port 988 [ 4608.847531] Key type lgssc registered [ 4609.857616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4616.320982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4616.342339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4617.772822] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4617.790500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4617.808123] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.17/24@tcp [ 4620.878836] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4627.937942] LustreError: 131991:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4627.944852] LustreError: 131991:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 4628.103989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4628.198243] Lustre: 132690:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4628.372591] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4628.403882] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.17/24@tcp [ 4628.443856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4631.778935] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4637.237969] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4641.414632] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4647.528780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4647.710128] Lustre: 133575:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4647.864398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4652.381539] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4657.660113] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4662.196756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4672.483819] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4672.486878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4672.489291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4672.493840] Lustre: Skipped 1 previous similar message [ 4672.499405] Lustre: Skipped 1 previous similar message [ 4677.146284] Lustre: server umount lustre-OST0000 complete [ 4682.724079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4682.726465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4682.729869] Lustre: Skipped 1 previous similar message [ 4682.734404] Lustre: Skipped 1 previous similar message [ 4686.027986] Lustre: server umount lustre-MDT0000 complete [ 4687.843226] LustreError: 131986:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4687.849844] LustreError: 131986:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 4688.957337] LustreError: 131969:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743501356 with bad export cookie 13985225519225022975 [ 4688.958190] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4688.964039] LustreError: 131969:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4689.204596] Lustre: server umount lustre-MDT0001 complete [ 4704.093969] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 4707.177499] Key type lgssc unregistered [ 4707.559732] LNet: 135220:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4708.581521] LNet: Removed LNI 192.168.206.117@tcp [ 4709.442524] Key type .llcrypt unregistered [ 4709.444060] Key type ._llcrypt unregistered [ 4735.678067] Key type ._llcrypt registered [ 4735.680182] Key type .llcrypt registered [ 4735.778556] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 4754.198635] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 4755.954822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4755.986251] alg: No test for adler32 (adler32-zlib) [ 4757.268985] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 4757.506718] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 4757.510050] LNet: Accept secure, port 988 [ 4759.176480] Key type lgssc registered [ 4760.123541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4769.080397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4775.882074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4782.118082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4788.968418] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4794.310476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4794.376148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4794.389929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4795.594059] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 4795.617945] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4795.661398] Lustre: lustre-MDT0000: new disk, initializing [ 4795.728302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4795.745256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4798.410840] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4806.465545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4806.532063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4806.642678] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 4806.659421] Lustre: 139342:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4806.688682] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4806.691527] Lustre: Skipped 1 previous similar message [ 4806.735786] Lustre: lustre-MDT0001: new disk, initializing [ 4806.819690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4806.852056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4806.860347] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4809.835635] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4816.570570] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4820.864122] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4826.476288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4826.546384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4826.729251] Lustre: 140294:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4826.758057] Lustre: lustre-OST0000: new disk, initializing [ 4826.763462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4826.831167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4830.877559] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4835.875255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4835.880468] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4835.977852] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4836.944820] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4839.857712] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4840.039766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4842.474913] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4842.674996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4846.051153] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4846.051838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4846.055427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4846.058498] Lustre: Skipped 1 previous similar message [ 4846.060962] Lustre: Skipped 1 previous similar message [ 4851.174772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4851.177776] Lustre: Skipped 1 previous similar message [ 4856.296818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4856.299409] Lustre: Skipped 1 previous similar message [ 4858.335106] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4858.438028] Lustre: server umount lustre-OST0000 complete [ 4861.408966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4861.409790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4861.414047] Lustre: Skipped 1 previous similar message [ 4861.418520] Lustre: Skipped 1 previous similar message [ 4866.529277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4866.532499] Lustre: Skipped 1 previous similar message [ 4875.231470] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4875.351619] Lustre: server umount lustre-MDT0000 complete [ 4877.833465] LustreError: 138573:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743501545 with bad export cookie 1360949481064045738 [ 4877.837991] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4877.838478] LustreError: 138573:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4878.030709] Lustre: server umount lustre-MDT0001 complete [ 4883.445635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4883.712658] LustreError: 142375:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4883.722219] LustreError: 142375:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 4883.757544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4886.287338] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4889.059804] LustreError: 142375:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4892.160672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4892.238322] Lustre: 143072:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4892.403113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4894.938964] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4899.360779] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4903.261591] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4907.775065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4907.895983] Lustre: 143953:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4907.983618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4911.398565] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4915.601657] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4918.694851] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4918.863481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4921.374863] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4921.550607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4927.422383] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4927.495231] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4927.615366] Lustre: 145334:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 4927.638462] Lustre: lustre-OST0001: Not available for connect from 192.168.206.17@tcp (not set up) [ 4927.642652] Lustre: Skipped 2 previous similar messages [ 4927.644171] Lustre: lustre-OST0001: new disk, initializing [ 4927.649499] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4927.713840] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4931.281660] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4933.649986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4933.653701] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4933.732649] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4936.499211] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4936.668348] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4939.100580] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4939.283185] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4942.460418] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff966a00729000.ost_server_uuid 50 [ 4944.162551] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff966a00729000.ost_server_uuid in FULL state after 0 sec [ 4954.088410] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4954.088843] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4954.091109] LustreError: Skipped 1 previous similar message [ 4954.096776] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4954.100855] Lustre: Skipped 1 previous similar message [ 4955.477186] Lustre: server umount lustre-OST0001 complete [ 4958.176427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4958.180180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4958.185663] Lustre: Skipped 1 previous similar message [ 4959.205510] LustreError: 145339:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4959.210997] LustreError: 145339:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 4964.024700] Lustre: server umount lustre-OST0000 complete [ 4969.442444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4969.442864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4969.451982] Lustre: Skipped 2 previous similar messages [ 4972.542217] Lustre: server umount lustre-MDT0000 complete [ 4974.563976] LustreError: 142368:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4974.571630] LustreError: 142368:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 4974.937524] LustreError: 142356:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743501642 with bad export cookie 1360949481064046620 [ 4974.942576] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4974.943185] LustreError: 142356:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4975.143732] Lustre: server umount lustre-MDT0001 complete [ 4993.987636] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 4996.681498] Key type lgssc unregistered [ 4997.015765] LNet: 148192:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4998.053597] LNet: Removed LNI 192.168.206.117@tcp [ 4998.845492] Key type .llcrypt unregistered [ 4998.846847] Key type ._llcrypt unregistered [ 5020.173248] Key type ._llcrypt registered [ 5020.174758] Key type .llcrypt registered [ 5020.252959] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 5035.965173] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 5037.457049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5037.472640] alg: No test for adler32 (adler32-zlib) [ 5038.502693] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 5038.669919] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 5038.672803] LNet: Accept secure, port 988 [ 5040.311164] Key type lgssc registered [ 5041.097802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5047.703063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5053.043409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5057.832338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5062.923762] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5067.314869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5067.364208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5067.374879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5068.564759] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5068.591809] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5068.637785] Lustre: lustre-MDT0000: new disk, initializing [ 5068.697220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5068.713665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5071.145847] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5077.869579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5077.924883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5078.008291] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5078.027395] Lustre: 152886:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5078.056123] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5078.058244] Lustre: Skipped 1 previous similar message [ 5078.100083] Lustre: lustre-MDT0001: new disk, initializing [ 5078.165793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5078.188887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5078.193886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5080.516140] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5086.126863] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5089.506968] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5093.885577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5093.962750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5094.128820] Lustre: 153838:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5094.150402] Lustre: lustre-OST0000: new disk, initializing [ 5094.153621] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5094.208137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5097.365609] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5102.947494] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5103.649173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5103.655375] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5103.749413] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5105.956455] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5106.134754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5108.456924] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5108.634316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5113.826812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5113.827411] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5113.832167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5113.837128] Lustre: Skipped 1 previous similar message [ 5116.043089] Lustre: server umount lustre-OST0000 complete [ 5118.945329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5118.946356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5118.950629] Lustre: Skipped 2 previous similar messages [ 5124.065294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5124.067866] Lustre: Skipped 2 previous similar messages [ 5124.533369] Lustre: server umount lustre-MDT0000 complete [ 5126.887034] LustreError: 152115:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743501794 with bad export cookie 2306761373048048845 [ 5126.890895] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5126.892167] LustreError: 152115:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5127.118139] Lustre: server umount lustre-MDT0001 complete [ 5135.437303] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 06:03:21 (1743501801) [ 5145.373622] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 5154.906658] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 5163.083798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5168.258248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5172.989579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5178.129070] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5182.509779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5182.564475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5182.749057] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5182.779597] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5182.821466] Lustre: lustre-MDT0000: new disk, initializing [ 5182.878122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5182.893096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5185.276792] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5192.182996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5192.244643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5192.333693] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5192.348725] Lustre: 159176:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5192.373625] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5192.376259] Lustre: Skipped 1 previous similar message [ 5192.418864] Lustre: lustre-MDT0001: new disk, initializing [ 5192.486828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5192.511439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5192.515320] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5194.852611] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5200.344763] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5203.709174] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5208.499487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5208.560711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5208.738502] Lustre: 160125:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5208.775220] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 5208.778178] Lustre: lustre-OST0000: new disk, initializing [ 5208.783889] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5208.865260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5212.516816] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5214.198151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5214.203052] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5214.263592] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5218.229443] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5221.877259] Lustre: Failing over lustre-OST0000 [ 5221.976770] Lustre: server umount lustre-OST0000 complete [ 5237.544221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5237.648266] Lustre: 161264:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5237.724283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5237.729852] Lustre: Skipped 1 previous similar message [ 5237.744359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5237.753966] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5240.470412] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5240.799166] Lustre: 150344:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743501892/real 1743501892] req@0000000093b883bf x1828194044934272/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1743501908 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 5240.809099] Lustre: 150344:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 5241.106656] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5242.872067] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.206.117@tcp (at 0@lo) [ 5242.872306] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 5243.368933] Lustre: Failing over lustre-MDT0000 [ 5243.492472] Lustre: server umount lustre-MDT0000 complete [ 5243.873062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5243.876600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5245.983110] Lustre: 150342:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743501897/real 1743501897] req@000000003f83e466 x1828194044935552/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1743501913 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 5247.972097] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5247.976338] Lustre: Skipped 1 previous similar message [ 5250.699252] LustreError: 159594:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5250.707386] LustreError: 159594:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 6 previous similar messages [ 5251.039306] Lustre: 150344:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743501902/real 1743501902] req@00000000ad2ad597 x1828194044936704/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1743501918 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 5253.088622] LustreError: 159205:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5253.095709] LustreError: 159205:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 5255.821196] LustreError: 161926:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5258.208353] LustreError: 159205:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5258.214405] LustreError: 159205:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 5259.231309] Lustre: 158417:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743501911/real 1743501911] req@0000000089dbaa8b x1828194044943872/t0(0) o101->MGC192.168.206.117@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1743501927 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 5259.240982] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5259.245958] Lustre: 158417:0:(mgc_request.c:1858:mgc_process_log()) MGC192.168.206.117@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 5259.397373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5264.352245] LustreError: 158425:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5264.359213] LustreError: 158425:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 6 previous similar messages [ 5269.472089] LustreError: 150340:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e45206e0 x1828194044948864/t0(0) o250->MGC192.168.206.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 5269.663055] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5271.179810] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5272.061540] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5275.118444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.117@tcp (at 0@lo) [ 5275.121912] Lustre: Skipped 1 previous similar message [ 5275.132820] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5275.151185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5278.688862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5278.691297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5278.697241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5283.957301] Lustre: server umount lustre-OST0000 complete [ 5290.465458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5290.466365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5290.467696] LustreError: Skipped 1 previous similar message [ 5290.470928] Lustre: Skipped 3 previous similar messages [ 5292.336593] Lustre: server umount lustre-MDT0000 complete [ 5294.619524] LustreError: 158407:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743501962 with bad export cookie 2306761373048050945 [ 5294.624383] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5294.624635] LustreError: 158407:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5294.813836] Lustre: server umount lustre-MDT0001 complete [ 5308.266899] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 5311.013492] Key type lgssc unregistered [ 5311.313923] LNet: 163637:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5312.357530] LNet: Removed LNI 192.168.206.117@tcp [ 5313.127832] Key type .llcrypt unregistered [ 5313.129478] Key type ._llcrypt unregistered [ 5349.670201] Key type ._llcrypt registered [ 5349.671619] Key type .llcrypt registered [ 5349.755960] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 5351.230744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5351.308536] alg: No test for adler32 (adler32-zlib) [ 5352.470767] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 5352.724683] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 5352.727546] LNet: Accept secure, port 988 [ 5354.391200] Key type lgssc registered [ 5355.212192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5360.492523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5360.507660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5361.851939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5364.544633] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5370.675252] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5371.361882] LustreError: 165127:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5371.368839] LustreError: 165127:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 5374.175853] LustreError: 165156:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5374.178587] LustreError: 165156:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5374.181234] LustreError: 165156:0:(lod_dev.c:506:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 0, failed: rc = -5 [ 5374.380912] Lustre: server umount lustre-MDT0000 complete [ 5383.506914] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 5393.152695] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 5401.340767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5406.711980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5411.588994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5416.718509] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5420.978368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5421.033569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5421.210224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5421.233848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5421.277621] Lustre: lustre-MDT0000: new disk, initializing [ 5421.335138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5421.351206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5423.614376] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5430.153212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5430.210610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5430.286955] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5430.300862] Lustre: 169064:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5430.333686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5430.335978] Lustre: Skipped 1 previous similar message [ 5430.337974] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5430.378234] Lustre: lustre-MDT0001: new disk, initializing [ 5430.441056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5430.464920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5430.468711] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5432.722642] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5438.451110] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5441.994969] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5446.448078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5446.511046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5446.683097] Lustre: 170017:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5446.705143] Lustre: lustre-OST0000: new disk, initializing [ 5446.709894] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5446.774363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5450.264936] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5454.885909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5454.890611] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5454.978670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5456.037948] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5459.810612] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5461.580418] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 06:08:47 (1743502127) [ 5608.417338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5608.418478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5608.423258] Lustre: Skipped 1 previous similar message [ 5613.199384] Lustre: server umount lustre-MDT0000 complete [ 5613.536249] LustreError: 169489:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5613.543401] LustreError: 169489:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 5615.474239] LustreError: 169078:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743502283 with bad export cookie 16983896963985978581 [ 5615.478840] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5615.480464] LustreError: 169078:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5618.655608] LustreError: 168310:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5618.655734] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5618.665062] Lustre: Skipped 1 previous similar message [ 5618.667014] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5618.668964] Lustre: Skipped 2 previous similar messages [ 5620.704076] LustreError: 168310:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5620.704077] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5621.736673] Lustre: server umount lustre-MDT0001 complete [ 5630.653178] Lustre: server umount lustre-OST0000 complete [ 5636.618419] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 5646.565622] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 5655.780569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5661.256839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5665.848400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5670.789311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5675.066990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5675.123670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5675.296975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5675.325114] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5675.369455] Lustre: lustre-MDT0000: new disk, initializing [ 5675.423347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5675.436375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5677.774444] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5684.640946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5684.709940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5684.796963] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5684.812410] Lustre: 175287:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5684.838125] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5684.840135] Lustre: Skipped 1 previous similar message [ 5684.884494] Lustre: lustre-MDT0001: new disk, initializing [ 5684.956966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5684.983659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5684.988757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5687.531381] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5693.325175] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5696.925488] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5701.677732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5701.737067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5701.909383] Lustre: 176240:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5701.931615] Lustre: lustre-OST0000: new disk, initializing [ 5701.935936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5701.996238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5705.679474] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5711.384481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5711.388404] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5711.470922] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5711.596414] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5714.519147] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5714.710385] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5717.322352] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5717.500470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5721.570261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5721.571354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5721.575021] LustreError: Skipped 1 previous similar message [ 5721.582604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5721.585617] Lustre: Skipped 1 previous similar message [ 5731.810348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5731.812902] Lustre: Skipped 3 previous similar messages [ 5732.831138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5732.931699] Lustre: server umount lustre-OST0000 complete [ 5736.416216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5736.419226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5736.423345] Lustre: Skipped 1 previous similar message [ 5741.550635] Lustre: server umount lustre-MDT0000 complete [ 5742.048421] LustreError: 175308:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5742.054505] LustreError: 175308:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 5743.788086] LustreError: 174517:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743502411 with bad export cookie 16983896963986137411 [ 5743.791925] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5743.793222] LustreError: 174517:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5743.982606] Lustre: server umount lustre-MDT0001 complete [ 5752.830572] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:13:38 (1743502418) [ 5763.601042] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 5773.762949] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 5782.177372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5787.912671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5793.044730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5798.251245] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5802.952813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5803.018376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5803.202612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5803.229806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5803.273671] Lustre: lustre-MDT0000: new disk, initializing [ 5803.337429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5803.351280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5805.884280] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5812.855623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5812.912674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5813.015044] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 5813.032845] Lustre: 181583:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5813.197054] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5815.735611] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5821.908102] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5825.467989] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5830.139547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5830.196590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5830.370921] Lustre: 182537:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5833.940857] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5837.325505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5837.329163] Lustre: Skipped 1 previous similar message [ 5837.331093] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5837.414713] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5839.836271] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5847.525730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5847.526420] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5847.530865] Lustre: Skipped 1 previous similar message [ 5847.536969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5847.539652] Lustre: Skipped 3 previous similar messages [ 5850.623588] Lustre: server umount lustre-OST0000 complete [ 5854.176556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5854.182423] LustreError: Skipped 1 previous similar message [ 5857.763519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5857.768621] Lustre: Skipped 2 previous similar messages [ 5859.116554] Lustre: server umount lustre-MDT0000 complete [ 5861.364470] LustreError: 182566:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743502529 with bad export cookie 16983896963986138293 [ 5861.368296] LustreError: 182566:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5861.369277] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5861.563960] Lustre: server umount lustre-MDT0001 complete [ 5869.764508] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:15:35 (1743502535) [ 5880.590832] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 5891.012880] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 5899.265126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5904.574571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5909.253768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5914.160947] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5918.247073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5918.302073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5918.484632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 5918.507477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5918.509144] Lustre: Skipped 3 previous similar messages [ 5918.550898] Lustre: lustre-MDT0000: new disk, initializing [ 5918.552982] Lustre: Skipped 2 previous similar messages [ 5918.618119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5918.621122] Lustre: Skipped 2 previous similar messages [ 5918.637455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5920.973724] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5927.739111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5927.801485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5927.905688] Lustre: 187492:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5928.086834] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5930.599840] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5936.444403] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5940.126936] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5944.780736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5944.849330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5948.699299] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 5954.372750] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5954.640966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5959.653168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5959.654186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5959.657739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5959.659786] LustreError: Skipped 1 previous similar message [ 5959.663181] Lustre: Skipped 4 previous similar messages [ 5961.082476] Lustre: server umount lustre-OST0000 complete [ 5963.381480] Lustre: server umount lustre-MDT0000 complete [ 5963.745103] LustreError: 186741:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5965.676664] LustreError: 186720:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743502633 with bad export cookie 16983896963986139378 [ 5965.679889] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5965.682497] LustreError: 186720:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5971.970695] Lustre: server umount lustre-MDT0001 complete [ 5985.210281] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 5987.979494] Key type lgssc unregistered [ 5988.269825] LNet: 190211:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5989.286607] LNet: Removed LNI 192.168.206.117@tcp [ 5990.111026] Key type .llcrypt unregistered [ 5990.112566] Key type ._llcrypt unregistered [ 6014.090590] Key type ._llcrypt registered [ 6014.092181] Key type .llcrypt registered [ 6014.180464] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 6015.442455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6015.516210] alg: No test for adler32 (adler32-zlib) [ 6016.675576] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 6016.874530] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 6016.879420] LNet: Accept secure, port 988 [ 6018.519183] Key type lgssc registered [ 6019.322810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6024.107322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6024.117692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6025.372619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6027.542892] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6032.430640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6032.493820] Lustre: 192294:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6032.637836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6034.905951] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6038.871811] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6042.043257] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6046.206118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6046.306178] Lustre: 193177:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6046.396848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6049.778325] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6050.539250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6053.824137] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6073.315244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6073.315640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6073.323446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6073.325571] Lustre: Skipped 1 previous similar message [ 6078.435357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6078.436990] Lustre: Skipped 1 previous similar message [ 6078.707587] Lustre: server umount lustre-OST0000 complete [ 6083.553347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6083.554211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6083.557791] Lustre: Skipped 2 previous similar messages [ 6088.672105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6088.674219] Lustre: Skipped 2 previous similar messages [ 6093.792503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6093.794501] Lustre: Skipped 1 previous similar message [ 6095.327102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6095.408945] Lustre: server umount lustre-MDT0000 complete [ 6097.611380] LustreError: 194049:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743502765 with bad export cookie 17022147785249178628 [ 6097.615649] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6097.615762] LustreError: 194049:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6097.773037] Lustre: server umount lustre-MDT0001 complete [ 6111.059365] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 6113.807645] Key type lgssc unregistered [ 6114.119623] LNet: 194798:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6115.173452] LNet: Removed LNI 192.168.206.117@tcp [ 6115.943320] Key type .llcrypt unregistered [ 6115.945065] Key type ._llcrypt unregistered [ 6139.537169] Key type ._llcrypt registered [ 6139.540310] Key type .llcrypt registered [ 6139.625476] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 6141.104327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6141.131130] alg: No test for adler32 (adler32-zlib) [ 6142.276853] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 6142.474476] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 6142.476087] LNet: Accept secure, port 988 [ 6144.103246] Key type lgssc registered [ 6144.909625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6149.612487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6149.626251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6150.892622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6153.173860] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6158.299135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6158.347802] Lustre: 196878:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6158.490098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6160.882101] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6164.781763] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6168.022142] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6172.359702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6172.466705] Lustre: 197759:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6172.554161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6175.662760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6175.968626] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6180.006334] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6189.026500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6189.027043] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6189.032560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6189.037832] Lustre: Skipped 1 previous similar message [ 6190.449025] Lustre: server umount lustre-OST0000 complete [ 6194.309662] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9669f0ea1000.ost_server_uuid 50 [ 6210.648468] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6223.841334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6223.842622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6223.846934] Lustre: Skipped 2 previous similar messages [ 6228.580676] Lustre: server umount lustre-MDT0000 complete [ 6228.961129] LustreError: 196182:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6228.965284] LustreError: 196182:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 6230.891318] LustreError: 198800:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743502898 with bad export cookie 4128638689751398828 [ 6230.894498] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6230.896772] LustreError: 198800:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6231.097972] Lustre: server umount lustre-MDT0001 complete [ 6239.311133] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 06:21:45 (1743502905) [ 6243.483848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6243.732562] LustreError: 199672:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6243.774888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6246.027191] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6248.929930] LustreError: 199672:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6250.938116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6250.997014] Lustre: 200371:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6251.124627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6253.347808] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6257.218545] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6260.154137] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6264.889292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6265.002603] Lustre: 201252:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6265.100355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6268.205603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6268.732613] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6273.189693] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6277.464337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6277.534953] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6277.653481] Lustre: 202144:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6277.670394] Lustre: lustre-OST0001: new disk, initializing [ 6277.673145] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6280.887518] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6284.818724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6284.823199] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6284.887872] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6286.414244] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6288.863990] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6288.866193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6288.870166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6288.873264] Lustre: Skipped 1 previous similar message [ 6292.633341] Lustre: lustre-OST0000: Not available for connect from 192.168.206.17@tcp (stopping) [ 6292.636089] Lustre: Skipped 2 previous similar messages [ 6294.790115] Lustre: server umount lustre-OST0000 complete [ 6295.011235] LustreError: 202714:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6295.016796] LustreError: 202714:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 6298.143332] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6298.316942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6300.133610] LustreError: 201256:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6300.139306] LustreError: 201256:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 6300.643464] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6300.811841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6310.370148] LustreError: 202714:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6310.375726] LustreError: 202714:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 5 previous similar messages [ 6315.489856] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6315.490242] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6315.492897] LustreError: Skipped 1 previous similar message [ 6315.498163] Lustre: Skipped 1 previous similar message [ 6315.500039] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6315.502903] Lustre: Skipped 1 previous similar message [ 6318.647575] Lustre: server umount lustre-OST0001 complete [ 6322.656023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6322.658825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6325.729177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6325.731022] Lustre: Skipped 2 previous similar messages [ 6327.022376] Lustre: server umount lustre-MDT0000 complete [ 6329.218392] LustreError: 199652:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743502997 with bad export cookie 4128638689751399899 [ 6329.222101] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6329.223752] LustreError: 199652:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6329.475251] Lustre: server umount lustre-MDT0001 complete [ 6332.922947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6335.274543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6337.516291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6339.937069] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6347.320454] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 06:23:33 (1743503013) [ 6351.863278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6352.066654] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6352.095304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6352.213969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6352.215883] Lustre: Skipped 1 previous similar message [ 6354.509561] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6359.653957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6359.716713] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6359.742942] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6359.757315] Lustre: 206165:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6362.370224] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6366.454307] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6369.812825] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6374.175638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6374.282378] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6377.828435] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6381.937129] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6384.171666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6386.607614] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6386.654979] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6386.699481] Lustre: 207971:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6386.703916] Lustre: 207971:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6386.794861] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6386.797188] Lustre: Skipped 2 previous similar messages [ 6390.296513] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6394.561839] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6396.899648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6396.903384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6396.904744] Lustre: Skipped 2 previous similar messages [ 6411.231152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6411.283117] Lustre: server umount lustre-OST0000 complete [ 6412.259632] LustreError: 207072:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6412.266258] LustreError: 207072:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 6427.617651] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6427.619113] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6427.626849] Lustre: Skipped 1 previous similar message [ 6430.674322] Lustre: server umount lustre-OST0001 complete [ 6436.832431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6436.835715] LustreError: Skipped 1 previous similar message [ 6436.838384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6436.840261] Lustre: Skipped 10 previous similar messages [ 6439.279570] Lustre: server umount lustre-MDT0000 complete [ 6441.655398] LustreError: 206180:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743503109 with bad export cookie 4128638689751401299 [ 6441.658995] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6441.660901] LustreError: 206180:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6441.853938] Lustre: server umount lustre-MDT0001 complete [ 6445.362355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6448.012221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6450.371062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6452.859832] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6459.790902] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 06:25:25 (1743503125) [ 6470.149349] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 6482.346843] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 6491.650617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6496.801304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6501.433088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6506.350629] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6510.550479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6510.621223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6510.801581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6510.824557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6510.867305] Lustre: lustre-MDT0000: new disk, initializing [ 6510.920957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6510.934189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6513.327395] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6519.739555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6519.791929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6519.870677] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 6519.885258] Lustre: 213836:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6519.904730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6519.907204] Lustre: Skipped 1 previous similar message [ 6519.949041] Lustre: lustre-MDT0001: new disk, initializing [ 6520.030235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6520.034664] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6522.202468] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6527.664611] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6531.123314] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6535.545665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6535.608553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6535.767913] Lustre: lustre-OST0000: new disk, initializing [ 6535.770966] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6539.095533] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6543.367612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6543.370989] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6543.439963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6544.671385] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6547.546389] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6547.718493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6550.014369] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6550.185811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6553.571079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6553.572443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6553.574267] LustreError: Skipped 1 previous similar message [ 6553.579039] Lustre: Skipped 2 previous similar messages [ 6553.581805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6553.585020] Lustre: Skipped 3 previous similar messages [ 6557.438471] Lustre: server umount lustre-OST0000 complete [ 6565.871260] Lustre: server umount lustre-MDT0000 complete [ 6568.170401] LustreError: 213067:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743503236 with bad export cookie 4128638689751402853 [ 6568.176714] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6568.370314] Lustre: server umount lustre-MDT0001 complete [ 6573.698776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6573.928234] LustreError: 216864:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6573.933698] LustreError: 216864:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 8 previous similar messages [ 6576.112750] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6581.406530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6583.779474] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6587.714382] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6590.899588] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6594.812426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6594.922607] Lustre: 218443:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6594.927317] Lustre: 218443:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 2 previous similar messages [ 6598.258567] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6602.002481] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6604.627240] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6604.788529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6607.236480] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6607.399257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6609.376439] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6609.378652] LustreError: Skipped 1 previous similar message [ 6614.645959] Lustre: server umount lustre-OST0000 complete [ 6617.908717] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6620.351954] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6646.413549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6646.597232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6646.599161] Lustre: Skipped 5 previous similar messages [ 6649.976505] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6651.748827] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6651.755145] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.206.117@tcp (at 0@lo) [ 6653.962646] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6661.988141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6661.991494] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.206.117@tcp (at 0@lo) [ 6667.126572] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 10 sec [ 6672.352977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6672.358251] Lustre: Skipped 5 previous similar messages [ 6675.172860] Lustre: server umount lustre-OST0000 complete [ 6682.592237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6682.594266] Lustre: Skipped 11 previous similar messages [ 6683.564620] Lustre: server umount lustre-MDT0000 complete [ 6685.811292] LustreError: 216845:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743503353 with bad export cookie 4128638689751403735 [ 6685.814140] LustreError: 216845:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6685.815942] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6686.000617] Lustre: server umount lustre-MDT0001 complete [ 6693.416847] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 06:29:19 (1743503359) [ 6697.524991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6697.752326] LustreError: 222292:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6697.758079] LustreError: 222292:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 6700.101974] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6704.948325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6707.290328] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6711.399216] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6714.930584] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6719.296167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6722.966232] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6726.943835] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6729.840956] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6730.015458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6732.398121] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6732.565293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6736.644872] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6736.724842] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6736.832139] Lustre: 225251:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6736.835285] Lustre: 225251:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 3 previous similar messages [ 6736.850281] Lustre: lustre-OST0001: new disk, initializing [ 6736.853311] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6740.150266] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6742.551249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6742.554135] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6742.624682] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6745.346382] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6747.964198] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6748.137800] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6750.418634] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6750.580504] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6752.739185] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6752.745393] LustreError: Skipped 4 previous similar messages [ 6757.773982] Lustre: server umount lustre-OST0001 complete [ 6761.461687] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6761.636897] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6763.955403] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6764.168375] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6790.300860] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6793.767349] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6795.751922] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6795.755739] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6795.755843] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 192.168.206.117@tcp (at 0@lo) [ 6797.907955] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6799.749887] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6805.993285] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6805.996994] Lustre: Skipped 5 previous similar messages [ 6807.171903] Lustre: server umount lustre-OST0001 complete [ 6826.467319] LustreError: 223876:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6826.471676] LustreError: 223876:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 29 previous similar messages [ 6827.316937] Lustre: server umount lustre-OST0000 complete [ 6835.640805] Lustre: server umount lustre-MDT0000 complete [ 6838.111870] LustreError: 222318:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743503505 with bad export cookie 4128638689751404701 [ 6838.116636] LustreError: 222318:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6838.117451] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6844.337239] Lustre: server umount lustre-MDT0001 complete [ 6847.446827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6849.793177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6852.133361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6854.722735] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6862.230738] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 06:32:07 (1743503527) [ 6866.358058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6866.549069] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6866.563109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6868.909747] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6873.839481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6873.887561] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6876.224187] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6880.195179] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6883.597407] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6888.047371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6888.148118] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6891.564402] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6895.819377] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6902.835481] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6902.885843] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6903.025805] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6903.028712] Lustre: Skipped 8 previous similar messages [ 6906.379407] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6910.465877] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6913.284207] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6915.862648] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6916.023579] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6918.275142] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6918.448883] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6921.188919] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9669f9829000.ost_server_uuid 50 [ 6922.826186] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9669f9829000.ost_server_uuid in FULL state after 0 sec [ 6923.822825] Lustre: Permanently deactivating lustre-OST0001 [ 6923.830461] Lustre: Setting parameter lustre-OST0001-osc.osc.active in log lustre-client [ 6923.833625] Lustre: Skipped 1 previous similar message [ 6925.808543] Lustre: Permanently reactivating lustre-OST0001 [ 6930.123170] Lustre: server umount lustre-OST0001 complete [ 6931.941967] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6931.944364] LustreError: Skipped 2 previous similar messages [ 6938.364456] Lustre: server umount lustre-OST0000 complete [ 6940.640915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6940.643495] Lustre: Skipped 15 previous similar messages [ 6946.669772] Lustre: server umount lustre-MDT0000 complete [ 6948.890354] LustreError: 231822:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743503616 with bad export cookie 4128638689751405975 [ 6948.894294] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6948.895227] LustreError: 231822:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6949.087460] Lustre: server umount lustre-MDT0001 complete [ 6952.418054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6954.757293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6956.982068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6959.139652] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6965.831989] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 06:33:51 (1743503631) [ 6966.882393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6970.736095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6970.902113] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6970.918757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 6970.920992] Lustre: Skipped 5 previous similar messages [ 6973.213343] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6978.088289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6978.136647] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6980.505650] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6984.487569] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6987.914802] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6992.446629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6992.513355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6992.622906] Lustre: Permanently deactivating lustre-OST0000 [ 6993.790671] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6996.079090] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7000.200794] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7004.652257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7004.728459] Lustre: 238812:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7004.733287] Lustre: 238812:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 6 previous similar messages [ 7008.011246] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7010.390545] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7012.178108] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7014.491997] Lustre: Permanently reactivating lustre-OST0000 [ 7014.499421] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [ 7014.501556] Lustre: Skipped 4 previous similar messages [ 7017.601796] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7017.638734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7029.698560] Lustre: server umount lustre-OST0001 complete [ 7046.111224] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7046.202515] Lustre: server umount lustre-OST0000 complete [ 7054.705682] Lustre: server umount lustre-MDT0000 complete [ 7056.899324] LustreError: 236333:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743503724 with bad export cookie 4128638689751407585 [ 7056.902991] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7056.904388] LustreError: 236333:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7057.099138] Lustre: server umount lustre-MDT0001 complete [ 7069.214842] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 7071.791899] Key type lgssc unregistered [ 7072.070495] LNet: 240658:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7073.125497] LNet: Removed LNI 192.168.206.117@tcp [ 7073.853601] Key type .llcrypt unregistered [ 7073.854500] Key type ._llcrypt unregistered [ 7097.423523] Key type ._llcrypt registered [ 7097.424571] Key type .llcrypt registered [ 7097.514142] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 7099.056698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7099.071256] alg: No test for adler32 (adler32-zlib) [ 7100.078704] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 7100.239980] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 7100.242323] LNet: Accept secure, port 988 [ 7101.871231] Key type lgssc registered [ 7102.657846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7104.936078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7108.652731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7108.665183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7109.962459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7112.195802] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7117.465613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7117.524070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7117.573990] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7117.593284] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7117.596960] Lustre: Permanently deactivating lustre-MDT0001 [ 7117.624757] Lustre: 242793:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7117.786636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7118.114794] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7118.120301] LustreError: 242125:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7118.125581] LustreError: 242125:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7118.129358] LustreError: 242125:0:(lod_dev.c:506:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 1, failed: rc = -108 [ 7120.041122] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7124.261846] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7127.522789] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7131.846589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7131.963178] Lustre: 243694:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7132.057958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7135.141029] LustreError: 243700:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7135.145511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7135.147501] LustreError: 243700:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 7135.651070] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7139.580761] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7140.321624] LustreError: 243700:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7142.371883] LustreError: 243701:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7142.377774] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7142.381195] LustreError: 243701:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 7144.119859] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7144.163644] Lustre: 244584:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7144.280162] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7147.568964] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7149.564622] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7149.570532] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7151.471954] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7154.135656] Lustre: Permanently reactivating lustre-MDT0001 [ 7154.143562] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 7154.146435] Lustre: Skipped 3 previous similar messages [ 7158.422420] Lustre: lustre-MDT0001: Client ff995b19-1383-4634-9bc7-c3ba332ce825 (at 192.168.206.17@tcp) reconnecting [ 7161.495405] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7163.135655] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7166.246772] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7166.418543] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7168.901779] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7169.082495] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7170.631756] Lustre: Permanently deactivating lustre-MDT0001 [ 7170.638620] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active in log lustre-client [ 7170.641491] Lustre: Skipped 2 previous similar messages [ 7171.348502] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7184.351935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7184.353838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7184.358712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7185.379104] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7185.379490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7185.382631] Lustre: Skipped 2 previous similar messages [ 7185.385019] Lustre: Skipped 2 previous similar messages [ 7190.193106] Lustre: server umount lustre-MDT0000 complete [ 7190.500663] LustreError: 242089:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7190.504963] LustreError: 242089:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 7192.356940] LustreError: 242077:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743503860 with bad export cookie 17563886950225164793 [ 7192.361845] LustreError: 242077:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7192.362896] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7192.560894] Lustre: server umount lustre-MDT0001 complete [ 7200.905954] Lustre: server umount lustre-OST0000 complete [ 7210.925764] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 06:37:56 (1743503876) [ 7217.440307] Lustre: server umount lustre-OST0001 complete [ 7221.420148] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 7230.951514] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 7240.644799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7246.163372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7250.554682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7255.090127] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7259.018913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7259.055894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7259.202323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7259.204969] Lustre: Skipped 2 previous similar messages [ 7259.226417] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7259.268173] Lustre: lustre-MDT0000: new disk, initializing [ 7259.320483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7259.332107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7261.405801] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7267.680235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7267.739458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7267.804696] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7267.817904] Lustre: 250482:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7267.835482] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7267.837066] Lustre: Skipped 1 previous similar message [ 7267.870094] Lustre: lustre-MDT0001: new disk, initializing [ 7267.922071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7267.942143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7267.946130] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7270.021226] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7275.423587] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7278.380832] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7282.590836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7282.650091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7282.794524] Lustre: 251430:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7282.813016] Lustre: lustre-OST0000: new disk, initializing [ 7282.815728] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7286.100279] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7291.206069] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7291.397416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7291.400176] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7291.475431] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7293.147708] LustreError: 250197:0:(mdt_handler.c:3118:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7298.925192] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7298.984476] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7299.083219] Lustre: 252441:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7299.103269] Lustre: lustre-OST0001: new disk, initializing [ 7299.106544] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7299.151262] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7299.154300] Lustre: Skipped 1 previous similar message [ 7302.323020] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7303.247066] LustreError: 250197:0:(mdt_handler.c:3118:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7303.256841] LustreError: 250197:0:(mdt_handler.c:3118:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7307.485705] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7308.821828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7308.826556] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7308.901549] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7309.167266] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7313.351116] LustreError: 250197:0:(mdt_handler.c:3118:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7315.604553] Lustre: lustre-OST0001: Not available for connect from 192.168.206.17@tcp (stopping) [ 7319.010922] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7319.012042] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7319.014152] LustreError: Skipped 1 previous similar message [ 7319.019183] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7319.022211] Lustre: Skipped 1 previous similar message [ 7320.453934] Lustre: server umount lustre-OST0001 complete [ 7320.716067] LustreError: 251440:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7329.253675] LustreError: 251438:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7329.259661] LustreError: 251438:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 7334.373229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7334.375134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7334.376946] Lustre: Skipped 2 previous similar messages [ 7334.381253] Lustre: Skipped 1 previous similar message [ 7340.340104] Lustre: server umount lustre-OST0000 complete [ 7344.609634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7344.610424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7344.615049] Lustre: Skipped 1 previous similar message [ 7344.616620] Lustre: Skipped 2 previous similar messages [ 7344.620890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7348.655920] Lustre: server umount lustre-MDT0000 complete [ 7349.729450] LustreError: 249732:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7349.733795] LustreError: 249732:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 5 previous similar messages [ 7350.736590] LustreError: 249712:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743504018 with bad export cookie 17563886950225167593 [ 7350.741959] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7365.087097] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7365.243304] Lustre: server umount lustre-MDT0001 complete [ 7377.453362] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 7380.035353] Key type lgssc unregistered [ 7380.314494] LNet: 254344:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7381.349393] LNet: Removed LNI 192.168.206.117@tcp [ 7382.090517] Key type .llcrypt unregistered [ 7382.091573] Key type ._llcrypt unregistered [ 7401.790556] Key type ._llcrypt registered [ 7401.791737] Key type .llcrypt registered [ 7401.872966] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 7416.596250] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 7417.990090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7418.003420] alg: No test for adler32 (adler32-zlib) [ 7419.080938] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 7419.239832] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 7419.241903] LNet: Accept secure, port 988 [ 7420.879235] Key type lgssc registered [ 7421.777806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7429.346230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7435.493376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7440.777655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7446.480531] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7450.964255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7451.015123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7451.026465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7452.207071] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 7452.233053] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7452.276404] Lustre: lustre-MDT0000: new disk, initializing [ 7452.329584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7452.342893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7454.731345] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7461.589755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7461.655667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7461.742121] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 7461.758077] Lustre: 259037:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7461.784621] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7461.786486] Lustre: Skipped 1 previous similar message [ 7461.828029] Lustre: lustre-MDT0001: new disk, initializing [ 7461.902067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7461.927944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7461.931263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7464.509135] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7470.347310] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7473.732953] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7478.266716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7478.351161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7478.528500] Lustre: 259988:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7478.550820] Lustre: lustre-OST0000: new disk, initializing [ 7478.555726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7478.613744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7482.126365] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7484.436875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7484.440402] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7484.525793] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7488.123112] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7491.190642] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7491.377314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7493.836677] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7494.030298] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7499.745910] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7499.747460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7499.750033] LustreError: Skipped 1 previous similar message [ 7499.754628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7499.758187] Lustre: Skipped 1 previous similar message [ 7501.327512] Lustre: server umount lustre-OST0000 complete [ 7504.866786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7504.867807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7504.870511] Lustre: Skipped 2 previous similar messages [ 7504.875340] Lustre: Skipped 1 previous similar message [ 7509.897568] Lustre: server umount lustre-MDT0000 complete [ 7512.304571] LustreError: 258268:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743504180 with bad export cookie 6565966035528391474 [ 7512.309159] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7512.309292] LustreError: 258268:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7512.515158] Lustre: server umount lustre-MDT0001 complete [ 7521.946966] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 06:43:07 (1743504187) [ 7526.417856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7526.693928] LustreError: 262261:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7526.703242] LustreError: 262261:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 7526.740759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7529.115432] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7532.002313] LustreError: 262261:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7534.292408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7534.343896] Lustre: 262957:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7534.477552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7536.732862] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7540.959900] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7544.392558] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7548.883335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7548.985242] Lustre: 263838:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7549.064774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7552.646137] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7556.914582] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7605.729293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7605.731254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7605.738563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7605.740612] Lustre: Skipped 1 previous similar message [ 7610.851064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7610.852970] Lustre: Skipped 1 previous similar message [ 7611.013447] Lustre: server umount lustre-OST0000 complete [ 7613.759228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7621.438699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7621.554934] Lustre: 265453:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7621.648163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7625.352862] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7629.774173] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7636.007045] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7636.011448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7636.011637] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.206.117@tcp (at 0@lo) [ 7636.017305] Lustre: Skipped 1 previous similar message [ 7669.731417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7669.732777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7669.738036] Lustre: Skipped 2 previous similar messages [ 7673.997493] Lustre: server umount lustre-OST0000 complete [ 7677.920474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7677.922413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7677.926061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7677.927698] Lustre: Skipped 1 previous similar message [ 7682.276725] Lustre: server umount lustre-MDT0000 complete [ 7684.373481] LustreError: 262241:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743504352 with bad export cookie 6565966035528392356 [ 7684.378900] LustreError: 262241:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7684.382828] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7684.581242] Lustre: server umount lustre-MDT0001 complete [ 7700.234690] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 7702.701936] Key type lgssc unregistered [ 7702.987597] LNet: 267177:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7704.037426] LNet: Removed LNI 192.168.206.117@tcp [ 7704.717242] Key type .llcrypt unregistered [ 7704.718224] Key type ._llcrypt unregistered [ 7730.564629] Key type ._llcrypt registered [ 7730.565725] Key type .llcrypt registered [ 7730.647409] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 7732.048303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7732.135972] alg: No test for adler32 (adler32-zlib) [ 7733.217392] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 7733.371804] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 7733.374610] LNet: Accept secure, port 988 [ 7734.999183] Key type lgssc registered [ 7735.744021] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7740.206921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7740.217986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7741.458399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7743.555194] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7748.131558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7748.181104] Lustre: 269260:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7748.327890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7750.393825] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7754.087302] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7757.779838] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7762.141912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7762.253773] Lustre: 270144:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7762.343126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7765.332047] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7766.448529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7769.374663] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7789.026104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7789.028047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7789.029897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7789.033532] Lustre: Skipped 1 previous similar message [ 7789.247356] Lustre: server umount lustre-OST0000 complete [ 7791.422867] Lustre: server umount lustre-MDT0000 complete [ 7793.587478] LustreError: 268544:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743504461 with bad export cookie 4287607872007618251 [ 7793.595885] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7799.873940] Lustre: server umount lustre-MDT0001 complete [ 7814.918060] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 7817.537195] Key type lgssc unregistered [ 7817.835574] LNet: 272509:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7818.853408] LNet: Removed LNI 192.168.206.117@tcp [ 7819.524803] Key type .llcrypt unregistered [ 7819.525775] Key type ._llcrypt unregistered [ 7838.834259] Key type ._llcrypt registered [ 7838.835623] Key type .llcrypt registered [ 7838.906156] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 7840.306607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7840.331440] alg: No test for adler32 (adler32-zlib) [ 7841.419398] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 7841.572993] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 7841.575333] LNet: Accept secure, port 988 [ 7843.199207] Key type lgssc registered [ 7843.946961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7848.723431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7848.737063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7850.032927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7852.345507] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7857.479701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7857.536186] Lustre: 274397:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7857.689075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7859.996648] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7864.004325] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7867.881794] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7872.203744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7872.304577] Lustre: 275278:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7872.379138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7874.431845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7875.733425] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7879.804509] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7892.965255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7892.967720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7892.971930] Lustre: Skipped 1 previous similar message [ 7892.977369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7897.293762] Lustre: server umount lustre-OST0000 complete [ 7903.200807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7903.201811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7903.204276] Lustre: Skipped 1 previous similar message [ 7903.207028] Lustre: Skipped 1 previous similar message [ 7905.637278] Lustre: server umount lustre-MDT0000 complete [ 7907.758258] LustreError: 273682:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743504575 with bad export cookie 13546592213332902669 [ 7907.761166] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7907.763685] LustreError: 273682:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7907.936782] Lustre: server umount lustre-MDT0001 complete [ 7923.029420] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 7925.595804] Key type lgssc unregistered [ 7925.855437] LNet: 277022:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7926.885437] LNet: Removed LNI 192.168.206.117@tcp [ 7927.603413] Key type .llcrypt unregistered [ 7927.604825] Key type ._llcrypt unregistered [ 7954.641771] Key type ._llcrypt registered [ 7954.642741] Key type .llcrypt registered [ 7954.724249] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 7956.095152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7956.122620] alg: No test for adler32 (adler32-zlib) [ 7957.367789] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 7957.574759] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 7957.577093] LNet: Accept secure, port 988 [ 7959.199145] Key type lgssc registered [ 7959.873357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7964.546657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7964.558730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7965.835350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7968.303556] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7973.778201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7973.832976] Lustre: 279105:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7974.004199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7976.587101] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7981.217386] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7985.365764] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7990.227816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7990.307668] LustreError: 279993:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7990.316024] LustreError: 279993:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 7990.343824] Lustre: 279986:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 7990.427176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7993.935706] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7995.880706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7998.331207] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8015.222629] Lustre: server umount lustre-OST0000 complete [ 8019.935921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8019.938401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8019.942567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8020.448695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8020.449352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8025.568161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8025.570223] Lustre: Skipped 2 previous similar messages [ 8027.659243] Lustre: server umount lustre-MDT0000 complete [ 8029.762479] LustreError: 278388:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743504697 with bad export cookie 11289671644898088757 [ 8029.766211] LustreError: 278388:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8029.766239] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8040.013382] Lustre: server umount lustre-MDT0001 complete [ 8054.374878] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 8057.101689] Key type lgssc unregistered [ 8057.381662] LNet: 282356:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8058.405419] LNet: Removed LNI 192.168.206.117@tcp [ 8059.116709] Key type .llcrypt unregistered [ 8059.117674] Key type ._llcrypt unregistered [ 8078.023782] Key type ._llcrypt registered [ 8078.025308] Key type .llcrypt registered [ 8078.095126] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 8079.500250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8079.546192] alg: No test for adler32 (adler32-zlib) [ 8080.506880] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 8080.663026] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 8080.665055] LNet: Accept secure, port 988 [ 8082.287172] Key type lgssc registered [ 8083.046694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8087.658495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8087.667661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8088.932370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8091.222788] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8096.292033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8096.349661] Lustre: 284284:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8096.511707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8098.866965] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8102.784343] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8106.507357] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8110.786958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8110.888756] Lustre: 285164:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8110.969634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8114.029115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8114.174489] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8117.984225] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8131.042411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 8131.045138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8131.050068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8131.555569] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8131.556131] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8131.564786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8135.756175] Lustre: server umount lustre-OST0000 complete [ 8141.794367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8141.795647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8141.797536] Lustre: Skipped 1 previous similar message [ 8141.800179] Lustre: Skipped 1 previous similar message [ 8144.315570] Lustre: server umount lustre-MDT0000 complete [ 8146.780638] LustreError: 286158:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743504814 with bad export cookie 18145077128054783729 [ 8146.785849] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8146.789893] LustreError: 286158:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8147.031293] Lustre: server umount lustre-MDT0001 complete [ 8161.470716] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 8163.911299] Key type lgssc unregistered [ 8164.178576] LNet: 286905:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8165.221483] LNet: Removed LNI 192.168.206.117@tcp [ 8165.924114] Key type .llcrypt unregistered [ 8165.925593] Key type ._llcrypt unregistered [ 8174.972287] Key type ._llcrypt registered [ 8174.974212] Key type .llcrypt registered [ 8175.061257] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8185.784584] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 8200.545902] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 8201.902171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8201.919653] alg: No test for adler32 (adler32-zlib) [ 8202.983430] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 8203.130941] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 8203.133048] LNet: Accept secure, port 988 [ 8204.767213] Key type lgssc registered [ 8205.539700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8212.065240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8216.986913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8221.172624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8225.849305] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8230.043262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8230.089613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8230.099021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8231.274564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8231.296539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8231.335207] Lustre: lustre-MDT0000: new disk, initializing [ 8231.386129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8231.400411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8233.604283] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8240.095856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8240.150643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8240.228603] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8240.242953] Lustre: 291150:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8240.264197] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8240.266177] Lustre: Skipped 1 previous similar message [ 8240.303757] Lustre: lustre-MDT0001: new disk, initializing [ 8240.368957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8240.390068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8240.393331] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8242.593596] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8247.886661] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8251.218247] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8255.341830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8255.395600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8255.541480] Lustre: 292102:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8255.563118] Lustre: lustre-OST0000: new disk, initializing [ 8255.567531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8255.617868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8258.808991] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8264.250487] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8264.710641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8264.715262] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8264.779802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8266.922495] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8267.091141] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8269.416824] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8269.569802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8274.916940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8274.917135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8274.918449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8274.920187] LustreError: Skipped 1 previous similar message [ 8274.925592] Lustre: Skipped 1 previous similar message [ 8276.797714] Lustre: server umount lustre-OST0000 complete [ 8280.033148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8280.034448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8280.039054] Lustre: Skipped 1 previous similar message [ 8280.040918] Lustre: Skipped 1 previous similar message [ 8285.152731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8285.155127] Lustre: Skipped 2 previous similar messages [ 8290.272576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8290.274739] Lustre: Skipped 1 previous similar message [ 8293.343168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8293.426435] Lustre: server umount lustre-MDT0000 complete [ 8295.579484] LustreError: 291164:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743504963 with bad export cookie 17624023686931068471 [ 8295.582180] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8295.582789] LustreError: 291164:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8295.755625] Lustre: server umount lustre-MDT0001 complete [ 8304.037446] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 06:56:09 (1743504969) [ 8308.145489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8308.381515] LustreError: 294375:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8308.389267] LustreError: 294375:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 8308.420477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8310.604343] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8313.824235] LustreError: 294375:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8315.725517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8315.791965] Lustre: 295072:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8315.945961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8318.160383] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8322.056071] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8325.308331] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8329.640713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8329.770026] Lustre: 295952:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8329.866633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8333.029855] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8336.948434] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8346.595386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8346.595522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8346.596325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8346.606963] Lustre: Skipped 1 previous similar message [ 8352.563001] Lustre: server umount lustre-OST0000 complete [ 8356.832574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8356.835025] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8356.835853] LustreError: 294375:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8356.839659] Lustre: server umount lustre-MDT0000 complete [ 8356.840358] Lustre: Skipped 1 previous similar message [ 8356.846620] LustreError: 294375:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 8358.987444] LustreError: 294355:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743505026 with bad export cookie 17624023686931069339 [ 8358.991569] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8358.991856] LustreError: 294355:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8359.174720] Lustre: server umount lustre-MDT0001 complete [ 8371.334080] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 8373.708816] Key type lgssc unregistered [ 8373.974579] LNet: 297533:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8375.015445] LNet: Removed LNI 192.168.206.117@tcp [ 8375.825604] Key type .llcrypt unregistered [ 8375.827031] Key type ._llcrypt unregistered [ 8396.349959] Key type ._llcrypt registered [ 8396.351069] Key type .llcrypt registered [ 8396.439198] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 8414.113388] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 8416.214223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8416.276662] alg: No test for adler32 (adler32-zlib) [ 8417.490602] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 8417.810915] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 8417.812940] LNet: Accept secure, port 988 [ 8419.583190] Key type lgssc registered [ 8420.329357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8426.978438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8432.216058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8436.887118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8441.854610] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8445.751499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8450.086740] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8454.297893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8454.352321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8454.359633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8455.538451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8455.560368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8455.603131] Lustre: lustre-MDT0000: new disk, initializing [ 8455.651684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8455.663134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8457.913028] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8464.215709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8464.290223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8464.356444] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8464.369345] Lustre: 302081:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8464.391355] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8464.393205] Lustre: Skipped 1 previous similar message [ 8464.432222] Lustre: lustre-MDT0001: new disk, initializing [ 8464.492890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8464.511831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8464.516313] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8466.817611] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8472.637380] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8476.434537] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8481.232358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8481.326594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8481.502015] Lustre: 303036:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8481.522150] Lustre: lustre-OST2710: new disk, initializing [ 8481.526387] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8481.586556] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8485.056450] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8491.034847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8491.040244] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8491.135108] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8491.223258] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8496.124473] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8496.191117] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8496.302882] Lustre: 303993:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8496.320589] Lustre: lustre-OST03e8: new disk, initializing [ 8496.323647] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8496.384746] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8499.045841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8499.050511] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8499.143648] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8500.031731] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8504.550534] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8509.350285] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8509.537371] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8511.964879] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8512.156795] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8514.649472] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8514.825561] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8517.373931] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8517.569181] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8520.672207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8520.676540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8520.683581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8524.770981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8524.777134] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8524.782608] Lustre: Skipped 2 previous similar messages [ 8526.198186] Lustre: server umount lustre-MDT0000 complete [ 8528.510953] LustreError: 301312:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743505196 with bad export cookie 12100071873341241411 [ 8528.517470] LustreError: 301312:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8528.518213] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8528.738723] Lustre: server umount lustre-MDT0001 complete [ 8541.579631] Lustre: server umount lustre-OST2710 complete [ 8554.197701] Lustre: server umount lustre-OST03e8 complete [ 8566.618903] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 8578.947662] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 8588.510735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8594.144222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8598.606426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8603.293697] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8607.264272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8607.317839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8607.482349] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8607.503661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8607.542219] Lustre: lustre-MDT0000: new disk, initializing [ 8607.591753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8607.602986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8609.700334] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8616.107067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8616.159962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8616.224202] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8616.235874] Lustre: 309796:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8616.258823] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8616.261338] Lustre: Skipped 1 previous similar message [ 8616.300283] Lustre: lustre-MDT0001: new disk, initializing [ 8616.359664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8616.381551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8616.384485] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8618.499881] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8624.009811] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8627.391730] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8631.457340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8631.526474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8631.661969] Lustre: 310750:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8634.718038] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8638.467609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8638.471825] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8638.535651] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8640.125767] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8642.724090] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8642.862086] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8645.152468] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8645.355966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8648.677058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8648.677456] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8648.680828] LustreError: Skipped 1 previous similar message [ 8648.687760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8648.690892] Lustre: Skipped 4 previous similar messages [ 8652.739171] Lustre: server umount lustre-OST0000 complete [ 8657.376187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8657.378322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8657.381707] Lustre: Skipped 1 previous similar message [ 8657.383492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8661.228666] Lustre: server umount lustre-MDT0000 complete [ 8663.808673] LustreError: 309029:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743505331 with bad export cookie 12100071873341243091 [ 8663.812385] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8663.814819] LustreError: 309029:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8664.021620] Lustre: server umount lustre-MDT0001 complete [ 8673.603100] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 07:02:19 (1743505339) [ 8675.850982] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8678.195347] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 07:02:23 (1743505343) [ 8680.162561] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 8685.568595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8687.823049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8690.395530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8692.728520] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8693.407500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8697.226504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8697.399414] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8697.414096] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8697.505507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8697.508316] Lustre: Skipped 1 previous similar message [ 8699.680422] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8704.597744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8704.647679] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8704.674257] Lustre: 315177:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8707.067678] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8711.277846] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8714.790467] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8719.438262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8719.501795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8719.616852] Lustre: Denying initial registration attempt from nid 192.168.206.117@tcp, specified as failover [ 8719.620699] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8719.625171] LustreError: 316075:0:(tgt_mount.c:2266:server_fill_super()) Unable to start targets: -99 [ 8719.628641] LustreError: 316075:0:(tgt_mount.c:1761:server_put_super()) no obd lustre-OST0000 [ 8719.631719] LustreError: 316075:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 8719.684372] Lustre: server umount lustre-OST0000 complete [ 8719.686487] LustreError: 316075:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8725.472101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8725.474837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8725.479034] Lustre: Skipped 1 previous similar message [ 8725.481089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8725.483054] Lustre: Skipped 2 previous similar messages [ 8729.634940] Lustre: server umount lustre-MDT0000 complete [ 8731.914149] LustreError: 314459:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743505399 with bad export cookie 12100071873341243973 [ 8731.917398] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8731.918885] LustreError: 314459:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8732.114276] Lustre: server umount lustre-MDT0001 complete [ 8745.520941] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 8748.107751] Key type lgssc unregistered [ 8748.370575] LNet: 317188:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8749.413536] LNet: Removed LNI 192.168.206.117@tcp [ 8750.105931] Key type .llcrypt unregistered [ 8750.107326] Key type ._llcrypt unregistered [ 8760.448849] Key type ._llcrypt registered [ 8760.450446] Key type .llcrypt registered [ 8760.528639] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 8761.597592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8761.668121] alg: No test for adler32 (adler32-zlib) [ 8762.648533] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 8762.810984] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 8762.813705] LNet: Accept secure, port 988 [ 8764.439168] Key type lgssc registered [ 8765.158400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8769.341366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8771.764940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8773.978418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8776.271101] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8776.932639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8794.087695] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 8800.396932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8800.410396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8801.562049] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8801.573454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8801.649111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8803.715445] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8808.366923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8808.416430] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8808.432272] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 8808.447428] Lustre: 320630:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8808.583648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8810.884432] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8814.686517] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8818.175669] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8822.297579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8822.363232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8822.453263] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8822.480974] Lustre: 321526:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8822.568786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8825.620597] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8829.682653] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8834.530779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8834.530847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8834.532935] LustreError: Skipped 1 previous similar message [ 8834.536281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8834.538202] Lustre: Skipped 1 previous similar message [ 8837.303633] Lustre: server umount lustre-OST0000 complete [ 8839.649339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8839.649823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8839.650900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8839.659514] Lustre: Skipped 2 previous similar messages [ 8844.769282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8844.772131] Lustre: Skipped 2 previous similar messages [ 8845.556882] Lustre: server umount lustre-MDT0000 complete [ 8847.593528] LustreError: 321551:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743505515 with bad export cookie 11547220477758504439 [ 8847.597061] LustreError: 321551:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8847.598260] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8847.769680] Lustre: server umount lustre-MDT0001 complete [ 8860.811970] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 8863.295896] Key type lgssc unregistered [ 8863.542434] LNet: 323137:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8864.613416] LNet: Removed LNI 192.168.206.117@tcp [ 8865.292282] Key type .llcrypt unregistered [ 8865.293198] Key type ._llcrypt unregistered [ 8891.526350] Key type ._llcrypt registered [ 8891.528220] Key type .llcrypt registered [ 8891.611115] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 8893.210566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8893.265440] alg: No test for adler32 (adler32-zlib) [ 8894.396820] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 8894.573692] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 8894.575438] LNet: Accept secure, port 988 [ 8896.207202] Key type lgssc registered [ 8897.044690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8902.452663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8902.468694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8903.747666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8906.126322] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8911.478054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8911.545344] Lustre: 325218:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8911.696993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8914.187986] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8918.568284] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8922.774547] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8927.584602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8927.696794] Lustre: 326101:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8927.783217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8931.009852] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8935.202923] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8942.053733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8942.055989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8942.059216] Lustre: Skipped 2 previous similar messages [ 8942.063056] Lustre: Skipped 2 previous similar messages [ 8942.405834] Lustre: server umount lustre-MDT0000 complete [ 8944.846368] LustreError: 324500:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743505612 with bad export cookie 715810826651275749 [ 8944.854993] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8947.168063] LustreError: 326212:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8947.168388] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8947.172737] LustreError: 326212:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 8947.181700] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8947.183675] Lustre: Skipped 1 previous similar message [ 8949.216100] LustreError: 326212:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8949.216270] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8949.221413] LustreError: 326212:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 8949.225059] Lustre: Skipped 1 previous similar message [ 8951.092565] Lustre: server umount lustre-MDT0001 complete [ 8955.152910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8960.310080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8969.695852] LustreError: 323836:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004dd4ffe7 x1828198088205568/t0(0) o250->MGC192.168.206.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 [ 8969.874835] LustreError: 326118:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8969.923481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8969.952594] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8972.233818] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8975.073888] LustreError: 327787:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8975.081961] LustreError: 327787:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 8977.691371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8977.743087] Lustre: 328460:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8977.896609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8979.946440] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.206.117@tcp (at 0@lo) [ 8980.321941] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8984.665472] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8986.702408] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8990.018160] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8992.369253] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8995.808382] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8995.811171] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8995.815617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8998.369315] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8998.371678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9000.418516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9000.421442] Lustre: Skipped 2 previous similar messages [ 9001.618246] Lustre: server umount lustre-OST0000 complete [ 9005.537508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9010.656646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9010.659761] Lustre: Skipped 3 previous similar messages [ 9018.335139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9018.417066] Lustre: server umount lustre-MDT0000 complete [ 9020.691470] LustreError: 324501:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743505688 with bad export cookie 715810826651282889 [ 9020.695065] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9020.695802] LustreError: 324501:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9020.876518] Lustre: server umount lustre-MDT0001 complete [ 9035.098436] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 9037.914483] Key type lgssc unregistered [ 9038.212729] LNet: 330190:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9039.269391] LNet: Removed LNI 192.168.206.117@tcp [ 9040.078483] Key type .llcrypt unregistered [ 9040.080430] Key type ._llcrypt unregistered [ 9070.521536] Key type ._llcrypt registered [ 9070.522685] Key type .llcrypt registered [ 9070.601620] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 9072.266493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9072.313872] alg: No test for adler32 (adler32-zlib) [ 9073.491591] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 9073.670086] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 9073.672051] LNet: Accept secure, port 988 [ 9075.327191] Key type lgssc registered [ 9076.233526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9081.589714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9081.605774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9082.896420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9085.277061] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9090.892220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9090.969723] Lustre: 332274:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9091.125276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9093.596110] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9098.267833] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9101.907453] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9104.014993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9104.016476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9104.021024] Lustre: Skipped 1 previous similar message [ 9109.472565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9109.474556] Lustre: Skipped 2 previous similar messages [ 9110.181356] Lustre: server umount lustre-MDT0000 complete [ 9112.432166] LustreError: 331556:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743505780 with bad export cookie 15177488334826305894 [ 9112.435212] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9112.436674] LustreError: 331556:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9112.629941] Lustre: server umount lustre-MDT0001 complete [ 9121.984649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9122.172411] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9122.193689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9122.321130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9124.865285] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9130.502766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9130.558378] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9130.575733] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9130.590471] Lustre: 334634:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9130.731542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9133.202571] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9137.548688] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9141.218067] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9146.217931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9146.305194] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9146.308067] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9146.313136] LustreError: 335577:0:(tgt_mount.c:2266:server_fill_super()) Unable to start targets: -2 [ 9146.316984] LustreError: 335577:0:(tgt_mount.c:1761:server_put_super()) no obd lustre-OST0000 [ 9146.320269] LustreError: 335577:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 9146.377595] Lustre: server umount lustre-OST0000 complete [ 9146.378791] LustreError: 335577:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9151.117698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9151.219765] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9151.255262] Lustre: 336036:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9151.353252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9154.427685] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9158.316556] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9159.723718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9163.085843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9163.147462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9163.209689] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9163.236581] Lustre: 336987:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9163.255536] Lustre: lustre-OST0001: new disk, initializing [ 9163.258885] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9163.309454] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9166.404818] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9168.414593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9168.417485] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9168.505619] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9170.655939] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9173.474675] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9173.477485] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9173.477920] LustreError: Skipped 1 previous similar message [ 9173.484694] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9173.487576] Lustre: Skipped 1 previous similar message [ 9174.799449] Lustre: server umount lustre-OST0001 complete [ 9178.594838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9178.597683] LustreError: 336651:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9178.598629] Lustre: Skipped 2 previous similar messages [ 9178.600259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9178.603057] LustreError: 336651:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 9183.716257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9183.716727] LustreError: 336043:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9183.718193] Lustre: Skipped 2 previous similar messages [ 9183.728111] LustreError: 336043:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 9188.836424] LustreError: 336043:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9188.843027] LustreError: 336043:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 9191.391156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9191.491038] Lustre: server umount lustre-OST0000 complete [ 9193.952984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9193.958199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9193.961266] Lustre: Skipped 3 previous similar messages [ 9208.287230] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9208.376733] Lustre: server umount lustre-MDT0000 complete [ 9209.312891] LustreError: 334663:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9209.317834] LustreError: 334663:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 9210.753716] LustreError: 333921:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743505878 with bad export cookie 15177488334826306447 [ 9210.756150] LustreError: 333921:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9210.756385] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9210.992991] Lustre: server umount lustre-MDT0001 complete [ 9226.266563] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 9229.187956] Key type lgssc unregistered [ 9229.539199] LNet: 338834:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9230.565569] LNet: Removed LNI 192.168.206.117@tcp [ 9231.412189] Key type .llcrypt unregistered [ 9231.413084] Key type ._llcrypt unregistered [ 9253.006494] Key type ._llcrypt registered [ 9253.007967] Key type .llcrypt registered [ 9253.079796] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 9270.418897] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 9272.494849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9272.524487] alg: No test for adler32 (adler32-zlib) [ 9273.662376] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 9273.902804] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 9273.904296] LNet: Accept secure, port 988 [ 9275.527179] Key type lgssc registered [ 9276.294584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9282.970550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9288.439333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9292.968286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9297.976338] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9302.056595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9302.104558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9302.114741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9303.282892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9303.307335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9303.343423] Lustre: lustre-MDT0000: new disk, initializing [ 9303.400344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9303.412205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9305.673160] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9312.327754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9312.395644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9312.473436] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9312.485946] Lustre: 343527:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9312.504818] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9312.506501] Lustre: Skipped 1 previous similar message [ 9312.542330] Lustre: lustre-MDT0001: new disk, initializing [ 9312.602423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9312.622378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9312.625921] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9314.804949] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9320.449537] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9324.254698] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9328.900602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9328.965953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9329.131275] Lustre: 344478:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9329.152466] Lustre: lustre-OST0000: new disk, initializing [ 9329.156444] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9329.207648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9332.727494] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9335.819044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9335.824221] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9335.901414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9338.571258] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9341.513889] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9341.685629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9344.249342] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9344.438552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9346.017314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9346.020845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9346.021859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9346.024490] Lustre: Skipped 1 previous similar message [ 9351.139494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9351.142058] Lustre: Skipped 2 previous similar messages [ 9351.686919] Lustre: server umount lustre-OST0000 complete [ 9356.257465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9356.258364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9356.260988] Lustre: Skipped 1 previous similar message [ 9356.263937] Lustre: Skipped 1 previous similar message [ 9360.253383] Lustre: server umount lustre-MDT0000 complete [ 9362.585067] LustreError: 345879:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743506030 with bad export cookie 4614060164503630188 [ 9362.586735] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9362.590945] LustreError: 345879:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9362.792353] Lustre: server umount lustre-MDT0001 complete [ 9371.589576] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 07:13:57 (1743506037) [ 9376.606943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9381.771110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9396.621807] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 9406.371844] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 9414.588789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9420.181964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9425.083094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9430.419949] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9434.692349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9434.749600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9434.930143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 9434.960082] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9435.014642] Lustre: lustre-MDT0000: new disk, initializing [ 9435.069809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9435.082524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9437.426463] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9444.717393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9444.791221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9444.867466] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 9444.882626] Lustre: 350600:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9444.911853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9444.913634] Lustre: Skipped 1 previous similar message [ 9444.958036] Lustre: lustre-MDT0001: new disk, initializing [ 9445.038189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9445.060722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9445.064764] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9447.611800] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9453.367968] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9456.783256] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9461.348622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9461.420746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9461.640776] Lustre: 351552:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9461.665433] Lustre: lustre-OST0000: new disk, initializing [ 9461.672650] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9461.747319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9465.462048] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9467.404065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9467.408174] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9467.507535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9471.501343] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9474.835813] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9475.059889] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9477.516556] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9477.696733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9482.724699] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9482.725228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9482.726940] LustreError: Skipped 2 previous similar messages [ 9482.728503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9482.732484] Lustre: Skipped 1 previous similar message [ 9482.736829] Lustre: Skipped 1 previous similar message [ 9485.013037] Lustre: server umount lustre-OST0000 complete [ 9487.841034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9487.841959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9487.849468] Lustre: Skipped 1 previous similar message [ 9493.703449] Lustre: server umount lustre-MDT0000 complete [ 9496.067477] LustreError: 349830:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743506163 with bad export cookie 4614060164503631070 [ 9496.071339] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9496.073435] LustreError: 349830:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9496.262234] Lustre: server umount lustre-MDT0001 complete [ 9504.745224] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 07:16:10 (1743506170) [ 9513.491014] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 07:16:19 (1743506179) [ 9519.190849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9519.447468] LustreError: 354165:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9519.456688] LustreError: 354165:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 9519.492480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9521.748150] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9524.705365] LustreError: 354165:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9527.496830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9527.553945] Lustre: 354863:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9530.344382] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9535.059806] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9538.377517] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9542.906278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9543.011119] Lustre: 355743:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9546.711033] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9550.887875] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9558.352573] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9560.257487] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9562.197666] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9563.881705] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9565.280796] Lustre: Failing over lustre-MDT0000 [ 9565.504741] Lustre: server umount lustre-MDT0000 complete [ 9567.714083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9567.715957] LustreError: 354161:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9567.717224] Lustre: Skipped 2 previous similar messages [ 9567.721993] LustreError: 354161:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 9572.833402] LustreError: 354165:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9572.837505] LustreError: 354165:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 9577.957469] LustreError: 356307:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9577.961429] LustreError: 356307:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 9581.397224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9581.459833] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9581.579855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9581.581980] Lustre: Skipped 2 previous similar messages [ 9581.613690] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9583.709418] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9583.755540] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9586.665686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.117@tcp (at 0@lo) [ 9586.675567] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9586.694139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9589.755221] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9591.634614] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9595.319448] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9596.791060] Lustre: Failing over lustre-MDT0000 [ 9596.897948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9596.898491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9596.900183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9596.900188] Lustre: Skipped 3 previous similar messages [ 9596.907722] Lustre: Skipped 2 previous similar messages [ 9596.972211] Lustre: server umount lustre-MDT0000 complete [ 9599.122702] LustreError: 356307:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9599.131094] LustreError: 356307:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [ 9612.718083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9612.794665] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9612.988891] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9614.477534] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9615.524022] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9618.407986] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.206.117@tcp (at 0@lo) [ 9618.410092] Lustre: Skipped 2 previous similar messages [ 9618.419305] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9618.438514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9621.723127] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9624.299556] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9633.763120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9633.763533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9633.764704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9633.764708] Lustre: Skipped 2 previous similar messages [ 9633.766796] Lustre: Skipped 1 previous similar message [ 9634.874438] Lustre: server umount lustre-MDT0000 complete [ 9637.246949] LustreError: 357660:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743506305 with bad export cookie 4614060164503633751 [ 9637.254200] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9637.462423] Lustre: server umount lustre-MDT0001 complete [ 9650.601425] Lustre: server umount lustre-OST0000 complete [ 9660.646816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9660.903494] LustreError: 359914:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9660.909084] LustreError: 359914:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 11 previous similar messages [ 9660.938659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9660.942064] Lustre: Skipped 1 previous similar message [ 9663.554705] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9669.716535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9669.773287] Lustre: 360610:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9672.560515] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9677.397974] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9681.179666] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9686.148069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9690.199833] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9694.654494] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9705.836114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9715.168543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9715.170682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9715.173718] Lustre: Skipped 1 previous similar message [ 9715.175343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9715.176760] Lustre: Skipped 2 previous similar messages [ 9720.325851] Lustre: server umount lustre-OST0000 complete [ 9726.434113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9726.437152] LustreError: Skipped 1 previous similar message [ 9737.183202] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9737.293115] Lustre: server umount lustre-MDT0000 complete [ 9740.059633] LustreError: 359893:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743506407 with bad export cookie 4614060164503634486 [ 9740.064315] LustreError: 359893:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9740.069981] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9740.318826] Lustre: server umount lustre-MDT0001 complete [ 9754.050838] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 9757.347503] Key type lgssc unregistered [ 9757.707840] LNet: 363113:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9758.761460] LNet: Removed LNI 192.168.206.117@tcp [ 9759.499749] Key type .llcrypt unregistered [ 9759.501468] Key type ._llcrypt unregistered [ 9786.675814] Key type ._llcrypt registered [ 9786.677329] Key type .llcrypt registered [ 9786.758445] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 9788.377044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9788.400932] alg: No test for adler32 (adler32-zlib) [ 9789.488345] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 9789.657516] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 9789.659778] LNet: Accept secure, port 988 [ 9791.303203] Key type lgssc registered [ 9792.107296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9797.199819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9797.214414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9798.471130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9800.865832] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9806.422921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9806.475167] Lustre: 365295:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9806.625900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9808.968364] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9813.412318] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9816.917612] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9821.391948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9821.497804] Lustre: 366179:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9821.591388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9823.671759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9825.116216] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9829.434239] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9842.148101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9842.151632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9842.155312] Lustre: Skipped 2 previous similar messages [ 9842.158167] Lustre: Skipped 1 previous similar message [ 9847.266099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9847.269571] Lustre: Skipped 2 previous similar messages [ 9847.811489] Lustre: server umount lustre-MDT0000 complete [ 9850.197342] LustreError: 364579:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743506518 with bad export cookie 14316106904553327662 [ 9850.202382] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9850.203155] LustreError: 364579:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9850.394355] Lustre: server umount lustre-MDT0001 complete [ 9863.250218] Lustre: server umount lustre-OST0000 complete [ 9871.936083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9872.180625] LustreError: 367983:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9872.185645] LustreError: 367983:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [ 9872.218572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9874.568847] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9877.472692] LustreError: 367983:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9880.578578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9880.636796] Lustre: 368680:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9880.796843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9883.573693] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9888.164160] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9892.564187] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9897.420078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9897.540125] Lustre: 369560:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9897.645558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9898.668490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9901.244047] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9905.562923] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9916.387052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9916.389273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9916.393217] Lustre: Skipped 1 previous similar message [ 9921.508714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9921.511839] Lustre: Skipped 4 previous similar messages [ 9922.114182] Lustre: server umount lustre-MDT0000 complete [ 9924.529963] LustreError: 367964:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743506592 with bad export cookie 14316106904553328775 [ 9924.534594] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9924.535908] LustreError: 367964:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9924.712559] Lustre: server umount lustre-MDT0001 complete [ 9937.558917] Lustre: server umount lustre-OST0000 complete [ 9947.297431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9947.579968] LustreError: 371539:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9947.626218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9950.157330] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9952.736518] LustreError: 371539:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9955.335598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9955.393133] Lustre: 372235:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9957.969914] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9963.036283] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9967.414990] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9972.218144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9972.338544] Lustre: 373119:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9973.486210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9975.958096] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9980.433675] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9986.018230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9986.021049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9986.023643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9986.026489] Lustre: Skipped 2 previous similar messages [ 9986.031885] Lustre: Skipped 1 previous similar message [ 9986.367013] Lustre: server umount lustre-OST0000 complete [ 9988.763330] Lustre: server umount lustre-MDT0000 complete [ 9991.021418] LustreError: 371519:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743506658 with bad export cookie 14316106904553329797 [ 9991.026610] LustreError: 371519:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9991.027529] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9997.304987] Lustre: server umount lustre-MDT0001 complete [10016.432337] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [10019.464470] Key type lgssc unregistered [10019.786628] LNet: 374716:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10020.837457] LNet: Removed LNI 192.168.206.117@tcp [10021.781706] Key type .llcrypt unregistered [10021.782787] Key type ._llcrypt unregistered [10054.000165] Key type ._llcrypt registered [10054.001180] Key type .llcrypt registered [10054.088101] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [10056.343840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10056.441042] alg: No test for adler32 (adler32-zlib) [10057.725216] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [10057.961344] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [10057.963292] LNet: Accept secure, port 988 [10059.615183] Key type lgssc registered [10060.572237] Lustre: Echo OBD driver; http://www.lustre.org/ [10065.656953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10065.669853] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10065.672661] LustreError: 376128:0:(osd_handler.c:8368:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10065.682743] LustreError: 376128:0:(obd_config.c:780:class_setup()) setup lustre-MDT0000-osd failed (-22) [10065.684814] LustreError: 376128:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10065.688398] LustreError: 376128:0:(tgt_mount.c:2234:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10065.692956] LustreError: 376128:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [10070.613295] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [10070.616365] LustreError: 376524:0:(osd_handler.c:8368:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10070.624902] LustreError: 376524:0:(obd_config.c:780:class_setup()) setup lustre-OST0000-osd failed (-22) [10070.629099] LustreError: 376524:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10070.632189] LustreError: 376524:0:(tgt_mount.c:2234:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10070.634784] LustreError: 376524:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [10090.163655] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [10092.927473] Key type lgssc unregistered [10093.211558] LNet: 377524:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10093.220440] LNet: Removed LNI 192.168.206.117@tcp [10093.928665] Key type .llcrypt unregistered [10093.929990] Key type ._llcrypt unregistered [10106.823296] Key type ._llcrypt registered [10106.824502] Key type .llcrypt registered [10106.899288] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [10124.749630] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [10127.069921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10127.096764] alg: No test for adler32 (adler32-zlib) [10128.450241] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [10128.867924] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [10128.871926] LNet: Accept secure, port 988 [10130.591179] Key type lgssc registered [10131.395257] Lustre: Echo OBD driver; http://www.lustre.org/ [10138.716340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10144.854956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10150.181400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10155.752431] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10160.300319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10160.350180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10160.359779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10161.531275] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10161.552399] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10161.591404] Lustre: lustre-MDT0000: new disk, initializing [10161.644230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10161.658510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10163.995016] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10170.592093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10170.687492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10170.771762] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10170.791322] Lustre: 381400:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [10170.821150] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10170.822769] Lustre: Skipped 1 previous similar message [10170.863889] Lustre: lustre-MDT0001: new disk, initializing [10170.945521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10170.977076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10170.981477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10173.587546] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10179.638707] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10183.127698] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10187.552475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10187.616943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10187.783294] Lustre: 382352:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [10187.804689] Lustre: lustre-OST0000: new disk, initializing [10187.808901] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10187.865834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10191.206802] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10193.426614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10193.432196] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10193.507617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10197.404734] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10200.313550] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10200.470103] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10203.102528] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10203.268587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10208.740455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10208.742438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10208.744589] LustreError: Skipped 1 previous similar message [10208.748792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10208.752280] Lustre: Skipped 1 previous similar message [10210.572403] Lustre: server umount lustre-OST0000 complete [10213.857252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10213.858097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10213.860758] Lustre: Skipped 2 previous similar messages [10213.863409] Lustre: Skipped 1 previous similar message [10218.976879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10218.979316] Lustre: Skipped 1 previous similar message [10224.097550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10224.100145] Lustre: Skipped 1 previous similar message [10227.167205] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10227.261892] Lustre: server umount lustre-MDT0000 complete [10229.552601] LustreError: 382376:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743506897 with bad export cookie 13485707022152385518 [10229.556362] LustreError: 382376:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [10229.556850] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10229.740438] Lustre: server umount lustre-MDT0001 complete [10238.519247] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10240.356490] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 07:28:26 (1743506906) [10245.411867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10245.697164] LustreError: 384675:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10245.702283] LustreError: 384675:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [10245.733826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10248.337878] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10251.234892] LustreError: 384675:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10253.751895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10253.816516] Lustre: 385374:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [10253.958133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10256.667189] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10261.238642] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10264.705748] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10269.210324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10269.306419] Lustre: 386253:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [10269.390536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10273.111115] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10277.485815] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10282.301987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10282.381077] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10282.497285] Lustre: 387146:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [10282.515659] Lustre: lustre-OST0001: new disk, initializing [10282.518643] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10286.611327] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10289.697176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10289.702275] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10289.782175] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10293.164986] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10299.884115] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10299.884195] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10299.886131] LustreError: Skipped 1 previous similar message [10299.891537] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10299.893165] Lustre: Skipped 1 previous similar message [10303.184620] Lustre: server umount lustre-OST0001 complete [10304.996840] LustreError: 386259:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10305.000711] LustreError: 386259:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [10310.116722] LustreError: 386259:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10310.124214] LustreError: 386259:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [10315.237381] LustreError: 386259:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10315.242509] LustreError: 386259:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [10320.357374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10320.359652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10320.361865] Lustre: Skipped 2 previous similar messages [10320.365577] Lustre: Skipped 2 previous similar messages [10323.724100] Lustre: server umount lustre-OST0000 complete [10330.593307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10330.593541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10330.602995] Lustre: Skipped 1 previous similar message [10332.542807] Lustre: server umount lustre-MDT0000 complete [10335.021704] LustreError: 384655:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743507002 with bad export cookie 13485707022152386386 [10335.027540] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10341.355390] Lustre: server umount lustre-MDT0001 complete [10355.927253] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [10358.768318] Key type lgssc unregistered [10359.088518] LNet: 389023:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10360.101579] LNet: Removed LNI 192.168.206.117@tcp [10361.045416] Key type .llcrypt unregistered [10361.046450] Key type ._llcrypt unregistered [10382.396428] Key type ._llcrypt registered [10382.397602] Key type .llcrypt registered [10382.471066] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [10399.246411] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [10401.803117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10401.835980] alg: No test for adler32 (adler32-zlib) [10403.141091] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [10403.503848] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [10403.507402] LNet: Accept secure, port 988 [10405.319297] Key type lgssc registered [10406.332906] Lustre: Echo OBD driver; http://www.lustre.org/ [10413.971097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10419.721050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10424.764875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10430.158284] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10434.540096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10434.607786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10434.616273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10435.805232] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10435.832057] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10435.876899] Lustre: lustre-MDT0000: new disk, initializing [10435.927548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10435.940598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10438.483581] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10445.249842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10445.319586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10445.415912] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10445.432974] Lustre: 393716:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [10445.456507] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10445.458311] Lustre: Skipped 1 previous similar message [10445.495774] Lustre: lustre-MDT0001: new disk, initializing [10445.563975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10445.591275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10445.596140] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10448.119387] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10453.660019] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10457.472189] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10462.105607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10462.171360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10462.349974] Lustre: 394666:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [10462.373222] Lustre: lustre-OST0000: new disk, initializing [10462.376506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10462.451292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10463.054556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10463.058542] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10463.197605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10466.029871] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10471.825077] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10474.762862] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10474.928773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10477.572576] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10477.790550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10483.683653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10483.684056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10483.685847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10483.687039] LustreError: Skipped 1 previous similar message [10483.693176] Lustre: Skipped 1 previous similar message [10485.110651] Lustre: server umount lustre-OST0000 complete [10488.801702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10488.803164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10488.806727] Lustre: Skipped 1 previous similar message [10488.809016] Lustre: Skipped 1 previous similar message [10493.676768] Lustre: server umount lustre-MDT0000 complete [10495.999761] LustreError: 392948:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743507163 with bad export cookie 16787164659457151656 [10496.003299] LustreError: 392948:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [10496.004342] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10496.217633] Lustre: server umount lustre-MDT0001 complete [10505.520322] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 07:32:51 (1743507171) [10511.434654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10516.663821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10516.881184] Lustre: lustre-MDT0000: new disk, initializing [10516.905884] LustreError: 397537:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10516.914282] LustreError: 397537:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [10516.952687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10519.297857] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10522.081362] LustreError: 397537:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10524.633418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10524.692111] Lustre: 398233:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [10524.831797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10527.330593] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10532.117553] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10536.042128] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10537.865699] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10537.866874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10537.872509] Lustre: Skipped 1 previous similar message [10543.072375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10543.074505] Lustre: Skipped 2 previous similar messages [10548.192451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10548.194511] Lustre: Skipped 1 previous similar message [10552.287165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10552.378185] Lustre: server umount lustre-MDT0000 complete [10554.336477] LustreError: 398242:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10554.344371] LustreError: 398242:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [10554.829858] LustreError: 397517:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743507222 with bad export cookie 16787164659457152538 [10554.835649] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10554.840943] LustreError: 397517:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [10555.039624] Lustre: server umount lustre-MDT0001 complete [10564.729825] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 07:33:50 (1743507230) [10569.657071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10569.908774] LustreError: 399791:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10569.949459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10572.379419] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10575.330030] LustreError: 399791:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10577.627053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10577.696446] Lustre: 400489:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [10580.250767] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10585.005806] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10588.629508] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10593.202537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10593.322543] Lustre: 401369:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [10593.413765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10593.416131] Lustre: Skipped 1 previous similar message [10596.795877] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10601.030671] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10607.050614] Lustre: Permanently deactivating lustre-OST0000 [10607.058162] Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client [10616.837017] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10618.903875] LustreError: 402560:0:(mgs_llog.c:1661:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002020 [10618.908330] LustreError: 402560:0:(mgs_llog.c:1715:mgs_replace_nids()) Only MGS is allowed to be started [10618.912123] LustreError: 402560:0:(mgs_handler.c:943:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10620.672946] LustreError: 402608:0:(mgs_llog.c:1715:mgs_replace_nids()) Only MGS is allowed to be started [10620.675164] LustreError: 402608:0:(mgs_handler.c:943:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10627.969796] Lustre: server umount lustre-OST0000 complete [10629.672477] LustreError: 402857:0:(mgs_llog.c:1715:mgs_replace_nids()) Only MGS is allowed to be started [10629.674820] LustreError: 402857:0:(mgs_handler.c:943:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10634.208330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10634.210605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10634.214744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10634.218202] Lustre: Skipped 2 previous similar messages [10640.934789] Lustre: server umount lustre-MDT0000 complete [10643.257183] LustreError: 399772:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743507311 with bad export cookie 16787164659457153077 [10643.261295] LustreError: 399772:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [10643.262645] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10653.780421] Lustre: server umount lustre-MDT0001 complete [10659.160505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10661.742945] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10663.404748] LustreError: 403994:0:(mgs_handler.c:943:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10667.144052] Lustre: 404334:0:(mgs_llog.c:1422:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.117@tcp [10668.599318] Lustre: server umount MGS complete [10676.153343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10676.405855] LustreError: 405058:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10676.416688] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10676.418634] Lustre: Skipped 1 previous similar message [10676.452534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10679.084977] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10684.694471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10684.753096] Lustre: 405755:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [10684.856879] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10687.412907] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10691.880300] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10695.871171] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10697.640503] Lustre: Permanently reactivating lustre-OST0000 [10697.648658] Lustre: Modifying parameter lustre-OST0000-osc.osc.active in log lustre-client [10697.650634] Lustre: Skipped 2 previous similar messages [10727.391139] LustreError: 406701:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10733.025305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10733.129251] Lustre: 407761:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [10736.717447] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10740.661958] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10751.968355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10751.971761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10751.976944] Lustre: Skipped 1 previous similar message [10751.979612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10751.982263] Lustre: Skipped 4 previous similar messages [10756.306956] Lustre: server umount lustre-OST0000 complete [10761.696104] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10761.699489] LustreError: Skipped 1 previous similar message [10761.701596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10761.707896] Lustre: Skipped 1 previous similar message [10764.922166] Lustre: server umount lustre-MDT0000 complete [10767.218618] LustreError: 406398:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743507435 with bad export cookie 16787164659457154022 [10767.221521] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10767.222901] LustreError: 406398:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [10767.441250] Lustre: server umount lustre-MDT0001 complete [10784.488795] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [10787.344867] Key type lgssc unregistered [10787.672678] LNet: 409357:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10788.709433] LNet: Removed LNI 192.168.206.117@tcp [10789.543801] Key type .llcrypt unregistered [10789.545722] Key type ._llcrypt unregistered [10802.391543] Key type ._llcrypt registered [10802.393281] Key type .llcrypt registered [10802.491082] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [10822.234941] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [10823.906834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10823.922395] alg: No test for adler32 (adler32-zlib) [10824.933826] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [10825.096750] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [10825.098669] LNet: Accept secure, port 988 [10826.727204] Key type lgssc registered [10827.550302] Lustre: Echo OBD driver; http://www.lustre.org/ [10834.705552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10840.311292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10845.414291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10850.460413] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10854.824643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10854.871966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10854.883567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10856.085389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [10856.108111] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10856.148733] Lustre: lustre-MDT0000: new disk, initializing [10856.214904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10856.231075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10858.636613] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10865.496631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10865.553924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10865.622136] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [10865.635455] Lustre: 413236:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [10865.657569] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10865.659177] Lustre: Skipped 1 previous similar message [10865.696485] Lustre: lustre-MDT0001: new disk, initializing [10865.753584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10865.775100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10865.777945] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10868.202721] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10874.152260] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10877.576293] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10882.564460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10882.636039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10882.813426] Lustre: 414185:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [10882.833874] Lustre: lustre-OST0000: new disk, initializing [10882.837552] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10882.894459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10886.366772] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10889.745166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10889.748650] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10889.848140] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10892.031619] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10894.830917] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10895.018927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10897.624941] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10897.829616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10899.938394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10899.940294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10899.941877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10899.943866] Lustre: Skipped 1 previous similar message [10905.059633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10905.061410] Lustre: Skipped 2 previous similar messages [10910.178969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10910.182099] Lustre: Skipped 1 previous similar message [10913.247114] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10913.335498] Lustre: server umount lustre-OST0000 complete [10916.832428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10916.835926] LustreError: Skipped 1 previous similar message [10916.837199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10916.841729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10920.417591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10921.653976] Lustre: server umount lustre-MDT0000 complete [10923.930435] LustreError: 413249:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743507591 with bad export cookie 5858680337948993566 [10923.935552] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10924.134731] Lustre: server umount lustre-MDT0001 complete [10934.303704] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10936.673197] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 07:40:02 (1743507602) [10941.373244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10941.657897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10943.886885] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10947.040577] LustreError: 416527:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10947.045548] LustreError: 416527:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [10949.014116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10949.068453] Lustre: 417207:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [10949.209084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10951.595685] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10956.057737] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10960.443159] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10965.298881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10965.404942] Lustre: 418090:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [10965.497869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10968.782603] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10973.016675] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10974.781465] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10977.195894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10981.856168] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10981.858518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10981.862538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10981.864099] Lustre: Skipped 2 previous similar messages [10984.931147] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10986.416348] Lustre: server umount lustre-OST0000 complete [10990.049098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10990.049312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10990.050125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10990.050130] Lustre: Skipped 2 previous similar messages [10990.057201] Lustre: Skipped 2 previous similar messages [10994.735957] Lustre: server umount lustre-MDT0000 complete [10995.169660] LustreError: 416509:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10995.178883] LustreError: 416509:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [10996.853899] LustreError: 416490:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743507664 with bad export cookie 5858680337948994448 [10996.858438] LustreError: 416490:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 5 previous similar messages [10996.859693] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10997.032526] Lustre: server umount lustre-MDT0001 complete [11012.275996] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [11015.082252] Key type lgssc unregistered [11015.398394] LNet: 419878:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11016.421530] LNet: Removed LNI 192.168.206.117@tcp [11017.183682] Key type .llcrypt unregistered [11017.184698] Key type ._llcrypt unregistered [11041.237090] Key type ._llcrypt registered [11041.238109] Key type .llcrypt registered [11041.320160] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [11041.579768] Key type .llcrypt unregistered [11041.581039] Key type ._llcrypt unregistered [11061.753650] Key type ._llcrypt registered [11061.756652] Key type .llcrypt registered [11061.844648] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [11063.407928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11063.425157] alg: No test for adler32 (adler32-zlib) [11064.537681] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [11064.769461] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [11064.771186] LNet: Accept secure, port 988 [11066.407162] Key type lgssc registered [11067.343578] Lustre: Echo OBD driver; http://www.lustre.org/ [11072.369550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11072.383957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11073.681774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11076.045212] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11081.280057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11081.382233] Lustre: 422728:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [11081.453473] LustreError: 422033:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11081.457848] LustreError: 422033:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [11081.470550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11083.553568] LustreError: 422032:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11083.560731] LustreError: 422032:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 2 previous similar messages [11083.562101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11084.726649] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11088.865391] LustreError: 422032:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11088.871426] LustreError: 422032:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [11089.319460] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11093.987692] LustreError: 422028:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11093.991839] LustreError: 422028:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [11094.139965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11094.202944] Lustre: 423621:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [11094.387718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11096.726752] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11109.344338] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11109.349202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11109.356064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11109.861961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11109.862149] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11109.867630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11112.122911] Lustre: server umount lustre-OST0000 complete [11114.976806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11114.977620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11114.978122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11114.980581] Lustre: Skipped 1 previous similar message [11120.096738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11120.098481] Lustre: Skipped 2 previous similar messages [11120.374664] Lustre: server umount lustre-MDT0000 complete [11122.632338] LustreError: 422013:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743507790 with bad export cookie 3808996873334103837 [11122.635146] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11122.635509] LustreError: 422013:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [11122.843861] Lustre: server umount lustre-MDT0001 complete [11136.524689] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [11139.384544] Key type lgssc unregistered [11139.688551] LNet: 425019:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11140.709533] LNet: Removed LNI 192.168.206.117@tcp [11141.413111] Key type .llcrypt unregistered [11141.414929] Key type ._llcrypt unregistered [11166.601072] Key type ._llcrypt registered [11166.602144] Key type .llcrypt registered [11166.680450] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [11168.070399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11168.084246] alg: No test for adler32 (adler32-zlib) [11169.112682] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [11169.271951] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [11169.273818] LNet: Accept secure, port 988 [11170.895147] Key type lgssc registered [11171.606898] Lustre: Echo OBD driver; http://www.lustre.org/ [11176.323623] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11176.333075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11193.823554] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11199.967291] LustreError: 426379:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.117@tcp: failed processing log, type 4: rc = -110 [11229.663394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11232.948288] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11236.794696] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11241.089462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11241.359318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11241.392506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11243.602533] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11248.610665] LustreError: 426405:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11248.615202] LustreError: 426405:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 5 previous similar messages [11248.722353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11248.777053] Lustre: 428002:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [11248.916428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11251.128705] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11255.783560] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11259.444163] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11264.484838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11264.488347] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11264.488745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11264.492297] Lustre: Skipped 1 previous similar message [11269.605357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11269.607472] Lustre: Skipped 1 previous similar message [11269.905333] Lustre: server umount lustre-OST0000 complete [11273.186743] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11273.190441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11273.196560] Lustre: Skipped 1 previous similar message [11273.199318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11274.105911] Lustre: server umount lustre-MDT0000 complete [11274.722558] LustreError: 427301:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11274.722855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11276.283707] LustreError: 427328:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743507944 with bad export cookie 8671045225304344777 [11276.286244] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11276.286803] LustreError: 427328:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [11276.479449] Lustre: server umount lustre-MDT0001 complete [11290.452949] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [11293.329430] Key type lgssc unregistered [11293.615711] LNet: 429592:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11294.629519] LNet: Removed LNI 192.168.206.117@tcp [11295.343119] Key type .llcrypt unregistered [11295.344407] Key type ._llcrypt unregistered [11319.693908] Key type ._llcrypt registered [11319.695295] Key type .llcrypt registered [11319.768412] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [11321.279110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11321.314626] alg: No test for adler32 (adler32-zlib) [11322.355423] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [11322.508545] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [11322.510367] LNet: Accept secure, port 988 [11324.135219] Key type lgssc registered [11324.932723] Lustre: Echo OBD driver; http://www.lustre.org/ [11329.872241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11329.888087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11331.199408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11333.754395] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11339.114054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11339.172633] Lustre: 431674:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [11339.324298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11341.567394] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11345.900732] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11349.415862] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11354.305896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11354.411636] Lustre: 432557:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [11354.488474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11356.530664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11357.879739] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11362.094190] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11369.623249] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.17@tcp (stopping) [11369.956054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11369.959210] Lustre: Skipped 2 previous similar messages [11369.960632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11371.889454] Lustre: server umount lustre-MDT0000 complete [11373.622099] LustreError: 430957:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.17@tcp arrived at 1743508041 with bad export cookie 16420858820106978416 [11373.626022] LustreError: 430957:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [11374.911555] LustreError: 430958:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743508042 with bad export cookie 16420858820106977527 [11374.916285] LustreError: 430958:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [11374.916534] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11375.072501] LustreError: 430978:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11375.073096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11375.073324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11375.076543] LustreError: 430978:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [11375.080646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11375.085717] Lustre: Skipped 3 previous similar messages [11379.104085] LustreError: 430978:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11379.104545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11379.108039] LustreError: 430978:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [11381.055873] Lustre: server umount lustre-OST0000 complete [11384.224881] LustreError: 430978:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11384.229281] LustreError: 430978:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [11385.029631] Lustre: server umount lustre-MDT0001 complete [11398.572929] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [11401.123860] Key type lgssc unregistered [11401.421704] LNet: 434307:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11402.469585] LNet: Removed LNI 192.168.206.117@tcp [11403.266103] Key type .llcrypt unregistered [11403.267164] Key type ._llcrypt unregistered [11427.951829] Key type ._llcrypt registered [11427.953581] Key type .llcrypt registered [11428.043404] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [11430.516024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11430.543905] alg: No test for adler32 (adler32-zlib) [11431.919725] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [11432.119991] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [11432.122271] LNet: Accept secure, port 988 [11433.775193] Key type lgssc registered [11434.701585] Lustre: Echo OBD driver; http://www.lustre.org/ [11439.333101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11439.344926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11440.599404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11443.023618] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11448.453369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11448.505278] Lustre: 436390:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [11448.656643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11450.937926] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11455.261515] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11458.230764] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11462.196302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11462.311726] Lustre: 437275:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [11462.404284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11465.517470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11465.617246] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11469.894050] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11475.940098] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11475.944170] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11475.945648] Lustre: Skipped 1 previous similar message [11477.131624] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.17@tcp (stopping) [11479.837147] Lustre: server umount lustre-MDT0001 complete [11481.057114] LustreError: 435693:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11481.064284] LustreError: 435693:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [11486.176952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11486.177758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11486.177834] LustreError: 435693:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11486.187118] Lustre: Skipped 1 previous similar message [11486.189098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11489.844615] Lustre: server umount lustre-OST0000 complete [11491.298519] LustreError: 436597:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11492.340975] Lustre: server umount lustre-MDT0000 complete [11508.147857] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [11511.006272] Key type lgssc unregistered [11511.311076] LNet: 439024:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11512.357435] LNet: Removed LNI 192.168.206.117@tcp [11513.049921] Key type .llcrypt unregistered [11513.050941] Key type ._llcrypt unregistered [11532.940409] Key type ._llcrypt registered [11532.942640] Key type .llcrypt registered [11533.043689] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [11552.931246] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [11555.056667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11555.087835] alg: No test for adler32 (adler32-zlib) [11556.188715] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [11556.420553] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [11556.422682] LNet: Accept secure, port 988 [11558.063134] Key type lgssc registered [11558.813116] Lustre: Echo OBD driver; http://www.lustre.org/ [11565.109958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11570.219900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11575.391988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11581.249887] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11595.355820] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [11604.077443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11607.951271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11608.003430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11608.011680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11609.193561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [11609.217105] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11609.258386] Lustre: lustre-MDT0000: new disk, initializing [11609.309248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11609.321225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11611.589257] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11617.629086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11621.745355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11621.839678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11622.007091] Lustre: 443971:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [11622.031218] Lustre: lustre-OST0000: new disk, initializing [11622.035155] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11622.037312] Lustre: Skipped 1 previous similar message [11622.085325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11625.538881] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11628.080683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11628.083523] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11628.097520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11631.048183] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11637.917733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11642.028262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11642.109693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11642.201234] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [11642.231325] Lustre: 445190:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [11642.256259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11642.296891] Lustre: lustre-MDT0001: new disk, initializing [11642.417353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11642.445316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11642.448513] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11644.802026] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11650.532178] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11650.694945] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11653.135340] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11653.299090] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11656.161025] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11656.161854] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11656.165368] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11656.169261] Lustre: Skipped 1 previous similar message [11656.172383] Lustre: Skipped 1 previous similar message [11658.387133] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.17@tcp (stopping) [11661.282212] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11661.284442] Lustre: Skipped 1 previous similar message [11662.238785] Lustre: server umount lustre-MDT0001 complete [11671.523120] LustreError: 442999:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11671.530677] LustreError: 442999:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 5 previous similar messages [11676.642895] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11676.643440] LustreError: 443006:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11676.646858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11681.761823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11681.761882] LustreError: 443006:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11682.805232] Lustre: server umount lustre-OST0000 complete [11691.332973] Lustre: server umount lustre-MDT0000 complete [11707.430489] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [11710.500266] Key type lgssc unregistered [11710.870987] LNet: 447558:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11711.909459] LNet: Removed LNI 192.168.206.117@tcp [11712.613441] Key type .llcrypt unregistered [11712.614876] Key type ._llcrypt unregistered [11753.933455] Key type ._llcrypt registered [11753.935044] Key type .llcrypt registered [11754.012338] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [11754.261539] Key type .llcrypt unregistered [11754.263450] Key type ._llcrypt unregistered [11774.133751] Key type ._llcrypt registered [11774.134890] Key type .llcrypt registered [11774.215157] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [11775.789803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11775.855306] alg: No test for adler32 (adler32-zlib) [11776.832538] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [11776.979970] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [11776.981876] LNet: Accept secure, port 988 [11778.615166] Key type lgssc registered [11779.364915] Lustre: Echo OBD driver; http://www.lustre.org/ [11785.832870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11791.163602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11795.949857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11800.067237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11800.126389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11800.134941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11801.324133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [11801.346501] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11801.384853] Lustre: lustre-MDT0000: new disk, initializing [11801.435097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11801.448115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11803.680659] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11810.219810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11810.299364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11810.376534] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [11810.394330] Lustre: 451648:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [11810.419464] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11810.421054] Lustre: Skipped 1 previous similar message [11810.460348] Lustre: lustre-MDT0001: new disk, initializing [11810.543562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11810.569485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11810.575360] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11812.850913] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11818.739848] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11822.579963] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11826.935199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11826.993083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11827.140239] Lustre: 452600:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [11827.162753] Lustre: lustre-OST0000: new disk, initializing [11827.165852] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11827.227610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11830.627416] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11836.025369] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11836.420221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11836.423194] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11836.488954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11841.512918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11841.515347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11841.515796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11841.516600] Lustre: Skipped 1 previous similar message [11846.581802] Lustre: server umount lustre-MDT0000 complete [11848.742827] LustreError: 450879:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743508516 with bad export cookie 13720800869890529071 [11848.748186] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11848.926982] Lustre: server umount lustre-MDT0001 complete [11860.301205] Lustre: server umount lustre-OST0000 complete [11872.152258] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 07:55:37 (1743508537) [11873.285152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11877.568030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11877.848263] LustreError: 454541:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11877.853461] LustreError: 454541:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 4 previous similar messages [11877.890484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11880.002955] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11882.977455] LustreError: 454541:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11884.934417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11884.986867] Lustre: 455238:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [11885.111172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11887.408459] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11891.780740] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11895.418085] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11899.678527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11899.744437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11899.839914] Lustre: Found index 0 for lustre-OST0000, updating log [11899.864609] Lustre: 456119:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [11899.944273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11903.007252] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11904.043106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11906.796542] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11914.210574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11914.211528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11914.214283] LustreError: Skipped 1 previous similar message [11914.215278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11914.217976] Lustre: Skipped 2 previous similar messages [11914.219300] Lustre: Skipped 2 previous similar messages [11915.235293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11915.236982] Lustre: Skipped 2 previous similar messages [11916.400366] Lustre: server umount lustre-OST0000 complete [11917.914373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11920.353080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11920.353912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11920.356397] Lustre: Skipped 1 previous similar message [11925.089430] Lustre: server umount lustre-MDT0000 complete [11926.496254] LustreError: 454541:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11926.500987] LustreError: 454541:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 1 previous similar message [11927.117353] LustreError: 454520:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743508594 with bad export cookie 13720800869890530464 [11927.120466] LustreError: 454520:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 7 previous similar messages [11927.120965] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11927.301485] Lustre: server umount lustre-MDT0001 complete [11937.283444] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 07:56:42 (1743508602) [11938.387820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11939.011607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11939.603431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11940.142855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11940.691539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11941.259081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11941.820972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11942.387414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11942.965667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11943.556764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11944.158222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11944.779975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11945.392449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11945.999205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11946.600502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11947.245093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11947.834668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11948.371894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11948.911074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11949.477680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11950.062667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11950.663785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11951.265386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11951.851042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11952.470452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11953.074298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11953.672065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11954.225403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11954.769635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11955.314133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11955.847230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11956.449416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11957.010263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11957.610550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11958.212457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11958.743630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11959.314738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11959.910783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11960.522971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11961.104221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11961.746849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11962.363197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11962.971796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11963.538617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11964.120037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11964.701599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11965.240437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11965.738357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11966.272610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11966.796527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11967.320537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11967.908807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11968.522699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11969.127398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11969.693402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11970.264268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11970.807794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11971.411433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11971.945642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11972.469241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11973.040473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11973.601803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11974.139746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11974.658476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11975.237086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11975.827934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11976.408265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11976.986147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11977.585838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11978.173803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11978.730274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11979.280337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11979.849364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11980.395063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11980.932796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11981.501287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11982.065759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11982.631658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11983.217069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11983.751465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11984.345829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11984.949031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11985.516855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11986.067421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11986.595189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11987.152476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11987.711278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11988.218757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11988.729869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11989.260535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11989.781240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11990.309433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11990.867307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11991.423947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11992.004589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11992.579631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11993.192975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11993.780203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11994.374723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11994.899891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11995.432267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11996.002419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11996.522630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11997.074568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11997.622371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11998.197307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11998.800824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12000.554723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12009.828076] Lustre: DEBUG MARKER: == conf-sanity test 74: Test per-device adaptive timeout parameters ========================================================== 07:57:55 (1743508675) [12014.373339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12014.610365] LustreError: 463828:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12014.647484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12016.796080] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [12019.680405] LustreError: 463828:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12021.431265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12021.486150] Lustre: 464524:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [12023.609180] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [12027.877686] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12031.960562] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12036.486212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12036.564495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12036.665446] Lustre: Found index 0 for lustre-OST0000, updating log [12036.695226] Lustre: 465405:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [12040.051715] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [12041.900752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [12043.813245] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12054.778671] loop0: detected capacity change from 0 to 204800000 [12055.056977] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12055.060594] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12055.064577] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12055.068687] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12055.097588] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12055.127659] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12055.154468] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12055.197077] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12055.235437] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12055.279098] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12059.909305] loop0: detected capacity change from 0 to 204800000 [12060.159312] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12060.165281] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12060.171658] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12060.259900] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12063.789182] loop0: detected capacity change from 0 to 204800000 [12067.476181] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12067.540671] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12067.620291] Lustre: 467079:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for fs15246-sptlrpc, check MGS config [12067.637108] Lustre: fs15246-OST0000: new disk, initializing [12067.639947] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [12067.672292] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12067.675857] Lustre: Skipped 2 previous similar messages [12070.969130] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [12076.579289] loop1: detected capacity change from 0 to 204800000 [12080.277292] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12080.358515] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12080.425066] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall in log fs15246-MDT0000 [12080.456155] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [12080.495981] Lustre: fs15246-MDT0000: new disk, initializing [12080.632050] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12080.780970] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12081.824249] Lustre: 468143:0:(client.c:1515:after_reply()) @@@ resending request on EINPROGRESS req@00000000ad390376 x1828201110283392/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1743508765 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 [12083.143528] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [12090.913877] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12090.916453] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12090.923931] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12104.680975] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12104.685209] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12104.687348] Lustre: Skipped 3 previous similar messages [12106.214855] Lustre: server umount fs15246-MDT0000 complete [12109.803134] LustreError: 465440:0:(ldlm_lib.c:1095:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12111.734128] Lustre: server umount fs15246-OST0000 complete [12119.008831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12119.010850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12119.017036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12132.319147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12132.422806] Lustre: server umount lustre-OST0000 complete [12137.441153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12137.443246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12137.444125] Lustre: Skipped 2 previous similar messages [12137.446804] Lustre: Skipped 7 previous similar messages [12140.664896] Lustre: server umount lustre-MDT0000 complete [12142.756195] LustreError: 463809:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743508810 with bad export cookie 13720800869890531542 [12142.759888] LustreError: 463809:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [12142.760560] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12142.959340] Lustre: server umount lustre-MDT0001 complete [12185.966449] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [12188.950828] Key type lgssc unregistered [12189.272720] LNet: 471219:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12190.309445] LNet: Removed LNI 192.168.206.117@tcp [12191.005507] Key type .llcrypt unregistered [12191.006539] Key type ._llcrypt unregistered [12214.484085] Key type ._llcrypt registered [12214.485068] Key type .llcrypt registered [12214.559668] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [12216.098247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12216.137795] alg: No test for adler32 (adler32-zlib) [12217.216520] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [12217.371563] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [12217.374727] LNet: Accept secure, port 988 [12219.007130] Key type lgssc registered [12219.707655] Lustre: Echo OBD driver; http://www.lustre.org/ [12224.444042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12227.672464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12231.186334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12234.489785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12243.933470] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [12253.029327] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [12260.600197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12265.469336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12270.146783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12275.755183] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12279.924559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12279.973257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12279.981890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12281.165897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [12281.190857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12281.236023] Lustre: lustre-MDT0000: new disk, initializing [12281.298055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12281.310560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12283.685532] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [12290.545388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12290.624425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12290.704421] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [12290.721237] Lustre: 476395:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [12290.743511] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12290.745155] Lustre: Skipped 1 previous similar message [12290.787384] Lustre: lustre-MDT0001: new disk, initializing [12290.863934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12290.886060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12290.889850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12293.169238] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [12299.212021] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12302.665336] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12306.606588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12306.667364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12306.804655] Lustre: 477347:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.117@tcp: no remote llog for lustre-sptlrpc, check MGS config [12306.823520] Lustre: lustre-OST0000: new disk, initializing [12306.826556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12306.876523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12310.043482] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [12315.151546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12315.156938] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12315.245791] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12315.406845] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12317.795560] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12317.980380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12320.134427] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12320.349512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12325.347403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12325.348386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12325.350697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12325.355644] Lustre: Skipped 1 previous similar message [12325.359652] Lustre: Skipped 1 previous similar message [12327.643079] Lustre: server umount lustre-OST0000 complete [12330.465630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12330.466502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12330.468604] Lustre: Skipped 1 previous similar message [12330.473356] Lustre: Skipped 1 previous similar message [12335.585187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12335.586833] Lustre: Skipped 1 previous similar message [12340.704546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12340.706162] Lustre: Skipped 1 previous similar message [12344.287149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12344.375972] Lustre: server umount lustre-MDT0000 complete [12346.623840] LustreError: 476409:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743509014 with bad export cookie 15810736006457822090 [12346.628636] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12346.809780] Lustre: server umount lustre-MDT0001 complete [12364.823871] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [12374.500347] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [12382.615051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12388.783188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12393.396882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12398.655848] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12400.518103] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11817 sec ========= 08:04:26 (1743509066) [12402.819144] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:04:28 (1743509068) === [12407.725096] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:04:33 (1743509073) === [12439.086505] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [12441.445232] Key type lgssc unregistered [12441.705356] LNet: 483805:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12442.725483] LNet: Removed LNI 192.168.206.117@tcp [12443.381808] Key type .llcrypt unregistered [12443.382796] Key type ._llcrypt unregistered