[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 696330041 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.003202] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010161] LSM: Security Framework initializing [ 0.011054] Yama: becoming mindful. [ 0.012040] SELinux: Initializing. [ 0.013064] *** VALIDATE selinux *** [ 0.022056] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027546] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028183] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029126] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030147] *** VALIDATE tmpfs *** [ 0.031682] *** VALIDATE proc *** [ 0.032323] *** VALIDATE cgroup *** [ 0.033016] *** VALIDATE cgroup2 *** [ 0.034348] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036196] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037018] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042152] debug: unmapping init [mem 0xffffffff8ec59000-0xffffffff8ec60fff] [ 0.045225] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046771] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047029] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049014] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052020] ... fixed-purpose events: 3 [ 0.053027] ... event mask: 000000070000000f [ 0.054408] rcu: Hierarchical SRCU implementation. [ 0.057234] smp: Bringing up secondary CPUs ... [ 0.058946] x86: Booting SMP configuration: [ 0.059051] .... node #0, CPUs: #1 #2 #3 [ 0.217162] smp: Brought up 1 node, 4 CPUs [ 0.219091] smpboot: Max logical packages: 1 [ 0.220012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.237000] node 0 deferred pages initialised in 14ms [ 0.238240] devtmpfs: initialized [ 0.239000] x86/mm: Memory block size: 128MB [ 0.239000] gcov: version magic: 0x41383552 [ 0.246293] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.247178] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.248000] pinctrl core: initialized pinctrl subsystem [ 0.252275] [ 0.252944] ************************************************************* [ 0.255012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.258011] ** ** [ 0.261010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.264012] ** ** [ 0.267011] ** This means that this kernel is built to expose internal ** [ 0.269010] ** IOMMU data structures, which may compromise security on ** [ 0.272009] ** your system. ** [ 0.274009] ** ** [ 0.276007] ** If you see this message and you are not debugging the ** [ 0.279010] ** kernel, report this immediately to your vendor! ** [ 0.282012] ** ** [ 0.288014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.292011] ************************************************************* [ 0.296398] NET: Registered protocol family 16 [ 0.299647] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.302055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.305051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.309056] cpuidle: using governor menu [ 0.312105] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.315580] PCI: Using configuration type 1 for base access [ 0.318149] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.329065] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.332061] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.334371] cryptd: max_cpu_qlen set to 1000 [ 0.337319] ACPI: Added _OSI(Module Device) [ 0.339011] ACPI: Added _OSI(Processor Device) [ 0.341072] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.343014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.350587] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.360347] ACPI: Interpreter enabled [ 0.362066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.364014] ACPI: Using IOAPIC for interrupt routing [ 0.367201] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.370464] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.386190] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.389043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.393022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.398145] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.404100] acpiphp: Slot [2] registered [ 0.406386] acpiphp: Slot [5] registered [ 0.409320] acpiphp: Slot [6] registered [ 0.411145] acpiphp: Slot [7] registered [ 0.413154] acpiphp: Slot [8] registered [ 0.415209] acpiphp: Slot [9] registered [ 0.417210] acpiphp: Slot [10] registered [ 0.419165] acpiphp: Slot [3] registered [ 0.421327] acpiphp: Slot [4] registered [ 0.423232] acpiphp: Slot [11] registered [ 0.425163] acpiphp: Slot [12] registered [ 0.427277] acpiphp: Slot [13] registered [ 0.430148] acpiphp: Slot [14] registered [ 0.432125] acpiphp: Slot [15] registered [ 0.434145] acpiphp: Slot [16] registered [ 0.435086] acpiphp: Slot [17] registered [ 0.437174] acpiphp: Slot [18] registered [ 0.439184] acpiphp: Slot [19] registered [ 0.441100] acpiphp: Slot [20] registered [ 0.443113] acpiphp: Slot [21] registered [ 0.445137] acpiphp: Slot [22] registered [ 0.447142] acpiphp: Slot [23] registered [ 0.449126] acpiphp: Slot [24] registered [ 0.451135] acpiphp: Slot [25] registered [ 0.453104] acpiphp: Slot [26] registered [ 0.455133] acpiphp: Slot [27] registered [ 0.457295] acpiphp: Slot [28] registered [ 0.460117] acpiphp: Slot [29] registered [ 0.462254] acpiphp: Slot [30] registered [ 0.464123] acpiphp: Slot [31] registered [ 0.466099] PCI host bridge to bus 0000:00 [ 0.468023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.471024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.474040] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.477021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.480022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.484029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.486363] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.490548] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.494000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.507018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.514060] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.517083] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.519013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.521009] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.523769] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.526954] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.530040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.532786] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.538014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.550023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.555010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.562921] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.606016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.619028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.647018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.661000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.674015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.694020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.748019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.767000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.780016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.789017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.825017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.842000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.873032] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.878022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.918017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.938054] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.952016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.968020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.012017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.017000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.035015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.050021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.079018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.101000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.104536] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.107518] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.110480] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.113224] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.119053] iommu: Default domain type: Passthrough [ 1.120534] SCSI subsystem initialized [ 1.121155] ACPI: bus type USB registered [ 1.122100] usbcore: registered new interface driver usbfs [ 1.124049] usbcore: registered new interface driver hub [ 1.126070] usbcore: registered new device driver usb [ 1.128151] pps_core: LinuxPPS API ver. 1 registered [ 1.130009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.134080] PTP clock support registered [ 1.136346] EDAC MC: Ver: 3.0.0 [ 1.137000] PCI: Using ACPI for IRQ routing [ 1.138000] NetLabel: Initializing [ 1.140008] NetLabel: domain hash size = 128 [ 1.142007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.144071] NetLabel: unlabeled traffic allowed by default [ 1.147323] vgaarb: loaded [ 1.149043] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.151014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.156317] clocksource: Switched to clocksource kvm-clock [ 1.298583] VFS: Disk quotas dquot_6.6.0 [ 1.300215] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.303242] *** VALIDATE ramfs *** [ 1.304597] *** VALIDATE hugetlbfs *** [ 1.306321] pnp: PnP ACPI init [ 1.308986] pnp: PnP ACPI: found 6 devices [ 1.328366] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.331612] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.334282] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.336551] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.339507] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.342294] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.345334] NET: Registered protocol family 2 [ 1.348333] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.354867] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.359316] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.365865] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.369859] TCP: Hash tables configured (established 65536 bind 65536) [ 1.373206] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.376805] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.379944] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.383583] NET: Registered protocol family 1 [ 1.386673] RPC: Registered named UNIX socket transport module. [ 1.389646] RPC: Registered udp transport module. [ 1.391569] RPC: Registered tcp transport module. [ 1.393455] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.396077] NET: Registered protocol family 44 [ 1.397753] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.400451] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.403534] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.406570] PCI: CLS 0 bytes, default 64 [ 1.409338] Unpacking initramfs... [ 3.430942] debug: unmapping init [mem 0xffff8dd6fcc54000-0xffff8dd6fffbffff] [ 3.438749] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.441589] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.448038] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.094264] Initialise system trusted keyrings [ 4.096297] Key type blacklist registered [ 4.100970] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.122412] zbud: loaded [ 4.126775] *** VALIDATE nfs *** [ 4.128024] *** VALIDATE nfs4 *** [ 4.130024] pstore: using deflate compression [ 4.134323] Platform Keyring initialized [ 4.269518] NET: Registered protocol family 38 [ 4.271528] Key type asymmetric registered [ 4.273073] Asymmetric key parser 'x509' registered [ 4.275016] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.279396] io scheduler mq-deadline registered [ 4.285458] io scheduler kyber registered [ 4.293424] io scheduler bfq registered [ 4.304785] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.308290] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.310761] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.313889] ACPI: Power Button [PWRF] [ 4.319970] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.328645] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.344199] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.358553] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.379785] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.412524] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.443957] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.450965] Non-volatile memory driver v1.3 [ 4.453306] Linux agpgart interface v0.103 [ 4.490442] virtio_blk virtio1: [vda] 136000 512-byte logical blocks (69.6 MB/66.4 MiB) [ 4.493391] vda: detected capacity change from 0 to 69632000 [ 4.509385] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.512620] vdb: detected capacity change from 0 to 1073741824 [ 4.556081] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.563479] vdc: detected capacity change from 0 to 2621440000 [ 4.582031] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.587181] vdd: detected capacity change from 0 to 2621440000 [ 4.622219] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.625142] vde: detected capacity change from 0 to 4294967296 [ 4.651795] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.658752] vdf: detected capacity change from 0 to 4294967296 [ 4.753916] libphy: Fixed MDIO Bus: probed [ 4.763858] usbcore: registered new interface driver usbserial_generic [ 4.766774] usbserial: USB Serial support registered for generic [ 4.769377] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.776550] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.785147] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.789741] mousedev: PS/2 mouse device common for all mice [ 4.795967] rtc_cmos 00:05: RTC can wake from S4 [ 4.802444] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.804425] rtc_cmos 00:05: registered as rtc0 [ 4.808993] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.812131] intel_pstate: CPU model not supported [ 4.815903] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.818401] hid: raw HID events driver (C) Jiri Kosina [ 4.823616] usbcore: registered new interface driver usbhid [ 4.826013] usbhid: USB HID core driver [ 4.827621] drop_monitor: Initializing network drop monitor service [ 4.830456] Initializing XFRM netlink socket [ 4.832540] NET: Registered protocol family 10 [ 4.836552] Segment Routing with IPv6 [ 4.838296] NET: Registered protocol family 17 [ 4.839138] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.844179] mpls_gso: MPLS GSO support [ 4.858151] RAS: Correctable Errors collector initialized. [ 4.861386] AVX version of gcm_enc/dec engaged. [ 4.863823] AES CTR mode by8 optimization enabled [ 4.994845] sched_clock: Marking stable (4994763932, 0)->(6086599478, -1091835546) [ 4.998936] registered taskstats version 1 [ 5.001248] Loading compiled-in X.509 certificates [ 5.003124] zswap: loaded using pool lzo/zbud [ 5.031666] Key type big_key registered [ 5.046413] Key type encrypted registered [ 5.048428] ima: No TPM chip found, activating TPM-bypass! [ 5.050283] ima: Allocated hash algorithm: sha1 [ 5.051931] ima: No architecture policies found [ 5.053681] evm: Initialising EVM extended attributes: [ 5.055471] evm: security.selinux [ 5.056826] evm: security.ima [ 5.057853] evm: security.capability [ 5.059081] evm: HMAC attrs: 0x1 [ 5.062256] rtc_cmos 00:05: setting system clock to 2026-05-29 20:18:37 UTC (1780085917) [ 5.069396] debug: unmapping init [mem 0xffffffff8fc03000-0xffffffff8fdfffff] [ 5.072712] debug: unmapping init [mem 0xffffffff8e982000-0xffffffff8ec58fff] [ 5.079279] Write protecting the kernel read-only data: 28672k [ 5.083465] debug: unmapping init [mem 0xffffffff8d003000-0xffffffff8d1fffff] [ 5.086381] debug: unmapping init [mem 0xffffffff8d914000-0xffffffff8d9fffff] [ 5.133996] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.145697] systemd[1]: Detected virtualization kvm. [ 5.148600] systemd[1]: Detected architecture x86-64. [ 5.150940] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.183643] systemd[1]: No hostname configured. [ 5.185967] systemd[1]: Set hostname to . [ 5.188734] random: systemd: uninitialized urandom read (16 bytes read) [ 5.192499] systemd[1]: Initializing machine ID from random generator. [ 5.320040] random: ln: uninitialized urandom read (6 bytes read) [ 5.642439] random: systemd: uninitialized urandom read (16 bytes read) [ 5.645788] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 5.680472] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.687378] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.482907] device-mapper: uevent: version 1.0.3 [ 7.485745] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.860609] virtio_net virtio0 ens2: renamed from eth0 [ 8.883589] random: fast init done [ 9.200189] scsi host0: ata_piix [ 9.237381] scsi host1: ata_piix [ 9.238514] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.243743] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.594673] random: crng init done [ 13.597150] random: 7 urandom warning(s) missed due to ratelimiting [ 16.430502] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 17.583108] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.662645] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.058450] SELinux: Disabled at runtime. [ 20.173815] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.182410] systemd[1]: Detected virtualization kvm. [ 20.184286] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.025776] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.030379] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.036572] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.042145] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.048394] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.061556] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.076665] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 21.216127] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 22.202794] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.037905] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.064101] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.045849] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.458803] EDAC sbridge: Ver: 1.1.2 [ 27.666584] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 28.076849] NFS: Registering the id_resolver key type [ 28.079159] Key type id_resolver registered [ 28.080431] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg149-server login: [ 67.020014] hrtimer: interrupt took 4413774 ns [ 71.262513] libcfs: loading out-of-tree module taints kernel. [ 71.288488] Key type ._llcrypt registered [ 71.289486] Key type .llcrypt registered [ 71.349358] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 81.673186] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 82.362955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.369277] alg: No test for adler32 (adler32-zlib) [ 83.426918] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 83.887862] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 85.578211] Key type lgssc registered [ 86.444108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 94.642116] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 114.922180] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 121.918766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 121.944753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 123.061778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 123.078809] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 123.132939] Lustre: lustre-MDT0000: new disk, initializing [ 123.180979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 123.194675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 125.546582] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 133.305481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 133.358977] Lustre: 6479:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 133.373697] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 133.375956] Lustre: Skipped 1 previous similar message [ 133.427875] Lustre: lustre-MDT0001: new disk, initializing [ 133.461446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 133.482159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 133.487391] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 135.683438] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 138.748820] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 144.591276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 144.744445] Lustre: lustre-OST0000: new disk, initializing [ 144.745993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 144.751857] Lustre: 8381:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 144.789642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 146.589097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 146.598923] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 146.668181] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 148.124585] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 156.263105] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 156.345557] Lustre: lustre-OST0001: new disk, initializing [ 156.348332] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 156.352242] Lustre: 9437:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 156.399974] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 159.914159] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 162.309729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 162.313874] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 162.366974] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 167.834845] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 173.720634] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 177.236894] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing check_logdir /tmp/testlogs/ [ 179.923861] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing yml_node [ 182.232687] Lustre: DEBUG MARKER: Client: 2.17.53.53 [ 183.711260] Lustre: DEBUG MARKER: MDS: 2.17.53.53 [ 185.155756] Lustre: DEBUG MARKER: OSS: 2.17.53.53 [ 186.089617] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri May 29 16:21:38 EDT 2026 [ 196.409822] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 197.306799] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 210.404567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 210.410920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 210.418982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 213.473158] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 213.473488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 216.617886] Lustre: server umount lustre-MDT0000 complete [ 218.868906] LustreError: 6471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780086131 with bad export cookie 5050112719427868118 [ 218.872516] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 218.873115] LustreError: 6471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 223.713245] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 223.714038] LustreError: 9875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 223.719358] Lustre: Skipped 2 previous similar messages [ 223.720538] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 223.740080] LustreError: 9875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 223.744944] Lustre: Skipped 4 previous similar messages [ 225.138547] Lustre: server umount lustre-MDT0001 complete [ 227.733131] Lustre: server umount lustre-OST0000 complete [ 230.007536] Lustre: server umount lustre-OST0001 complete [ 233.825424] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 238.852743] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 260.497858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 260.663435] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 260.683280] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 260.728692] Lustre: lustre-MDT0000: new disk, initializing [ 260.806398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 260.819239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 263.110597] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 269.454893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 269.535754] Lustre: 14940:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 269.580606] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 269.584287] Lustre: Skipped 1 previous similar message [ 269.669494] Lustre: lustre-MDT0001: new disk, initializing [ 269.747696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 269.776618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 269.785318] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 272.191606] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 277.015425] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 279.290273] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 283.603060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 283.779567] Lustre: 16680:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 285.725245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 285.839065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 287.210766] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 292.186660] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 294.778331] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 294.935637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 297.454214] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 297.578624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 301.025321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 301.033430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.044907] Lustre: Skipped 1 previous similar message [ 301.051949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 304.853292] Lustre: server umount lustre-OST0000 complete [ 310.757270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 310.763897] LustreError: Skipped 1 previous similar message [ 310.765814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.771645] Lustre: Skipped 1 previous similar message [ 310.775168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 310.777096] Lustre: Skipped 1 previous similar message [ 313.323364] Lustre: server umount lustre-MDT0000 complete [ 315.438617] LustreError: 14932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780086227 with bad export cookie 5050112719427869658 [ 315.446939] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 315.448719] LustreError: 14932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 315.620241] Lustre: server umount lustre-MDT0001 complete [ 318.042706] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 16:23:50 (1780086230) [ 319.299242] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 320.251875] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 16:23:52 (1780086232) [ 324.083249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 324.342942] LustreError: 18938:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.402293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 324.406279] Lustre: Skipped 1 previous similar message [ 326.504890] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 329.699517] LustreError: 18939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 331.475331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 334.149749] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 337.534403] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 339.586939] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 343.497508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 346.932261] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 350.670130] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 364.521755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 364.530180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.542275] Lustre: Skipped 1 previous similar message [ 364.547762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 364.551821] Lustre: Skipped 2 previous similar messages [ 370.433084] Lustre: server umount lustre-OST0000 complete [ 377.831647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 377.840963] LustreError: Skipped 1 previous similar message [ 378.988862] Lustre: server umount lustre-MDT0000 complete [ 381.351098] LustreError: 18918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780086293 with bad export cookie 5050112719427870540 [ 381.354477] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 381.361508] LustreError: 18918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 381.588430] Lustre: server umount lustre-MDT0001 complete [ 388.650526] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 390.728219] Key type lgssc unregistered [ 390.952520] LNet: 22136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 390.956622] LNetError: 22136:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 390.974887] LNet: Removed LNI 192.168.201.149@tcp [ 391.574203] Key type .llcrypt unregistered [ 391.575391] Key type ._llcrypt unregistered [ 405.533838] Key type ._llcrypt registered [ 405.534693] Key type .llcrypt registered [ 405.616762] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 406.462312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 406.520430] alg: No test for adler32 (adler32-zlib) [ 407.478241] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 407.623899] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 409.281274] Key type lgssc registered [ 409.995395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 415.666876] loop: module loaded [ 415.702900] loop0: detected capacity change from 0 to 204800000 [ 415.916461] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 415.925212] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 416.115357] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 419.538184] loop0: detected capacity change from 0 to 204800000 [ 419.698157] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 419.710681] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 419.718973] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 423.178725] loop0: detected capacity change from 0 to 204800000 [ 426.671216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 426.688840] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 427.891483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 427.929362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 427.975827] Lustre: lustre-MDT0000: new disk, initializing [ 428.021656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 428.031240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 430.488835] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 436.430230] loop1: detected capacity change from 0 to 204800000 [ 440.016822] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 440.176662] Lustre: lustre-OST07c6: new disk, initializing [ 440.179490] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 440.187737] Lustre: Skipped 1 previous similar message [ 440.191816] Lustre: 25263:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 440.246381] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 440.992894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 441.006624] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 441.037334] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 444.147413] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 447.578072] Lustre: 24270:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 447.582241] Lustre: 24270:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 447.585881] Lustre: 24270:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 447.589444] Lustre: 24270:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 447.593032] Lustre: 24270:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 447.596126] Lustre: 24270:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 466.908836] Lustre: 25854:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 466.915619] Lustre: 25854:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 466.924671] Lustre: 25854:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 466.932190] Lustre: 25854:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 466.937654] Lustre: 25854:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 466.942621] Lustre: 25854:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 471.068198] Lustre: 25854:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 471.071908] Lustre: 25854:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 471.077681] Lustre: 25854:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 471.083328] Lustre: 25854:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 471.088783] Lustre: 25854:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 471.094248] Lustre: 25854:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 471.097898] Lustre: 25854:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 471.101432] Lustre: 25854:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 471.104325] Lustre: 25854:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 471.106354] Lustre: 25854:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 471.109886] Lustre: 25854:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 471.114501] Lustre: 25854:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 474.233861] Lustre: 24268:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 474.238354] Lustre: 24268:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 474.241783] Lustre: 24268:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 474.245501] Lustre: 24268:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 474.249298] Lustre: 24268:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 474.253244] Lustre: 24268:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 474.256264] Lustre: 24268:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 474.259783] Lustre: 24268:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 474.263699] Lustre: 24268:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 474.267232] Lustre: 24268:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 474.271035] Lustre: 24268:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 474.275156] Lustre: 24268:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 481.228028] Lustre: 24268:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 481.241058] Lustre: 24268:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 481.247067] Lustre: 24268:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 481.252108] Lustre: 24268:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 481.257255] Lustre: 24268:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 481.263903] Lustre: 24268:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 481.270209] Lustre: 24268:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 481.273083] Lustre: 24268:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 481.276126] Lustre: 24268:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 481.280356] Lustre: 24268:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 481.284147] Lustre: 24268:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 481.287302] Lustre: 24268:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 491.386608] Lustre: 24269:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 491.394896] Lustre: 24269:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 491.400147] Lustre: 24269:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 491.405953] Lustre: 24269:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 491.410465] Lustre: 24269:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 491.414735] Lustre: 24269:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 491.417454] Lustre: 24269:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 491.426543] Lustre: 24269:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 491.430368] Lustre: 24269:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 491.434489] Lustre: 24269:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 491.439484] Lustre: 24269:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 491.444976] Lustre: 24269:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 533.476511] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 533.484648] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 533.491551] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 538.287127] Lustre: server umount lustre-OST07c6 complete [ 543.101517] Lustre: server umount lustre-MDT0000 complete [ 550.269386] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 551.615970] Key type lgssc unregistered [ 551.766317] LNet: 27783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 551.770039] LNetError: 27783:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 551.783455] LNet: Removed LNI 192.168.201.149@tcp [ 552.190843] Key type .llcrypt unregistered [ 552.195409] Key type ._llcrypt unregistered [ 561.392336] Key type ._llcrypt registered [ 561.393818] Key type .llcrypt registered [ 561.443991] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 561.962220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 561.971598] alg: No test for adler32 (adler32-zlib) [ 562.849676] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 562.955381] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 564.568201] Key type lgssc registered [ 565.047672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 568.670970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 568.680234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.970860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 571.676201] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 575.784661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.971261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 577.844512] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 580.451790] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 582.158079] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 585.389288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 585.565691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 587.987285] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 590.569158] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 594.793537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 605.153670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 605.154729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 605.159203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 605.163640] Lustre: Skipped 1 previous similar message [ 608.388855] Lustre: server umount lustre-OST0000 complete [ 610.804334] Lustre: server umount lustre-MDT0000 complete [ 611.813235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 611.818983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 611.830039] Lustre: Skipped 1 previous similar message [ 611.834775] LustreError: 29014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 611.845761] LustreError: 29014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 613.009257] LustreError: 28996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780086525 with bad export cookie 8874673480571428547 [ 613.020373] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 619.286501] Lustre: server umount lustre-MDT0001 complete [ 624.285319] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 625.513315] Key type lgssc unregistered [ 625.672453] LNet: 32388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 625.677144] LNetError: 32388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 626.727841] LNet: Removed LNI 192.168.201.149@tcp [ 627.106993] Key type .llcrypt unregistered [ 627.108740] Key type ._llcrypt unregistered [ 635.152862] Key type ._llcrypt registered [ 635.154183] Key type .llcrypt registered [ 635.192697] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 635.532541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 635.575170] alg: No test for adler32 (adler32-zlib) [ 636.459053] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 636.568546] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 638.160190] Key type lgssc registered [ 638.659136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 641.908074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 641.917442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 643.134680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 644.586759] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 648.179571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 648.337088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 649.929762] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 651.956277] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 653.452416] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 656.569349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 656.787919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 659.187170] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 661.597946] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 668.075507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 678.373145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 678.373231] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 678.374970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 678.381413] Lustre: Skipped 1 previous similar message [ 682.724268] Lustre: server umount lustre-OST0000 complete [ 684.512816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 684.515906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 684.521832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 684.524640] Lustre: Skipped 1 previous similar message [ 688.609783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 688.609902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 690.365794] Lustre: server umount lustre-MDT0000 complete [ 692.036433] LustreError: 35878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780086604 with bad export cookie 4087618779048524201 [ 692.038480] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 692.045958] LustreError: 35878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 692.247705] Lustre: server umount lustre-MDT0001 complete [ 696.732574] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 698.001056] Key type lgssc unregistered [ 698.147941] LNet: 36813:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.152715] LNetError: 36813:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 698.163531] LNet: Removed LNI 192.168.201.149@tcp [ 698.532325] Key type .llcrypt unregistered [ 698.533992] Key type ._llcrypt unregistered [ 708.972406] Key type ._llcrypt registered [ 708.973739] Key type .llcrypt registered [ 709.020441] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 709.455502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 709.504700] alg: No test for adler32 (adler32-zlib) [ 710.398386] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 710.507637] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 712.096773] Key type lgssc registered [ 712.614327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 715.671486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 715.687971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 716.901950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 718.542828] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 722.191447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 722.341518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 723.847776] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 726.160613] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 727.727361] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 730.554554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 730.689492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 732.816058] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 734.904207] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 741.990606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 748.001916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 748.005071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 748.009402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 750.589354] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.49@tcp (stopping) [ 752.610099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 752.611811] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 752.612825] Lustre: Skipped 1 previous similar message [ 752.618389] Lustre: Skipped 1 previous similar message [ 754.081339] Lustre: server umount lustre-MDT0000 complete [ 755.531299] LustreError: 38171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780086667 with bad export cookie 16806522570968209659 [ 755.534177] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 755.535257] LustreError: 38171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 755.703128] LustreError: 39401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 755.710023] LustreError: 39401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 755.713151] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.49@tcp (stopping) [ 755.718630] Lustre: Skipped 1 previous similar message [ 757.729380] LustreError: 38184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 757.729660] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 760.736793] LustreError: 39401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 760.737082] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 760.745833] Lustre: Skipped 1 previous similar message [ 761.706611] Lustre: server umount lustre-MDT0001 complete [ 779.741152] Lustre: server umount lustre-OST0000 complete [ 786.540805] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 787.669115] Key type lgssc unregistered [ 787.804541] LNet: 41708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.810225] LNetError: 41708:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 787.818356] LNet: Removed LNI 192.168.201.149@tcp [ 788.152854] Key type .llcrypt unregistered [ 788.154332] Key type ._llcrypt unregistered [ 798.278772] Key type ._llcrypt registered [ 798.280219] Key type .llcrypt registered [ 798.327831] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 798.721835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 798.781458] alg: No test for adler32 (adler32-zlib) [ 799.649962] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 799.738757] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 801.328158] Key type lgssc registered [ 801.734249] Lustre: Echo OBD driver; http://www.lustre.org/ [ 804.823680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 804.836178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 806.025164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 807.627061] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 811.373115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 811.595227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 813.539549] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 816.130256] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 817.670883] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 820.562180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 820.702595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 823.038279] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 825.121349] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 830.953787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 836.067788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 836.068058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 836.069431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 836.073850] Lustre: Skipped 1 previous similar message [ 836.606204] Lustre: lustre-OST0000: Not available for connect from 192.168.201.49@tcp (stopping) [ 836.609020] Lustre: Skipped 1 previous similar message [ 839.760809] Lustre: server umount lustre-OST0000 complete [ 852.839791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 852.845377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 855.906027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 855.906140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 861.153834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 861.156252] Lustre: Skipped 1 previous similar message [ 867.296132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 867.361660] Lustre: server umount lustre-MDT0000 complete [ 868.952177] LustreError: 43065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780086781 with bad export cookie 16422107380749131287 [ 868.956303] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 868.958030] LustreError: 43065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 875.166493] Lustre: server umount lustre-MDT0001 complete [ 879.831831] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 881.294707] Key type lgssc unregistered [ 881.456215] LNet: 46436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.460149] LNetError: 46436:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.473599] LNet: Removed LNI 192.168.201.149@tcp [ 881.874178] Key type .llcrypt unregistered [ 881.875555] Key type ._llcrypt unregistered [ 892.052703] Key type ._llcrypt registered [ 892.054197] Key type .llcrypt registered [ 892.108497] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 892.476233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 892.528738] alg: No test for adler32 (adler32-zlib) [ 893.395597] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 893.487465] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 895.080191] Key type lgssc registered [ 895.519287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 898.551799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 898.561058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 899.827442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 901.573501] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 905.436346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 905.624149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 907.271369] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 909.641196] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 911.237294] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 913.903223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 914.034583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 916.104937] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 918.172503] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 925.288573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 932.053697] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 933.486588] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 944.349618] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-05-2916h34m16s [ 945.023722] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 946.147509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 946.148279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 946.154511] Lustre: Skipped 2 previous similar messages [ 946.156793] Lustre: Skipped 1 previous similar message [ 947.170643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 947.174676] Lustre: Skipped 3 previous similar messages [ 950.271501] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.49@tcp (stopping) [ 952.117421] Lustre: server umount lustre-MDT0000 complete [ 953.318159] LustreError: 47812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 953.335376] LustreError: 47812:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 953.710468] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 956.342842] LustreError: 47806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 956.722833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 956.789239] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 956.902662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 956.935081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 957.927378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 958.554769] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 960.101284] Lustre: DEBUG MARKER: Wait for df (35947) ... [ 960.818596] Lustre: DEBUG MARKER: done [ 961.171626] LustreError: 47794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.49@tcp arrived at 1780086873 with bad export cookie 9562456818248558059 [ 961.177466] LustreError: 47794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 962.529122] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 962.532791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 962.538578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 962.541232] Lustre: Skipped 1 previous similar message [ 968.164345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 968.170181] Lustre: Skipped 3 previous similar messages [ 976.352300] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 976.419226] Lustre: server umount lustre-OST0000 complete [ 978.401941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 978.402247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 978.403880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 978.403885] Lustre: Skipped 2 previous similar messages [ 978.414934] Lustre: Skipped 2 previous similar messages [ 992.224132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 992.291140] Lustre: server umount lustre-MDT0000 complete [ 993.972165] LustreError: 50122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780086906 with bad export cookie 9562456818248558528 [ 993.978070] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 993.981524] LustreError: 50122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 994.133101] Lustre: server umount lustre-MDT0001 complete [ 998.800538] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 1000.146243] Key type lgssc unregistered [ 1000.304768] LNet: 52380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1000.308730] LNetError: 52380:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1000.321458] LNet: Removed LNI 192.168.201.149@tcp [ 1000.699137] Key type .llcrypt unregistered [ 1000.701725] Key type ._llcrypt unregistered [ 1018.735420] Key type ._llcrypt registered [ 1018.737082] Key type .llcrypt registered [ 1018.777795] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1019.062220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1019.086569] alg: No test for adler32 (adler32-zlib) [ 1019.977524] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 1020.073803] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1021.664165] Key type lgssc registered [ 1022.110531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1025.306880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1025.316295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1026.432574] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1026.442237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1026.517781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1028.015616] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1031.459987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1031.522979] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1031.538785] Lustre: 54563:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1031.663050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1033.300179] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1035.627517] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1037.074935] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1040.046897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1040.166801] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1040.266132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1041.208802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1042.743667] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1045.033458] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1048.688781] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-05-2916h36m01s [ 1049.326703] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1051.895297] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1052.853123] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1052.870127] LustreError: 54569:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1073.157381] Lustre: lustre-MDT0000: Client f11fdb7f-e7cc-4c55-a431-d67a1049be7e (at 192.168.201.49@tcp) reconnecting [ 1092.912628] LustreError: 54569:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1093.855271] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1094.824213] Lustre: DEBUG MARKER: done [ 1097.699889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1097.699949] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1097.700639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1097.705684] Lustre: Skipped 1 previous similar message [ 1097.709137] LustreError: Skipped 1 previous similar message [ 1102.748077] Lustre: server umount lustre-OST0000 complete [ 1107.937705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1107.940406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1107.941718] Lustre: Skipped 1 previous similar message [ 1107.946172] Lustre: Skipped 2 previous similar messages [ 1108.962099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1108.965336] Lustre: Skipped 1 previous similar message [ 1110.369686] Lustre: server umount lustre-MDT0000 complete [ 1111.746267] LustreError: 54553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087024 with bad export cookie 9355891318194192804 [ 1111.748674] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1111.752951] LustreError: 54553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1111.872012] Lustre: server umount lustre-MDT0001 complete [ 1115.597736] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 1116.663173] Key type lgssc unregistered [ 1116.808358] LNet: 58528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1116.811346] LNetError: 58528:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1116.820426] LNet: Removed LNI 192.168.201.149@tcp [ 1117.128150] Key type .llcrypt unregistered [ 1117.129160] Key type ._llcrypt unregistered [ 1134.264479] Key type ._llcrypt registered [ 1134.266074] Key type .llcrypt registered [ 1134.310774] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1134.650569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1134.718942] alg: No test for adler32 (adler32-zlib) [ 1135.583462] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 1135.672840] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1137.256158] Key type lgssc registered [ 1137.633821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1138.786951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1138.792717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1139.887312] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1139.894884] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1139.962575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1140.316708] Lustre: Failing over lustre-MDT0000 [ 1140.418271] Lustre: server umount lustre-MDT0000 complete [ 1142.575324] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 16:37:35 (1780087055) [ 1144.792794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1144.931404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1146.341564] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1149.451320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1149.491966] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1149.507462] Lustre: 61285:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1149.609667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1151.007974] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1152.878703] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1154.101520] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1156.530304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1156.610709] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1156.692092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1158.830506] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1159.633417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1160.867376] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1164.079343] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1169.893167] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1169.895571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1169.898182] Lustre: Skipped 2 previous similar messages [ 1169.900108] Lustre: Skipped 1 previous similar message [ 1172.438602] Lustre: server umount lustre-MDT0000 complete [ 1173.849285] LustreError: 61275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087086 with bad export cookie 13946645224400288745 [ 1173.852552] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1173.853641] LustreError: 61275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1173.997191] Lustre: server umount lustre-MDT0001 complete [ 1175.272768] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1176.958795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1180.344297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1184.160834] LustreError: 60206:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000025f58866 x1866556556474624/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1184.322520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1184.358989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1185.739183] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1188.969218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1189.082385] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1190.181647] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1190.668898] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1191.200816] Lustre: 60209:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780087087/real 1780087087] req@0000000099aa7ef7 x1866556556474240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780087103 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1192.730641] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1194.006346] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1194.272146] Lustre: 60210:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780087090/real 1780087090] req@000000000d68cfbb x1866556556474368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780087106 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1197.025716] Lustre: 60207:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780087092/real 1780087092] req@00000000f4ec3df2 x1866556556474496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780087108 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1199.585491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1199.590292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1199.596498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1199.598778] Lustre: Skipped 1 previous similar message [ 1200.096209] Lustre: 60208:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780087096/real 1780087096] req@000000008292c5b5 x1866556556474752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780087112 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1200.609972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1200.612418] Lustre: Skipped 2 previous similar messages [ 1202.149199] Lustre: server umount lustre-MDT0000 complete [ 1203.902179] LustreError: 62002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087116 with bad export cookie 13946645224400291013 [ 1203.905804] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1203.906059] LustreError: 62002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1204.047477] Lustre: server umount lustre-MDT0001 complete [ 1206.989803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1210.689805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1213.856643] LustreError: 60206:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fcf387ec x1866556556494592/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1213.974591] LustreError: 62906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1213.982332] LustreError: 62906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1214.001132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1214.003897] Lustre: Skipped 1 previous similar message [ 1214.036277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1215.548488] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1219.042733] LustreError: 62907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1219.044750] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1219.054902] Lustre: Skipped 2 previous similar messages [ 1219.063868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1220.737426] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1221.288181] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1221.290680] Lustre: Skipped 1 previous similar message [ 1222.112088] Lustre: 60208:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780087118/real 1780087118] req@000000000bdfeed0 x1866556556494208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780087134 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1222.687858] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1223.946946] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1226.722434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1226.723636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1226.726473] Lustre: Skipped 2 previous similar messages [ 1226.728193] Lustre: Skipped 1 previous similar message [ 1231.842113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1231.844127] Lustre: Skipped 2 previous similar messages [ 1231.910125] Lustre: server umount lustre-MDT0000 complete [ 1233.225977] LustreError: 61276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087145 with bad export cookie 13946645224400292392 [ 1233.229080] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1233.230019] LustreError: 61276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1233.340849] Lustre: server umount lustre-MDT0001 complete [ 1235.537192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1236.770116] Lustre: DEBUG MARKER: files compared the same [ 1246.755455] Lustre: server umount lustre-OST0000 complete [ 1252.386461] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 1253.538038] Key type lgssc unregistered [ 1253.691813] LNet: 70186:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.695281] LNetError: 70186:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1253.706366] LNet: Removed LNI 192.168.201.149@tcp [ 1254.051651] Key type .llcrypt unregistered [ 1254.052664] Key type ._llcrypt unregistered [ 1263.025322] Key type ._llcrypt registered [ 1263.026692] Key type .llcrypt registered [ 1263.064663] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1263.402214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1263.451655] alg: No test for adler32 (adler32-zlib) [ 1264.341448] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 1264.420939] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1266.000177] Key type lgssc registered [ 1266.388044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1269.086145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1269.093148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1270.248494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1271.546851] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug +malloc all [ 1274.496438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1274.594387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1275.940913] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug +malloc all [ 1277.750906] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1278.948538] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1281.381848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1281.483807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1283.441057] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug +malloc all [ 1285.296637] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1294.819233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1299.938229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1299.938610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1299.944381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1299.947223] LustreError: Skipped 1 previous similar message [ 1299.949104] Lustre: Skipped 1 previous similar message [ 1304.470376] Lustre: server umount lustre-OST0000 complete [ 1310.176795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1310.181781] Lustre: Skipped 1 previous similar message [ 1310.183648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1310.186413] Lustre: Skipped 1 previous similar message [ 1311.827933] Lustre: server umount lustre-MDT0000 complete [ 1313.054557] LustreError: 71543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087225 with bad export cookie 2378517720577826507 [ 1313.055687] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1313.057589] LustreError: 71543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1313.166171] Lustre: server umount lustre-MDT0001 complete [ 1316.613823] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 1317.639032] Key type lgssc unregistered [ 1317.765386] LNet: 74757:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1317.768590] LNetError: 74757:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1317.777509] LNet: Removed LNI 192.168.201.149@tcp [ 1318.084181] Key type .llcrypt unregistered [ 1318.085143] Key type ._llcrypt unregistered [ 1326.858712] Key type ._llcrypt registered [ 1326.859915] Key type .llcrypt registered [ 1326.898687] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1327.196847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1327.204023] alg: No test for adler32 (adler32-zlib) [ 1328.049188] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 1328.130738] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1329.704161] Key type lgssc registered [ 1330.047267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1332.585719] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1332.592988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1348.064322] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1353.184171] LustreError: 76121:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.149@tcp: failed processing log, type 4: rc = -110 [ 1379.808163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1381.552107] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1383.245711] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1385.620028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1385.658483] Lustre: *** cfs_fail_loc=706, val=0*** [ 1385.803123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1385.828424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1386.960209] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1389.802296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1389.935816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1391.257612] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1393.005296] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1394.177040] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1395.170299] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1395.170369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1395.175371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1395.179104] Lustre: Skipped 1 previous similar message [ 1396.936351] Lustre: server umount lustre-OST0000 complete [ 1400.289485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1400.289981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1400.298106] Lustre: Skipped 1 previous similar message [ 1400.300251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1400.301991] Lustre: Skipped 1 previous similar message [ 1404.240758] Lustre: server umount lustre-MDT0000 complete [ 1405.408942] LustreError: 77095:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1405.414597] LustreError: 77095:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1405.437764] LustreError: 76128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087317 with bad export cookie 10421461738997499459 [ 1405.439130] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1405.443261] LustreError: 76128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1405.547875] Lustre: server umount lustre-MDT0001 complete [ 1409.123907] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 1410.242570] Key type lgssc unregistered [ 1410.381441] LNet: 79372:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1410.384392] LNetError: 79372:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1410.392316] LNet: Removed LNI 192.168.201.149@tcp [ 1410.674526] Key type .llcrypt unregistered [ 1410.675810] Key type ._llcrypt unregistered [ 1420.100079] Key type ._llcrypt registered [ 1420.101153] Key type .llcrypt registered [ 1420.142109] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1420.446082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1420.474910] alg: No test for adler32 (adler32-zlib) [ 1421.321775] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 1421.400950] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1422.976212] Key type lgssc registered [ 1423.341378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1425.848475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1425.855512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1428.159314] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1430.611797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1430.775669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1432.030680] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1434.818937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1434.940727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1436.830781] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1439.732677] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1439.738758] Lustre: MGS: Client 27d1d9e4-08e0-4e6b-9332-6d835c3efbc5 (at 0@lo) reconnecting [ 1439.828762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1439.861133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1441.148046] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1449.953182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1449.955922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1449.961405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1451.491736] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1451.491878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1451.495212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1451.497476] Lustre: Skipped 1 previous similar message [ 1454.930041] Lustre: server umount lustre-OST0000 complete [ 1456.610328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1456.610483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1456.611014] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.614762] Lustre: Skipped 1 previous similar message [ 1461.729625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1461.732434] Lustre: Skipped 2 previous similar messages [ 1462.274852] Lustre: server umount lustre-MDT0000 complete [ 1463.578472] Lustre: server umount lustre-MDT0001 complete [ 1464.774636] Lustre: server umount MGS complete [ 1468.347151] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 16:43:01 (1780087381) [ 1477.800116] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 1480.822910] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1492.097449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1493.434483] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1497.226294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1497.318312] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1497.333111] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1497.365542] Lustre: lustre-MDT0001: new disk, initializing [ 1497.389058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1497.396898] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1497.399490] Lustre: 88473:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1497.402778] LustreError: 88473:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1498.649123] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1502.745598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1502.855791] Lustre: lustre-OST0000: new disk, initializing [ 1502.857673] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1502.859844] Lustre: 89226:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1502.880191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1504.666615] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1506.080812] Lustre: 89632:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000001759fb10 x1866556855321344/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1780087434 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1507.758969] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1510.012045] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1510.017450] Lustre: MGS: Client 74d269c0-8478-487a-8712-84c38f9382b8 (at 0@lo) reconnecting [ 1510.032229] Lustre: 90158:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1510.041813] LustreError: 88459:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1510.044850] LustreError: 88459:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1510.051160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1510.084681] Lustre: lustre-MDT0000: new disk, initializing [ 1510.140216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1511.200864] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1511.201399] Lustre: 90184:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000954a8dbc x1866556855326976/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1780087439 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1511.213841] Lustre: 90184:0:(client.c:1638:after_reply()) Skipped 1 previous similar message [ 1511.503644] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1515.171921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1515.174789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1516.259756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1516.262624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1516.263631] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1535.969032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1535.971189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1535.975958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1541.193616] Lustre: server umount lustre-OST0000 complete [ 1545.186596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1545.186720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1545.190993] Lustre: Skipped 1 previous similar message [ 1545.193398] LustreError: Skipped 1 previous similar message [ 1545.195245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1545.197718] Lustre: Skipped 4 previous similar messages [ 1548.619739] Lustre: server umount lustre-MDT0000 complete [ 1556.078513] Lustre: server umount lustre-MDT0001 complete [ 1557.263132] Lustre: server umount MGS complete [ 1560.797660] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 16:44:33 (1780087473) [ 1566.632699] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 1567.627954] Key type lgssc unregistered [ 1567.750438] LNet: 92643:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1567.752581] LNetError: 92643:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1567.761375] LNet: Removed LNI 192.168.201.149@tcp [ 1568.069147] Key type .llcrypt unregistered [ 1568.070664] Key type ._llcrypt unregistered [ 1574.845061] Key type ._llcrypt registered [ 1574.846139] Key type .llcrypt registered [ 1574.884342] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1575.200946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1575.208032] alg: No test for adler32 (adler32-zlib) [ 1576.062655] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 1576.143880] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1577.720156] Key type lgssc registered [ 1578.069824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1580.551124] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1580.557978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1581.715880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1582.948678] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1585.856591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1585.990475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1587.297381] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1589.051478] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1590.212851] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1592.442834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1592.563512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1594.445964] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1596.311823] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1606.883592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1616.353051] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1616.355407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1616.359898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1617.378294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1617.378894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1617.379702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1620.375302] Lustre: server umount lustre-OST0000 complete [ 1621.984777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1621.987656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1621.992502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1621.995247] Lustre: Skipped 1 previous similar message [ 1627.617444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1627.620226] Lustre: Skipped 3 previous similar messages [ 1627.799164] Lustre: server umount lustre-MDT0000 complete [ 1629.033075] LustreError: 95983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087541 with bad export cookie 15099521992826929148 [ 1629.036086] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1629.037096] LustreError: 95983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1629.136158] Lustre: server umount lustre-MDT0001 complete [ 1632.807341] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 1633.826143] Key type lgssc unregistered [ 1633.946309] LNet: 97020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1633.948691] LNetError: 97020:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1633.955308] LNet: Removed LNI 192.168.201.149@tcp [ 1634.247346] Key type .llcrypt unregistered [ 1634.248520] Key type ._llcrypt unregistered [ 1640.588789] Key type ._llcrypt registered [ 1640.589794] Key type .llcrypt registered [ 1640.634941] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1641.035880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1641.090171] alg: No test for adler32 (adler32-zlib) [ 1641.961672] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 1642.046454] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1643.632175] Key type lgssc registered [ 1643.981957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1646.035431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1646.038858] LustreError: 98063:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1646.039411] LustreError: 98064:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1646.043250] LustreError: 98063:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1646.045341] LustreError: 98063:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1646.048475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1646.048937] LustreError: 98063:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1647.220445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1649.734016] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1649.822728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1651.054850] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1652.627220] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing lsmod [ 1653.546715] LustreError: 99234:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1653.547714] LustreError: 99235:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1653.550635] LustreError: 99234:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4998 [ 1653.552633] LustreError: 99234:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1653.556063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1653.557646] LustreError: 99234:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1653.686657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1654.566707] LustreError: 98130:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1654.569729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1654.570459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1654.571308] LustreError: 98130:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1654.574601] Lustre: Skipped 1 previous similar message [ 1654.580099] LustreError: 98130:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 1659.681701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1659.683353] Lustre: Skipped 4 previous similar messages [ 1660.690862] Lustre: server umount lustre-MDT0000 complete [ 1661.923568] LustreError: 98075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087574 with bad export cookie 7445501105031483064 [ 1661.926048] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1661.929965] LustreError: 98075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1663.776255] LustreError: 98532:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b00650cb x1866557087038592/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1663.825190] Lustre: server umount lustre-MDT0001 complete [ 1675.284072] Lustre: server umount lustre-OST0000 complete [ 1677.804956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1677.946454] LustreError: 100299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1677.953069] LustreError: 100299:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1677.967214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1679.142871] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1681.810595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1683.133812] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1684.772352] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1685.887540] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1688.075865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1688.172156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1688.174498] Lustre: Skipped 1 previous similar message [ 1689.942693] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1691.613473] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1703.526815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1709.025243] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1709.027022] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1709.028395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1709.032314] Lustre: Skipped 2 previous similar messages [ 1714.146166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1714.148359] Lustre: Skipped 2 previous similar messages [ 1719.266304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1719.267832] Lustre: Skipped 1 previous similar message [ 1722.848127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1722.898330] Lustre: server umount lustre-OST0000 complete [ 1724.385299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1724.390016] Lustre: Skipped 1 previous similar message [ 1729.504909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1729.506948] Lustre: Skipped 3 previous similar messages [ 1730.195542] Lustre: server umount lustre-MDT0000 complete [ 1731.377030] LustreError: 100324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087643 with bad export cookie 7445501105031483743 [ 1731.379207] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1731.380278] LustreError: 100324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1731.501789] Lustre: server umount lustre-MDT0001 complete [ 1735.031544] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 1736.047726] Key type lgssc unregistered [ 1736.181442] LNet: 103495:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1736.184769] LNetError: 103495:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.195432] LNet: Removed LNI 192.168.201.149@tcp [ 1736.516406] Key type .llcrypt unregistered [ 1736.517912] Key type ._llcrypt unregistered [ 1745.153162] Key type ._llcrypt registered [ 1745.154624] Key type .llcrypt registered [ 1745.189970] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1745.502656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1745.552298] alg: No test for adler32 (adler32-zlib) [ 1746.413812] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 1746.483077] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1748.064122] Key type lgssc registered [ 1748.402918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1750.943740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1750.950235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1752.096931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1753.284059] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1756.017299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1756.139416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1757.363930] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1758.988495] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1760.176128] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1762.322950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1762.427786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1764.198571] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1765.865533] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1776.806061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1783.062264] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1783.722645] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1787.362854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1787.364558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1787.365274] LustreError: Skipped 1 previous similar message [ 1787.365891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1787.371776] Lustre: Skipped 1 previous similar message [ 1787.375904] Lustre: Skipped 1 previous similar message [ 1790.737667] Lustre: server umount lustre-OST0000 complete [ 1792.482045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1792.483101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1792.486134] Lustre: Skipped 1 previous similar message [ 1797.601558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1797.603555] Lustre: Skipped 2 previous similar messages [ 1798.239967] Lustre: server umount lustre-MDT0000 complete [ 1799.533141] LustreError: 104850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087711 with bad export cookie 15046297721998720217 [ 1799.536558] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1799.537782] LustreError: 104850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1799.655664] Lustre: server umount lustre-MDT0001 complete [ 1803.437096] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 1804.480224] Key type lgssc unregistered [ 1804.618339] LNet: 108170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1804.620809] LNetError: 108170:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1804.628358] LNet: Removed LNI 192.168.201.149@tcp [ 1804.945647] Key type .llcrypt unregistered [ 1804.947285] Key type ._llcrypt unregistered [ 1812.347244] Key type ._llcrypt registered [ 1812.348298] Key type .llcrypt registered [ 1812.387182] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1812.709544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1812.724206] alg: No test for adler32 (adler32-zlib) [ 1813.574745] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 1813.653312] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1815.232108] Key type lgssc registered [ 1815.602471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1818.055642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1818.061030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1819.197854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1820.402416] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1823.132653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1823.245150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1824.427878] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1826.091754] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1827.159643] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1829.189376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1829.289388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1830.311329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1830.967093] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1832.543647] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1838.563415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1838.564226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1838.564702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1838.572314] Lustre: Skipped 1 previous similar message [ 1842.187301] Lustre: server umount lustre-OST0000 complete [ 1843.681471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1843.681677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1843.682229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1843.682232] Lustre: Skipped 1 previous similar message [ 1843.692586] Lustre: Skipped 1 previous similar message [ 1848.801288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1848.802767] Lustre: Skipped 2 previous similar messages [ 1849.494993] Lustre: server umount lustre-MDT0000 complete [ 1850.731102] LustreError: 109333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087763 with bad export cookie 7343664008124344541 [ 1850.733921] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1850.735285] LustreError: 109333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1850.845596] Lustre: server umount lustre-MDT0001 complete [ 1854.231322] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 1855.196781] Key type lgssc unregistered [ 1855.311467] LNet: 112526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.315194] LNetError: 112526:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1855.323312] LNet: Removed LNI 192.168.201.149@tcp [ 1855.595838] Key type .llcrypt unregistered [ 1855.597218] Key type ._llcrypt unregistered [ 1864.072163] Key type ._llcrypt registered [ 1864.073666] Key type .llcrypt registered [ 1864.107128] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1864.516567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1864.527171] alg: No test for adler32 (adler32-zlib) [ 1865.371136] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 1865.447616] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1867.024141] Key type lgssc registered [ 1867.349948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1869.743466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1869.750626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.894858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1872.115625] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1874.817846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1874.946249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1876.204284] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1877.859563] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1878.960456] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1881.003287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1881.059279] LustreError: 115493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1881.063195] LustreError: 115493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1881.128065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1882.845853] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1884.520465] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1885.223781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1893.003637] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1894.114071] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1894.115885] Lustre: Skipped 2 previous similar messages [ 1895.380593] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1895.382987] Lustre: Skipped 2 previous similar messages [ 1901.052248] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1901.063078] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1906.041239] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.49@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1906.045187] Lustre: Skipped 2 previous similar messages [ 1908.278605] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.49@tcp 192.168.0.[2,10]@tcp [ 1911.024952] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.49/24@tcp in log lustre-MDT0000 [ 1911.028112] Lustre: Skipped 2 previous similar messages [ 1914.357654] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.49/24@tcp [ 1914.360389] Lustre: Skipped 1 previous similar message [ 1917.922510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1917.925512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1917.930560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1920.993241] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1920.994467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1920.996061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1923.344213] Lustre: server umount lustre-OST0000 complete [ 1926.112992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1926.120032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1926.122548] Lustre: Skipped 2 previous similar messages [ 1930.643056] Lustre: server umount lustre-MDT0000 complete [ 1931.233907] LustreError: 113903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1931.238938] LustreError: 113903:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1931.797947] LustreError: 113883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087844 with bad export cookie 13290749976266491273 [ 1931.801262] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1931.802225] LustreError: 113883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1931.903211] Lustre: server umount lustre-MDT0001 complete [ 1935.192841] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 1936.090911] Key type lgssc unregistered [ 1936.223293] LNet: 118569:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1936.225411] LNetError: 118569:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1936.236267] LNet: Removed LNI 192.168.201.149@tcp [ 1936.504578] Key type .llcrypt unregistered [ 1936.506288] Key type ._llcrypt unregistered [ 1946.636993] Key type ._llcrypt registered [ 1946.638503] Key type .llcrypt registered [ 1946.675272] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 1946.995940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1947.055839] alg: No test for adler32 (adler32-zlib) [ 1947.912515] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 1947.985148] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 1949.568163] Key type lgssc registered [ 1949.888600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1952.279237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1952.287437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1953.434415] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1953.440469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1953.443647] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.49/24@tcp [ 1954.654986] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1957.405523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1957.518882] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1957.528281] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.49/24@tcp [ 1957.548498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1958.838452] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1960.613667] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1961.838725] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1964.019784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1964.147445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1965.883641] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 1967.502146] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1978.471369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 1988.065072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1988.068215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1988.072694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1989.090302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1989.091297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1989.091671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1992.076785] Lustre: server umount lustre-OST0000 complete [ 1993.696988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1993.698806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1993.701958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1993.703520] Lustre: Skipped 1 previous similar message [ 1999.329081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1999.330980] Lustre: Skipped 3 previous similar messages [ 1999.445500] Lustre: server umount lustre-MDT0000 complete [ 2000.622074] LustreError: 120072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087913 with bad export cookie 6415796788074006515 [ 2000.625028] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2000.627122] LustreError: 120072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2000.742690] Lustre: server umount lustre-MDT0001 complete [ 2004.129373] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 2005.105984] Key type lgssc unregistered [ 2005.234364] LNet: 123339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2005.236569] LNetError: 123339:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2005.245296] LNet: Removed LNI 192.168.201.149@tcp [ 2005.537681] Key type .llcrypt unregistered [ 2005.538731] Key type ._llcrypt unregistered [ 2013.834753] Key type ._llcrypt registered [ 2013.835759] Key type .llcrypt registered [ 2013.869086] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 2019.861975] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 2020.281305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2020.359768] alg: No test for adler32 (adler32-zlib) [ 2021.218549] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 2021.313654] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 2022.904142] Key type lgssc registered [ 2023.251138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2033.901195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2033.907089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2034.990191] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2035.001903] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2035.041072] Lustre: lustre-MDT0000: new disk, initializing [ 2035.066095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2035.073041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2036.266447] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2040.246521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2040.284884] Lustre: 126677:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2040.309562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2040.311383] Lustre: Skipped 1 previous similar message [ 2040.344575] Lustre: lustre-MDT0001: new disk, initializing [ 2040.374827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2040.386712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2040.390137] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2041.591978] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2044.499862] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2045.561291] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2047.611300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2047.705912] Lustre: lustre-OST0000: new disk, initializing [ 2047.708383] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2047.711902] Lustre: 128416:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2047.739020] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2049.441584] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2052.406780] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2053.112674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2053.116286] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2053.147829] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2053.739862] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2053.807630] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2055.023632] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2055.093821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2058.210534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2058.211052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.217790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2058.220544] Lustre: Skipped 1 previous similar message [ 2061.836993] Lustre: server umount lustre-OST0000 complete [ 2063.329282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2063.329811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2063.332613] Lustre: Skipped 2 previous similar messages [ 2068.449252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2068.450698] Lustre: Skipped 2 previous similar messages [ 2069.143614] Lustre: server umount lustre-MDT0000 complete [ 2070.371239] LustreError: 126668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780087982 with bad export cookie 7482965472766682896 [ 2070.374159] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2070.376662] LustreError: 126668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2070.496655] Lustre: server umount lustre-MDT0001 complete [ 2073.448772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.663635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2075.094686] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2077.979491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2078.109773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2079.486502] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2081.302677] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2082.567959] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2084.921393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2086.921071] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2088.667286] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2090.023052] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2100.290888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 2101.624260] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2101.704106] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2105.219790] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2105.287755] Lustre: lustre-OST0001: new disk, initializing [ 2105.290341] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2105.293902] Lustre: 133548:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2105.325838] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2105.329011] Lustre: Skipped 1 previous similar message [ 2107.240067] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2110.497492] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2112.003611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2112.008401] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2112.046302] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2112.605926] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2113.953617] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2114.029434] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2115.227654] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f804735c000.ost_server_uuid 50 [ 2115.742198] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f804735c000.ost_server_uuid in FULL state after 0 sec [ 2122.215789] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2122.216169] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2122.219395] LustreError: Skipped 2 previous similar messages [ 2122.224961] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2122.227672] Lustre: Skipped 1 previous similar message [ 2123.843859] Lustre: server umount lustre-OST0001 complete [ 2125.280835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2125.283702] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2125.288461] Lustre: Skipped 1 previous similar message [ 2127.329256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2127.330780] Lustre: Skipped 1 previous similar message [ 2127.332309] LustreError: 132075:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2127.336094] LustreError: 132075:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2131.217117] Lustre: server umount lustre-OST0000 complete [ 2132.455203] LustreError: 124902:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000dd4948b3 x1866557484514048/t0(0) o8->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2132.455694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2132.462903] Lustre: Skipped 2 previous similar messages [ 2137.569293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2137.572202] Lustre: Skipped 4 previous similar messages [ 2138.588259] Lustre: server umount lustre-MDT0000 complete [ 2139.919506] LustreError: 130465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088052 with bad export cookie 7482965472766683764 [ 2139.923738] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2139.925022] LustreError: 130465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2140.069726] Lustre: server umount lustre-MDT0001 complete [ 2146.344559] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 2147.378244] Key type lgssc unregistered [ 2147.499536] LNet: 136398:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2147.501920] LNetError: 136398:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2147.513379] LNet: Removed LNI 192.168.201.149@tcp [ 2147.819118] Key type .llcrypt unregistered [ 2147.820474] Key type ._llcrypt unregistered [ 2160.030355] Key type ._llcrypt registered [ 2160.031407] Key type .llcrypt registered [ 2160.067829] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 2165.786090] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 2166.160864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2166.188628] alg: No test for adler32 (adler32-zlib) [ 2167.035149] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 2167.113126] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 2168.688183] Key type lgssc registered [ 2169.023517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2179.432459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2179.437729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2180.515581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2180.525459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2180.557780] Lustre: lustre-MDT0000: new disk, initializing [ 2180.579350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2180.585215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2181.793217] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2185.914119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2185.956385] Lustre: 140486:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2185.983859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2185.985663] Lustre: Skipped 1 previous similar message [ 2186.016339] Lustre: lustre-MDT0001: new disk, initializing [ 2186.046684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2186.056680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2186.059663] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2187.442194] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2190.524428] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2191.777327] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2194.187278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2194.311773] Lustre: lustre-OST0000: new disk, initializing [ 2194.313642] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2194.316548] Lustre: 142228:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2194.353114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2196.283698] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2197.174490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2197.177636] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2197.205693] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2199.401796] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2201.435745] Lustre: Failing over lustre-OST0000 [ 2201.511378] Lustre: server umount lustre-OST0000 complete [ 2214.322465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2214.431985] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2214.435564] Lustre: Skipped 1 previous similar message [ 2214.440225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2214.446635] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2215.933440] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2216.313911] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2217.606044] Lustre: Failing over lustre-MDT0000 [ 2217.711437] Lustre: server umount lustre-MDT0000 complete [ 2218.656177] Lustre: 138715:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780088115/real 1780088115] req@00000000357c39ab x1866557637550976/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1780088131 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2219.489468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2219.494425] Lustre: Skipped 2 previous similar messages [ 2223.584180] Lustre: 138717:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780088120/real 1780088120] req@0000000061430286 x1866557637552768/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1780088136 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2224.609614] LustreError: 140499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2224.613642] LustreError: 140499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2226.164718] LustreError: 140523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2228.704126] Lustre: 138718:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780088125/real 1780088125] req@0000000020a59f58 x1866557637553920/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1780088141 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2229.729891] LustreError: 140494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2229.735601] LustreError: 140494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2230.652410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2230.694548] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2230.698933] Lustre: 140491:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.149@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2230.775746] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2230.794074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2230.812354] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2231.284267] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2232.063949] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2235.880361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2235.887816] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2235.915190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2240.999675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2241.003082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2241.004532] LustreError: Skipped 1 previous similar message [ 2241.011393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2241.014412] Lustre: Skipped 1 previous similar message [ 2243.020334] Lustre: server umount lustre-OST0000 complete [ 2246.114127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2246.114356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2246.114880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2246.119137] Lustre: Skipped 2 previous similar messages [ 2250.460879] Lustre: server umount lustre-MDT0000 complete [ 2251.233117] LustreError: 140499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2251.238523] LustreError: 140499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2251.730140] LustreError: 140481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088164 with bad export cookie 7095694986903294285 [ 2251.732824] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2251.734630] LustreError: 140481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2251.853171] Lustre: server umount lustre-MDT0001 complete [ 2255.490947] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 2256.505491] Key type lgssc unregistered [ 2256.629700] LNet: 145708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2256.631963] LNetError: 145708:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2256.638302] LNet: Removed LNI 192.168.201.149@tcp [ 2256.928620] Key type .llcrypt unregistered [ 2256.929907] Key type ._llcrypt unregistered [ 2277.077513] Key type ._llcrypt registered [ 2277.078643] Key type .llcrypt registered [ 2277.121598] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 2277.536615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2277.575166] alg: No test for adler32 (adler32-zlib) [ 2278.417866] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 2278.511091] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 2280.096162] Key type lgssc registered [ 2280.448629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2282.786312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2282.791148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2283.950676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2285.220670] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2288.304170] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2290.280831] LustreError: 147240:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2290.285033] LustreError: 147240:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2290.287833] LustreError: 147240:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 2290.414234] Lustre: server umount lustre-MDT0000 complete [ 2295.414355] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 2298.647900] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 2310.304427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2310.389461] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2310.401789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2310.441315] Lustre: lustre-MDT0000: new disk, initializing [ 2310.463360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2310.470082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2311.690043] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2315.683729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2315.734442] Lustre: 150389:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2315.764723] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2315.768028] Lustre: Skipped 1 previous similar message [ 2315.808056] Lustre: lustre-MDT0001: new disk, initializing [ 2315.840382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2315.852026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2315.855634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2317.130984] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2320.255524] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2321.508491] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2323.824978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2323.914598] Lustre: lustre-OST0000: new disk, initializing [ 2323.915938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2323.917826] Lustre: 152130:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2323.940084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2325.561073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2325.564042] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2325.595809] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2325.689272] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2328.682771] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2330.488603] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2331.040368] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 16:57:23 (1780088243) [ 2397.152740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2397.152908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2397.158025] Lustre: Skipped 2 previous similar messages [ 2397.160120] Lustre: Skipped 1 previous similar message [ 2402.272561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2402.275644] Lustre: Skipped 3 previous similar messages [ 2402.467063] Lustre: server umount lustre-MDT0000 complete [ 2403.731457] LustreError: 150379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088316 with bad export cookie 3244082149573674035 [ 2403.732518] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2403.736146] LustreError: 150379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2407.392431] LustreError: 150396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2407.392598] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2407.399330] LustreError: 150396:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2407.403046] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2408.416498] LustreError: 150394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.536507] LustreError: 153775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.536573] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2413.546441] Lustre: Skipped 1 previous similar message [ 2418.144202] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2418.225512] Lustre: server umount lustre-MDT0001 complete [ 2425.608976] Lustre: server umount lustre-OST0000 complete [ 2428.637148] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 2431.867145] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 2443.541737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2443.628423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2443.637724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2443.671266] Lustre: lustre-MDT0000: new disk, initializing [ 2443.694369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2443.700957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2444.960626] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2449.012574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2449.055193] Lustre: 156608:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2449.082461] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2449.084790] Lustre: Skipped 1 previous similar message [ 2449.122291] Lustre: lustre-MDT0001: new disk, initializing [ 2449.169418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2449.174370] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2450.450695] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2453.447809] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2454.661569] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2456.992279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2457.095766] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2457.098151] Lustre: 158349:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2457.128980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2457.132031] Lustre: Skipped 1 previous similar message [ 2459.034883] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2462.033477] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2463.224438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2463.228194] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2463.260215] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2463.372598] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2463.445762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2464.768227] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2464.844045] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2468.324483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2468.325360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2468.327353] LustreError: Skipped 1 previous similar message [ 2468.333454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2468.335825] Lustre: Skipped 2 previous similar messages [ 2471.574275] Lustre: server umount lustre-OST0000 complete [ 2473.441596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2473.444936] Lustre: Skipped 2 previous similar messages [ 2478.560848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2478.562232] Lustre: Skipped 3 previous similar messages [ 2487.264122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2487.319977] Lustre: server umount lustre-MDT0000 complete [ 2487.777246] LustreError: 156618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.781527] LustreError: 156618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2488.556400] LustreError: 157386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088400 with bad export cookie 3244082149573832851 [ 2488.559059] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2488.559971] LustreError: 157386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2488.676068] Lustre: server umount lustre-MDT0001 complete [ 2491.297328] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:00:04 (1780088404) [ 2496.129448] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 2498.902979] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 2510.193853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2510.298333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2510.309013] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2510.347414] Lustre: lustre-MDT0000: new disk, initializing [ 2510.348765] Lustre: Skipped 1 previous similar message [ 2510.371087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2510.377695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2511.671727] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2515.949446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.989405] Lustre: 162917:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2516.082594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2517.320111] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2520.269591] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2521.364857] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2523.464714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2523.578304] Lustre: 164667:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2525.362347] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2528.454082] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2533.369696] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2533.403927] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2538.465521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2538.468235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2538.468572] LustreError: Skipped 1 previous similar message [ 2538.469324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2538.469327] Lustre: Skipped 2 previous similar messages [ 2538.473629] Lustre: Skipped 1 previous similar message [ 2541.650338] Lustre: server umount lustre-OST0000 complete [ 2549.013748] Lustre: server umount lustre-MDT0000 complete [ 2550.279432] LustreError: 163700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088462 with bad export cookie 3244082149573833719 [ 2550.282181] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2550.284216] LustreError: 163700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2550.403352] Lustre: server umount lustre-MDT0001 complete [ 2553.070248] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:01:05 (1780088465) [ 2557.965947] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 2560.799121] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 2571.980996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2572.065301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2572.075714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2572.078040] Lustre: Skipped 3 previous similar messages [ 2572.111106] Lustre: lustre-MDT0000: new disk, initializing [ 2572.112797] Lustre: Skipped 2 previous similar messages [ 2572.135121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2572.137009] Lustre: Skipped 2 previous similar messages [ 2572.142943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2572.145835] Lustre: Skipped 2 previous similar messages [ 2573.384871] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2577.334888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2577.378853] Lustre: 168851:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2577.471759] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2578.678171] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2581.548251] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2582.597594] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2584.615330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2584.707063] Lustre: 170603:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2586.394416] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2588.633206] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2589.375272] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2593.763260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2593.765597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2593.766613] LustreError: Skipped 1 previous similar message [ 2593.770954] Lustre: Skipped 2 previous similar messages [ 2593.773930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2593.776705] Lustre: Skipped 6 previous similar messages [ 2597.335708] Lustre: server umount lustre-OST0000 complete [ 2604.762600] Lustre: server umount lustre-MDT0000 complete [ 2606.023181] LustreError: 169634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088518 with bad export cookie 3244082149573834804 [ 2606.026070] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2606.027639] LustreError: 169634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2606.138144] Lustre: server umount lustre-MDT0001 complete [ 2609.865372] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 2610.832638] Key type lgssc unregistered [ 2610.960360] LNet: 172322:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2610.962766] LNetError: 172322:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2610.971286] LNet: Removed LNI 192.168.201.149@tcp [ 2611.272502] Key type .llcrypt unregistered [ 2611.273907] Key type ._llcrypt unregistered [ 2620.488734] Key type ._llcrypt registered [ 2620.489794] Key type .llcrypt registered [ 2620.530304] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 2620.974528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2621.006357] alg: No test for adler32 (adler32-zlib) [ 2621.863198] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 2621.944232] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 2623.520155] Key type lgssc registered [ 2623.921823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2626.516119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2626.521641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2627.689300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2628.986193] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2631.874580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2632.003901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2633.420169] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2635.269995] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2636.552365] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2638.909577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2639.029369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2640.967745] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2642.865132] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2653.352788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2673.122319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2673.124786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2673.128424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2674.146579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2674.146863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2674.147109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2674.148522] Lustre: Skipped 1 previous similar message [ 2676.044716] Lustre: server umount lustre-OST0000 complete [ 2678.240703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2678.242716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2678.246454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2683.351201] Lustre: server umount lustre-MDT0000 complete [ 2684.385592] LustreError: 175856:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2684.391163] LustreError: 175856:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2684.654097] LustreError: 173678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088597 with bad export cookie 4387544160198961181 [ 2684.657542] LustreError: 173678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2684.657590] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2684.779644] Lustre: server umount lustre-MDT0001 complete [ 2688.400170] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 2689.393149] Key type lgssc unregistered [ 2689.525399] LNet: 176895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2689.527667] LNetError: 176895:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2689.534325] LNet: Removed LNI 192.168.201.149@tcp [ 2689.821947] Key type .llcrypt unregistered [ 2689.822917] Key type ._llcrypt unregistered [ 2698.820294] Key type ._llcrypt registered [ 2698.821538] Key type .llcrypt registered [ 2698.855352] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 2699.160327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2699.206273] alg: No test for adler32 (adler32-zlib) [ 2700.070055] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 2700.147572] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 2701.728153] Key type lgssc registered [ 2702.066707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2704.589735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2704.595692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2705.743465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2707.023755] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2709.824196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2709.957883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2711.185542] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2712.877397] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2714.063603] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2716.190994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2716.294462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2718.112491] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2719.782685] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2730.665607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2740.733888] Lustre: lustre-OST0000: Not available for connect from 192.168.201.49@tcp (stopping) [ 2741.219507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2741.219746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.222113] LustreError: Skipped 1 previous similar message [ 2743.886160] Lustre: server umount lustre-OST0000 complete [ 2745.702873] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9f8044985000.ost_server_uuid 50 [ 2746.248638] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9f8044985000.ost_server_uuid in DISCONN state after 0 sec [ 2746.834171] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2760.993802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2760.997313] Lustre: Skipped 1 previous similar message [ 2760.999243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2761.001211] Lustre: Skipped 3 previous similar messages [ 2763.038721] Lustre: server umount lustre-MDT0000 complete [ 2764.248182] LustreError: 178250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088676 with bad export cookie 14669783504223630976 [ 2764.251550] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2764.251995] LustreError: 178250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2764.359189] Lustre: server umount lustre-MDT0001 complete [ 2767.222701] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 17:04:40 (1780088680) [ 2769.455193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2769.622319] LustreError: 181796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.628351] LustreError: 181796:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2769.642408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2770.874892] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2773.603031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2773.729042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2775.015036] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2776.812629] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2778.000552] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2780.189098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2782.077084] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2783.817912] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2786.203439] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2786.270924] Lustre: lustre-OST0001: new disk, initializing [ 2786.272271] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2786.274257] Lustre: 184296:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2786.299859] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2786.301593] Lustre: Skipped 1 previous similar message [ 2788.006013] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2790.925774] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2791.419288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2791.421996] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2791.445740] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2795.560303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2800.096842] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2800.098802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2800.102687] Lustre: Skipped 1 previous similar message [ 2800.104531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2803.258123] Lustre: server umount lustre-OST0000 complete [ 2805.088055] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2805.155897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2805.731379] LustreError: 183389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2805.737893] LustreError: 183389:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2806.381692] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2806.451979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2806.773115] LustreError: 184900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2810.850219] LustreError: 183390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2810.855911] LustreError: 183390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2815.970137] LustreError: 184901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2815.975710] LustreError: 184901:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2816.993068] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2816.993190] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2816.995924] LustreError: Skipped 2 previous similar messages [ 2816.996505] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2816.999600] Lustre: Skipped 2 previous similar messages [ 2817.001324] Lustre: Skipped 2 previous similar messages [ 2821.093959] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2821.096101] Lustre: Skipped 1 previous similar message [ 2822.474671] Lustre: server umount lustre-OST0001 complete [ 2825.184784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2825.187147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2829.783267] Lustre: server umount lustre-MDT0000 complete [ 2830.956872] LustreError: 183411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088743 with bad export cookie 14669783504223632033 [ 2830.959916] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2830.959917] LustreError: 183411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2831.061298] Lustre: server umount lustre-MDT0001 complete [ 2838.339968] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 17:05:51 (1780088751) [ 2840.437868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2840.524662] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2840.535936] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2840.607860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2841.845304] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2844.615434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2844.642448] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2844.654575] Lustre: 187559:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2845.958787] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2847.631362] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2848.731489] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2850.811739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2850.870160] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2852.710739] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2854.371377] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2855.945622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 2856.386168] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2856.408985] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2858.159926] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2859.922212] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2861.025279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2861.025376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2861.027857] LustreError: Skipped 1 previous similar message [ 2861.031931] Lustre: Skipped 1 previous similar message [ 2861.034042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2861.036768] Lustre: Skipped 4 previous similar messages [ 2865.184055] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 2875.360228] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2875.400185] Lustre: server umount lustre-OST0000 complete [ 2880.484944] LustreError: 189184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2880.490787] LustreError: 189184:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2890.728705] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2890.729027] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2890.731336] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2890.731342] Lustre: Skipped 9 previous similar messages [ 2890.734218] Lustre: Skipped 1 previous similar message [ 2890.736449] LustreError: Skipped 1 previous similar message [ 2892.119643] Lustre: server umount lustre-OST0001 complete [ 2899.546184] Lustre: server umount lustre-MDT0000 complete [ 2900.849181] LustreError: 187553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088813 with bad export cookie 14669783504223633419 [ 2900.853377] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2900.853424] LustreError: 187553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2900.976518] Lustre: server umount lustre-MDT0001 complete [ 2908.883950] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 17:07:01 (1780088821) [ 2913.776275] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 2916.676174] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 2928.094933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2928.183498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2928.192899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2928.229098] Lustre: lustre-MDT0000: new disk, initializing [ 2928.250486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2928.252677] Lustre: Skipped 3 previous similar messages [ 2928.258717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2929.537089] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2933.518578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2933.556663] Lustre: 195140:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2933.578949] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2933.581279] Lustre: Skipped 1 previous similar message [ 2933.620525] Lustre: lustre-MDT0001: new disk, initializing [ 2933.670862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2933.674910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2934.851453] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2937.764965] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2938.866126] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2940.928454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2941.022660] Lustre: lustre-OST0000: new disk, initializing [ 2941.024397] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2941.026979] Lustre: 196880:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2942.854153] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2944.695754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2944.699680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2944.740592] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2945.829877] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2947.099726] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2947.167046] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2948.361211] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2948.427103] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2950.113731] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2950.115928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2950.116187] LustreError: Skipped 2 previous similar messages [ 2950.120131] Lustre: Skipped 2 previous similar messages [ 2950.121763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2950.123927] Lustre: Skipped 4 previous similar messages [ 2955.151430] Lustre: server umount lustre-OST0000 complete [ 2962.393963] Lustre: server umount lustre-MDT0000 complete [ 2963.544272] LustreError: 195131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088875 with bad export cookie 14669783504223635162 [ 2963.547142] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2963.548508] LustreError: 195131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2963.654378] Lustre: server umount lustre-MDT0001 complete [ 2966.369865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2966.529312] LustreError: 198946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.536838] LustreError: 198946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2967.855170] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2970.763635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2972.196106] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2973.932651] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2975.082574] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2977.224267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2979.120733] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 2980.795752] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2982.083096] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2991.313252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2992.581765] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2992.659220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2995.683186] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2995.686385] LustreError: Skipped 1 previous similar message [ 2999.373658] Lustre: server umount lustre-OST0000 complete [ 3001.231112] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3002.544102] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3025.999077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3026.094363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3026.097233] Lustre: Skipped 5 previous similar messages [ 3027.823312] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3029.491163] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3030.016420] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3036.131495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3036.133699] Lustre: Skipped 6 previous similar messages [ 3037.069223] Lustre: server umount lustre-OST0000 complete [ 3041.249667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3041.254069] Lustre: Skipped 4 previous similar messages [ 3044.309200] Lustre: server umount lustre-MDT0000 complete [ 3045.455154] LustreError: 198928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780088957 with bad export cookie 14669783504223636030 [ 3045.458200] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3045.458892] LustreError: 198928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3045.562938] Lustre: server umount lustre-MDT0001 complete [ 3048.152033] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 17:09:21 (1780088961) [ 3050.200605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3050.362518] LustreError: 204443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3051.553115] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3054.076523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3055.486906] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3057.129431] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3058.209339] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3060.265228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3062.052149] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3063.758848] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3065.121866] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3065.200054] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3066.514260] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3066.592890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3068.721933] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3068.806987] Lustre: lustre-OST0001: new disk, initializing [ 3068.809319] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3068.812631] Lustre: 207437:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3070.760806] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3073.756176] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3074.556460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3074.561079] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3074.587647] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3075.207684] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3075.284414] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3076.643265] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3076.718270] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3079.657321] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3079.660014] LustreError: Skipped 1 previous similar message [ 3083.452879] Lustre: server umount lustre-OST0001 complete [ 3085.411509] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3085.481770] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3086.816588] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3086.892736] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3110.559204] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3112.491502] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3114.268143] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3114.808952] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3121.527649] Lustre: server umount lustre-OST0001 complete [ 3124.196318] LustreError: 206039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3124.201351] LustreError: 206039:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 3139.597383] Lustre: server umount lustre-OST0000 complete [ 3147.033789] Lustre: server umount lustre-MDT0000 complete [ 3148.243071] LustreError: 206061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780089060 with bad export cookie 14669783504223636961 [ 3148.246303] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3148.247071] LustreError: 206061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3154.411740] Lustre: server umount lustre-MDT0001 complete [ 3162.117489] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 17:11:14 (1780089074) [ 3164.375973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3164.491378] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3164.501510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3164.586053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3164.588401] Lustre: Skipped 5 previous similar messages [ 3165.958929] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3168.830513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3168.866058] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3168.877205] Lustre: 212375:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3170.342850] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3172.278966] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3173.491238] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3175.803178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3175.874110] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3177.966746] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3179.846180] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3183.721306] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3185.692886] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3187.484152] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3188.014757] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3189.229798] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3189.295207] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3190.490732] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3190.554857] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3191.666172] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f8045465800.ost_server_uuid 50 [ 3192.151083] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f8045465800.ost_server_uuid in FULL state after 0 sec [ 3192.467749] Lustre: Permanently deactivating lustre-OST0001 [ 3192.472244] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3193.237363] Lustre: Permanently reactivating lustre-OST0001 [ 3195.362644] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3195.366030] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3195.366880] Lustre: Skipped 8 previous similar messages [ 3195.368764] Lustre: Skipped 18 previous similar messages [ 3200.252946] Lustre: server umount lustre-OST0001 complete [ 3207.516490] Lustre: server umount lustre-OST0000 complete [ 3210.208921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3210.213206] LustreError: Skipped 5 previous similar messages [ 3215.003157] Lustre: server umount lustre-MDT0000 complete [ 3216.270128] LustreError: 212365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780089128 with bad export cookie 14669783504223638221 [ 3216.273470] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3216.273573] LustreError: 212365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3216.399180] Lustre: server umount lustre-MDT0001 complete [ 3224.049442] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 17:12:16 (1780089136) [ 3226.543030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3226.635791] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3226.646307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3226.649310] Lustre: Skipped 5 previous similar messages [ 3227.947243] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3230.813108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3230.847899] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3230.850234] Lustre: Skipped 1 previous similar message [ 3230.861185] Lustre: 218473:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3232.346912] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3234.189271] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3235.436143] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3237.775218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3237.861480] Lustre: Permanently deactivating lustre-OST0000 [ 3237.866686] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3239.713758] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3241.479909] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3243.509454] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3243.512858] LustreError: 220958:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -108 [ 3243.754084] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3245.738724] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3247.608234] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3252.318186] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3254.108662] Lustre: Permanently reactivating lustre-OST0000 [ 3254.115643] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 3254.120535] Lustre: Skipped 2 previous similar messages [ 3260.946865] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3260.949693] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3260.960375] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3260.979243] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3260.987227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3273.627767] Lustre: server umount lustre-OST0001 complete [ 3277.806086] LustreError: 220091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3277.826892] LustreError: 220091:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 3283.527394] Lustre: server umount lustre-OST0000 complete [ 3302.368372] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3302.804403] Lustre: server umount lustre-MDT0000 complete [ 3308.228068] LustreError: 221007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780089220 with bad export cookie 14669783504223639852 [ 3308.232552] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3308.246553] LustreError: 221007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3308.902810] Lustre: server umount lustre-MDT0001 complete [ 3319.461435] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 3321.257633] Key type lgssc unregistered [ 3321.454813] LNet: 222876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3321.458966] LNetError: 222876:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3322.472366] LNet: Removed LNI 192.168.201.149@tcp [ 3323.098482] Key type .llcrypt unregistered [ 3323.100562] Key type ._llcrypt unregistered [ 3337.701658] Key type ._llcrypt registered [ 3337.702989] Key type .llcrypt registered [ 3337.758622] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 3338.391569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3338.410144] alg: No test for adler32 (adler32-zlib) [ 3339.329795] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 3339.502300] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 3341.120233] Key type lgssc registered [ 3341.659622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3346.876240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3346.888526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3348.183991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3350.455122] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3354.943896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3355.015533] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3355.030840] Lustre: 224290:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3355.037442] Lustre: Permanently deactivating lustre-MDT0001 [ 3355.044469] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3355.217466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3357.116386] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3360.487387] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3360.777232] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3362.936958] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3366.796721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3367.004980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3370.086841] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3373.156550] LustreError: 225918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3373.172316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3373.172468] LustreError: 225918:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3373.464614] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3377.437262] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3377.571271] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3379.633785] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3379.639470] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 3379.655737] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3381.042385] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3384.363657] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3387.558549] Lustre: Permanently reactivating lustre-MDT0001 [ 3387.565672] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3387.567727] Lustre: Skipped 2 previous similar messages [ 3391.194514] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3391.202336] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 3391.213750] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 3391.219811] LustreError: 227412:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 3391.220241] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3391.225191] LustreError: 227412:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 3391.231359] LustreError: 227412:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.149@tcp: cfg command failed: rc = -108 [ 3391.235892] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 3391.235892] [ 3391.243412] LustreError: 224295:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3394.575477] Lustre: lustre-MDT0001: Client 4b521bfa-1458-40b3-a11f-5f26fc937f78 (at 192.168.201.49@tcp) reconnecting [ 3396.768885] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3397.591887] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3400.288366] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3400.394816] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3402.707259] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3402.832929] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3404.219370] Lustre: Permanently deactivating lustre-MDT0001 [ 3404.226491] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3404.228654] Lustre: Skipped 2 previous similar messages [ 3412.502093] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3416.548203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3416.551158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3416.572510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3421.669836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3421.677421] Lustre: Skipped 6 previous similar messages [ 3422.881979] Lustre: server umount lustre-MDT0000 complete [ 3425.474154] LustreError: 226814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780089337 with bad export cookie 11582239652877804035 [ 3425.480836] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3425.486412] LustreError: 226814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3425.702123] Lustre: server umount lustre-MDT0001 complete [ 3434.189197] Lustre: server umount lustre-OST0000 complete [ 3440.708551] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 17:15:53 (1780089353) [ 3447.317503] Lustre: server umount lustre-OST0001 complete [ 3451.145802] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 3456.310365] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 3474.994819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3475.163592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3475.168960] Lustre: Skipped 2 previous similar messages [ 3475.189193] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3475.240421] Lustre: lustre-MDT0000: new disk, initializing [ 3475.281086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3475.289284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3477.317719] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3482.898617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3482.948928] Lustre: 231936:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3482.977789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3482.980109] Lustre: Skipped 1 previous similar message [ 3483.019519] Lustre: lustre-MDT0001: new disk, initializing [ 3483.073860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3483.077606] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3484.743310] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3488.855661] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3490.666364] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3493.954688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3494.125872] Lustre: lustre-OST0000: new disk, initializing [ 3494.128330] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3494.131488] Lustre: 233678:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3494.163602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3494.166653] Lustre: Skipped 1 previous similar message [ 3496.702811] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3500.710781] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3503.151273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3503.156530] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3503.211211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3507.150613] LustreError: 231943:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3512.166343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3512.249863] Lustre: lustre-OST0001: new disk, initializing [ 3512.251525] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3512.254540] Lustre: 234681:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3512.289585] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3515.187609] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3517.168197] LustreError: 231943:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3517.186230] LustreError: 231944:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3518.987824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3518.998196] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3519.058851] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3519.556801] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3520.521931] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3527.288120] LustreError: 231944:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3529.187572] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3529.189637] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3529.191354] LustreError: Skipped 1 previous similar message [ 3529.196374] Lustre: Skipped 3 previous similar messages [ 3529.199151] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3529.201571] Lustre: Skipped 1 previous similar message [ 3534.305296] Lustre: server umount lustre-OST0001 complete [ 3534.311209] LustreError: 234284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3534.318475] LustreError: 234284:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3539.429546] LustreError: 233661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3539.436772] LustreError: 233661:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3544.552022] LustreError: 233662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3544.558565] LustreError: 233662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3549.154517] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3549.155641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3549.159071] LustreError: Skipped 1 previous similar message [ 3549.159826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3549.159830] Lustre: Skipped 1 previous similar message [ 3549.182425] Lustre: Skipped 1 previous similar message [ 3549.674210] LustreError: 233661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3549.689530] LustreError: 233661:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3553.559554] Lustre: server umount lustre-OST0000 complete [ 3559.909938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3559.913801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3559.922958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3559.935415] Lustre: Skipped 2 previous similar messages [ 3561.775727] Lustre: server umount lustre-MDT0000 complete [ 3563.847119] LustreError: 232723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780089476 with bad export cookie 11582239652877807087 [ 3563.848887] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3563.856661] LustreError: 232723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3578.337073] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3578.503804] Lustre: server umount lustre-MDT0001 complete [ 3584.457709] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 3585.838569] Key type lgssc unregistered [ 3586.018398] LNet: 236583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3586.022831] LNetError: 236583:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3586.038927] LNet: Removed LNI 192.168.201.149@tcp [ 3586.472036] Key type .llcrypt unregistered [ 3586.473690] Key type ._llcrypt unregistered [ 3607.631027] Key type ._llcrypt registered [ 3607.632696] Key type .llcrypt registered [ 3607.695539] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 3608.200870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3608.212068] alg: No test for adler32 (adler32-zlib) [ 3609.101335] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 3609.235137] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 3610.832200] Key type lgssc registered [ 3611.428391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3616.111313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3616.128666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3617.290099] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3617.316144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3617.433556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3619.521874] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3624.127055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3624.181102] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3624.199530] Lustre: 238752:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3624.335418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3626.417549] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3628.783573] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3630.671628] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3633.880808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3633.973637] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3634.060125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3637.184439] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3640.109908] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3642.978841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3688.930145] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3688.933368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3688.940132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3689.442718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3689.443650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3689.444272] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3693.038927] Lustre: server umount lustre-OST0000 complete [ 3695.441981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3702.396573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3702.549282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3705.134846] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3707.871130] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3709.733740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3709.738043] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3714.862153] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3714.870639] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3745.763911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3745.771171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3745.773419] LustreError: Skipped 1 previous similar message [ 3745.782555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3745.788729] Lustre: Skipped 2 previous similar messages [ 3748.547833] Lustre: server umount lustre-OST0000 complete [ 3750.882771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3750.884268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3750.888920] Lustre: Skipped 2 previous similar messages [ 3750.894051] Lustre: Skipped 1 previous similar message [ 3756.001486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3756.006568] Lustre: Skipped 1 previous similar message [ 3756.747794] Lustre: server umount lustre-MDT0000 complete [ 3758.519272] LustreError: 239485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780089670 with bad export cookie 16697972930604682294 [ 3758.522109] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3758.524608] LustreError: 239485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3758.721112] Lustre: server umount lustre-MDT0001 complete [ 3763.509256] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 3764.832841] Key type lgssc unregistered [ 3764.974315] LNet: 243704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3764.977180] LNetError: 243704:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3764.987334] LNet: Removed LNI 192.168.201.149@tcp [ 3765.327855] Key type .llcrypt unregistered [ 3765.329548] Key type ._llcrypt unregistered [ 3775.667095] Key type ._llcrypt registered [ 3775.668773] Key type .llcrypt registered [ 3775.707920] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 3776.108792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3776.131085] alg: No test for adler32 (adler32-zlib) [ 3776.993605] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 3777.085844] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 3778.672185] Key type lgssc registered [ 3779.106816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3782.497451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3782.509849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3783.797090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3785.878813] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3789.878654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3790.046995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3791.730704] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3794.200170] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3796.035349] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3799.081892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3799.263432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3801.870959] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3804.537700] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3809.510892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3820.003755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3820.003935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3820.014105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3820.016828] Lustre: Skipped 1 previous similar message [ 3823.010346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3823.012975] Lustre: Skipped 1 previous similar message [ 3825.554472] Lustre: server umount lustre-OST0000 complete [ 3828.193511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3828.194168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3828.197245] Lustre: Skipped 2 previous similar messages [ 3833.245785] Lustre: server umount lustre-MDT0000 complete [ 3833.313876] LustreError: 245801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3833.323558] LustreError: 245801:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3834.762475] LustreError: 245062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780089747 with bad export cookie 18207604967631005631 [ 3834.765248] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3834.767874] LustreError: 245062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3834.911885] Lustre: server umount lustre-MDT0001 complete [ 3839.246636] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 3840.394417] Key type lgssc unregistered [ 3840.520430] LNet: 249025:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3840.524229] LNetError: 249025:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3840.532642] LNet: Removed LNI 192.168.201.149@tcp [ 3840.888583] Key type .llcrypt unregistered [ 3840.890361] Key type ._llcrypt unregistered [ 3848.288749] Key type ._llcrypt registered [ 3848.290204] Key type .llcrypt registered [ 3848.343072] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 3848.767495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3848.808394] alg: No test for adler32 (adler32-zlib) [ 3849.681870] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 3849.782042] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 3851.368151] Key type lgssc registered [ 3851.773724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3854.587433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3854.598794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3855.769897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3857.215303] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3860.420696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3860.611144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3862.322744] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3864.669429] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3866.185660] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3869.265810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3869.476674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3871.837937] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3874.221153] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3880.743902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3891.170547] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3891.173274] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3891.174015] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3891.177449] Lustre: Skipped 1 previous similar message [ 3891.182606] Lustre: Skipped 1 previous similar message [ 3895.397446] Lustre: server umount lustre-OST0000 complete [ 3901.411317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3901.414959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3901.419706] Lustre: Skipped 1 previous similar message [ 3903.202521] Lustre: server umount lustre-MDT0000 complete [ 3905.053869] LustreError: 250187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780089817 with bad export cookie 8094492102382531564 [ 3905.056748] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3905.062899] LustreError: 250187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3905.222750] Lustre: server umount lustre-MDT0001 complete [ 3910.553691] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 3911.954904] Key type lgssc unregistered [ 3912.121485] LNet: 253548:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3912.127227] LNetError: 253548:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3912.141408] LNet: Removed LNI 192.168.201.149@tcp [ 3912.561226] Key type .llcrypt unregistered [ 3912.562851] Key type ._llcrypt unregistered [ 3924.232555] Key type ._llcrypt registered [ 3924.233889] Key type .llcrypt registered [ 3924.288438] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 3924.803348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3924.812957] alg: No test for adler32 (adler32-zlib) [ 3925.711623] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 3925.818815] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 3927.416824] Key type lgssc registered [ 3928.031408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3931.478757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3931.487335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3932.800401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3934.571496] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3938.299813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3938.516387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3940.797408] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3943.335770] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3944.756859] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3947.470523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3947.660503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3949.951249] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 3952.227224] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3957.931119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3968.484369] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3968.485194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3968.488111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3968.496803] Lustre: Skipped 1 previous similar message [ 3969.496461] Lustre: server umount lustre-OST0000 complete [ 3973.283246] Lustre: server umount lustre-MDT0000 complete [ 3973.604890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3973.606071] LustreError: 256538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3973.609354] Lustre: Skipped 1 previous similar message [ 3973.617978] LustreError: 256538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3975.173919] LustreError: 254905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780089887 with bad export cookie 3994322964168251372 [ 3975.189438] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3975.379724] Lustre: server umount lustre-MDT0001 complete [ 3979.690333] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 3980.811477] Key type lgssc unregistered [ 3980.954710] LNet: 258864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3980.961322] LNetError: 258864:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3980.972090] LNet: Removed LNI 192.168.201.149@tcp [ 3981.316869] Key type .llcrypt unregistered [ 3981.318525] Key type ._llcrypt unregistered [ 3989.494094] Key type ._llcrypt registered [ 3989.495315] Key type .llcrypt registered [ 3989.545904] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 3989.996420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3990.080825] alg: No test for adler32 (adler32-zlib) [ 3990.991375] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 3991.103933] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 3992.713560] Key type lgssc registered [ 3993.290523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3997.107649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3997.120838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3998.391480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4000.160135] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4004.046898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4004.211723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4005.992768] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4008.372473] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4010.046599] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4013.262076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4013.499383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4015.961335] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4018.459483] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4022.704403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4029.924272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4029.927688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4029.933980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4032.993997] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4032.994622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4032.997081] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4033.007612] Lustre: Skipped 1 previous similar message [ 4038.114471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4038.117582] Lustre: Skipped 1 previous similar message [ 4042.208104] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4042.278499] Lustre: server umount lustre-OST0000 complete [ 4045.280945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4045.284189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4045.289895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4048.355333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4053.473898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4053.476165] Lustre: Skipped 3 previous similar messages [ 4058.080449] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4058.148962] Lustre: server umount lustre-MDT0000 complete [ 4058.593452] LustreError: 260070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4058.601440] LustreError: 260070:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4059.796415] LustreError: 260028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780089972 with bad export cookie 6069076443497077212 [ 4059.799526] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4059.804688] LustreError: 260028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4059.951390] Lustre: server umount lustre-MDT0001 complete [ 4064.495372] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 4065.792513] Key type lgssc unregistered [ 4065.952915] LNet: 263398:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4065.955442] LNetError: 263398:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4065.963526] LNet: Removed LNI 192.168.201.149@tcp [ 4066.346991] Key type .llcrypt unregistered [ 4066.348284] Key type ._llcrypt unregistered [ 4070.615155] Key type ._llcrypt registered [ 4070.616437] Key type .llcrypt registered [ 4070.666191] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4077.363419] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 4084.333792] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 4084.703523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4084.753888] alg: No test for adler32 (adler32-zlib) [ 4085.626744] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 4085.719453] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 4087.328224] Key type lgssc registered [ 4087.764834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4100.820455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4100.827748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4101.967910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4101.982812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4102.026100] Lustre: lustre-MDT0000: new disk, initializing [ 4102.065492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4102.074501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4103.611939] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4108.319232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4108.371229] Lustre: 266871:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4108.398834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4108.400943] Lustre: Skipped 1 previous similar message [ 4108.440579] Lustre: lustre-MDT0001: new disk, initializing [ 4108.475081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4108.488043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4108.492283] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4109.936077] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4113.381824] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4114.812618] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4117.698852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4117.842753] Lustre: lustre-OST0000: new disk, initializing [ 4117.845282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4117.848041] Lustre: 268613:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4117.892203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4120.062125] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4123.555543] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4125.113548] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4125.178475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4125.182598] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4125.221886] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4126.225304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4127.814309] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4127.901559] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4130.273478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4130.276646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4130.279738] LustreError: Skipped 1 previous similar message [ 4130.280615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4130.284510] Lustre: Skipped 1 previous similar message [ 4134.815810] Lustre: server umount lustre-OST0000 complete [ 4139.489361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4139.492863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4139.499080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4139.501027] Lustre: Skipped 1 previous similar message [ 4140.513827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4140.514107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4140.523565] Lustre: Skipped 1 previous similar message [ 4142.557905] Lustre: server umount lustre-MDT0000 complete [ 4143.927223] LustreError: 267648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780090056 with bad export cookie 4171409515040633708 [ 4143.930647] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4143.932503] LustreError: 267648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4144.058609] Lustre: server umount lustre-MDT0001 complete [ 4147.146525] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 17:27:39 (1780090059) [ 4149.737941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4149.959313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4151.525587] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4155.180809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4155.385842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4157.294897] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4160.812247] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4164.849283] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4172.686664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4173.141419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4178.421828] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4185.717962] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4200.936190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4200.941164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4200.977922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4200.987101] LustreError: Skipped 1 previous similar message [ 4201.001834] Lustre: Skipped 1 previous similar message [ 4202.888166] Lustre: server umount lustre-OST0000 complete [ 4211.175685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4211.189169] Lustre: Skipped 1 previous similar message [ 4211.194990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4211.200971] Lustre: Skipped 1 previous similar message [ 4214.713696] Lustre: server umount lustre-MDT0000 complete [ 4216.302765] LustreError: 271999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4216.325676] LustreError: 271999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4219.479073] LustreError: 271599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780090131 with bad export cookie 4171409515040634590 [ 4219.482410] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4219.501244] LustreError: 271599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4220.499536] Lustre: server umount lustre-MDT0001 complete [ 4237.980954] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 4240.758738] Key type lgssc unregistered [ 4240.999911] LNet: 274061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4241.004729] LNetError: 274061:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4241.023606] LNet: Removed LNI 192.168.201.149@tcp [ 4241.688645] Key type .llcrypt unregistered [ 4241.691361] Key type ._llcrypt unregistered [ 4262.854687] Key type ._llcrypt registered [ 4262.860281] Key type .llcrypt registered [ 4262.944188] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 4274.662257] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 4275.407652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4275.474371] alg: No test for adler32 (adler32-zlib) [ 4276.497439] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 4276.634837] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 4278.256157] Key type lgssc registered [ 4279.103848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4314.024591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4314.047527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4315.386747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4315.439814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4315.529187] Lustre: lustre-MDT0000: new disk, initializing [ 4315.618338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4315.635842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4319.069448] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4327.543488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4327.656819] Lustre: 277832:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4327.702390] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4327.707176] Lustre: Skipped 1 previous similar message [ 4327.762549] Lustre: lustre-MDT0001: new disk, initializing [ 4327.848649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4327.879663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4327.885987] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4330.796729] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4336.638480] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4339.423916] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4344.108956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4344.331568] Lustre: lustre-OST2710: new disk, initializing [ 4344.334772] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4344.339365] Lustre: 279571:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4344.433684] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4349.186536] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4354.642574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4354.654028] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4354.814306] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4355.304957] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4360.462173] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4360.604241] Lustre: lustre-OST03e8: new disk, initializing [ 4360.612317] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4360.617336] Lustre: lustre-OST03e8: Not available for connect from 0@lo (not set up) [ 4360.624539] Lustre: 280521:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4360.709514] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4364.903913] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4365.824724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4365.834196] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4365.876894] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4368.672390] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4374.965664] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4375.138854] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4379.077806] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4379.287565] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4383.704647] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4383.996597] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4388.030500] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4388.208897] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4391.398515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4391.401015] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4391.418592] Lustre: Skipped 1 previous similar message [ 4391.429179] Lustre: Skipped 3 previous similar messages [ 4396.517662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4396.524483] Lustre: Skipped 1 previous similar message [ 4397.460148] Lustre: server umount lustre-MDT0000 complete [ 4400.726071] LustreError: 277823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780090313 with bad export cookie 10246440164287959684 [ 4400.733923] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4400.737334] LustreError: 277823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4401.102949] Lustre: server umount lustre-MDT0001 complete [ 4415.162922] Lustre: server umount lustre-OST2710 complete [ 4427.808605] Lustre: server umount lustre-OST03e8 complete [ 4443.598848] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 4451.625212] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 4483.742907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4484.000607] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4484.027879] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4484.160651] Lustre: lustre-MDT0000: new disk, initializing [ 4484.220148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4484.232381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4488.134120] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4498.090617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4498.207755] Lustre: 285561:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4498.244167] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4498.247160] Lustre: Skipped 1 previous similar message [ 4498.292563] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4498.297393] Lustre: Skipped 2 previous similar messages [ 4498.317867] Lustre: lustre-MDT0001: new disk, initializing [ 4498.387239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4498.448450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4498.457140] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4502.793787] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4509.992395] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4514.310663] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4520.854837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4521.156962] Lustre: lustre-OST0000: new disk, initializing [ 4521.164855] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4521.174244] Lustre: 287304:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4521.263684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4527.370917] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4531.290603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4531.298601] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4531.511598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4535.296914] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4539.898165] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4540.076260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4545.228448] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4545.463571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4552.164447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4552.166053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4552.175094] LustreError: Skipped 1 previous similar message [ 4552.178001] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4552.208884] Lustre: Skipped 3 previous similar messages [ 4554.128705] Lustre: server umount lustre-OST0000 complete [ 4559.841521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4559.845742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4562.412395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4562.415750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4562.417823] Lustre: Skipped 2 previous similar messages [ 4564.438686] Lustre: server umount lustre-MDT0000 complete [ 4567.750743] LustreError: 285553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780090480 with bad export cookie 10246440164287961357 [ 4567.758340] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4567.758983] LustreError: 285553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4568.012780] Lustre: server umount lustre-MDT0001 complete [ 4575.797663] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 17:34:47 (1780090487) [ 4577.184621] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4578.705993] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 17:34:50 (1780090490) [ 4581.340422] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 4604.904320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4605.246451] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4605.270747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4605.412638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4609.373344] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4617.709107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4617.807133] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4617.841293] Lustre: 290995:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4622.008746] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4642.669688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4642.846438] Lustre: Denying initial registration attempt from nid 192.168.201.149@tcp, specified as failover [ 4642.857602] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4642.869373] LustreError: 292414:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 4642.874385] LustreError: 292414:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 4642.879700] LustreError: 292414:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4642.956574] Lustre: server umount lustre-OST0000 complete [ 4642.959522] LustreError: 292414:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4648.930503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4648.938648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4648.953866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4648.969754] Lustre: Skipped 1 previous similar message [ 4663.264099] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4663.401131] Lustre: server umount lustre-MDT0000 complete [ 4665.313543] LustreError: 291005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4665.324000] LustreError: 291005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4666.537173] LustreError: 290990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780090578 with bad export cookie 10246440164287962246 [ 4666.546173] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4666.812937] Lustre: server umount lustre-MDT0001 complete [ 4676.764589] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 4679.287673] Key type lgssc unregistered [ 4679.597208] LNet: 293519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4679.607344] LNetError: 293519:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4680.679729] LNet: Removed LNI 192.168.201.149@tcp [ 4681.496746] Key type .llcrypt unregistered [ 4681.499487] Key type ._llcrypt unregistered [ 4691.225155] Key type ._llcrypt registered [ 4691.226511] Key type .llcrypt registered [ 4691.301107] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 4692.258653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4692.284770] alg: No test for adler32 (adler32-zlib) [ 4693.336344] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 4693.519626] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 4695.176186] Key type lgssc registered [ 4696.196638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4736.086274] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 4743.732944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4743.763945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4745.193322] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4745.212138] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4745.336733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4749.465494] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4757.475994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4757.566728] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4757.638988] Lustre: 296229:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4757.943391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4762.308893] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4767.893471] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4771.132503] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4790.514041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4790.703334] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4790.859030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4796.845575] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4801.751812] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4806.627203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4806.634231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4806.650460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4807.139916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4807.154268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4807.158308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4807.160573] Lustre: Skipped 1 previous similar message [ 4809.654498] Lustre: server umount lustre-OST0000 complete [ 4814.306414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4814.314302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4814.321362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4817.379957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4817.380459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4817.411168] Lustre: Skipped 1 previous similar message [ 4818.716876] Lustre: server umount lustre-MDT0000 complete [ 4821.363654] LustreError: 296223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780090733 with bad export cookie 274483941551921229 [ 4821.366304] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4821.374320] LustreError: 296223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4821.648717] Lustre: server umount lustre-MDT0001 complete [ 4829.221431] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 4831.212266] Key type lgssc unregistered [ 4831.445074] LNet: 299447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4831.449670] LNetError: 299447:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4831.467753] LNet: Removed LNI 192.168.201.149@tcp [ 4832.068416] Key type .llcrypt unregistered [ 4832.070058] Key type ._llcrypt unregistered [ 4850.250683] Key type ._llcrypt registered [ 4850.253675] Key type .llcrypt registered [ 4850.333574] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 4851.217148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4851.225582] alg: No test for adler32 (adler32-zlib) [ 4852.181074] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 4852.362677] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 4854.010423] Key type lgssc registered [ 4854.799899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4860.604392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4860.626341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4861.934384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4864.996893] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4870.800446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4871.041894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4873.872375] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4877.625660] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4880.267250] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4886.836988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4887.156551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4891.747963] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4895.336748] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4901.859226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4901.861939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4901.869580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4902.935594] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.49@tcp (stopping) [ 4902.945440] Lustre: Skipped 2 previous similar messages [ 4906.982669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4906.987911] Lustre: Skipped 2 previous similar messages [ 4912.097542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4912.104380] Lustre: Skipped 1 previous similar message [ 4916.192137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4916.362489] Lustre: server umount lustre-MDT0000 complete [ 4917.219556] LustreError: 300826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4917.233414] LustreError: 300826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4918.265174] LustreError: 300820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4918.984134] LustreError: 300807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780090831 with bad export cookie 12463328040828066513 [ 4918.986396] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4918.991511] LustreError: 300807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4922.337251] LustreError: 300822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4922.337665] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4922.370387] Lustre: Skipped 2 previous similar messages [ 4922.373574] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4922.376845] Lustre: Skipped 1 previous similar message [ 4925.233756] Lustre: server umount lustre-MDT0001 complete [ 4929.905283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4936.283289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4943.776957] LustreError: 300136:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000855f8ffd x1866560452982272/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4944.010431] LustreError: 302431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4944.025977] LustreError: 302431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4944.087558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4944.134956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 4947.334719] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4948.198512] LustreError: 302432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4948.205652] LustreError: 302432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4953.319643] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4953.484781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4953.761704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4957.058318] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 4959.211820] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4960.816498] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4965.252815] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4968.181409] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4969.508341] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4974.562123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4974.565796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4974.569442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4974.569449] Lustre: Skipped 2 previous similar messages [ 4974.600098] Lustre: Skipped 1 previous similar message [ 4987.361880] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4987.520740] Lustre: server umount lustre-OST0000 complete [ 4995.046353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4995.049477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4995.058572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4995.063573] Lustre: Skipped 1 previous similar message [ 4995.076505] Lustre: Skipped 5 previous similar messages [ 5005.280155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5005.488890] Lustre: server umount lustre-MDT0000 complete [ 5008.424237] LustreError: 300806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780090920 with bad export cookie 12463328040828073716 [ 5008.428273] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5008.430823] LustreError: 300806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5008.670853] Lustre: server umount lustre-MDT0001 complete [ 5017.064164] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 5019.063435] Key type lgssc unregistered [ 5019.306812] LNet: 306507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5019.311280] LNetError: 306507:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5019.342528] LNet: Removed LNI 192.168.201.149@tcp [ 5020.092311] Key type .llcrypt unregistered [ 5020.094282] Key type ._llcrypt unregistered [ 5038.405624] Key type ._llcrypt registered [ 5038.406738] Key type .llcrypt registered [ 5038.573089] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 5039.592662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5039.633064] alg: No test for adler32 (adler32-zlib) [ 5040.592196] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 5040.797085] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 5042.416168] Key type lgssc registered [ 5043.237372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5049.692187] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5049.716351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5051.182304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5054.558479] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5060.587295] LustreError: 307887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5060.619660] LustreError: 307887:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5061.841815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5062.167685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5065.569224] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5070.190992] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5073.086758] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5077.474879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5077.475046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5077.478674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5077.497452] Lustre: Skipped 1 previous similar message [ 5081.747681] Lustre: server umount lustre-MDT0000 complete [ 5082.594314] LustreError: 307886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5082.609952] LustreError: 307886:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5084.316215] LustreError: 307868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780090996 with bad export cookie 13716809176456341494 [ 5084.326940] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5084.328679] LustreError: 307868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5084.564954] Lustre: server umount lustre-MDT0001 complete [ 5094.516456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5094.681199] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5094.698306] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5094.885661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5097.909532] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5103.814941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5103.875238] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5103.911817] Lustre: 310245:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5104.080534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5107.275814] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5111.575294] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5114.090533] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5121.520486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5121.668884] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5121.676019] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5121.681381] LustreError: 311900:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 5121.685603] LustreError: 311900:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 5121.689635] LustreError: 311900:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5121.721879] Lustre: server umount lustre-OST0000 complete [ 5121.723463] LustreError: 311900:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5129.455502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5129.584192] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5129.711084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5133.697854] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5137.504699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 5137.983419] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5143.713474] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5143.770687] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5143.824737] Lustre: lustre-OST0001: new disk, initializing [ 5143.827919] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5143.833118] Lustre: 313330:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5143.891861] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5148.407792] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5152.813487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5152.826598] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5152.916460] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5152.961733] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5158.376256] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5158.378256] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5158.378959] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5158.378974] Lustre: Skipped 1 previous similar message [ 5158.418851] Lustre: Skipped 1 previous similar message [ 5161.194564] Lustre: server umount lustre-OST0001 complete [ 5163.492477] LustreError: 312366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5163.507191] LustreError: 312366:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5168.097818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5168.102024] LustreError: Skipped 1 previous similar message [ 5168.104599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5168.117258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5168.126646] Lustre: Skipped 1 previous similar message [ 5168.621470] LustreError: 312368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5168.632742] LustreError: 312368:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5170.386519] Lustre: server umount lustre-OST0000 complete [ 5173.730404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5173.743574] Lustre: Skipped 1 previous similar message [ 5173.750910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5173.755088] Lustre: Skipped 2 previous similar messages [ 5178.861171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5178.869300] Lustre: Skipped 1 previous similar message [ 5179.371940] Lustre: server umount lustre-MDT0000 complete [ 5181.888491] LustreError: 313328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780091094 with bad export cookie 13716809176456342040 [ 5181.894958] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5181.902224] LustreError: 313328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5182.197425] Lustre: server umount lustre-MDT0001 complete [ 5189.267731] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 5191.016293] Key type lgssc unregistered [ 5191.226628] LNet: 315156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5191.231857] LNetError: 315156:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5191.241794] LNet: Removed LNI 192.168.201.149@tcp [ 5191.779342] Key type .llcrypt unregistered [ 5191.780996] Key type ._llcrypt unregistered [ 5265.672821] Key type ._llcrypt registered [ 5265.674865] Key type .llcrypt registered [ 5265.852441] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 5286.458783] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 5287.741286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5288.275146] alg: No test for adler32 (adler32-zlib) [ 5289.959785] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 5290.407940] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 5292.217716] Key type lgssc registered [ 5294.841240] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5341.884207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5341.954228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5343.259494] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5343.308550] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5343.440669] Lustre: lustre-MDT0000: new disk, initializing [ 5343.558636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5343.586589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5348.149850] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5359.463045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5359.651935] Lustre: 320062:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5359.730871] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5359.734411] Lustre: Skipped 1 previous similar message [ 5359.848409] Lustre: lustre-MDT0001: new disk, initializing [ 5359.959253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5360.021963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5360.036346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5364.423720] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5372.281526] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5377.049258] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5384.051555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5384.287926] Lustre: lustre-OST0000: new disk, initializing [ 5384.292179] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5384.297833] Lustre: 321802:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5384.364244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5390.499744] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5390.900448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5390.909143] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5391.022995] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5398.540314] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5403.235274] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5403.444535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5407.428435] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5407.661769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5411.809976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5411.816996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5411.840682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5415.479706] Lustre: server umount lustre-OST0000 complete [ 5421.540871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5421.553037] LustreError: Skipped 1 previous similar message [ 5421.560047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5421.577305] Lustre: Skipped 1 previous similar message [ 5421.586816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5421.600532] Lustre: Skipped 1 previous similar message [ 5425.643604] Lustre: server umount lustre-MDT0000 complete [ 5429.388913] LustreError: 321812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780091341 with bad export cookie 3304752726090892170 [ 5429.398555] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5429.401058] LustreError: 321812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5429.904608] Lustre: server umount lustre-MDT0001 complete [ 5438.856148] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 17:49:10 (1780091350) [ 5447.004615] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 17:49:18 (1780091358) [ 5454.977413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5455.292487] LustreError: 324412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5455.308668] LustreError: 324412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5455.371852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5459.243432] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5460.451454] LustreError: 324413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5465.573815] LustreError: 324412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5468.081422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5468.411781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5472.670843] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5478.021951] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5481.690073] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5487.897983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5488.227986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5493.867124] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5499.269667] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5510.921045] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5512.511949] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5514.137288] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5515.890524] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5517.786910] Lustre: Failing over lustre-MDT0000 [ 5518.170171] Lustre: server umount lustre-MDT0000 complete [ 5519.846896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5519.864650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5519.881349] Lustre: Skipped 1 previous similar message [ 5519.887943] LustreError: 324412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5522.404680] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5522.423256] Lustre: Skipped 1 previous similar message [ 5525.495979] LustreError: 324409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5525.514996] LustreError: 324409:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5535.738342] LustreError: 324407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5535.755062] LustreError: 324407:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5536.096395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5536.236943] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5536.427266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5536.468041] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5540.660490] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5540.857328] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5541.878197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5541.908711] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5541.961834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5548.226389] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5549.905542] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5553.516338] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5555.581894] Lustre: Failing over lustre-MDT0000 [ 5555.949306] Lustre: server umount lustre-MDT0000 complete [ 5556.248054] LustreError: 324408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5557.217696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5557.223697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5557.230117] Lustre: Skipped 2 previous similar messages [ 5573.538835] Lustre: 318285:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780091469/real 1780091469] req@00000000754a600d x1866560912312704/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780091485 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5573.574109] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5574.410920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5583.842928] LustreError: 318283:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000512828c x1866560912318080/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5584.306323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5584.381418] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5586.942023] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5588.542584] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5589.496197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5589.503623] Lustre: Skipped 2 previous similar messages [ 5589.553771] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5589.590704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5596.349367] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5598.514629] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5604.836469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5604.844918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5604.872465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5604.877592] Lustre: Skipped 2 previous similar messages [ 5609.957377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5610.892648] Lustre: server umount lustre-MDT0000 complete [ 5614.965764] LustreError: 324392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780091527 with bad export cookie 3304752726090894886 [ 5614.987714] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5614.995045] LustreError: 324392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5615.075814] LustreError: 325545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5615.075839] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5615.107044] LustreError: 325545:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 27 previous similar messages [ 5615.110918] Lustre: Skipped 2 previous similar messages [ 5615.444596] Lustre: server umount lustre-MDT0001 complete [ 5621.963669] Lustre: server umount lustre-OST0000 complete [ 5638.662462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5643.538685] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5652.370868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5652.872558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5652.877377] Lustre: Skipped 1 previous similar message [ 5657.252156] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5663.698337] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5667.609412] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5673.673388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5679.549442] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5684.218841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5684.516624] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5689.317977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5689.318463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5689.330343] Lustre: Skipped 3 previous similar messages [ 5689.341823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5690.965722] Lustre: server umount lustre-OST0000 complete [ 5700.880392] Lustre: server umount lustre-MDT0000 complete [ 5704.675336] LustreError: 331196:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5704.716961] LustreError: 331196:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5705.269636] LustreError: 330189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780091617 with bad export cookie 3304752726090895628 [ 5705.284383] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5705.657414] Lustre: server umount lustre-MDT0001 complete [ 5716.677435] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 5719.913958] Key type lgssc unregistered [ 5720.165956] LNet: 333383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5720.178718] LNetError: 333383:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5721.256295] LNet: Removed LNI 192.168.201.149@tcp [ 5721.905362] Key type .llcrypt unregistered [ 5721.907781] Key type ._llcrypt unregistered [ 5751.850354] Key type ._llcrypt registered [ 5751.852218] Key type .llcrypt registered [ 5751.956685] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 5753.453908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5753.499050] alg: No test for adler32 (adler32-zlib) [ 5754.598649] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 5754.792931] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 5756.432175] Key type lgssc registered [ 5757.425783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5765.053602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5765.093901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5766.458539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5770.758972] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5775.844189] LustreError: 334886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5775.872977] LustreError: 334886:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5779.611160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5780.038618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5784.445672] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5790.289417] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5793.949527] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5800.264256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5800.522119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5805.851244] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5810.801566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5811.057309] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5820.902764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5820.911488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5820.925683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5821.413746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5821.425895] Lustre: Skipped 1 previous similar message [ 5821.432984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5821.441221] Lustre: Skipped 2 previous similar messages [ 5822.226379] Lustre: server umount lustre-MDT0000 complete [ 5825.422788] LustreError: 334867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780091737 with bad export cookie 9019523128683111856 [ 5825.437279] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5825.827517] Lustre: server umount lustre-MDT0001 complete [ 5840.114328] Lustre: server umount lustre-OST0000 complete [ 5854.614568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5855.084289] LustreError: 338255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5855.172142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5860.328156] LustreError: 338256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5860.933109] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5865.448356] LustreError: 338255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5870.388308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5870.796311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5875.827472] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5882.696393] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5887.378800] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5895.613285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5895.969452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5901.102665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5902.293319] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5909.150552] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5921.769960] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5921.791852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5921.799103] Lustre: Skipped 2 previous similar messages [ 5926.883129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5926.886667] Lustre: Skipped 4 previous similar messages [ 5927.102787] Lustre: server umount lustre-MDT0000 complete [ 5931.001720] LustreError: 338235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780091843 with bad export cookie 9019523128683112969 [ 5931.020255] LustreError: 338235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5931.036193] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5931.302816] Lustre: server umount lustre-MDT0001 complete [ 5944.942557] Lustre: server umount lustre-OST0000 complete [ 5960.321941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5960.756186] LustreError: 341826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5960.763277] LustreError: 341826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5960.809789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5964.928978] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5971.426498] LustreError: 341827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5971.442355] LustreError: 341827:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5973.345961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5977.287639] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 5982.795817] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5987.141403] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5994.105170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5994.425244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5994.428123] Lustre: Skipped 1 previous similar message [ 6000.267387] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6005.688833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6006.067777] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6011.152443] Lustre: server umount lustre-OST0000 complete [ 6015.974814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6015.975910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6015.983601] Lustre: Skipped 1 previous similar message [ 6015.995560] Lustre: Skipped 1 previous similar message [ 6025.284712] Lustre: server umount lustre-MDT0000 complete [ 6026.209907] LustreError: 341821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6026.221293] LustreError: 341821:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6028.943920] LustreError: 341806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780091941 with bad export cookie 9019523128683113984 [ 6028.945952] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6028.950834] LustreError: 341806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6039.654297] Lustre: server umount lustre-MDT0001 complete [ 6052.009749] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 6054.958852] Key type lgssc unregistered [ 6055.306124] LNet: 345002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6055.324364] LNetError: 345002:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6055.364495] LNet: Removed LNI 192.168.201.149@tcp [ 6056.253536] Key type .llcrypt unregistered [ 6056.257468] Key type ._llcrypt unregistered [ 6082.649076] Key type ._llcrypt registered [ 6082.651387] Key type .llcrypt registered [ 6082.768460] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 6084.326167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6084.407088] alg: No test for adler32 (adler32-zlib) [ 6085.474197] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 6085.700204] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 6087.400193] Key type lgssc registered [ 6088.509525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6096.225878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6096.245436] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6096.260301] LustreError: 346430:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6096.296537] LustreError: 346430:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6096.304375] LustreError: 346430:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6096.313106] LustreError: 346430:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6096.321958] LustreError: 346430:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6103.285852] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6103.290989] LustreError: 346827:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6103.303321] LustreError: 346827:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6103.307909] LustreError: 346827:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6103.312704] LustreError: 346827:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6103.318415] LustreError: 346827:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6120.948917] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 6123.223640] Key type lgssc unregistered [ 6123.492678] LNet: 347813:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6123.500889] LNetError: 347813:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6123.514173] LNet: Removed LNI 192.168.201.149@tcp [ 6124.141590] Key type .llcrypt unregistered [ 6124.143774] Key type ._llcrypt unregistered [ 6141.644668] Key type ._llcrypt registered [ 6141.646848] Key type .llcrypt registered [ 6141.759306] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 6156.949817] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 6157.751940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6157.835315] alg: No test for adler32 (adler32-zlib) [ 6159.045927] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 6159.240487] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 6160.912135] Key type lgssc registered [ 6161.809537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6196.759169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6196.789179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6198.230608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6198.269956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6198.362435] Lustre: lustre-MDT0000: new disk, initializing [ 6198.417482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6198.428775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6202.165982] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6211.975821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6212.116069] Lustre: 350950:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6212.167041] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6212.170461] Lustre: Skipped 1 previous similar message [ 6212.232824] Lustre: lustre-MDT0001: new disk, initializing [ 6212.340823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6212.407115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6212.421174] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6216.152624] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6223.688793] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6227.352630] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6233.442890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6233.755715] Lustre: lustre-OST0000: new disk, initializing [ 6233.759212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6233.765681] Lustre: 352689:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6233.842584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6239.831118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6239.840442] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6239.950204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6239.982809] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6246.982301] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6251.360923] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6251.551861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6255.671478] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6255.822653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6260.710055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6260.711968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6260.719426] LustreError: Skipped 1 previous similar message [ 6260.721678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6260.732845] Lustre: Skipped 1 previous similar message [ 6260.757076] Lustre: Skipped 1 previous similar message [ 6263.772706] Lustre: server umount lustre-OST0000 complete [ 6268.898696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6268.905315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6268.926081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6270.945535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6270.946981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6276.067486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6276.074256] Lustre: Skipped 2 previous similar messages [ 6281.186551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6282.208176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6282.445066] Lustre: server umount lustre-MDT0000 complete [ 6285.753475] LustreError: 350940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780092198 with bad export cookie 4482449108164800652 [ 6285.756712] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6285.762778] LustreError: 350940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6286.087673] Lustre: server umount lustre-MDT0001 complete [ 6294.539452] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6296.591605] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 18:03:28 (1780092208) [ 6303.620045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6304.108534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6308.048344] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6309.347882] LustreError: 355003:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6309.364419] LustreError: 355003:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6314.468356] LustreError: 355004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6316.199678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6316.527325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6320.647123] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6325.948294] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6329.574985] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6336.537245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6336.880480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6342.131774] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6347.100226] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6352.475557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6352.610265] Lustre: lustre-OST0001: new disk, initializing [ 6352.613388] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6352.618267] Lustre: 357499:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6357.490937] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6362.709875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6362.725572] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6362.821989] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6364.411866] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6368.225932] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6368.235686] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6368.247513] LustreError: Skipped 1 previous similar message [ 6368.258925] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6368.275239] Lustre: Skipped 1 previous similar message [ 6368.281089] Lustre: Skipped 1 previous similar message [ 6370.328340] Lustre: server umount lustre-OST0001 complete [ 6371.360193] LustreError: 358054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6378.472857] LustreError: 358054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6378.498734] LustreError: 358054:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6383.589702] LustreError: 358054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6383.612784] LustreError: 358054:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6387.169770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6387.177897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6387.204096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6387.208789] Lustre: Skipped 1 previous similar message [ 6392.831523] Lustre: server umount lustre-OST0000 complete [ 6398.442748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6398.446826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6398.459247] Lustre: Skipped 1 previous similar message [ 6403.678878] Lustre: server umount lustre-MDT0000 complete [ 6404.075262] LustreError: 355004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6404.093422] LustreError: 355004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6408.653939] LustreError: 354984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780092321 with bad export cookie 4482449108164801541 [ 6408.654952] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6408.667634] LustreError: 354984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6415.137638] Lustre: server umount lustre-MDT0001 complete [ 6426.343837] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 6428.756201] Key type lgssc unregistered [ 6428.995714] LNet: 359359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6429.008677] LNetError: 359359:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6429.042476] LNet: Removed LNI 192.168.201.149@tcp [ 6429.744271] Key type .llcrypt unregistered [ 6429.746245] Key type ._llcrypt unregistered [ 6478.745598] Key type ._llcrypt registered [ 6478.747903] Key type .llcrypt registered [ 6478.850562] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 6480.388169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6480.416489] alg: No test for adler32 (adler32-zlib) [ 6481.441850] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 6481.726822] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 6483.424230] Key type lgssc registered [ 6484.485131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6492.309616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6492.341572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6493.656641] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6493.708926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6493.893263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6498.547727] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6507.507851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6507.635698] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6507.673276] Lustre: 362147:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6507.929174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6512.142526] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6517.301313] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6520.962609] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6523.364845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6523.377477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6523.386344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6526.952551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6526.959273] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6529.359484] Lustre: server umount lustre-MDT0000 complete [ 6532.869952] LustreError: 362879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780092445 with bad export cookie 13515650740392883715 [ 6532.871834] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6532.877926] LustreError: 362879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6533.125362] Lustre: server umount lustre-MDT0001 complete [ 6542.101750] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 18:07:33 (1780092453) [ 6547.701560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6547.995810] LustreError: 364430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6548.017487] LustreError: 364430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6548.050477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6552.145903] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6553.059294] LustreError: 364430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6553.078400] LustreError: 364430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6558.182342] LustreError: 364431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6560.002377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6560.268956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6564.089681] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6569.059342] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6573.062928] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6578.938852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6579.050207] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6579.139645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6583.560900] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6588.061880] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6595.887253] Lustre: Permanently deactivating lustre-OST0000 [ 6595.898582] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6603.283773] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6604.813737] LustreError: 367075:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6604.826444] LustreError: 367075:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 6604.835549] LustreError: 367075:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6607.009331] LustreError: 367124:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 6607.013453] LustreError: 367124:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6614.862616] Lustre: server umount lustre-OST0000 complete [ 6616.914822] LustreError: 367373:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 6616.919431] LustreError: 367373:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6621.667502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6621.669109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6621.685835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6621.685843] Lustre: Skipped 1 previous similar message [ 6626.785219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6626.789057] Lustre: Skipped 2 previous similar messages [ 6627.893717] Lustre: server umount lustre-MDT0000 complete [ 6630.815705] LustreError: 365145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780092543 with bad export cookie 13515650740392884296 [ 6630.826573] LustreError: 365145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6630.828319] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6641.290400] Lustre: server umount lustre-MDT0001 complete [ 6648.029229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6651.375976] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6653.675579] LustreError: 368513:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6658.755736] Lustre: 368852:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.149@tcp [ 6660.536552] Lustre: server umount MGS complete [ 6669.958207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6670.265508] LustreError: 369575:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6670.283127] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6670.286682] Lustre: Skipped 1 previous similar message [ 6670.331652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6673.150546] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6675.425680] LustreError: 369576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6679.670546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6679.951356] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6683.434356] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6688.086892] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6691.314777] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6692.914808] Lustre: Permanently reactivating lustre-OST0000 [ 6692.923153] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6692.925411] Lustre: Skipped 2 previous similar messages [ 6715.872105] LustreError: 371032:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6721.194618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6721.458358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6721.463288] Lustre: Skipped 1 previous similar message [ 6726.030719] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6730.131601] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6740.965597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6740.970434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6740.973343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6740.982830] Lustre: Skipped 2 previous similar messages [ 6745.032539] Lustre: server umount lustre-OST0000 complete [ 6751.204524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6751.205710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6751.226165] Lustre: Skipped 1 previous similar message [ 6751.228696] Lustre: Skipped 1 previous similar message [ 6753.926573] Lustre: server umount lustre-MDT0000 complete [ 6756.323448] LustreError: 369575:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6756.338258] LustreError: 369575:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6756.822813] LustreError: 369555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780092669 with bad export cookie 13515650740392885346 [ 6756.825743] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6756.831499] LustreError: 369555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6757.076748] Lustre: server umount lustre-MDT0001 complete [ 6765.096757] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 6767.237248] Key type lgssc unregistered [ 6767.529863] LNet: 373532:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6767.536335] LNetError: 373532:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6767.550932] LNet: Removed LNI 192.168.201.149@tcp [ 6768.180213] Key type .llcrypt unregistered [ 6768.182875] Key type ._llcrypt unregistered [ 6782.105447] Key type ._llcrypt registered [ 6782.108068] Key type .llcrypt registered [ 6782.195807] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 6794.137808] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 6794.862711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6794.897182] alg: No test for adler32 (adler32-zlib) [ 6795.957166] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 6796.221805] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 6797.872328] Key type lgssc registered [ 6798.654599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6825.247125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6825.287586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6826.492106] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6826.504571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6826.556655] Lustre: lustre-MDT0000: new disk, initializing [ 6826.623629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6826.640226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6829.783943] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6837.748810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6837.840633] Lustre: 376638:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6837.876985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6837.880729] Lustre: Skipped 1 previous similar message [ 6837.921275] Lustre: lustre-MDT0001: new disk, initializing [ 6837.989365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6838.019962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6838.030989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6841.170844] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6846.611462] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6849.272711] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6854.226282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6854.398709] Lustre: lustre-OST0000: new disk, initializing [ 6854.401597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6854.407718] Lustre: 378377:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6854.472633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6858.021665] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6862.355104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6862.359166] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6862.431849] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6863.687343] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6866.695730] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6866.837576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6869.761200] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6869.905615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6872.547805] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6872.549138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6872.553676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6872.572551] Lustre: Skipped 1 previous similar message [ 6877.358755] Lustre: server umount lustre-OST0000 complete [ 6882.786899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6882.792901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6882.795693] Lustre: Skipped 1 previous similar message [ 6882.804964] Lustre: Skipped 2 previous similar messages [ 6887.908178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6887.916173] Lustre: Skipped 1 previous similar message [ 6893.026927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6893.032795] Lustre: Skipped 1 previous similar message [ 6894.048153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6894.176499] Lustre: server umount lustre-MDT0000 complete [ 6896.676120] LustreError: 376630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780092809 with bad export cookie 10608214014836158096 [ 6896.677179] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6896.686448] LustreError: 376630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6896.942624] Lustre: server umount lustre-MDT0001 complete [ 6902.813824] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6903.980865] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 18:13:36 (1780092816) [ 6908.944288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6909.252616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6912.320813] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6914.535756] LustreError: 380689:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6914.558576] LustreError: 380689:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6918.615704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6918.630820] LustreError: 380688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6918.932458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6921.580883] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6925.552981] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6928.469292] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6933.687040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6933.947490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6938.768743] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 6943.379854] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6944.723372] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6946.867794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6954.470322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6954.479130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6954.486034] Lustre: Skipped 1 previous similar message [ 6954.486367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6954.486370] LustreError: Skipped 1 previous similar message [ 6954.514425] Lustre: Skipped 1 previous similar message [ 6956.769072] Lustre: server umount lustre-OST0000 complete [ 6959.585319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6959.596678] Lustre: Skipped 1 previous similar message [ 6964.707205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6964.712084] Lustre: Skipped 3 previous similar messages [ 6965.625639] Lustre: server umount lustre-MDT0000 complete [ 6968.124314] LustreError: 380670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780092880 with bad export cookie 10608214014836158971 [ 6968.126544] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6968.144900] LustreError: 380670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6968.347918] Lustre: server umount lustre-MDT0001 complete [ 6975.819084] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 6977.447161] Key type lgssc unregistered [ 6977.659746] LNet: 384050:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6977.673477] LNetError: 384050:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6977.695224] LNet: Removed LNI 192.168.201.149@tcp [ 6978.120724] Key type .llcrypt unregistered [ 6978.122750] Key type ._llcrypt unregistered [ 6994.011196] Key type ._llcrypt registered [ 6994.012728] Key type .llcrypt registered [ 6994.084248] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 6994.247520] Key type .llcrypt unregistered [ 6994.250236] Key type ._llcrypt unregistered [ 7007.495976] Key type ._llcrypt registered [ 7007.497817] Key type .llcrypt registered [ 7007.559900] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 7008.395906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7008.425902] alg: No test for adler32 (adler32-zlib) [ 7009.482296] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 7009.680651] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 7011.360643] Key type lgssc registered [ 7012.248841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7017.523336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7017.533806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7018.871382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7021.746987] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7028.048698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7028.245262] LustreError: 386208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7028.258874] LustreError: 386208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7028.268458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7029.288057] LustreError: 386202:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7029.297494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7032.472744] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7034.348228] LustreError: 386207:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7034.369544] LustreError: 386207:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7037.174659] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7039.467771] LustreError: 386208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7039.488442] LustreError: 386208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7042.643264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7042.918533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7046.033140] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7058.403384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7058.410832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7058.424971] Lustre: Skipped 1 previous similar message [ 7058.435103] Lustre: Skipped 1 previous similar message [ 7062.259314] Lustre: server umount lustre-OST0000 complete [ 7068.647229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7068.655246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7068.659042] Lustre: Skipped 1 previous similar message [ 7068.669246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7073.764853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7073.769594] Lustre: Skipped 2 previous similar messages [ 7077.861700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7077.869473] Lustre: Skipped 1 previous similar message [ 7079.392173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7079.563446] Lustre: server umount lustre-MDT0000 complete [ 7082.757972] LustreError: 386187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780092995 with bad export cookie 10082876244288399472 [ 7082.762532] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7082.772947] LustreError: 386187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7083.021527] Lustre: server umount lustre-MDT0001 complete [ 7090.386866] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 7092.246928] Key type lgssc unregistered [ 7092.479656] LNet: 389191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7092.490565] LNetError: 389191:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7092.500724] LNet: Removed LNI 192.168.201.149@tcp [ 7093.132763] Key type .llcrypt unregistered [ 7093.134781] Key type ._llcrypt unregistered [ 7110.229517] Key type ._llcrypt registered [ 7110.234986] Key type .llcrypt registered [ 7110.342810] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 7111.327135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7111.344301] alg: No test for adler32 (adler32-zlib) [ 7112.385345] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 7112.620244] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 7114.265922] Key type lgssc registered [ 7115.201310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7120.894892] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7120.917392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7136.480500] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7141.600761] LustreError: 390546:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.149@tcp: failed processing log, type 4: rc = -110 [ 7168.288990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7172.792485] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7177.045494] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7182.160479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7182.519564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7182.580163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7185.730164] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7188.967938] LustreError: 391470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7188.986878] LustreError: 391470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7192.004514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7192.297147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7195.159692] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7199.266299] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7202.607939] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7213.031508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7213.036174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7213.049939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7213.062126] Lustre: Skipped 1 previous similar message [ 7214.585805] Lustre: server umount lustre-OST0000 complete [ 7217.417229] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7217.432717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7217.444308] Lustre: Skipped 1 previous similar message [ 7217.455275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7223.265683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7223.271297] Lustre: Skipped 3 previous similar messages [ 7223.681790] Lustre: server umount lustre-MDT0000 complete [ 7226.615819] LustreError: 390554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780093139 with bad export cookie 14262341346862932652 [ 7226.625134] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7226.631779] LustreError: 390554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7226.850718] Lustre: server umount lustre-MDT0001 complete [ 7235.842274] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 7237.718764] Key type lgssc unregistered [ 7237.945708] LNet: 393755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7237.954318] LNetError: 393755:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7237.970609] LNet: Removed LNI 192.168.201.149@tcp [ 7238.570491] Key type .llcrypt unregistered [ 7238.572825] Key type ._llcrypt unregistered [ 7254.707838] Key type ._llcrypt registered [ 7254.709980] Key type .llcrypt registered [ 7254.774389] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 7255.248285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7255.276432] alg: No test for adler32 (adler32-zlib) [ 7256.236257] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 7256.368986] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 7258.024486] Key type lgssc registered [ 7258.658596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7263.519577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7263.543278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7264.846201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7267.666991] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7272.913039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7273.138572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7275.359370] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7278.883760] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7281.452572] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7285.851665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7286.064538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7289.635856] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7290.162191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7293.521588] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7298.533765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7298.539157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7298.551552] Lustre: Skipped 2 previous similar messages [ 7298.553391] Lustre: Skipped 1 previous similar message [ 7301.125691] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.49@tcp (stopping) [ 7301.132670] Lustre: Skipped 1 previous similar message [ 7303.658745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7303.776100] Lustre: server umount lustre-MDT0000 complete [ 7305.419850] LustreError: 396747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.49@tcp arrived at 1780093217 with bad export cookie 903688697801703479 [ 7305.433203] LustreError: 396747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7306.946890] LustreError: 395112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780093219 with bad export cookie 903688697801702618 [ 7306.965944] LustreError: 395112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7306.967517] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7308.771070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7308.771417] LustreError: 395133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7308.786672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7308.802869] LustreError: 395133:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7308.806868] Lustre: Skipped 2 previous similar messages [ 7311.781335] LustreError: 395128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7311.809178] LustreError: 395128:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7313.160194] Lustre: server umount lustre-OST0000 complete [ 7316.898240] LustreError: 395127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7316.916163] LustreError: 395127:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7319.072309] LustreError: 395844:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ee84cfae x1866562973756032/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7319.207088] Lustre: server umount lustre-MDT0001 complete [ 7326.445512] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 7328.298519] Key type lgssc unregistered [ 7328.505897] LNet: 398455:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7328.515638] LNetError: 398455:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7328.535588] LNet: Removed LNI 192.168.201.149@tcp [ 7329.075583] Key type .llcrypt unregistered [ 7329.077380] Key type ._llcrypt unregistered [ 7344.112259] Key type ._llcrypt registered [ 7344.114419] Key type .llcrypt registered [ 7344.194911] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 7344.835453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7344.933854] alg: No test for adler32 (adler32-zlib) [ 7345.871461] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 7346.024751] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 7347.649289] Key type lgssc registered [ 7348.367892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7353.345233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7353.355031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7354.596430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7357.694651] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7363.292770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7363.494378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7366.138569] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7369.745787] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7372.573915] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7377.155436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7377.371447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7380.468104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7381.064106] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7384.572351] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7390.694843] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7390.695762] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7390.708827] Lustre: Skipped 1 previous similar message [ 7391.225915] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.49@tcp (stopping) [ 7391.232933] Lustre: Skipped 1 previous similar message [ 7393.518367] Lustre: server umount lustre-MDT0001 complete [ 7395.819785] LustreError: 399832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7395.829062] LustreError: 399832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7400.931194] LustreError: 399832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7400.931472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7400.932595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7400.936111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7403.215462] Lustre: server umount lustre-OST0000 complete [ 7405.555794] Lustre: server umount lustre-MDT0000 complete [ 7414.077693] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 7415.569745] Key type lgssc unregistered [ 7415.740563] LNet: 403155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7415.746565] LNetError: 403155:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7415.762840] LNet: Removed LNI 192.168.201.149@tcp [ 7416.262561] Key type .llcrypt unregistered [ 7416.263873] Key type ._llcrypt unregistered [ 7431.173564] Key type ._llcrypt registered [ 7431.175455] Key type .llcrypt registered [ 7431.252989] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 7440.465801] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 7440.973071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7441.014462] alg: No test for adler32 (adler32-zlib) [ 7441.979840] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 7442.134387] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 7443.760105] Key type lgssc registered [ 7444.421589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7466.684855] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 7476.341573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7476.356678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7477.546948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7477.585210] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7477.686228] Lustre: lustre-MDT0000: new disk, initializing [ 7477.748599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7477.757881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7480.602505] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7490.034950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7490.213837] Lustre: lustre-OST0000: new disk, initializing [ 7490.216563] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7490.219544] Lustre: Skipped 1 previous similar message [ 7490.223676] Lustre: 408095:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7490.264951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7493.002777] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7495.184194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7495.190919] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7495.203277] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7497.479457] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7506.347634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7506.446902] Lustre: 407096:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7506.500461] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7506.543935] Lustre: lustre-MDT0001: new disk, initializing [ 7506.608888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7506.631162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 7506.644832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 7508.802187] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7514.008592] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7514.122469] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7516.195610] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7516.341729] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7520.268278] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.49@tcp (stopping) [ 7520.741716] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7520.741836] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7520.747790] Lustre: Skipped 1 previous similar message [ 7525.266530] Lustre: server umount lustre-MDT0001 complete [ 7530.978622] LustreError: 407107:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7530.992751] LustreError: 407107:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7536.098727] LustreError: 407108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7536.105093] LustreError: 407108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7541.218122] LustreError: 407107:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7541.218184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7541.220407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7541.242841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7541.246462] Lustre: Skipped 2 previous similar messages [ 7545.627845] Lustre: server umount lustre-OST0000 complete [ 7546.341793] LustreError: 407108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7553.552387] Lustre: server umount lustre-MDT0000 complete [ 7560.789658] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 7562.155405] Key type lgssc unregistered [ 7562.290444] LNet: 411655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7562.293537] LNetError: 411655:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7562.300456] LNet: Removed LNI 192.168.201.149@tcp [ 7562.688390] Key type .llcrypt unregistered [ 7562.692519] Key type ._llcrypt unregistered [ 7580.806838] Key type ._llcrypt registered [ 7580.808260] Key type .llcrypt registered [ 7580.849455] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 7580.957343] Key type .llcrypt unregistered [ 7580.958571] Key type ._llcrypt unregistered [ 7588.579802] Key type ._llcrypt registered [ 7588.581452] Key type .llcrypt registered [ 7588.624799] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 7589.039614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7589.062193] alg: No test for adler32 (adler32-zlib) [ 7589.957968] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 7590.063080] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 7591.664118] Key type lgssc registered [ 7592.223854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7606.017840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7606.040903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7607.179518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7607.198648] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7607.243900] Lustre: lustre-MDT0000: new disk, initializing [ 7607.279304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7607.287434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7609.419569] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7615.897396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7615.969835] Lustre: 414971:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7616.016177] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7616.021018] Lustre: Skipped 1 previous similar message [ 7616.060490] Lustre: lustre-MDT0001: new disk, initializing [ 7616.109724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7616.134442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7616.140932] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7618.134175] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7622.134963] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7624.040566] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7627.503837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7627.657315] Lustre: lustre-OST0000: new disk, initializing [ 7627.661068] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7627.664323] Lustre: 416713:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7627.716993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7630.716361] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7634.967548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7634.980211] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7635.054491] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7635.199845] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7640.036053] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7640.037200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7640.049146] Lustre: Skipped 2 previous similar messages [ 7640.055627] Lustre: Skipped 2 previous similar messages [ 7645.018730] Lustre: server umount lustre-MDT0000 complete [ 7647.014289] LustreError: 414963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780093559 with bad export cookie 464709608619107457 [ 7647.019376] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7647.019515] LustreError: 414963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7647.198487] Lustre: server umount lustre-MDT0001 complete [ 7659.435090] Lustre: server umount lustre-OST0000 complete [ 7665.153599] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 18:26:17 (1780093577) [ 7669.073915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7669.321459] LustreError: 418626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7669.337885] LustreError: 418626:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7669.358716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7671.342150] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7674.849648] LustreError: 418627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7675.189133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7675.383172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7677.598492] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7680.435427] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7682.232246] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7698.983159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7699.092876] Lustre: Found index 0 for lustre-OST0000, updating log [ 7699.201854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7702.112447] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 7704.826803] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7715.563529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7726.049470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7726.049784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7726.052806] LustreError: Skipped 1 previous similar message [ 7726.053365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7726.057687] Lustre: Skipped 1 previous similar message [ 7726.067149] Lustre: Skipped 1 previous similar message [ 7727.082670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7727.089127] Lustre: Skipped 1 previous similar message [ 7728.861134] Lustre: server umount lustre-OST0000 complete [ 7743.458125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7743.458686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7743.463139] Lustre: Skipped 1 previous similar message [ 7748.299989] Lustre: server umount lustre-MDT0000 complete [ 7748.585245] LustreError: 418622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7748.592333] LustreError: 418622:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7750.158782] LustreError: 419343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780093662 with bad export cookie 464709608619108836 [ 7750.162076] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7750.162807] LustreError: 419343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7750.327239] Lustre: server umount lustre-MDT0001 complete [ 7754.155442] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 18:27:46 (1780093666) [ 7764.128294] LNet: 1 local NIs in recovery (showing 1): 192.168.201.149@tcp [ 8337.871226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8338.044792] LustreError: 424714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8338.069271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8339.586824] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8342.625766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8342.776963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8344.234751] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8346.177129] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8347.438665] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8362.979432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8363.078213] Lustre: Found index 0 for lustre-OST0000, updating log [ 8363.197902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8365.278013] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8365.288639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 8367.207699] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8368.994256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8368.999630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8369.001221] Lustre: Skipped 1 previous similar message [ 8370.017526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8370.021251] Lustre: Skipped 1 previous similar message [ 8374.093390] Lustre: server umount lustre-OST0000 complete [ 8376.417724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8376.418770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8376.422336] Lustre: Skipped 1 previous similar message [ 8378.465437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8378.468264] Lustre: Skipped 4 previous similar messages [ 8381.527524] Lustre: server umount lustre-MDT0000 complete [ 8381.539828] LustreError: 424714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8381.545693] LustreError: 424714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8382.889629] LustreError: 426384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780094295 with bad export cookie 464709608619109907 [ 8382.892598] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8382.900322] LustreError: 426384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8383.001071] Lustre: server umount lustre-MDT0001 complete [ 8391.840269] LNet: 1 local NIs in recovery (showing 1): 192.168.201.149@tcp [ 8400.562208] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 8404.084374] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8416.705940] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 18:38:49 (1780094329) [ 8422.644387] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8423.770483] Key type lgssc unregistered [ 8426.977380] LNet: 430973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8426.981041] LNetError: 430973:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8492.518328] LNet: Removed LNI 192.168.201.149@tcp [ 8492.804962] Key type .llcrypt unregistered [ 8492.806049] Key type ._llcrypt unregistered [ 8499.223788] Key type ._llcrypt registered [ 8499.225432] Key type .llcrypt registered [ 8499.266802] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8499.604896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8499.650597] alg: No test for adler32 (adler32-zlib) [ 8500.497317] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 8500.580529] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 8502.160151] Key type lgssc registered [ 8502.483660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8503.818682] LNet: Added LNI 192.168.201.149@tcp100 [8/256/0/180] [ 8504.093178] LNet: Added LNI 192.168.201.149@tcp101 [8/256/0/180] [ 8504.385641] LNet: Added LNI 192.168.201.149@tcp102 [8/256/0/180] [ 8504.668765] LNet: Added LNI 192.168.201.149@tcp103 [8/256/0/180] [ 8504.972729] LNet: Added LNI 192.168.201.149@tcp104 [8/256/0/180] [ 8505.261696] LNet: Added LNI 192.168.201.149@tcp105 [8/256/0/180] [ 8505.549464] LNet: Added LNI 192.168.201.149@tcp106 [8/256/0/180] [ 8505.832445] LNet: Added LNI 192.168.201.149@tcp107 [8/256/0/180] [ 8506.102153] LNet: Added LNI 192.168.201.149@tcp108 [8/256/0/180] [ 8506.382932] LNet: Added LNI 192.168.201.149@tcp109 [8/256/0/180] [ 8506.653254] LNet: Added LNI 192.168.201.149@tcp110 [8/256/0/180] [ 8506.935324] LNet: Added LNI 192.168.201.149@tcp111 [8/256/0/180] [ 8507.230570] LNet: Added LNI 192.168.201.149@tcp112 [8/256/0/180] [ 8507.516210] LNet: Added LNI 192.168.201.149@tcp113 [8/256/0/180] [ 8507.786368] LNet: Added LNI 192.168.201.149@tcp114 [8/256/0/180] [ 8508.072885] LNet: Added LNI 192.168.201.149@tcp115 [8/256/0/180] [ 8508.346961] LNet: Added LNI 192.168.201.149@tcp116 [8/256/0/180] [ 8508.613116] LNet: Added LNI 192.168.201.149@tcp117 [8/256/0/180] [ 8508.899070] LNet: Added LNI 192.168.201.149@tcp118 [8/256/0/180] [ 8509.177556] LNet: Added LNI 192.168.201.149@tcp119 [8/256/0/180] [ 8509.459534] LNet: Added LNI 192.168.201.149@tcp120 [8/256/0/180] [ 8509.728827] LNet: Added LNI 192.168.201.149@tcp121 [8/256/0/180] [ 8510.004301] LNet: Added LNI 192.168.201.149@tcp122 [8/256/0/180] [ 8510.283271] LNet: Added LNI 192.168.201.149@tcp123 [8/256/0/180] [ 8510.570679] LNet: Added LNI 192.168.201.149@tcp124 [8/256/0/180] [ 8510.842461] LNet: Added LNI 192.168.201.149@tcp125 [8/256/0/180] [ 8511.128786] LNet: Added LNI 192.168.201.149@tcp126 [8/256/0/180] [ 8511.413339] LNet: Added LNI 192.168.201.149@tcp127 [8/256/0/180] [ 8511.686968] LNet: Added LNI 192.168.201.149@tcp128 [8/256/0/180] [ 8511.974485] LNet: Added LNI 192.168.201.149@tcp129 [8/256/0/180] [ 8512.271198] LNet: Added LNI 192.168.201.149@tcp130 [8/256/0/180] [ 8512.554685] LNet: Added LNI 192.168.201.149@tcp131 [8/256/0/180] [ 8512.843302] LNet: Added LNI 192.168.201.149@tcp132 [8/256/0/180] [ 8513.146713] LNet: Added LNI 192.168.201.149@tcp133 [8/256/0/180] [ 8513.445533] LNet: Added LNI 192.168.201.149@tcp134 [8/256/0/180] [ 8513.729701] LNet: Added LNI 192.168.201.149@tcp135 [8/256/0/180] [ 8515.875471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8515.881281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8516.920943] Lustre: 433953:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.149@tcp: can't add NID 192.168.201.149@tcp130: rc = -75 [ 8516.974445] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8516.986049] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8517.028248] Lustre: lustre-MDT0000: new disk, initializing [ 8517.056110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8517.062066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8518.305928] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8522.204129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8522.254275] Lustre: 433969:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8522.279690] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8522.281231] Lustre: Skipped 1 previous similar message [ 8522.310916] Lustre: lustre-MDT0001: new disk, initializing [ 8522.323804] Lustre: 434749:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.149@tcp: can't add NID 192.168.201.149@tcp130: rc = -75 [ 8522.326352] Lustre: 434749:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 8522.361566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8522.371044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8522.374177] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8523.589273] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8526.520887] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8527.701407] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8529.981428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8530.075978] Lustre: lustre-OST0000: new disk, initializing [ 8530.077659] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8530.080147] Lustre: 435710:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8530.109782] Lustre: 435715:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.149@tcp: can't add NID 192.168.201.149@tcp130: rc = -75 [ 8530.112046] Lustre: 435715:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [ 8530.126392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8531.902790] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8534.954948] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8537.301604] Lustre: server umount lustre-OST0000 complete [ 8539.122173] Lustre: 436741:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.149@tcp: can't add NID 192.168.201.149@tcp130: rc = -75 [ 8539.125237] Lustre: 436741:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 8539.134449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8539.135313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8539.137743] Lustre: Skipped 1 previous similar message [ 8539.140089] Lustre: Skipped 1 previous similar message [ 8544.225026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8544.227691] Lustre: Skipped 1 previous similar message [ 8544.597985] Lustre: server umount lustre-MDT0000 complete [ 8545.848147] LustreError: 433962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780094458 with bad export cookie 15429798967369021481 [ 8545.850897] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8545.851962] LustreError: 433962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8545.957469] Lustre: server umount lustre-MDT0001 complete [ 8549.481602] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8550.505261] Key type lgssc unregistered [ 8550.659948] LNet: 437484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8550.663258] LNetError: 437484:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8550.666550] LNet: Removed LNI 192.168.201.149@tcp [ 8551.712130] LNet: Removed LNI 192.168.201.149@tcp100 [ 8551.713926] LNetError: 437484:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8551.716519] LNetError: 437484:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8551.719548] LNet: Removed LNI 192.168.201.149@tcp101 [ 8552.736137] LNet: Removed LNI 192.168.201.149@tcp102 [ 8552.737647] LNetError: 437484:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8552.740553] LNetError: 437484:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8552.742975] LNet: Removed LNI 192.168.201.149@tcp103 [ 8553.760143] LNet: Removed LNI 192.168.201.149@tcp104 [ 8554.784156] LNet: Removed LNI 192.168.201.149@tcp105 [ 8554.785276] LNetError: 437484:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8554.786906] LNetError: 437484:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 8554.788910] LNet: Removed LNI 192.168.201.149@tcp106 [ 8554.790408] LNet: Removed LNI 192.168.201.149@tcp107 [ 8554.791806] LNet: Removed LNI 192.168.201.149@tcp108 [ 8554.793299] LNet: Removed LNI 192.168.201.149@tcp109 [ 8554.794692] LNet: Removed LNI 192.168.201.149@tcp110 [ 8554.796362] LNet: Removed LNI 192.168.201.149@tcp111 [ 8554.797785] LNet: Removed LNI 192.168.201.149@tcp112 [ 8554.799202] LNet: Removed LNI 192.168.201.149@tcp113 [ 8554.800764] LNet: Removed LNI 192.168.201.149@tcp114 [ 8554.802239] LNet: Removed LNI 192.168.201.149@tcp115 [ 8554.803543] LNet: Removed LNI 192.168.201.149@tcp116 [ 8554.805247] LNet: Removed LNI 192.168.201.149@tcp117 [ 8554.807089] LNet: Removed LNI 192.168.201.149@tcp118 [ 8554.808836] LNet: Removed LNI 192.168.201.149@tcp119 [ 8554.810054] LNet: Removed LNI 192.168.201.149@tcp120 [ 8554.811661] LNet: Removed LNI 192.168.201.149@tcp121 [ 8554.813386] LNet: Removed LNI 192.168.201.149@tcp122 [ 8554.815066] LNet: Removed LNI 192.168.201.149@tcp123 [ 8554.816547] LNet: Removed LNI 192.168.201.149@tcp124 [ 8554.817823] LNet: Removed LNI 192.168.201.149@tcp125 [ 8554.819098] LNet: Removed LNI 192.168.201.149@tcp126 [ 8554.820443] LNet: Removed LNI 192.168.201.149@tcp127 [ 8554.822283] LNet: Removed LNI 192.168.201.149@tcp128 [ 8554.823679] LNet: Removed LNI 192.168.201.149@tcp129 [ 8554.824845] LNet: Removed LNI 192.168.201.149@tcp130 [ 8554.826061] LNet: Removed LNI 192.168.201.149@tcp131 [ 8554.827309] LNet: Removed LNI 192.168.201.149@tcp132 [ 8554.828275] LNet: Removed LNI 192.168.201.149@tcp133 [ 8554.829639] LNet: Removed LNI 192.168.201.149@tcp134 [ 8554.836493] LNet: Removed LNI 192.168.201.149@tcp135 [ 8555.177817] Key type .llcrypt unregistered [ 8555.179180] Key type ._llcrypt unregistered [ 8562.454746] Key type ._llcrypt registered [ 8562.456202] Key type .llcrypt registered [ 8562.494608] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 8568.470708] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8568.904569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8568.918227] alg: No test for adler32 (adler32-zlib) [ 8569.776781] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 8569.860977] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 8571.472152] Key type lgssc registered [ 8571.844506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8583.082697] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 18:41:35 (1780094495) [ 8589.218434] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8590.253729] Key type lgssc unregistered [ 8590.377362] LNet: 441487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8590.380362] LNetError: 441487:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8590.389444] LNet: Removed LNI 192.168.201.149@tcp [ 8590.678418] Key type .llcrypt unregistered [ 8590.679567] Key type ._llcrypt unregistered [ 8597.457376] Key type ._llcrypt registered [ 8597.458335] Key type .llcrypt registered [ 8597.496902] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8597.765651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8597.772728] alg: No test for adler32 (adler32-zlib) [ 8598.639927] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 8598.728335] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 8600.312162] Key type lgssc registered [ 8600.693173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8603.970304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8603.976453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8605.064404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8605.076222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8605.111044] Lustre: lustre-MDT0000: new disk, initializing [ 8605.135043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8605.143405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8606.430437] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8610.676900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8610.716880] Lustre: 442658:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8610.744181] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8610.745945] Lustre: Skipped 1 previous similar message [ 8610.783803] Lustre: lustre-MDT0001: new disk, initializing [ 8610.820605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8610.833491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8610.836625] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8612.125353] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8615.205454] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8616.398895] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8618.662205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8618.768327] Lustre: lustre-OST0000: new disk, initializing [ 8618.770820] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8618.774405] Lustre: 444400:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8618.804215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8620.763026] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8620.923561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8620.927434] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8620.966145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8624.081484] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8625.318115] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing lnet_if_list [ 8626.897928] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 8626.910601] LNet: Added LNI 192.168.201.149@tcp10 [8/256/0/180] [ 8628.750781] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 8628.764839] LNet: Added LNI 192.168.201.149@tcp11 [8/256/0/180] [ 8630.059408] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 8630.069641] LNet: Added LNI 192.168.201.149@tcp12 [8/256/0/180] [ 8631.308063] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 8631.318383] LNet: Added LNI 192.168.201.149@tcp13 [8/256/0/180] [ 8632.565846] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 8632.575362] LNet: Added LNI 192.168.201.149@tcp14 [8/256/0/180] [ 8633.863885] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 8633.873811] LNet: Added LNI 192.168.201.149@tcp15 [8/256/0/180] [ 8635.205532] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 8635.217414] LNet: Added LNI 192.168.201.149@tcp16 [8/256/0/180] [ 8636.533349] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 8636.547058] LNet: Added LNI 192.168.201.149@tcp17 [8/256/0/180] [ 8638.003415] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 8638.014313] LNet: Added LNI 192.168.201.149@tcp18 [8/256/0/180] [ 8639.558623] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 8639.575355] LNet: Added LNI 192.168.201.149@tcp19 [8/256/0/180] [ 8641.063844] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 8641.072963] LNet: Added LNI 192.168.201.149@tcp20 [8/256/0/180] [ 8642.598963] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 8642.612925] LNet: Added LNI 192.168.201.149@tcp21 [8/256/0/180] [ 8644.198725] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 8644.214343] LNet: Added LNI 192.168.201.149@tcp22 [8/256/0/180] [ 8645.844257] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8645.853129] LNet: Added LNI 192.168.201.149@tcp23 [8/256/0/180] [ 8647.468501] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8647.478423] LNet: Added LNI 192.168.201.149@tcp24 [8/256/0/180] [ 8649.179663] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8649.191866] LNet: Added LNI 192.168.201.149@tcp25 [8/256/0/180] [ 8651.031411] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8651.043516] LNet: Added LNI 192.168.201.149@tcp26 [8/256/0/180] [ 8652.864953] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8652.882192] LNet: Added LNI 192.168.201.149@tcp27 [8/256/0/180] [ 8654.622456] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8654.634298] LNet: Added LNI 192.168.201.149@tcp28 [8/256/0/180] [ 8656.445860] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8656.458529] LNet: Added LNI 192.168.201.149@tcp29 [8/256/0/180] [ 8658.283918] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8658.294571] LNet: Added LNI 192.168.201.149@tcp30 [8/256/0/180] [ 8660.128963] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8660.141707] LNet: Added LNI 192.168.201.149@tcp31 [8/256/0/180] [ 8662.211164] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8662.225722] LNet: Added LNI 192.168.201.149@tcp32 [8/256/0/180] [ 8664.314748] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8664.326321] LNet: Added LNI 192.168.201.149@tcp33 [8/256/0/180] [ 8666.534443] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8666.545823] LNet: Added LNI 192.168.201.149@tcp34 [8/256/0/180] [ 8668.720516] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8668.732777] LNet: Added LNI 192.168.201.149@tcp35 [8/256/0/180] [ 8670.848630] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8670.860179] LNet: Added LNI 192.168.201.149@tcp36 [8/256/0/180] [ 8673.000560] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8673.013264] LNet: Added LNI 192.168.201.149@tcp37 [8/256/0/180] [ 8675.301208] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8675.313930] LNet: Added LNI 192.168.201.149@tcp38 [8/256/0/180] [ 8677.544413] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8677.555734] LNet: Added LNI 192.168.201.149@tcp39 [8/256/0/180] [ 8679.795505] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8679.808014] LNet: Added LNI 192.168.201.149@tcp40 [8/256/0/180] [ 8682.142577] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 8682.156532] LNet: Added LNI 192.168.201.149@tcp41 [8/256/0/180] [ 8684.704594] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8684.723123] LNet: Added LNI 192.168.201.149@tcp42 [8/256/0/180] [ 8687.268089] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 8687.279576] LNet: Added LNI 192.168.201.149@tcp43 [8/256/0/180] [ 8690.169299] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 8690.181569] LNet: Added LNI 192.168.201.149@tcp44 [8/256/0/180] [ 8692.874983] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 8692.893870] LNet: Added LNI 192.168.201.149@tcp45 [8/256/0/180] [ 8695.594676] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 8695.607610] LNet: Added LNI 192.168.201.149@tcp46 [8/256/0/180] [ 8698.424663] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 8698.437663] LNet: Added LNI 192.168.201.149@tcp47 [8/256/0/180] [ 8701.564024] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 8701.579876] LNet: Added LNI 192.168.201.149@tcp48 [8/256/0/180] [ 8704.782966] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 8704.798250] LNet: Added LNI 192.168.201.149@tcp49 [8/256/0/180] [ 8707.719203] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 8707.732018] LNet: Added LNI 192.168.201.149@tcp50 [8/256/0/180] [ 8718.307137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8718.307159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8718.312205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8718.314474] Lustre: Skipped 1 previous similar message [ 8721.377179] Lustre: server umount lustre-MDT0000 complete [ 8722.663249] LustreError: 443437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780094635 with bad export cookie 4943343323531390007 [ 8722.666449] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8722.672256] LustreError: 443437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8722.798100] Lustre: server umount lustre-MDT0001 complete [ 8730.426195] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8733.403093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8738.784109] Lustre: 441933:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780094635/real 1780094635] req@00000000663d17c6 x1866564382047744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780094651 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8738.794723] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8738.799730] Lustre: Skipped 1 previous similar message [ 8742.880133] Lustre: 441936:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780094639/real 1780094639] req@0000000023537235 x1866564382047872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780094655 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8743.904216] Lustre: 441936:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780094640/real 1780094640] req@00000000ab110fc2 x1866564382048000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780094656 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8748.000811] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x449a488fd1e3345d [ 8748.003764] Lustre: MGC192.168.201.149@tcp: Connection restored to 0@lo (at 0@lo) [ 8748.110687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8749.024089] Lustre: 441933:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780094645/real 1780094645] req@000000004e9c91b2 x1866564382048256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780094661 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8749.395913] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8752.167774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8752.280224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8753.552391] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8754.272168] Lustre: 441933:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780094650/real 1780094650] req@0000000096bd101f x1866564382048384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780094666 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8755.354217] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8756.570627] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8757.733332] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8758.755567] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8767.969465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8767.972077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8767.977198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8767.979441] Lustre: Skipped 2 previous similar messages [ 8768.994269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8768.996359] Lustre: Skipped 1 previous similar message [ 8770.068822] Lustre: server umount lustre-OST0000 complete [ 8773.088741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8773.090893] LustreError: Skipped 1 previous similar message [ 8773.092066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8773.095300] Lustre: Skipped 1 previous similar message [ 8773.097385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8777.501061] Lustre: server umount lustre-MDT0000 complete [ 8778.825250] LustreError: 443437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780094691 with bad export cookie 4943343323531392093 [ 8778.828349] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8778.833586] LustreError: 443437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8778.944167] Lustre: server umount lustre-MDT0001 complete [ 8782.792903] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8783.838810] Key type lgssc unregistered [ 8783.992096] LNet: 470291:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8783.995184] LNetError: 470291:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8783.997882] LNet: Removed LNI 192.168.201.149@tcp [ 8783.999409] LNet: Removed LNI 192.168.201.149@tcp10 [ 8784.000704] LNet: Removed LNI 192.168.201.149@tcp11 [ 8784.001776] LNet: Removed LNI 192.168.201.149@tcp12 [ 8784.002780] LNet: Removed LNI 192.168.201.149@tcp13 [ 8784.004219] LNet: Removed LNI 192.168.201.149@tcp14 [ 8784.005760] LNet: Removed LNI 192.168.201.149@tcp15 [ 8784.007277] LNet: Removed LNI 192.168.201.149@tcp16 [ 8784.008843] LNet: Removed LNI 192.168.201.149@tcp17 [ 8784.010439] LNet: Removed LNI 192.168.201.149@tcp18 [ 8784.011567] LNet: Removed LNI 192.168.201.149@tcp19 [ 8784.012657] LNet: Removed LNI 192.168.201.149@tcp20 [ 8784.013731] LNet: Removed LNI 192.168.201.149@tcp21 [ 8784.014819] LNet: Removed LNI 192.168.201.149@tcp22 [ 8784.015917] LNet: Removed LNI 192.168.201.149@tcp23 [ 8784.016989] LNet: Removed LNI 192.168.201.149@tcp24 [ 8784.018062] LNet: Removed LNI 192.168.201.149@tcp25 [ 8784.019561] LNet: Removed LNI 192.168.201.149@tcp26 [ 8784.020724] LNet: Removed LNI 192.168.201.149@tcp27 [ 8784.021914] LNet: Removed LNI 192.168.201.149@tcp28 [ 8784.023112] LNet: Removed LNI 192.168.201.149@tcp29 [ 8784.024751] LNet: Removed LNI 192.168.201.149@tcp30 [ 8784.026277] LNet: Removed LNI 192.168.201.149@tcp31 [ 8784.027504] LNet: Removed LNI 192.168.201.149@tcp32 [ 8784.029196] LNet: Removed LNI 192.168.201.149@tcp33 [ 8784.030686] LNet: Removed LNI 192.168.201.149@tcp34 [ 8784.032419] LNet: Removed LNI 192.168.201.149@tcp35 [ 8784.033687] LNet: Removed LNI 192.168.201.149@tcp36 [ 8784.034962] LNet: Removed LNI 192.168.201.149@tcp37 [ 8784.036756] LNet: Removed LNI 192.168.201.149@tcp38 [ 8784.038630] LNet: Removed LNI 192.168.201.149@tcp39 [ 8784.040451] LNet: Removed LNI 192.168.201.149@tcp40 [ 8784.042165] LNet: Removed LNI 192.168.201.149@tcp41 [ 8784.043400] LNet: Removed LNI 192.168.201.149@tcp42 [ 8784.044825] LNet: Removed LNI 192.168.201.149@tcp43 [ 8784.046408] LNet: Removed LNI 192.168.201.149@tcp44 [ 8784.047853] LNet: Removed LNI 192.168.201.149@tcp45 [ 8784.049649] LNet: Removed LNI 192.168.201.149@tcp46 [ 8784.050893] LNet: Removed LNI 192.168.201.149@tcp47 [ 8784.052036] LNet: Removed LNI 192.168.201.149@tcp48 [ 8784.053441] LNet: Removed LNI 192.168.201.149@tcp49 [ 8784.060318] LNet: Removed LNI 192.168.201.149@tcp50 [ 8784.321390] Key type .llcrypt unregistered [ 8784.323116] Key type ._llcrypt unregistered [ 8790.869288] Key type ._llcrypt registered [ 8790.870511] Key type .llcrypt registered [ 8790.915960] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 8796.786076] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8797.071780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8797.094454] alg: No test for adler32 (adler32-zlib) [ 8797.945246] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 8798.021432] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 8799.600160] Key type lgssc registered [ 8799.938852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8811.366795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8811.373529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8812.462868] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8812.475906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8812.515859] Lustre: lustre-MDT0000: new disk, initializing [ 8812.541330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8812.548448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8813.923425] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8818.060824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8818.104136] Lustre: 473426:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8818.127811] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8818.130151] Lustre: Skipped 1 previous similar message [ 8818.170102] Lustre: lustre-MDT0001: new disk, initializing [ 8818.203490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8818.215710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8818.220146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8819.498794] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8822.509886] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8823.715242] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8826.232809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8826.304145] Lustre: *** cfs_fail_loc=90f, val=0*** [ 8826.330970] Lustre: lustre-OST0000: new disk, initializing [ 8826.332473] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8826.335713] Lustre: 475216:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8826.365488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8827.967437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8827.972036] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8828.014446] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8828.385703] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8831.567053] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8838.114868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8838.115058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8838.117813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8838.121430] Lustre: Skipped 1 previous similar message [ 8838.124928] Lustre: Skipped 1 previous similar message [ 8840.025507] Lustre: server umount lustre-OST0000 complete [ 8843.233535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8843.234332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8843.237537] Lustre: Skipped 1 previous similar message [ 8847.511361] Lustre: server umount lustre-MDT0000 complete [ 8848.717578] LustreError: 475215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780094761 with bad export cookie 14753751110347266693 [ 8848.720143] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8848.720737] LustreError: 475215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8848.834536] Lustre: server umount lustre-MDT0001 complete [ 8852.259301] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8853.288410] Key type lgssc unregistered [ 8853.414452] LNet: 476986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8853.417947] LNetError: 476986:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8853.431377] LNet: Removed LNI 192.168.201.149@tcp [ 8853.716663] Key type .llcrypt unregistered [ 8853.718229] Key type ._llcrypt unregistered [ 8860.918618] Key type ._llcrypt registered [ 8860.920140] Key type .llcrypt registered [ 8860.954117] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8861.273588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8861.326444] alg: No test for adler32 (adler32-zlib) [ 8862.176700] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 8862.263246] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 8863.840170] Key type lgssc registered [ 8864.189165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8866.802083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8866.809701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8867.962620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8869.226229] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8872.024275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8872.148532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8873.394164] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8875.055483] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8876.165557] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8878.403869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8878.512320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8880.319628] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8882.139225] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8886.659982] loop0: detected capacity change from 0 to 204800000 [ 8886.723782] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8886.728942] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8886.732078] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8888.643547] loop0: detected capacity change from 0 to 204800000 [ 8888.765081] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8888.768922] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8888.773911] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8890.377025] loop0: detected capacity change from 0 to 204800000 [ 8892.108457] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8892.154469] Lustre: fs15246-OST0000: new disk, initializing [ 8892.156229] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8892.158503] Lustre: 481466:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8892.175236] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8893.912498] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8897.683817] loop1: detected capacity change from 0 to 204800000 [ 8899.478850] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8899.522025] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8899.544986] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8899.582708] Lustre: fs15246-MDT0000: new disk, initializing [ 8899.639844] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8899.650049] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8899.681793] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8900.704757] Lustre: 482571:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000dff3ef8d x1866564657771136/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1780094829 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8900.874110] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8909.792300] LustreError: 482571:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 8909.795765] LustreError: 482571:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 8909.798695] LustreError: 482571:0:(osp_precreate.c:460:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 8912.357263] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8912.361063] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8914.320589] Lustre: server umount fs15246-MDT0000 complete [ 8917.416073] Lustre: server umount fs15246-OST0000 complete [ 8922.595077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8922.597305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8922.602878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8922.604575] Lustre: Skipped 1 previous similar message [ 8927.713345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8927.715264] Lustre: Skipped 1 previous similar message [ 8932.833493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8932.835767] Lustre: Skipped 1 previous similar message [ 8934.880122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8934.929663] Lustre: server umount lustre-OST0000 complete [ 8937.953598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8937.954319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8937.957904] Lustre: Skipped 2 previous similar messages [ 8937.961913] Lustre: Skipped 1 previous similar message [ 8942.305501] Lustre: server umount lustre-MDT0000 complete [ 8943.073146] LustreError: 478215:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8943.081066] LustreError: 478215:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8943.629334] LustreError: 478195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780094856 with bad export cookie 18240240400598482666 [ 8943.633111] LustreError: 478195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8943.633285] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8943.761882] Lustre: server umount lustre-MDT0001 complete [ 8947.246058] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 8948.239508] Key type lgssc unregistered [ 8948.359372] LNet: 485628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8948.362152] LNetError: 485628:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8948.369237] LNet: Removed LNI 192.168.201.149@tcp [ 8948.667504] Key type .llcrypt unregistered [ 8948.669059] Key type ._llcrypt unregistered [ 8955.541094] Key type ._llcrypt registered [ 8955.542246] Key type .llcrypt registered [ 8955.579332] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8956.007441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8956.019072] alg: No test for adler32 (adler32-zlib) [ 8956.865389] Lustre: Lustre: Build Version: 2.17.53_53_g8b67a55 [ 8956.949759] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 8958.536199] Key type lgssc registered [ 8958.945197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8970.796692] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 8973.698524] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 8984.969709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8984.975367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8986.063843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8986.075095] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8986.115298] Lustre: lustre-MDT0000: new disk, initializing [ 8986.137951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8986.147089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8987.339425] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8991.390723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8991.435888] Lustre: 490013:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8991.460278] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8991.462153] Lustre: Skipped 1 previous similar message [ 8991.493201] Lustre: lustre-MDT0001: new disk, initializing [ 8991.520632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8991.530689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8991.533824] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8992.705509] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 8995.805495] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8997.045974] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8999.522374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8999.638840] Lustre: lustre-OST0000: new disk, initializing [ 8999.640728] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8999.644408] Lustre: 491752:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8999.677695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9000.702683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9000.707522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9000.750225] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9001.553072] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug -1 all [ 9004.688383] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9006.039722] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9006.113551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9007.376483] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9007.447615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9010.130528] Lustre: server umount lustre-OST0000 complete [ 9012.192895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9012.194749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9012.199578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9016.289463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9016.289978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9021.409617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9021.411818] Lustre: Skipped 2 previous similar messages [ 9021.436082] Lustre: server umount lustre-MDT0000 complete [ 9022.653069] LustreError: 490789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780094935 with bad export cookie 9230992510401178094 [ 9022.657077] LustreError: 490789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9022.657349] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9033.231779] Lustre: server umount lustre-MDT0001 complete [ 9040.995531] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 9043.946992] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 9054.168132] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8867 sec ========== 18:49:27 (1780094967) [ 9054.718840] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:49:27 (1780094967) === [ 9055.781102] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:49:28 (1780094968) === [ 9068.664087] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 9069.635811] Key type lgssc unregistered [ 9069.748287] LNet: 498186:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9069.750412] LNetError: 498186:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9069.757291] LNet: Removed LNI 192.168.201.149@tcp [ 9070.035355] Key type .llcrypt unregistered [ 9070.036741] Key type ._llcrypt unregistered