[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 602026448 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001018] APIC: Switch to symmetric I/O mode setup [ 0.003345] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008354] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010018] pid_max: default: 32768 minimum: 301 [ 0.011156] LSM: Security Framework initializing [ 0.012059] Yama: becoming mindful. [ 0.013043] SELinux: Initializing. [ 0.014086] *** VALIDATE selinux *** [ 0.023624] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029108] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031149] *** VALIDATE tmpfs *** [ 0.032497] *** VALIDATE proc *** [ 0.034140] *** VALIDATE cgroup *** [ 0.035008] *** VALIDATE cgroup2 *** [ 0.037130] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038178] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039018] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041022] Spectre V2 : User space: Vulnerable [ 0.042012] Speculative Store Bypass: Vulnerable [ 0.045582] debug: unmapping init [mem 0xffffffffb0259000-0xffffffffb0260fff] [ 0.048231] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049732] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050032] ... version: 2 [ 0.051015] ... bit width: 48 [ 0.052010] ... generic registers: 4 [ 0.053012] ... value mask: 0000ffffffffffff [ 0.054015] ... max period: 00007fffffffffff [ 0.055020] ... fixed-purpose events: 3 [ 0.056013] ... event mask: 000000070000000f [ 0.057359] rcu: Hierarchical SRCU implementation. [ 0.059632] smp: Bringing up secondary CPUs ... [ 0.060712] x86: Booting SMP configuration: [ 0.061023] .... node #0, CPUs: #1 #2 #3 [ 0.066123] smp: Brought up 1 node, 4 CPUs [ 0.068020] smpboot: Max logical packages: 1 [ 0.069015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.161902] node 0 deferred pages initialised in 87ms [ 0.165216] devtmpfs: initialized [ 0.167347] x86/mm: Memory block size: 128MB [ 0.169921] gcov: version magic: 0x41383552 [ 0.171321] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.172122] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.173324] pinctrl core: initialized pinctrl subsystem [ 0.174269] [ 0.175000] ************************************************************* [ 0.176020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.177021] ** ** [ 0.178019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.179026] ** ** [ 0.180019] ** This means that this kernel is built to expose internal ** [ 0.181017] ** IOMMU data structures, which may compromise security on ** [ 0.182024] ** your system. ** [ 0.183021] ** ** [ 0.184019] ** If you see this message and you are not debugging the ** [ 0.185017] ** kernel, report this immediately to your vendor! ** [ 0.186020] ** ** [ 0.187017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.188014] ************************************************************* [ 0.189891] NET: Registered protocol family 16 [ 0.190542] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.191066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.192096] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.196046] cpuidle: using governor menu [ 0.199975] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.202670] PCI: Using configuration type 1 for base access [ 0.204144] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.213157] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.216050] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.220044] cryptd: max_cpu_qlen set to 1000 [ 0.224386] ACPI: Added _OSI(Module Device) [ 0.226016] ACPI: Added _OSI(Processor Device) [ 0.227013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.229015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.233029] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.240673] ACPI: Interpreter enabled [ 0.242072] ACPI: PM: (supports S0 S3 S4 S5) [ 0.243013] ACPI: Using IOAPIC for interrupt routing [ 0.245126] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.249481] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.260000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.260044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.263021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.266159] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.272554] acpiphp: Slot [2] registered [ 0.274168] acpiphp: Slot [5] registered [ 0.275123] acpiphp: Slot [6] registered [ 0.277188] acpiphp: Slot [7] registered [ 0.278151] acpiphp: Slot [8] registered [ 0.280197] acpiphp: Slot [9] registered [ 0.281205] acpiphp: Slot [10] registered [ 0.283151] acpiphp: Slot [3] registered [ 0.285139] acpiphp: Slot [4] registered [ 0.286133] acpiphp: Slot [11] registered [ 0.288093] acpiphp: Slot [12] registered [ 0.290163] acpiphp: Slot [13] registered [ 0.291096] acpiphp: Slot [14] registered [ 0.292091] acpiphp: Slot [15] registered [ 0.294100] acpiphp: Slot [16] registered [ 0.295088] acpiphp: Slot [17] registered [ 0.297110] acpiphp: Slot [18] registered [ 0.298086] acpiphp: Slot [19] registered [ 0.300095] acpiphp: Slot [20] registered [ 0.302111] acpiphp: Slot [21] registered [ 0.303114] acpiphp: Slot [22] registered [ 0.304099] acpiphp: Slot [23] registered [ 0.305151] acpiphp: Slot [24] registered [ 0.307118] acpiphp: Slot [25] registered [ 0.309109] acpiphp: Slot [26] registered [ 0.310115] acpiphp: Slot [27] registered [ 0.311108] acpiphp: Slot [28] registered [ 0.312098] acpiphp: Slot [29] registered [ 0.313093] acpiphp: Slot [30] registered [ 0.314205] acpiphp: Slot [31] registered [ 0.316075] PCI host bridge to bus 0000:00 [ 0.317023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.319027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.321034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.323028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.325031] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.328035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.329190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.332853] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.335469] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.344021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.350014] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.352023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.354022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.357019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.362496] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.366385] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.370060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.373952] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.380022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.394021] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.399022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.405268] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.411021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.417021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.434029] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.449000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.457021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.468026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.481016] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.489569] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.497020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.502024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.517025] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.529413] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.539022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.549020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.564028] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.576000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.581025] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.588030] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.605036] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.612000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.617015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.622020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.634019] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.649637] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.651490] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.654397] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.655000] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.656073] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.662616] iommu: Default domain type: Passthrough [ 0.664692] SCSI subsystem initialized [ 0.666166] ACPI: bus type USB registered [ 0.668111] usbcore: registered new interface driver usbfs [ 0.670087] usbcore: registered new interface driver hub [ 0.672092] usbcore: registered new device driver usb [ 0.673199] pps_core: LinuxPPS API ver. 1 registered [ 0.675013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.679522] PTP clock support registered [ 0.681148] EDAC MC: Ver: 3.0.0 [ 0.683134] PCI: Using ACPI for IRQ routing [ 0.686647] NetLabel: Initializing [ 0.688012] NetLabel: domain hash size = 128 [ 0.690011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.692107] NetLabel: unlabeled traffic allowed by default [ 0.696255] vgaarb: loaded [ 0.698298] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.700013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.706130] clocksource: Switched to clocksource kvm-clock [ 0.834957] VFS: Disk quotas dquot_6.6.0 [ 0.836660] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.839603] *** VALIDATE ramfs *** [ 0.840972] *** VALIDATE hugetlbfs *** [ 0.842776] pnp: PnP ACPI init [ 0.845777] pnp: PnP ACPI: found 6 devices [ 0.863446] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.867051] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.869355] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.871555] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.873946] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.876466] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.879247] NET: Registered protocol family 2 [ 0.881648] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.886854] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.890469] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.896369] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.899586] TCP: Hash tables configured (established 65536 bind 65536) [ 0.902351] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.905840] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.909125] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.912426] NET: Registered protocol family 1 [ 0.916052] RPC: Registered named UNIX socket transport module. [ 0.919871] RPC: Registered udp transport module. [ 0.921828] RPC: Registered tcp transport module. [ 0.924354] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.927931] NET: Registered protocol family 44 [ 0.929616] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.931997] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.933963] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.936137] PCI: CLS 0 bytes, default 64 [ 0.937954] Unpacking initramfs... [ 2.398558] debug: unmapping init [mem 0xffff9ba33cc54000-0xffff9ba33ffbffff] [ 2.409699] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.411743] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.414712] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.926647] Initialise system trusted keyrings [ 2.928208] Key type blacklist registered [ 2.929987] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.938964] zbud: loaded [ 2.941427] *** VALIDATE nfs *** [ 2.942459] *** VALIDATE nfs4 *** [ 2.943974] pstore: using deflate compression [ 2.946780] Platform Keyring initialized [ 3.081612] NET: Registered protocol family 38 [ 3.083282] Key type asymmetric registered [ 3.084661] Asymmetric key parser 'x509' registered [ 3.094750] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.102613] io scheduler mq-deadline registered [ 3.105082] io scheduler kyber registered [ 3.106917] io scheduler bfq registered [ 3.108720] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.112200] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.115115] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.117922] ACPI: Power Button [PWRF] [ 3.212980] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.305717] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.501273] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.594739] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.791534] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.822767] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.867358] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.875382] Non-volatile memory driver v1.3 [ 3.877941] Linux agpgart interface v0.103 [ 3.906100] virtio_blk virtio1: [vda] 133864 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.908855] vda: detected capacity change from 0 to 68538368 [ 3.926369] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.929587] vdb: detected capacity change from 0 to 1073741824 [ 3.965857] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.969764] vdc: detected capacity change from 0 to 2621440000 [ 4.013911] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.018257] vdd: detected capacity change from 0 to 2621440000 [ 4.037176] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.042390] vde: detected capacity change from 0 to 4294967296 [ 4.063352] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.067251] vdf: detected capacity change from 0 to 4294967296 [ 4.080231] libphy: Fixed MDIO Bus: probed [ 4.088399] usbcore: registered new interface driver usbserial_generic [ 4.090438] usbserial: USB Serial support registered for generic [ 4.092522] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.096791] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.098453] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.101184] mousedev: PS/2 mouse device common for all mice [ 4.105228] rtc_cmos 00:05: RTC can wake from S4 [ 4.108272] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.112962] rtc_cmos 00:05: registered as rtc0 [ 4.113046] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.113090] intel_pstate: CPU model not supported [ 4.121077] hid: raw HID events driver (C) Jiri Kosina [ 4.132986] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.133044] usbcore: registered new interface driver usbhid [ 4.136748] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.137660] usbhid: USB HID core driver [ 4.141294] drop_monitor: Initializing network drop monitor service [ 4.143191] Initializing XFRM netlink socket [ 4.144958] NET: Registered protocol family 10 [ 4.148541] Segment Routing with IPv6 [ 4.149621] NET: Registered protocol family 17 [ 4.151276] mpls_gso: MPLS GSO support [ 4.156596] RAS: Correctable Errors collector initialized. [ 4.158482] AVX version of gcm_enc/dec engaged. [ 4.160205] AES CTR mode by8 optimization enabled [ 4.237957] sched_clock: Marking stable (4237936141, 0)->(5308567668, -1070631527) [ 4.240940] registered taskstats version 1 [ 4.242592] Loading compiled-in X.509 certificates [ 4.246212] zswap: loaded using pool lzo/zbud [ 4.275478] Key type big_key registered [ 4.287772] Key type encrypted registered [ 4.288981] ima: No TPM chip found, activating TPM-bypass! [ 4.290401] ima: Allocated hash algorithm: sha1 [ 4.291718] ima: No architecture policies found [ 4.293197] evm: Initialising EVM extended attributes: [ 4.294733] evm: security.selinux [ 4.295782] evm: security.ima [ 4.296694] evm: security.capability [ 4.297744] evm: HMAC attrs: 0x1 [ 4.299929] rtc_cmos 00:05: setting system clock to 2025-11-25 21:24:06 UTC (1764105846) [ 4.306431] debug: unmapping init [mem 0xffffffffb1203000-0xffffffffb13fffff] [ 4.308807] debug: unmapping init [mem 0xffffffffaff82000-0xffffffffb0258fff] [ 4.318133] Write protecting the kernel read-only data: 28672k [ 4.321674] debug: unmapping init [mem 0xffffffffae603000-0xffffffffae7fffff] [ 4.324345] debug: unmapping init [mem 0xffffffffaef14000-0xffffffffaeffffff] [ 4.408220] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.422451] systemd[1]: Detected virtualization kvm. [ 4.427610] systemd[1]: Detected architecture x86-64. [ 4.429628] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.459874] systemd[1]: No hostname configured. [ 4.462591] systemd[1]: Set hostname to . [ 4.466359] random: systemd: uninitialized urandom read (16 bytes read) [ 4.470304] systemd[1]: Initializing machine ID from random generator. [ 4.668561] random: systemd: uninitialized urandom read (16 bytes read) [ 4.670632] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.677094] random: systemd: uninitialized urandom read (16 bytes read) [ 4.680312] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.690444] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.564549] device-mapper: uevent: version 1.0.3 [ 5.566654] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook...[ 6.381310] random: fast init done [ 6.444380] virtio_net virtio0 ens2: renamed from eth0 [ 6.701051] scsi host0: ata_piix [ 6.728569] scsi host1: ata_piix [ 6.732074] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.736174] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.830771] random: crng init done [ 11.832931] random: 7 urandom warning(s) missed due to ratelimiting [ 11.883158] dracut-initqueue[579]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 13.288519] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.749706] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.188585] SELinux: Disabled at runtime. [ 15.261502] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.273485] systemd[1]: Detected virtualization kvm. [ 15.275600] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.080153] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.091970] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.104213] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.111012] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.120959] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.133147] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.146947] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ 16.213522] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 17.001165] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.566794] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.676366] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.889324] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 17.950268] EDAC sbridge: Ver: 1.1.2 [ 20.570920] Key type dns_resolver registered [ 20.948268] NFS: Registering the id_resolver key type [ 20.950353] Key type id_resolver registered [ 20.952818] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg142-server login: [ 60.124308] libcfs: loading out-of-tree module taints kernel. [ 60.136058] Key type ._llcrypt registered [ 60.137500] Key type .llcrypt registered [ 60.181164] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 68.565408] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 69.355913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 69.366787] alg: No test for adler32 (adler32-zlib) [ 70.418142] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 70.759582] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 72.400203] Key type lgssc registered [ 73.241901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 81.064502] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 81.843275] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.131247] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 89.631321] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 93.101509] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 100.545522] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 106.048379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 106.083089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 106.098108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 107.241398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 107.259320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 107.311313] Lustre: lustre-MDT0000: new disk, initializing [ 107.374220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 107.393641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 109.093864] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 115.842390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 115.891791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 115.938235] Lustre: 6478:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 115.955446] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 115.958277] Lustre: Skipped 1 previous similar message [ 116.012456] Lustre: lustre-MDT0001: new disk, initializing [ 116.057691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 116.081857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 116.087311] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 117.985764] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 120.551376] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 124.535803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 124.570973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 124.693530] Lustre: lustre-OST0000: new disk, initializing [ 124.696236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 124.721325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 126.959382] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 130.553770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 130.558404] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 130.602153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 133.208404] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 133.236877] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 133.282036] Lustre: lustre-OST0001: new disk, initializing [ 133.284591] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 133.306432] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 135.580584] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 135.920995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 135.925955] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 135.935131] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 141.977407] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 149.175823] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 156.649630] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing check_logdir /tmp/testlogs/ [ 158.702382] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing yml_node [ 161.010033] Lustre: DEBUG MARKER: Client: 2.16.61.56 [ 162.212563] Lustre: DEBUG MARKER: MDS: 2.16.61.56 [ 163.404180] Lustre: DEBUG MARKER: OSS: 2.16.61.56 [ 164.227214] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Nov 25 16:26:45 EST 2025 [ 171.910138] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 172.626623] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 187.360952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 187.361603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 187.366378] Lustre: Skipped 3 previous similar messages [ 187.368981] Lustre: Skipped 2 previous similar messages [ 191.064423] LustreError: 12182:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 191.126876] Lustre: server umount lustre-MDT0000 complete [ 192.991304] LustreError: 6471:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764106035 with bad export cookie 6547170363246153695 [ 192.992204] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 192.997365] LustreError: 6471:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 197.601872] LustreError: 6487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 197.602686] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 197.603365] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 197.603370] Lustre: Skipped 1 previous similar message [ 197.608381] LustreError: 6487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 197.622157] Lustre: Skipped 1 previous similar message [ 202.720806] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 202.720808] LustreError: 6487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 202.731820] Lustre: Skipped 2 previous similar messages [ 203.744808] LustreError: 7929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 203.757270] LustreError: 7929:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 207.328698] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 207.381690] LustreError: 12384:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 207.384531] LustreError: 12384:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 207.465385] Lustre: server umount lustre-MDT0001 complete [ 209.403454] LustreError: 12586:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 209.406422] LustreError: 12586:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 209.432624] Lustre: server umount lustre-OST0000 complete [ 211.302235] Lustre: server umount lustre-OST0001 complete [ 214.334413] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 218.761474] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 223.531369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 226.732882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 229.821914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 233.041242] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 236.462136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 236.491736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 236.633376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 236.651628] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 236.697253] Lustre: lustre-MDT0000: new disk, initializing [ 236.742116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 236.752979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 238.606832] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 244.927250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 244.972637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 245.032202] Lustre: 14965:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 245.078912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 245.081823] Lustre: Skipped 1 previous similar message [ 245.129232] Lustre: lustre-MDT0001: new disk, initializing [ 245.198568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 245.219097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 245.226236] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 247.382445] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 251.279373] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 253.026395] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 255.910752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 255.948890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 257.612922] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 257.658858] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 258.377737] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 262.340685] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 264.388617] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 264.514581] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 266.547217] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 266.660514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 268.259607] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 268.260975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 268.264674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 268.268741] Lustre: Skipped 1 previous similar message [ 268.270845] Lustre: Skipped 2 previous similar messages [ 273.379204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 273.382852] Lustre: Skipped 1 previous similar message [ 273.722137] LustreError: 17902:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 273.728137] LustreError: 17902:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 273.794220] Lustre: server umount lustre-OST0000 complete [ 275.937236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 275.941636] LustreError: Skipped 1 previous similar message [ 275.944854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 283.620557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 283.625273] Lustre: Skipped 4 previous similar messages [ 290.272938] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 290.324512] LustreError: 18104:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 290.327044] LustreError: 18104:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 290.373513] Lustre: server umount lustre-MDT0000 complete [ 292.173379] LustreError: 16712:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764106134 with bad export cookie 6547170363246155228 [ 292.176320] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 292.178106] LustreError: 16712:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 292.332920] Lustre: server umount lustre-MDT0001 complete [ 294.213694] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 16:28:55 (1764106135) [ 295.412696] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 296.186835] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 16:28:57 (1764106137) [ 299.027916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 299.228855] LustreError: 18976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 299.263146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 299.265811] Lustre: Skipped 1 previous similar message [ 300.883086] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 304.433405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 306.199920] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 308.549671] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 310.101119] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 312.948965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 315.538086] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 317.914700] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 334.820165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 334.825162] Lustre: Skipped 1 previous similar message [ 334.829065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 334.831453] Lustre: Skipped 2 previous similar messages [ 334.842274] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 337.586784] LustreError: 21240:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 337.589224] LustreError: 21240:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 337.644642] Lustre: server umount lustre-OST0000 complete [ 345.784657] Lustre: server umount lustre-MDT0000 complete [ 347.602057] LustreError: 21141:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764106189 with bad export cookie 6547170363246156110 [ 347.606940] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 347.608265] LustreError: 21141:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 347.826453] Lustre: server umount lustre-MDT0001 complete [ 354.349042] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 355.841887] Key type lgssc unregistered [ 356.017773] LNet: 22177:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.021726] LNetError: 22177:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 356.032748] LNet: Removed LNI 192.168.201.142@tcp [ 356.521185] Key type .llcrypt unregistered [ 356.523081] Key type ._llcrypt unregistered [ 369.330171] Key type ._llcrypt registered [ 369.331849] Key type .llcrypt registered [ 369.393963] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 369.833430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 369.877235] alg: No test for adler32 (adler32-zlib) [ 370.788352] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 370.909880] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 372.520303] Key type lgssc registered [ 373.121594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 377.038632] loop: module loaded [ 377.077592] loop0: detected capacity change from 0 to 204800000 [ 377.167522] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 377.175877] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 377.323179] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 377.357061] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 379.930968] loop0: detected capacity change from 0 to 204800000 [ 379.999237] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 380.004587] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 380.011378] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 380.089064] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 382.249210] loop0: detected capacity change from 0 to 204800000 [ 384.607768] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 384.629637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 384.635227] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 385.747660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 385.778980] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 385.822634] Lustre: lustre-MDT0000: new disk, initializing [ 385.858502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 385.866240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 387.616733] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 392.288308] loop1: detected capacity change from 0 to 204800000 [ 394.738523] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 394.769571] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 394.894534] Lustre: lustre-OST07c6: new disk, initializing [ 394.896674] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 394.899502] Lustre: Skipped 1 previous similar message [ 394.927174] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 397.312097] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 400.411365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 400.415484] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 400.431727] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 405.461870] Lustre: 24315:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 405.465480] Lustre: 24315:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 405.468919] Lustre: 24315:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 405.473356] Lustre: 24315:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 405.477017] Lustre: 24315:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 405.480133] Lustre: 24315:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 418.292909] Lustre: 24314:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 418.297213] Lustre: 24314:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 418.300324] Lustre: 24314:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 418.303658] Lustre: 24314:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 418.307477] Lustre: 24314:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 418.311501] Lustre: 24314:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 422.280135] Lustre: 24315:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 422.285231] Lustre: 24315:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 422.288500] Lustre: 24315:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 422.292499] Lustre: 24315:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 422.297469] Lustre: 24315:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 422.301595] Lustre: 24315:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 422.307391] Lustre: 24315:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 422.313054] Lustre: 24315:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 422.317625] Lustre: 24315:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 422.322419] Lustre: 24315:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 422.327921] Lustre: 24315:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 422.332129] Lustre: 24315:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 425.437939] Lustre: 24315:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 425.442180] Lustre: 24315:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 425.445565] Lustre: 24315:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 425.447644] Lustre: 24315:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 425.449937] Lustre: 24315:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 425.453230] Lustre: 24315:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 425.455910] Lustre: 24315:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 425.459966] Lustre: 24315:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 425.463040] Lustre: 24315:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 425.466350] Lustre: 24315:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 425.469748] Lustre: 24315:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 425.473188] Lustre: 24315:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 432.854602] Lustre: 24315:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 432.861750] Lustre: 24315:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 3 previous similar messages [ 432.866770] Lustre: 24315:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 432.872223] Lustre: 24315:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 432.876130] Lustre: 24315:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 432.886749] Lustre: 24315:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 432.890952] Lustre: 24315:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 432.895015] Lustre: 24315:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 432.899235] Lustre: 24315:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 432.903391] Lustre: 24315:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 432.906632] Lustre: 24315:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 432.909324] Lustre: 24315:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 442.810298] Lustre: 24315:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 442.813911] Lustre: 24315:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 5 previous similar messages [ 442.817941] Lustre: 24315:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 442.821746] Lustre: 24315:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 442.825443] Lustre: 24315:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 442.828750] Lustre: 24315:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 442.832907] Lustre: 24315:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 442.836706] Lustre: 24315:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 442.840084] Lustre: 24315:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 442.843928] Lustre: 24315:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 442.848639] Lustre: 24315:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 442.852842] Lustre: 24315:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 487.393782] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 487.393979] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 487.402852] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 489.712098] LustreError: 26499:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 489.754122] Lustre: server umount lustre-OST07c6 complete [ 493.289751] LustreError: 26999:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 493.292417] LustreError: 26999:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 493.355739] Lustre: server umount lustre-MDT0000 complete [ 499.634771] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 500.873294] Key type lgssc unregistered [ 501.023525] LNet: 27832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 501.027122] LNetError: 27832:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 501.034393] LNet: Removed LNI 192.168.201.142@tcp [ 501.385700] Key type .llcrypt unregistered [ 501.387238] Key type ._llcrypt unregistered [ 510.211430] Key type ._llcrypt registered [ 510.213054] Key type .llcrypt registered [ 510.257423] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 510.653425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 510.725170] alg: No test for adler32 (adler32-zlib) [ 511.616227] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 511.721940] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 513.320181] Key type lgssc registered [ 513.772339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 517.070336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 517.081944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 518.301378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 520.023075] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 523.972553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 524.147627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 525.776704] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 528.008391] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 529.487260] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 532.485870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 532.646062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 535.006467] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 537.340685] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 542.903502] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 555.489815] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 555.493266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 555.499985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 556.006471] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 556.010067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.016302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 558.562137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 558.566844] Lustre: Skipped 1 previous similar message [ 559.663240] LustreError: 31529:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 559.718773] Lustre: server umount lustre-OST0000 complete [ 563.681962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 563.688120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 563.690331] Lustre: Skipped 1 previous similar message [ 567.621703] LustreError: 31731:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 567.624064] LustreError: 31731:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 567.657576] Lustre: server umount lustre-MDT0000 complete [ 568.801258] LustreError: 29062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.809577] LustreError: 29062:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 569.183447] LustreError: 31530:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764106411 with bad export cookie 10280652324762003524 [ 569.187491] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 569.187910] LustreError: 31530:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 569.269447] LustreError: 31932:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 569.273052] LustreError: 31932:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 569.344644] Lustre: server umount lustre-MDT0001 complete [ 573.872164] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 575.080869] Key type lgssc unregistered [ 575.233978] LNet: 32464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 575.238393] LNetError: 32464:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 575.254076] LNet: Removed LNI 192.168.201.142@tcp [ 575.598437] Key type .llcrypt unregistered [ 575.600259] Key type ._llcrypt unregistered [ 583.925492] Key type ._llcrypt registered [ 583.927197] Key type .llcrypt registered [ 583.971287] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 584.419724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 584.431375] alg: No test for adler32 (adler32-zlib) [ 585.315915] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 585.421930] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 587.016198] Key type lgssc registered [ 587.473742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 590.603439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 590.614918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 591.813725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 593.328491] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 596.905670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 597.079539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 598.641240] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 600.971736] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 602.406762] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 605.192865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 605.352914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 607.697960] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 609.983376] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 616.615561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 627.170954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 627.174707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 627.176381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 627.180139] Lustre: Skipped 1 previous similar message [ 631.856097] LustreError: 35959:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 631.898302] Lustre: server umount lustre-OST0000 complete [ 637.411444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 637.412599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 637.417774] Lustre: Skipped 1 previous similar message [ 637.421040] Lustre: Skipped 1 previous similar message [ 639.492321] LustreError: 36161:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 639.496082] LustreError: 36161:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 639.538642] Lustre: server umount lustre-MDT0000 complete [ 641.157692] LustreError: 36162:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764106483 with bad export cookie 9004525215309629132 [ 641.160662] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 641.161277] LustreError: 36162:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 641.227431] LustreError: 36364:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 641.230469] LustreError: 36364:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 641.291737] Lustre: server umount lustre-MDT0001 complete [ 645.562313] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 646.726288] Key type lgssc unregistered [ 646.858463] LNet: 36894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 646.861201] LNetError: 36894:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 646.868462] LNet: Removed LNI 192.168.201.142@tcp [ 647.205646] Key type .llcrypt unregistered [ 647.207132] Key type ._llcrypt unregistered [ 658.396625] Key type ._llcrypt registered [ 658.398173] Key type .llcrypt registered [ 658.447961] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 658.859973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 658.926849] alg: No test for adler32 (adler32-zlib) [ 659.816106] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 659.932569] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 661.537048] Key type lgssc registered [ 662.078813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 665.523976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 665.538174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 666.763264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 668.473265] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 672.601560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 672.784503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 674.532147] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 677.236880] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 678.916814] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 682.171902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 682.333357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 684.781602] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 687.009709] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 691.560766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 702.947024] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 702.948506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 702.952180] Lustre: Skipped 2 previous similar messages [ 702.954689] Lustre: Skipped 1 previous similar message [ 703.864589] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.42@tcp (stopping) [ 706.054645] LustreError: 40536:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 706.093129] Lustre: server umount lustre-MDT0000 complete [ 707.715367] LustreError: 38251:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764106549 with bad export cookie 3393165051304241468 [ 707.718555] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 707.720401] LustreError: 38251:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 708.065835] LustreError: 40428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 708.066118] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 708.071872] LustreError: 40428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 708.077920] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 708.967250] LustreError: 40428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 712.097117] LustreError: 38265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 712.097144] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 712.110474] Lustre: Skipped 1 previous similar message [ 713.866875] LustreError: 40739:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 713.870210] LustreError: 40739:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 713.939360] Lustre: server umount lustre-MDT0001 complete [ 725.040843] LustreError: 38251:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.42@tcp arrived at 1764106567 with bad export cookie 3393165051304242084 [ 725.047585] LustreError: 38251:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 740.320169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 740.342415] LustreError: 40961:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 740.344635] LustreError: 40961:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 740.386395] Lustre: server umount lustre-OST0000 complete [ 747.592456] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 748.887721] Key type lgssc unregistered [ 749.044394] LNet: 41798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.048333] LNetError: 41798:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 749.055504] LNet: Removed LNI 192.168.201.142@tcp [ 749.417188] Key type .llcrypt unregistered [ 749.419791] Key type ._llcrypt unregistered [ 762.248737] Key type ._llcrypt registered [ 762.250687] Key type .llcrypt registered [ 762.302452] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 762.740541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 762.823990] alg: No test for adler32 (adler32-zlib) [ 763.724466] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 763.839768] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 765.441075] Key type lgssc registered [ 765.876195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 769.376546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 769.391916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 770.617970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 772.303818] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 776.561240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 776.774170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 778.666870] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 781.158597] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 782.821486] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 785.977606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 786.024617] LustreError: 44770:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 786.033215] LustreError: 44770:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 786.159990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 788.619373] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 790.902992] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 795.631112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 800.739548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 800.741470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 800.744765] Lustre: Skipped 1 previous similar message [ 800.749847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 801.648654] Lustre: lustre-OST0000: Not available for connect from 192.168.201.42@tcp (stopping) [ 801.651724] Lustre: Skipped 1 previous similar message [ 805.858282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 805.860707] Lustre: Skipped 1 previous similar message [ 810.979364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 810.982157] Lustre: Skipped 2 previous similar messages [ 813.028201] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 813.056294] LustreError: 45417:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 813.118066] Lustre: server umount lustre-OST0000 complete [ 827.872881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 827.875418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 827.880128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 827.882147] Lustre: Skipped 1 previous similar message [ 831.972085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 832.706263] LustreError: 45795:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 832.709031] LustreError: 45795:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 832.742302] Lustre: server umount lustre-MDT0000 complete [ 834.255098] LustreError: 45544:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764106676 with bad export cookie 13720165639276480160 [ 834.255953] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 834.259170] LustreError: 45544:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 840.386973] LustreError: 45996:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 840.388986] LustreError: 45996:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 840.450704] Lustre: server umount lustre-MDT0001 complete [ 845.069579] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 846.302259] Key type lgssc unregistered [ 846.457873] LNet: 46529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.461668] LNetError: 46529:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 846.472809] LNet: Removed LNI 192.168.201.142@tcp [ 846.855729] Key type .llcrypt unregistered [ 846.857360] Key type ._llcrypt unregistered [ 858.151384] Key type ._llcrypt registered [ 858.152911] Key type .llcrypt registered [ 858.194732] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 858.670812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 858.684706] alg: No test for adler32 (adler32-zlib) [ 859.575085] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 859.673188] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 861.264217] Key type lgssc registered [ 861.711827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 864.830294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 864.842541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 866.046882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 867.719233] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 871.347584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 871.499106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 873.036452] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 875.092544] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 876.498955] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 879.207315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 879.357123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 881.693567] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 884.116625] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 892.648936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 898.320420] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 899.975925] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 910.903463] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-11-2516h39m12s [ 911.615611] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 913.380252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 913.382511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 913.385367] Lustre: Skipped 2 previous similar messages [ 913.387955] Lustre: Skipped 1 previous similar message [ 914.401988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 914.406425] Lustre: Skipped 3 previous similar messages [ 916.340641] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.42@tcp (stopping) [ 918.732206] LustreError: 50487:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 918.780147] Lustre: server umount lustre-MDT0000 complete [ 919.522946] LustreError: 47899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 919.529866] LustreError: 47899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 920.442482] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 920.550731] LustreError: 47899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 920.559792] LustreError: 47899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 922.407762] LustreError: 47898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 924.150573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.231531] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 924.353468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 924.388506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 925.423574] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 926.417490] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 928.213497] Lustre: DEBUG MARKER: Wait for df (36013) ... [ 928.975537] Lustre: DEBUG MARKER: done [ 930.786925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 930.789426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 930.792408] Lustre: Skipped 1 previous similar message [ 930.795696] Lustre: Skipped 1 previous similar message [ 935.905441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 935.909182] Lustre: Skipped 2 previous similar messages [ 936.628186] LustreError: 51541:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 936.631204] LustreError: 51541:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 936.683774] Lustre: server umount lustre-OST0000 complete [ 941.027587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 941.027738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 944.712222] LustreError: 51744:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 944.716152] LustreError: 51744:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 944.761539] Lustre: server umount lustre-MDT0000 complete [ 946.609328] LustreError: 47883:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764106788 with bad export cookie 17465340224223163541 [ 946.614317] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 946.618431] LustreError: 47883:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 946.712497] LustreError: 51945:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 946.715629] LustreError: 51945:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 946.785313] Lustre: server umount lustre-MDT0001 complete [ 951.353828] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 952.597447] Key type lgssc unregistered [ 952.750085] LNet: 52476:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.753544] LNetError: 52476:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.762619] LNet: Removed LNI 192.168.201.142@tcp [ 953.142278] Key type .llcrypt unregistered [ 953.144595] Key type ._llcrypt unregistered [ 968.593023] Key type ._llcrypt registered [ 968.594721] Key type .llcrypt registered [ 968.640380] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 976.481954] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 977.012762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 977.022527] alg: No test for adler32 (adler32-zlib) [ 977.952261] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 978.076523] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 979.688342] Key type lgssc registered [ 980.236239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 984.343228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 987.400558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 990.419117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 993.670424] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 996.954572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 996.984282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 996.999355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 998.121248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 998.152229] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 998.204531] Lustre: lustre-MDT0000: new disk, initializing [ 998.276111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 998.291766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1000.244520] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1005.561635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1005.598212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1005.653256] Lustre: 56402:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1005.688457] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1005.691185] Lustre: Skipped 1 previous similar message [ 1005.733987] Lustre: lustre-MDT0001: new disk, initializing [ 1005.780399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1005.799660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1005.803093] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1007.496252] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1011.238921] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1012.847983] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1016.051679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1016.082843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1016.224853] Lustre: lustre-OST0000: new disk, initializing [ 1016.227848] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1016.278731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1016.369211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1016.373396] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1016.428887] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1018.509322] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1021.886195] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1023.454825] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1023.540270] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1025.074492] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1025.161422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1026.532636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1026.534623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1026.536781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1026.537605] Lustre: Skipped 1 previous similar message [ 1031.651354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1031.653917] Lustre: Skipped 2 previous similar messages [ 1036.772890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1036.775945] Lustre: Skipped 1 previous similar message [ 1040.352301] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1040.379116] LustreError: 59339:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1040.430060] Lustre: server umount lustre-OST0000 complete [ 1047.009658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1047.010613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1047.015536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1047.018452] LustreError: Skipped 1 previous similar message [ 1047.021181] Lustre: Skipped 1 previous similar message [ 1048.196121] LustreError: 59541:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1048.199088] LustreError: 59541:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1048.235597] Lustre: server umount lustre-MDT0000 complete [ 1049.715436] LustreError: 56392:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764106891 with bad export cookie 14863409201079145642 [ 1049.718480] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1049.720406] LustreError: 56392:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1049.783185] LustreError: 59743:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1049.785802] LustreError: 59743:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1049.845135] Lustre: server umount lustre-MDT0001 complete [ 1053.192704] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 16:41:35 (1764106895) [ 1055.733612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1055.934209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1057.380876] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1060.641209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1060.798276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1062.414129] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1064.566578] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1065.899931] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1068.608637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1070.759795] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1072.799594] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1087.887701] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-11-2516h42m09s [ 1089.433962] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1093.181138] hrtimer: interrupt took 3122290 ns [ 1098.681995] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1102.992354] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1103.062286] LustreError: 60408:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1123.292657] Lustre: lustre-MDT0000: Client 58c02132-7a31-4400-818b-36644c9d54f2 (at 192.168.201.42@tcp) reconnecting [ 1143.073254] LustreError: 60408:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1146.703124] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1150.501165] Lustre: DEBUG MARKER: done [ 1158.117228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1158.120211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1158.121626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1158.139060] Lustre: Skipped 2 previous similar messages [ 1158.154200] Lustre: Skipped 1 previous similar message [ 1164.105203] LustreError: 63438:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1164.109866] LustreError: 63438:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1164.319638] Lustre: server umount lustre-OST0000 complete [ 1168.353877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1168.355047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1168.360422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1168.366984] LustreError: Skipped 1 previous similar message [ 1168.377683] Lustre: Skipped 3 previous similar messages [ 1182.177594] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1182.238761] LustreError: 63639:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1182.241304] LustreError: 63639:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1182.301716] Lustre: server umount lustre-MDT0000 complete [ 1185.069598] LustreError: 60391:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764107027 with bad export cookie 14863409201079146524 [ 1185.076488] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1185.078671] LustreError: 60391:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1185.329576] Lustre: server umount lustre-MDT0001 complete [ 1193.253412] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 1195.677239] Key type lgssc unregistered [ 1196.004403] LNet: 64374:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1196.009955] LNetError: 64374:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1196.039867] LNet: Removed LNI 192.168.201.142@tcp [ 1196.924188] Key type .llcrypt unregistered [ 1196.927864] Key type ._llcrypt unregistered [ 1220.120841] Key type ._llcrypt registered [ 1220.122555] Key type .llcrypt registered [ 1220.206805] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 1231.550335] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 1232.428846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1232.446908] alg: No test for adler32 (adler32-zlib) [ 1233.418017] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 1233.613109] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 1235.248359] Key type lgssc registered [ 1235.982870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1242.134236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1247.037621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1251.545873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1256.237673] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1261.334399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1261.399525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1261.426877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1262.641410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1262.668415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1262.729848] Lustre: lustre-MDT0000: new disk, initializing [ 1262.810705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1262.831415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1265.710464] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1273.046572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1273.106208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1273.186843] Lustre: 68300:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1273.240787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1273.244128] Lustre: Skipped 1 previous similar message [ 1273.290763] Lustre: lustre-MDT0001: new disk, initializing [ 1273.360097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1273.402483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1273.418920] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1275.985500] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1281.446542] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1283.696285] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1288.174269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1288.235474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1288.381691] Lustre: lustre-OST0000: new disk, initializing [ 1288.388865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1288.443205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1292.381851] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1297.095441] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1297.433372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1297.439901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1297.514337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1299.655982] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1299.805113] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1303.001632] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1303.183392] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1307.618621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1307.619613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1307.628441] LustreError: Skipped 1 previous similar message [ 1307.637146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1312.738661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1312.743254] Lustre: Skipped 1 previous similar message [ 1317.860967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1317.863095] Lustre: Skipped 1 previous similar message [ 1318.880184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1318.915362] LustreError: 71239:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1319.006564] Lustre: server umount lustre-OST0000 complete [ 1322.982659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1322.991365] Lustre: Skipped 1 previous similar message [ 1322.994715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1322.997829] Lustre: Skipped 1 previous similar message [ 1327.835641] LustreError: 71442:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1327.854750] LustreError: 71442:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1327.958149] Lustre: server umount lustre-MDT0000 complete [ 1330.558895] LustreError: 70047:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764107172 with bad export cookie 3770063150323110100 [ 1330.561930] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1330.565528] LustreError: 70047:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1330.676326] LustreError: 71644:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1330.684083] LustreError: 71644:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1330.776932] Lustre: server umount lustre-MDT0001 complete [ 1336.260783] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 16:46:17 (1764107177) [ 1337.338139] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1338.632280] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 16:46:20 (1764107180) [ 1346.946045] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 1349.622856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1349.931635] LustreError: 72690:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1349.947833] LustreError: 72690:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1349.968323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.713799] Lustre: Failing over lustre-MDT0000 [ 1350.719481] LustreError: 72719:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1350.724967] LustreError: 72719:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1350.731677] LustreError: 72719:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -5 [ 1350.825073] LustreError: 72771:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1350.829419] LustreError: 72771:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1350.966410] Lustre: server umount lustre-MDT0000 complete [ 1355.638508] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 16:46:37 (1764107197) [ 1359.812458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1360.087051] LustreError: 73338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1360.124272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1362.374307] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1365.476416] LustreError: 73338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1367.558752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1369.880131] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1373.074941] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1375.291924] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1379.325246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1379.588928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1379.592233] Lustre: Skipped 1 previous similar message [ 1383.406918] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1387.176930] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1395.239681] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1401.800667] LustreError: 75683:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1401.804631] LustreError: 75683:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1401.908073] Lustre: server umount lustre-MDT0000 complete [ 1402.338889] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1402.340328] LustreError: 74049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1402.344678] Lustre: Skipped 3 previous similar messages [ 1402.351904] LustreError: 74049:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1404.594411] LustreError: 73317:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764107246 with bad export cookie 3770063150323111255 [ 1404.599821] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1404.599821] LustreError: 73317:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1404.874714] Lustre: server umount lustre-MDT0001 complete [ 1407.769509] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1412.892774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1422.818599] Lustre: 66515:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764107249/real 1764107249] req@000000009d226b1f x1849800340340096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764107265 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1422.867932] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1422.959086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1424.929100] Lustre: 66516:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764107251/real 1764107251] req@00000000b147701f x1849800340340224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764107267 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1427.936164] Lustre: 66513:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764107254/real 1764107254] req@0000000092d60d62 x1849800340340352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764107270 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1429.025049] LustreError: 66512:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002e18d789 x1849800340341248/t0(0) o250->MGC192.168.201.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1429.576796] LustreError: 74944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1429.721073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1429.896922] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1430.709261] Lustre: 66514:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764107256/real 1764107256] req@000000007ec905d3 x1849800340340608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764107272 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1434.336370] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1435.809128] Lustre: 66516:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764107261/real 1764107261] req@000000008fd862eb x1849800340340736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764107277 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1438.178212] LustreError: 76742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1438.191180] LustreError: 76742:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1440.467188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1442.794518] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1443.346698] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1446.293722] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1448.479662] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1453.030556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1453.034897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1453.042371] Lustre: Skipped 1 previous similar message [ 1453.048953] Lustre: Skipped 3 previous similar messages [ 1457.607883] LustreError: 78147:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1457.610210] LustreError: 78147:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1457.656649] Lustre: server umount lustre-MDT0000 complete [ 1458.145888] LustreError: 77467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1458.152854] LustreError: 77467:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1459.867633] LustreError: 73317:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764107302 with bad export cookie 3770063150323113159 [ 1459.875552] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1459.886206] LustreError: 73317:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1460.097277] Lustre: server umount lustre-MDT0001 complete [ 1463.992363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1469.729641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1470.433291] LustreError: 66512:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000030c925bb x1849800340363392/t0(0) o250->MGC192.168.201.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1470.627849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1470.632766] Lustre: Skipped 1 previous similar message [ 1470.670433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1472.797630] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1476.067196] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1476.073147] Lustre: Skipped 1 previous similar message [ 1476.769565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1478.736289] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1479.008105] Lustre: 66516:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764107305/real 1764107305] req@00000000062732b3 x1849800340363008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764107321 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1479.019554] Lustre: 66516:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1481.667822] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1482.213976] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1482.217115] Lustre: Skipped 1 previous similar message [ 1483.240248] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1483.875236] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1488.362852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1488.365724] Lustre: Skipped 2 previous similar messages [ 1493.987419] LustreError: 80599:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1493.991414] LustreError: 80599:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1494.061315] Lustre: server umount lustre-MDT0000 complete [ 1497.092790] LustreError: 73317:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764107339 with bad export cookie 3770063150323114531 [ 1497.095323] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1497.103944] LustreError: 73317:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1497.334907] Lustre: server umount lustre-MDT0001 complete [ 1503.315900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1506.278805] Lustre: DEBUG MARKER: files compared the same [ 1517.868750] Lustre: server umount lustre-OST0000 complete [ 1526.164198] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 1527.510700] Key type lgssc unregistered [ 1527.681612] LNet: 82231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1527.686311] LNetError: 82231:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1527.696484] LNet: Removed LNI 192.168.201.142@tcp [ 1528.169500] Key type .llcrypt unregistered [ 1528.170820] Key type ._llcrypt unregistered [ 1540.734447] Key type ._llcrypt registered [ 1540.736109] Key type .llcrypt registered [ 1540.787505] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 1541.203333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1541.216909] alg: No test for adler32 (adler32-zlib) [ 1542.165762] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 1542.313180] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 1543.928691] Key type lgssc registered [ 1544.540956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1548.339353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1548.355058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1549.608448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1551.522109] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug +malloc all [ 1555.550052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1555.702728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1557.490939] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug +malloc all [ 1559.976402] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1561.674068] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1564.915092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1565.074846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1567.858477] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug +malloc all [ 1570.463510] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1574.312633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1584.609335] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1584.612096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1584.620938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1585.636151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1585.636278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1585.644633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1590.198144] LustreError: 85870:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1590.256729] Lustre: server umount lustre-OST0000 complete [ 1595.873064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1595.874335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1595.878983] Lustre: Skipped 1 previous similar message [ 1595.881529] Lustre: Skipped 1 previous similar message [ 1598.540240] LustreError: 86073:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1598.543809] LustreError: 86073:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1598.587695] Lustre: server umount lustre-MDT0000 complete [ 1600.260178] LustreError: 83587:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764107442 with bad export cookie 16899367775602520743 [ 1600.261816] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1600.264722] LustreError: 83587:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1600.334327] LustreError: 86275:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1600.337025] LustreError: 86275:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1600.398884] Lustre: server umount lustre-MDT0001 complete [ 1605.303980] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 1606.719981] Key type lgssc unregistered [ 1606.882820] LNet: 86807:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1606.886792] LNetError: 86807:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1606.901540] LNet: Removed LNI 192.168.201.142@tcp [ 1607.295050] Key type .llcrypt unregistered [ 1607.296541] Key type ._llcrypt unregistered [ 1618.788954] Key type ._llcrypt registered [ 1618.790571] Key type .llcrypt registered [ 1618.831925] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 1619.224777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1619.232054] alg: No test for adler32 (adler32-zlib) [ 1620.108431] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 1620.203494] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 1621.808141] Key type lgssc registered [ 1622.218221] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1625.199432] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1625.208912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1640.672340] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1645.792420] LustreError: 88155:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.142@tcp: failed processing log, type 4: rc = -110 [ 1672.416214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1674.309152] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1676.046512] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1678.569206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1678.599122] Lustre: *** cfs_fail_loc=706, val=0*** [ 1678.725460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1678.744090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1679.893670] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1682.756637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1682.905711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1684.374723] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1686.278620] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1687.571723] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1689.056979] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1689.059753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1689.063995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1692.132273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1692.133118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1694.319102] LustreError: 90478:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1694.359102] Lustre: server umount lustre-OST0000 complete [ 1697.249722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1697.250471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1697.253826] Lustre: Skipped 1 previous similar message [ 1697.256178] Lustre: Skipped 1 previous similar message [ 1701.692195] LustreError: 90681:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1701.694145] LustreError: 90681:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1701.724694] Lustre: server umount lustre-MDT0000 complete [ 1702.368879] LustreError: 89130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1702.374066] LustreError: 89130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1702.972317] LustreError: 88162:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764107545 with bad export cookie 1062117504028852812 [ 1702.975819] LustreError: 88162:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1702.977579] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1703.041608] LustreError: 90883:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1703.043466] LustreError: 90883:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1703.097331] Lustre: server umount lustre-MDT0001 complete [ 1706.395281] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 1707.385991] Key type lgssc unregistered [ 1707.502343] LNet: 91414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1707.504296] LNetError: 91414:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1707.510400] LNet: Removed LNI 192.168.201.142@tcp [ 1707.794770] Key type .llcrypt unregistered [ 1707.796100] Key type ._llcrypt unregistered [ 1716.878590] Key type ._llcrypt registered [ 1716.879921] Key type .llcrypt registered [ 1716.915121] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 1717.207598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1717.213804] alg: No test for adler32 (adler32-zlib) [ 1718.083156] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 1718.177873] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 1719.768227] Key type lgssc registered [ 1720.128168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1722.827842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1722.835441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1725.153629] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1727.915488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1728.110181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1729.473316] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1732.718519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1732.845181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1734.831781] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1737.955486] LustreError: 93480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1737.961054] LustreError: 93480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 1738.093120] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1738.099081] Lustre: MGS: Client 1814d1de-ae75-48d6-a39b-2ce436d4521d (at 0@lo) reconnecting [ 1738.190204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1738.221499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1739.644982] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1748.452566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1748.453435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1748.456053] Lustre: Skipped 1 previous similar message [ 1748.456305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1748.458936] Lustre: Skipped 1 previous similar message [ 1753.571033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1753.573103] Lustre: Skipped 1 previous similar message [ 1753.901073] LustreError: 95421:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1753.936984] Lustre: server umount lustre-OST0000 complete [ 1758.690106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1758.690295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1758.690889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1758.701677] Lustre: Skipped 1 previous similar message [ 1761.347752] LustreError: 95624:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1761.350479] LustreError: 95624:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1761.359887] Lustre: server umount lustre-MDT0000 complete [ 1762.734783] LustreError: 95826:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1762.736885] LustreError: 95826:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1762.784664] Lustre: server umount lustre-MDT0001 complete [ 1764.129227] Lustre: server umount MGS complete [ 1767.943286] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 16:53:29 (1764107609) [ 1778.287728] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 1781.658384] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 1785.273409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1787.606035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1789.844879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1792.274227] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1794.978287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1795.010283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.469564] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1800.531347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1800.552464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1800.633359] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1800.654114] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1800.693646] Lustre: lustre-MDT0001: new disk, initializing [ 1800.727733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1800.739834] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1800.743520] Lustre: 100533:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1800.747179] LustreError: 100533:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1802.141959] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1806.631309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1806.661641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1806.756752] Lustre: lustre-OST0000: new disk, initializing [ 1806.758838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1806.780770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1808.802344] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1811.104894] Lustre: 101726:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000008686c894 x1849800848868352/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1764107669 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1812.057684] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1814.549351] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1814.554845] Lustre: MGS: Client cf8c4a3f-9f24-4d7f-94ba-743bd898dc0c (at 0@lo) reconnecting [ 1814.571529] Lustre: 99820:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1814.580945] LustreError: 99820:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1814.595935] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1814.634077] Lustre: lustre-MDT0000: new disk, initializing [ 1814.699975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1815.712790] Lustre: 102244:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000008f226faf x1849800848873600/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1764107673 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1815.777779] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1816.134876] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1817.826949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1817.830374] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1817.840383] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1820.770819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1820.773712] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1840.610170] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1840.613416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1840.619532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1840.621715] Lustre: Skipped 2 previous similar messages [ 1846.193170] LustreError: 102765:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1846.195611] LustreError: 102765:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1846.242761] Lustre: server umount lustre-OST0000 complete [ 1848.802705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1848.802849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1848.806127] LustreError: Skipped 1 previous similar message [ 1848.806788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1848.814487] Lustre: Skipped 2 previous similar messages [ 1848.821423] Lustre: Skipped 3 previous similar messages [ 1853.827615] LustreError: 102969:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1853.829484] LustreError: 102969:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1853.838833] Lustre: server umount lustre-MDT0000 complete [ 1853.921174] LustreError: 100504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1853.927279] LustreError: 100504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1861.347187] Lustre: server umount lustre-MDT0001 complete [ 1862.644599] LustreError: 103373:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1862.646468] LustreError: 103373:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1862.667530] Lustre: server umount MGS complete [ 1866.578828] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 16:55:08 (1764107708) [ 1872.599202] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 1873.662941] Key type lgssc unregistered [ 1873.799579] LNet: 104706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1873.803474] LNetError: 104706:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1873.814607] LNet: Removed LNI 192.168.201.142@tcp [ 1874.138782] Key type .llcrypt unregistered [ 1874.140205] Key type ._llcrypt unregistered [ 1881.460317] Key type ._llcrypt registered [ 1881.461692] Key type .llcrypt registered [ 1881.505993] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 1881.839771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1881.864071] alg: No test for adler32 (adler32-zlib) [ 1882.722523] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 1882.814525] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 1884.400169] Key type lgssc registered [ 1884.785682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1887.696918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1887.705219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1888.881668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.291227] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1893.220882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1893.334738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1894.641280] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1896.455511] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1897.694626] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1899.968983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1900.086469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1902.138976] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1904.027903] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1914.409992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 1921.795135] LustreError: 108153:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1921.904624] Lustre: server umount lustre-OST0000 complete [ 1929.185257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1929.194123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1929.207200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1929.699929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1929.711157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1934.820874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1934.824255] Lustre: Skipped 1 previous similar message [ 1934.843366] LustreError: 108354:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1934.846889] LustreError: 108354:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1934.924846] Lustre: server umount lustre-MDT0000 complete [ 1938.016491] LustreError: 107505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764107780 with bad export cookie 5306809231393631966 [ 1938.033358] LustreError: 107505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1938.033940] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1948.657879] LustreError: 108556:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1948.670430] LustreError: 108556:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1948.875519] Lustre: server umount lustre-MDT0001 complete [ 1957.672858] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 1958.997945] Key type lgssc unregistered [ 1959.168415] LNet: 109089:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1959.175791] LNetError: 109089:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1959.183838] LNet: Removed LNI 192.168.201.142@tcp [ 1959.703513] Key type .llcrypt unregistered [ 1959.705575] Key type ._llcrypt unregistered [ 1967.176655] Key type ._llcrypt registered [ 1967.178038] Key type .llcrypt registered [ 1967.229246] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 1967.609538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1967.646095] alg: No test for adler32 (adler32-zlib) [ 1968.521678] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 1968.614109] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 1970.200189] Key type lgssc registered [ 1970.704127] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1973.189991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1973.190125] LustreError: 110131:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1973.191758] Lustre: Skipped 1 previous similar message [ 1973.194777] LustreError: 110130:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1973.197255] LustreError: 110131:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1973.198852] LustreError: 110131:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1973.202057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.202543] LustreError: 110131:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1974.410378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1977.584810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1977.681048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1979.134404] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 1981.201348] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing lsmod [ 1982.400051] LustreError: 111307:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1982.400627] LustreError: 111308:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1982.406493] LustreError: 111307:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1982.409522] LustreError: 111307:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1982.415494] LustreError: 111307:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1982.415622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1982.568510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1986.915141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1986.917053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1986.922143] Lustre: Skipped 2 previous similar messages [ 1986.928595] Lustre: Skipped 2 previous similar messages [ 1992.163337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1992.166324] Lustre: Skipped 2 previous similar messages [ 1997.282065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1997.284664] Lustre: Skipped 2 previous similar messages [ 1998.304135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1998.345440] LustreError: 111495:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1998.379374] Lustre: server umount lustre-MDT0000 complete [ 1999.952283] LustreError: 110146:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764107842 with bad export cookie 13521908191909191022 [ 1999.956383] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1999.956847] LustreError: 110146:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2001.248240] LustreError: 110603:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d09f9423 x1849801110996480/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2001.261358] LustreError: 111701:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2001.264613] LustreError: 111701:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2001.323939] Lustre: server umount lustre-MDT0001 complete [ 2013.112851] LustreError: 111902:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2013.116198] LustreError: 111902:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2013.174378] Lustre: server umount lustre-OST0000 complete [ 2016.579394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2016.778145] LustreError: 112379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.786217] LustreError: 112379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2016.808752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2018.292273] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2021.666757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2021.801271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2023.315317] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2025.445061] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2026.792128] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2029.363541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2031.603555] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2033.570547] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2041.772323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 2052.068146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2052.069843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2052.078962] Lustre: Skipped 1 previous similar message [ 2055.618294] LustreError: 114648:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2055.621046] LustreError: 114648:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2055.888476] Lustre: server umount lustre-OST0000 complete [ 2062.306334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2062.308757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2062.339911] Lustre: Skipped 1 previous similar message [ 2062.385430] Lustre: Skipped 2 previous similar messages [ 2066.590478] LustreError: 114849:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2066.598305] LustreError: 114849:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2066.681356] Lustre: server umount lustre-MDT0000 complete [ 2067.425774] LustreError: 112379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2067.440170] LustreError: 112379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2070.012249] LustreError: 113097:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764107912 with bad export cookie 13521908191909191708 [ 2070.015157] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2070.036124] LustreError: 113097:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2070.411976] Lustre: server umount lustre-MDT0001 complete [ 2082.529118] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 2085.660302] Key type lgssc unregistered [ 2086.020652] LNet: 115583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2086.031382] LNetError: 115583:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2087.078888] LNet: Removed LNI 192.168.201.142@tcp [ 2088.245384] Key type .llcrypt unregistered [ 2088.255451] Key type ._llcrypt unregistered [ 2105.486221] Key type ._llcrypt registered [ 2105.487587] Key type .llcrypt registered [ 2105.531413] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 2106.046236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2106.061116] alg: No test for adler32 (adler32-zlib) [ 2107.052404] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 2107.198141] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 2108.808248] Key type lgssc registered [ 2109.486668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2113.783191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2113.794548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.044716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2117.149057] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2121.170587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2121.336353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2122.878069] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2125.324961] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2126.960652] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2130.249050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2130.413543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2132.849807] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2135.572694] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2140.650258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 2143.338169] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2145.027014] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2150.881700] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2150.884252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2150.900931] Lustre: Skipped 1 previous similar message [ 2150.907154] Lustre: Skipped 1 previous similar message [ 2153.013628] LustreError: 119312:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2153.072766] Lustre: server umount lustre-OST0000 complete [ 2156.002682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2156.005330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2156.008791] Lustre: Skipped 1 previous similar message [ 2159.009435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2159.014833] Lustre: Skipped 2 previous similar messages [ 2161.544530] LustreError: 119513:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2161.546731] LustreError: 119513:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2161.590284] Lustre: server umount lustre-MDT0000 complete [ 2163.532781] LustreError: 119158:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764108005 with bad export cookie 12647686614434816013 [ 2163.537426] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2163.543538] LustreError: 119158:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2163.636288] LustreError: 119715:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2163.640507] LustreError: 119715:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2163.729796] Lustre: server umount lustre-MDT0001 complete [ 2169.416505] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 2170.795495] Key type lgssc unregistered [ 2170.983556] LNet: 120246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2170.988255] LNetError: 120246:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2171.000431] LNet: Removed LNI 192.168.201.142@tcp [ 2171.401546] Key type .llcrypt unregistered [ 2171.403145] Key type ._llcrypt unregistered [ 2180.743863] Key type ._llcrypt registered [ 2180.745291] Key type .llcrypt registered [ 2180.793322] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 2181.211737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2181.255352] alg: No test for adler32 (adler32-zlib) [ 2182.153962] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 2182.266215] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 2183.872173] Key type lgssc registered [ 2184.470676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2188.336668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2188.355674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2189.599737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2191.438983] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2195.783026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2195.978058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2197.761364] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2200.322046] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2202.174695] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2205.531545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2205.736638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2209.301288] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2212.725297] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2213.949337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 2224.102673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2224.112565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2224.143397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2229.140596] LustreError: 123684:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2229.233602] LustreError: 123022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2229.265030] LustreError: 123022:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2229.420641] Lustre: server umount lustre-OST0000 complete [ 2234.337922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2234.339288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2234.348651] Lustre: Skipped 1 previous similar message [ 2234.360702] Lustre: Skipped 2 previous similar messages [ 2239.286519] LustreError: 123887:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2239.293586] LustreError: 123887:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2239.387119] Lustre: server umount lustre-MDT0000 complete [ 2239.461089] LustreError: 121422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2239.481824] LustreError: 121422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2242.780926] LustreError: 121407:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764108084 with bad export cookie 10177444336442429905 [ 2242.789842] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2242.804808] LustreError: 121407:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2242.915900] LustreError: 124089:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2242.918758] LustreError: 124089:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2243.083226] Lustre: server umount lustre-MDT0001 complete [ 2250.773149] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 2252.485509] Key type lgssc unregistered [ 2252.649568] LNet: 124622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2252.655135] LNetError: 124622:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2252.664904] LNet: Removed LNI 192.168.201.142@tcp [ 2253.297618] Key type .llcrypt unregistered [ 2253.300149] Key type ._llcrypt unregistered [ 2270.744275] Key type ._llcrypt registered [ 2270.745762] Key type .llcrypt registered [ 2270.869996] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 2271.527113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2271.554215] alg: No test for adler32 (adler32-zlib) [ 2272.495628] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 2272.645074] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 2274.288563] Key type lgssc registered [ 2275.107899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2281.350387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2281.364446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2282.735775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2285.382564] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2292.161426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2292.197111] LustreError: 126000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.209294] LustreError: 126000:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2292.407719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2295.120367] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2298.998404] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2301.447588] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2306.555125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2306.820964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2307.829458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 2310.853995] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2314.516593] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2324.279093] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2326.919170] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2326.930098] Lustre: Skipped 2 previous similar messages [ 2330.062559] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2330.065435] Lustre: Skipped 2 previous similar messages [ 2331.691266] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2331.707596] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2339.174739] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.42@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2339.180426] Lustre: Skipped 2 previous similar messages [ 2340.198241] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.42@tcp 192.168.0.[2,10]@tcp [ 2348.952627] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.42/24@tcp in log lustre-MDT0000 [ 2348.957423] Lustre: Skipped 2 previous similar messages [ 2358.875376] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.42/24@tcp [ 2358.885610] Lustre: Skipped 1 previous similar message [ 2363.369288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2363.373640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2363.405476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2364.400211] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2364.404504] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2364.420348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2367.457389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2367.463325] Lustre: Skipped 1 previous similar message [ 2369.283265] LustreError: 129602:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2369.413703] Lustre: server umount lustre-OST0000 complete [ 2372.577879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2372.580493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2372.588227] Lustre: Skipped 1 previous similar message [ 2377.699228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2377.706311] Lustre: Skipped 2 previous similar messages [ 2378.722108] LustreError: 129803:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2378.725118] LustreError: 129803:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2378.828349] Lustre: server umount lustre-MDT0000 complete [ 2382.056899] LustreError: 128299:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764108224 with bad export cookie 11216309451152453234 [ 2382.060189] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2382.062931] LustreError: 128299:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2382.188554] LustreError: 130004:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2382.202441] LustreError: 130004:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2382.340907] Lustre: server umount lustre-MDT0001 complete [ 2391.256633] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 2394.326984] Key type lgssc unregistered [ 2394.631298] LNet: 130536:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2394.637728] LNetError: 130536:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2394.665376] LNet: Removed LNI 192.168.201.142@tcp [ 2395.692710] Key type .llcrypt unregistered [ 2395.694820] Key type ._llcrypt unregistered [ 2421.074476] Key type ._llcrypt registered [ 2421.075806] Key type .llcrypt registered [ 2421.133876] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 2421.981966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2422.030788] alg: No test for adler32 (adler32-zlib) [ 2422.978774] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 2423.161242] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 2424.776545] Key type lgssc registered [ 2425.834209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2432.937532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2432.965771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2434.412970] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2434.426029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2434.450875] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.42/24@tcp [ 2437.332507] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2444.603426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2444.773386] LustreError: 132059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2444.798069] LustreError: 132059:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2444.893300] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2444.930936] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.42/24@tcp [ 2444.960974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2448.033337] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2452.075375] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2454.748727] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2460.246698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2460.537505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2464.767755] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2468.873099] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2478.967718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 2486.246468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2486.253212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2486.260872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2489.316143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2489.318253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2489.324070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2491.080167] LustreError: 134377:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2491.160253] Lustre: server umount lustre-OST0000 complete [ 2494.434102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2494.436670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2494.453921] Lustre: Skipped 1 previous similar message [ 2494.470916] Lustre: Skipped 2 previous similar messages [ 2499.553509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2499.560024] Lustre: Skipped 1 previous similar message [ 2500.496414] LustreError: 134578:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2500.498945] LustreError: 134578:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2500.558829] Lustre: server umount lustre-MDT0000 complete [ 2503.195165] LustreError: 134579:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764108345 with bad export cookie 11005122949526132648 [ 2503.198721] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2503.208578] LustreError: 134579:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2503.329059] LustreError: 134781:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2503.333204] LustreError: 134781:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2503.430597] Lustre: server umount lustre-MDT0001 complete [ 2511.521208] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 2513.345897] Key type lgssc unregistered [ 2513.519551] LNet: 135315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2513.524421] LNetError: 135315:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2513.539890] LNet: Removed LNI 192.168.201.142@tcp [ 2514.036628] Key type .llcrypt unregistered [ 2514.040175] Key type ._llcrypt unregistered [ 2531.214435] Key type ._llcrypt registered [ 2531.216048] Key type .llcrypt registered [ 2531.314710] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 2542.053613] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 2542.818558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2542.831138] alg: No test for adler32 (adler32-zlib) [ 2543.777963] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 2543.965972] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 2545.592187] Key type lgssc registered [ 2546.404700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2552.765399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2557.416838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2562.024944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2567.659300] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2572.793321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2572.827529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2572.837765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2574.070117] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2574.106945] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2574.175497] Lustre: lustre-MDT0000: new disk, initializing [ 2574.249515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2574.272416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2576.902674] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2584.734124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2584.785743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2584.839870] Lustre: 138668:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2584.890677] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2584.893615] Lustre: Skipped 1 previous similar message [ 2584.965339] Lustre: lustre-MDT0001: new disk, initializing [ 2585.062658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2585.095351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2585.113918] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2588.371850] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2593.939291] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2596.396661] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2601.253452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2601.328978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2601.515489] Lustre: lustre-OST0000: new disk, initializing [ 2601.522015] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2601.607749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2605.795622] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2611.252897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2611.257789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2611.319456] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2611.396746] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2614.096372] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2614.237222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2616.978890] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2617.128580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2621.929624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2621.930191] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2621.935810] LustreError: Skipped 1 previous similar message [ 2621.951415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2621.957039] Lustre: Skipped 1 previous similar message [ 2624.572183] LustreError: 141607:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2624.698372] Lustre: server umount lustre-OST0000 complete [ 2631.140992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2631.152426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2631.170075] Lustre: Skipped 1 previous similar message [ 2631.181571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2632.161613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2633.582874] LustreError: 141811:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2633.592109] LustreError: 141811:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2633.685427] Lustre: server umount lustre-MDT0000 complete [ 2636.540086] LustreError: 138660:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764108478 with bad export cookie 8030492881867353324 [ 2636.545506] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2636.548094] LustreError: 138660:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2636.666538] LustreError: 142012:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2636.669624] LustreError: 142012:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2636.802762] Lustre: server umount lustre-MDT0001 complete [ 2642.907557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2643.268597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2645.729222] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2648.547595] LustreError: 142487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2648.563366] LustreError: 142487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2651.727526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2651.995456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2654.875708] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2658.764151] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2661.206872] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2665.934692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2666.156619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2669.900887] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2673.252770] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2675.526660] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2675.678396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2677.820719] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2677.924934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2683.211756] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2683.249462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2683.350846] Lustre: lustre-OST0001: new disk, initializing [ 2683.354970] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2683.409364] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2686.663749] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2690.122683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2690.133336] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2690.188850] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2691.993771] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2692.149340] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2694.690867] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2694.832111] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2697.086193] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92e007acd000.ost_server_uuid 50 [ 2698.018993] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92e007acd000.ost_server_uuid in FULL state after 0 sec [ 2704.120216] LustreError: 146734:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2704.122579] LustreError: 146734:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2704.196783] Lustre: server umount lustre-OST0001 complete [ 2705.894819] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2705.896557] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2705.901000] Lustre: Skipped 1 previous similar message [ 2705.904401] LustreError: 144085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.914463] LustreError: 144085:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2708.451336] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2708.458980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2708.470090] Lustre: Skipped 2 previous similar messages [ 2711.010965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2711.011430] LustreError: 144085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.013266] Lustre: Skipped 2 previous similar messages [ 2711.078684] LustreError: 144085:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2716.142183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2716.150131] LustreError: 146046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2716.171931] LustreError: 146046:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2721.248143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2721.253783] LustreError: 144085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2721.289182] LustreError: 144085:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2721.412918] LustreError: 146935:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2721.415978] LustreError: 146935:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2721.542726] Lustre: server umount lustre-OST0000 complete [ 2726.372258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2726.374994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2726.396028] Lustre: Skipped 3 previous similar messages [ 2726.414337] Lustre: Skipped 4 previous similar messages [ 2731.797571] LustreError: 147137:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2731.815166] LustreError: 147137:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2731.995799] Lustre: server umount lustre-MDT0000 complete [ 2736.257881] LustreError: 145489:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764108578 with bad export cookie 8030492881867354206 [ 2736.272713] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2736.275674] LustreError: 145489:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2736.883702] Lustre: server umount lustre-MDT0001 complete [ 2755.207899] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 2757.184527] Key type lgssc unregistered [ 2757.475878] LNet: 148328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2757.481260] LNetError: 148328:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2757.494563] LNet: Removed LNI 192.168.201.142@tcp [ 2758.120522] Key type .llcrypt unregistered [ 2758.122634] Key type ._llcrypt unregistered [ 2781.287137] Key type ._llcrypt registered [ 2781.288712] Key type .llcrypt registered [ 2781.402457] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 2791.557419] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 2792.149788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2792.262700] alg: No test for adler32 (adler32-zlib) [ 2793.245697] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 2793.371960] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 2794.992769] Key type lgssc registered [ 2795.655490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2800.899905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2805.097184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2809.054253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2813.040691] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2817.469906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2817.523715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2817.537592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2818.740740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2818.772411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2818.866840] Lustre: lustre-MDT0000: new disk, initializing [ 2819.007441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2819.034666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2822.320218] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2829.934688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2829.993241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2830.048462] Lustre: 152256:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2830.105767] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2830.113772] Lustre: Skipped 1 previous similar message [ 2830.176814] Lustre: lustre-MDT0001: new disk, initializing [ 2830.284991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2830.320938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2830.349652] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2833.415784] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2839.130875] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2841.498491] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2846.185927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2846.258793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2846.509267] Lustre: lustre-OST0000: new disk, initializing [ 2846.519244] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2846.587548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2850.246615] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2853.398547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2853.405851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2853.494786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2854.930358] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2857.415767] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2857.536246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2860.051590] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2860.210169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2863.591382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2863.591919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2863.604384] LustreError: Skipped 1 previous similar message [ 2863.605299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2863.613693] Lustre: Skipped 1 previous similar message [ 2867.515080] LustreError: 155190:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2867.573196] Lustre: server umount lustre-OST0000 complete [ 2871.267154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2871.270969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2871.276787] Lustre: Skipped 1 previous similar message [ 2871.279750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2873.825887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2873.831884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2878.945460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2878.949365] Lustre: Skipped 2 previous similar messages [ 2883.044967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2883.048214] Lustre: Skipped 1 previous similar message [ 2884.064114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2884.138649] LustreError: 155392:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2884.141867] LustreError: 155392:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2884.190249] Lustre: server umount lustre-MDT0000 complete [ 2886.644099] LustreError: 152247:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764108728 with bad export cookie 15573574583525774196 [ 2886.649254] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2886.652889] LustreError: 152247:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2886.770224] LustreError: 155596:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2886.776832] LustreError: 155596:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2886.899090] Lustre: server umount lustre-MDT0001 complete [ 2891.919953] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 17:12:13 (1764108733) [ 2901.013805] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 2906.258345] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 2912.412216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2916.242361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2919.840049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2923.529285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2927.412336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2927.447855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.588304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2927.619506] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2927.673656] Lustre: lustre-MDT0000: new disk, initializing [ 2927.746638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2927.766078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2929.781112] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2935.700810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2935.741466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2935.804948] Lustre: 158584:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2935.845930] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2935.849023] Lustre: Skipped 1 previous similar message [ 2935.893435] Lustre: lustre-MDT0001: new disk, initializing [ 2935.955294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2935.981150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2935.986559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2938.311612] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2943.713372] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2946.248148] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2950.341212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2950.414496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2950.566736] Lustre: lustre-OST0000: new disk, initializing [ 2950.570897] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2950.579380] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 2950.631952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2953.947115] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2955.775345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2955.783293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2955.826256] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2959.093739] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2962.007967] Lustre: Failing over lustre-OST0000 [ 2962.045257] LustreError: 161031:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2962.050851] LustreError: 161031:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2962.115367] Lustre: server umount lustre-OST0000 complete [ 2976.619910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2976.799177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2976.811420] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2980.041317] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 2980.896176] Lustre: 150469:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764108807/real 1764108807] req@00000000c6a997db x1849801976116992/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1764108823 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2980.913966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2980.933382] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2981.274897] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 2981.276846] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2982.305220] Lustre: 150468:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764108808/real 1764108808] req@000000008cd26640 x1849801976117888/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1764108824 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2982.557207] Lustre: Failing over lustre-MDT0000 [ 2982.658641] LustreError: 162011:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 2982.664218] LustreError: 162011:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2982.724121] Lustre: server umount lustre-MDT0000 complete [ 2985.955573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2985.960682] Lustre: Skipped 3 previous similar messages [ 2986.976163] Lustre: 150469:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764108813/real 1764108813] req@00000000ea082a15 x1849801976119040/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1764108829 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2991.073294] LustreError: 158597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2991.080175] LustreError: 158597:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2992.080164] Lustre: 150468:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764108818/real 1764108818] req@00000000aa7f9c75 x1849801976120320/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1764108834 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2996.195228] LustreError: 158597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2996.204430] LustreError: 158597:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2997.109195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2997.184641] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2997.356600] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2999.138624] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3000.183531] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3002.343113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3002.346184] Lustre: Skipped 1 previous similar message [ 3002.352219] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3002.373043] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3006.945912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3006.952373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3006.964145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3009.990324] LustreError: 162897:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3009.993065] LustreError: 162897:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3010.071243] Lustre: server umount lustre-OST0000 complete [ 3017.698246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3017.705287] LustreError: Skipped 1 previous similar message [ 3019.042248] Lustre: server umount lustre-MDT0000 complete [ 3022.588680] LustreError: 158577:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764108864 with bad export cookie 15573574583525776303 [ 3022.591964] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3022.599738] LustreError: 158577:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3022.854673] Lustre: server umount lustre-MDT0001 complete [ 3030.800935] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 3033.351411] Key type lgssc unregistered [ 3033.578634] LNet: 163832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3033.592531] LNetError: 163832:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3033.616358] LNet: Removed LNI 192.168.201.142@tcp [ 3034.300700] Key type .llcrypt unregistered [ 3034.307177] Key type ._llcrypt unregistered [ 3066.237403] Key type ._llcrypt registered [ 3066.243434] Key type .llcrypt registered [ 3066.347841] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 3067.573915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3067.628443] alg: No test for adler32 (adler32-zlib) [ 3068.703304] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 3068.909409] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 3070.560188] Key type lgssc registered [ 3071.899750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3077.800467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3077.812877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3079.168567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3081.583886] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3086.248850] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3088.751507] LustreError: 165360:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3088.757438] LustreError: 165360:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3088.762082] LustreError: 165360:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 3088.844237] LustreError: 165882:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3088.929770] Lustre: server umount lustre-MDT0000 complete [ 3097.065758] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 3102.941174] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 3108.814276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3112.565317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3116.016624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3119.378763] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3123.037528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3123.073740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3123.192920] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3123.214594] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3123.261399] Lustre: lustre-MDT0000: new disk, initializing [ 3123.330022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3123.346858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3125.431176] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3132.039910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3132.088747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3132.149458] Lustre: 168524:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3132.195136] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3132.200648] Lustre: Skipped 1 previous similar message [ 3132.250052] Lustre: lustre-MDT0001: new disk, initializing [ 3132.317757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3132.343055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3132.353141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3134.617821] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3138.794787] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3140.553054] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3144.201971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3144.242783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3144.378263] Lustre: lustre-OST0000: new disk, initializing [ 3144.381610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3144.438513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3147.448357] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3151.256019] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3153.421330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3153.426691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3153.468970] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3158.881641] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3159.794081] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 17:16:41 (1764109001) [ 3362.790792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3362.797847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3362.803709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3363.297121] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3363.315162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3363.320193] Lustre: Skipped 1 previous similar message [ 3365.791874] LustreError: 172050:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3365.810650] LustreError: 172050:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3365.958974] Lustre: server umount lustre-MDT0000 complete [ 3368.419456] LustreError: 169574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3368.433387] LustreError: 169574:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3369.370236] LustreError: 168518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764109211 with bad export cookie 6287084180990725252 [ 3369.373377] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3369.393492] LustreError: 168518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3373.541487] LustreError: 168534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3373.543782] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3373.628237] Lustre: Skipped 1 previous similar message [ 3373.631542] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3374.565053] LustreError: 168534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3375.502183] LustreError: 172252:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3375.506880] LustreError: 172252:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3375.610320] Lustre: server umount lustre-MDT0001 complete [ 3384.504606] LustreError: 172453:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3384.508275] LustreError: 172453:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3384.582259] Lustre: server umount lustre-OST0000 complete [ 3389.435742] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 3394.561469] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 3400.700516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3404.674097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3408.429957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3412.305263] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3416.545787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3416.590745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3416.773985] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3416.797894] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3416.857089] Lustre: lustre-MDT0000: new disk, initializing [ 3416.917411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3416.929348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3419.350421] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3426.056204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3426.089977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3426.141556] Lustre: 174790:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3426.176750] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3426.181267] Lustre: Skipped 1 previous similar message [ 3426.230339] Lustre: lustre-MDT0001: new disk, initializing [ 3426.292516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3426.316218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3426.322346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3428.630371] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3433.162322] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3435.149962] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3439.015072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3439.064260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3439.228210] Lustre: lustre-OST0000: new disk, initializing [ 3439.232893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3443.043614] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3444.772506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3444.778585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3444.838187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3448.711068] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3451.498525] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3451.627257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3454.078599] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3454.201112] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3460.069775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3460.070258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3460.075724] LustreError: Skipped 1 previous similar message [ 3460.077339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3460.081817] Lustre: Skipped 1 previous similar message [ 3460.084572] Lustre: Skipped 1 previous similar message [ 3461.369239] LustreError: 177729:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3461.378552] LustreError: 177729:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3461.444093] Lustre: server umount lustre-OST0000 complete [ 3465.187932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3465.195629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3465.195944] Lustre: Skipped 1 previous similar message [ 3465.199429] Lustre: Skipped 1 previous similar message [ 3469.987546] LustreError: 177932:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3469.989867] LustreError: 177932:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3470.091654] Lustre: server umount lustre-MDT0000 complete [ 3470.313522] LustreError: 174801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3470.320270] LustreError: 174801:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3472.471190] LustreError: 174783:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764109314 with bad export cookie 6287084180990884082 [ 3472.473721] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3472.476972] LustreError: 174783:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3472.670436] Lustre: server umount lustre-MDT0001 complete [ 3477.457133] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:21:59 (1764109319) [ 3486.034370] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 3490.885955] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 3496.918391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3500.962311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3504.501691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3508.484230] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3512.696582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3512.746584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3512.899162] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3512.924694] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3512.978668] Lustre: lustre-MDT0000: new disk, initializing [ 3513.046020] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3513.049387] Lustre: Skipped 1 previous similar message [ 3513.057447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3515.467626] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3521.691620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3521.733290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3521.791209] Lustre: 181120:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3521.950733] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3524.417231] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3530.131613] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3533.292341] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3539.881130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3539.972843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3546.961721] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3549.241275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3549.256164] Lustre: Skipped 1 previous similar message [ 3549.266720] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3549.363727] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3552.396301] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3559.905449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3559.916823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3559.925933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3559.929643] Lustre: Skipped 1 previous similar message [ 3562.486143] LustreError: 183664:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3562.488451] LustreError: 183664:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3562.559349] Lustre: server umount lustre-OST0000 complete [ 3579.360248] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3579.482967] Lustre: server umount lustre-MDT0000 complete [ 3580.387867] LustreError: 181131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3580.394906] LustreError: 181131:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3581.814715] LustreError: 181904:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764109424 with bad export cookie 6287084180990884964 [ 3581.823891] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3581.829531] LustreError: 181904:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3582.020434] Lustre: server umount lustre-MDT0001 complete [ 3586.893541] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:23:48 (1764109428) [ 3596.467892] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 3601.890416] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 3608.866842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3612.746596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3616.443472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3620.323614] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3624.362510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3624.399569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3624.538600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3624.557586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3624.560644] Lustre: Skipped 3 previous similar messages [ 3624.607676] Lustre: lustre-MDT0000: new disk, initializing [ 3624.609560] Lustre: Skipped 2 previous similar messages [ 3624.663956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3624.666593] Lustre: Skipped 2 previous similar messages [ 3624.682410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3627.594693] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3633.712879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3633.743273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3633.794593] Lustre: 187057:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3633.959227] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3636.052989] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3640.240122] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3642.191249] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3645.774409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3645.828804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3648.699818] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3649.409760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3652.706523] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3659.748762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3659.749416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3659.756132] LustreError: Skipped 1 previous similar message [ 3659.756966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3659.766167] Lustre: Skipped 4 previous similar messages [ 3659.772969] Lustre: Skipped 6 previous similar messages [ 3662.139143] LustreError: 189603:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3662.141883] LustreError: 189603:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3662.229805] Lustre: server umount lustre-OST0000 complete [ 3664.865637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3664.869444] LustreError: Skipped 1 previous similar message [ 3670.510663] Lustre: server umount lustre-MDT0000 complete [ 3672.386316] LustreError: 187050:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764109514 with bad export cookie 6287084180990886035 [ 3672.389994] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3672.392495] LustreError: 187050:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3672.553573] Lustre: server umount lustre-MDT0001 complete [ 3677.325373] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 3678.828711] Key type lgssc unregistered [ 3679.021822] LNet: 190538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3679.026316] LNetError: 190538:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3679.034723] LNet: Removed LNI 192.168.201.142@tcp [ 3679.541543] Key type .llcrypt unregistered [ 3679.543223] Key type ._llcrypt unregistered [ 3693.192627] Key type ._llcrypt registered [ 3693.194176] Key type .llcrypt registered [ 3693.250404] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 3693.854076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3693.864918] alg: No test for adler32 (adler32-zlib) [ 3694.817838] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 3694.955245] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 3696.568414] Key type lgssc registered [ 3697.163656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3702.007468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3702.024905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3703.284841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3705.530833] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3710.878814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3711.168776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3713.624733] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3717.210406] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3719.689649] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3724.158857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3724.422965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3727.938892] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3728.562756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3731.032902] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3749.347304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3749.351239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3749.359189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3752.418234] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3752.419649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3752.423157] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3755.057980] LustreError: 194180:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3755.112814] Lustre: server umount lustre-OST0000 complete [ 3757.540518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3757.541208] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3757.542114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3757.542120] Lustre: Skipped 1 previous similar message [ 3757.558304] Lustre: Skipped 1 previous similar message [ 3762.657538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3762.660225] Lustre: Skipped 2 previous similar messages [ 3763.300480] LustreError: 194381:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3763.303842] LustreError: 194381:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3763.370528] Lustre: server umount lustre-MDT0000 complete [ 3765.444166] LustreError: 191894:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764109607 with bad export cookie 4701845700803286922 [ 3765.445229] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3765.452180] LustreError: 191894:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3765.548663] LustreError: 194583:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3765.551707] LustreError: 194583:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3765.642632] Lustre: server umount lustre-MDT0001 complete [ 3771.467645] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 3773.071747] Key type lgssc unregistered [ 3773.251852] LNet: 195114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3773.255968] LNetError: 195114:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3773.271496] LNet: Removed LNI 192.168.201.142@tcp [ 3773.845960] Key type .llcrypt unregistered [ 3773.848347] Key type ._llcrypt unregistered [ 3786.780275] Key type ._llcrypt registered [ 3786.781511] Key type .llcrypt registered [ 3786.847315] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 3787.413519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3787.425368] alg: No test for adler32 (adler32-zlib) [ 3788.355207] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 3788.481538] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 3790.088278] Key type lgssc registered [ 3790.623662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3794.587604] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3794.599901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3795.883454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3797.781198] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3802.443474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3802.680148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3804.886608] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3807.838939] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3809.667483] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3812.995399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3813.195788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3815.833822] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3818.195325] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3821.419197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3832.804343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3832.809573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.810800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3832.815966] Lustre: Skipped 1 previous similar message [ 3833.910180] LustreError: 198758:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3834.001778] Lustre: server umount lustre-OST0000 complete [ 3837.772994] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff92e009a19000.ost_server_uuid 50 [ 3852.460916] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3863.269782] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3863.271437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3863.277864] Lustre: Skipped 1 previous similar message [ 3863.284796] Lustre: Skipped 2 previous similar messages [ 3868.641896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3868.644903] Lustre: Skipped 1 previous similar message [ 3869.403798] LustreError: 199128:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3869.406847] LustreError: 199128:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3869.475219] Lustre: server umount lustre-MDT0000 complete [ 3871.743794] LustreError: 196472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764109713 with bad export cookie 17556536426928518362 [ 3871.747749] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3871.752466] LustreError: 196472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3871.852267] LustreError: 199329:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3871.854729] LustreError: 199329:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3871.932724] Lustre: server umount lustre-MDT0001 complete [ 3875.932962] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 17:28:37 (1764109717) [ 3879.572755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3879.808789] LustreError: 199999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3879.815370] LustreError: 199999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3879.863375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3882.077979] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3885.027618] LustreError: 199999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3886.642853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3886.872936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3888.977363] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3892.173426] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3894.194932] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3898.392247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3898.613643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3902.361912] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3905.782875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3906.160309] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3910.936885] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3911.009260] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3911.150444] Lustre: lustre-OST0001: new disk, initializing [ 3911.153974] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3912.906614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3912.933965] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3913.078588] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3915.671702] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3921.846787] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3926.497145] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3926.500363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3926.506479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3927.521419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3932.518906] Lustre: lustre-OST0000: Not available for connect from 192.168.201.42@tcp (stopping) [ 3932.523581] Lustre: Skipped 4 previous similar messages [ 3938.272177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3938.303840] LustreError: 203203:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3938.306627] LustreError: 203203:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3938.332411] Lustre: server umount lustre-OST0000 complete [ 3938.789627] LustreError: 202589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3938.799167] LustreError: 202589:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3941.346389] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3941.473719] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3942.758839] LustreError: 202589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3943.725923] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3943.854916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3947.882405] LustreError: 201594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3947.896943] LustreError: 201594:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3959.265946] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3959.269079] LustreError: 202589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3959.269597] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3959.276101] LustreError: 202589:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3959.278803] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3959.282227] Lustre: Skipped 2 previous similar messages [ 3959.287404] Lustre: Skipped 3 previous similar messages [ 3961.398100] LustreError: 203920:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3961.408337] LustreError: 203920:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3961.471179] Lustre: server umount lustre-OST0001 complete [ 3963.873621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3963.877192] LustreError: Skipped 1 previous similar message [ 3963.879244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3969.615940] LustreError: 204121:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3969.618161] LustreError: 204121:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3969.661954] Lustre: server umount lustre-MDT0000 complete [ 3971.569670] LustreError: 201614:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764109813 with bad export cookie 17556536426928519433 [ 3971.571053] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3971.575770] LustreError: 201614:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3971.755336] Lustre: server umount lustre-MDT0001 complete [ 3974.476625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3976.399180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3978.526800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3980.784116] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3985.140921] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 17:30:26 (1764109826) [ 3989.168856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3989.323895] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3989.341421] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3989.464778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3989.468149] Lustre: Skipped 1 previous similar message [ 3991.860894] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 3996.989759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3997.056426] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3997.085218] Lustre: 205813:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3999.342106] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4001.946189] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4003.565692] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4006.954344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4007.096292] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4009.711509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4010.347725] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4013.164272] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4016.803084] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4016.844874] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4019.875730] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4022.771857] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4025.313380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4025.313645] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4025.316127] LustreError: Skipped 1 previous similar message [ 4025.321409] Lustre: Skipped 1 previous similar message [ 4025.324141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4025.327974] Lustre: Skipped 6 previous similar messages [ 4030.328427] LustreError: 209009:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4030.330797] LustreError: 209009:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4030.367072] Lustre: server umount lustre-OST0000 complete [ 4033.383827] LustreError: 207443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4043.746324] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4043.751569] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4048.272949] Lustre: server umount lustre-OST0001 complete [ 4056.436866] Lustre: server umount lustre-MDT0000 complete [ 4058.835954] LustreError: 205806:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764109901 with bad export cookie 17556536426928520826 [ 4058.839523] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4058.846870] LustreError: 205806:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4059.078384] Lustre: server umount lustre-MDT0001 complete [ 4062.737716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4065.096486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4067.402651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4069.628959] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4073.053394] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 17:31:54 (1764109914) [ 4081.653174] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 4086.352057] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 4090.796487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4093.737560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4096.546286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4099.334976] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4102.200250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4102.230557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4102.329808] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4102.348129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4102.389592] Lustre: lustre-MDT0000: new disk, initializing [ 4102.427765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4102.430954] Lustre: Skipped 3 previous similar messages [ 4102.439608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4104.106100] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4109.075741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4109.106101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4109.155894] Lustre: 213434:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4109.188916] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4109.191794] Lustre: Skipped 1 previous similar message [ 4109.238955] Lustre: lustre-MDT0001: new disk, initializing [ 4109.323019] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4109.328486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4111.147200] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4115.093537] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4116.986638] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4120.530109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4120.572334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4120.709942] Lustre: lustre-OST0000: new disk, initializing [ 4120.711956] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4123.485659] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4127.771834] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4130.349794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4130.362153] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4130.465570] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4130.683730] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4130.837179] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4133.381598] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4133.570954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4135.909533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4135.911435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4135.914744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4135.914757] Lustre: Skipped 9 previous similar messages [ 4135.915464] Lustre: Skipped 4 previous similar messages [ 4135.923904] LustreError: Skipped 3 previous similar messages [ 4136.824089] LustreError: 216371:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4136.828465] LustreError: 216371:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4136.891933] Lustre: server umount lustre-OST0000 complete [ 4153.312302] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4153.397037] Lustre: server umount lustre-MDT0000 complete [ 4155.109412] LustreError: 214214:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764109997 with bad export cookie 17556536426928522387 [ 4155.110319] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4155.116645] LustreError: 214214:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4155.273939] Lustre: server umount lustre-MDT0001 complete [ 4159.304317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4159.580866] LustreError: 217251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4159.587602] LustreError: 217251:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4161.761851] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4166.962542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4167.174925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4167.178991] Lustre: Skipped 3 previous similar messages [ 4169.134806] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4171.652263] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4173.229273] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4176.326482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4178.929596] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4181.430429] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4183.407656] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4185.569695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4187.614505] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4187.738603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4195.002858] Lustre: server umount lustre-OST0000 complete [ 4197.770506] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4199.584625] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4224.114951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4226.983641] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4229.429743] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4230.225463] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4230.442992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4230.450343] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4230.451158] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4230.460577] Lustre: Skipped 1 previous similar message [ 4235.745150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4235.748261] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4235.753864] LustreError: Skipped 3 previous similar messages [ 4235.760968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4235.770324] Lustre: Skipped 5 previous similar messages [ 4235.770548] Lustre: Skipped 10 previous similar messages [ 4237.877232] LustreError: 221642:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4237.879857] LustreError: 221642:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 4237.955997] Lustre: server umount lustre-OST0000 complete [ 4246.201044] Lustre: server umount lustre-MDT0000 complete [ 4248.084395] LustreError: 217231:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764110090 with bad export cookie 17556536426928523255 [ 4248.088620] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4248.090909] LustreError: 217231:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4248.272525] Lustre: server umount lustre-MDT0001 complete [ 4252.104993] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 17:34:53 (1764110093) [ 4255.272384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4255.517107] LustreError: 222716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4255.527266] LustreError: 222716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4257.213486] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4261.231086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4263.572820] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4266.182622] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4267.710670] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4270.684329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4273.679419] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4276.583575] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4278.645049] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4280.803510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4282.739312] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4282.861767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4286.736348] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4286.800873] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4286.885420] Lustre: lustre-OST0001: new disk, initializing [ 4286.889105] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4290.433185] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4294.184442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4294.188394] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4294.247961] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4295.502789] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4298.069655] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4298.193566] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4300.401363] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4300.535880] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4307.731566] Lustre: server umount lustre-OST0001 complete [ 4310.353401] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4310.457285] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4312.168776] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4312.272974] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4337.411028] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4337.527865] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4337.531507] Lustre: Skipped 6 previous similar messages [ 4340.322915] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4342.760947] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4342.770934] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4342.772133] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4342.781945] Lustre: Skipped 1 previous similar message [ 4343.084994] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4343.847808] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4359.136288] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4359.178626] Lustre: server umount lustre-OST0001 complete [ 4372.448989] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4372.452688] LustreError: Skipped 4 previous similar messages [ 4372.454830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4372.460420] Lustre: Skipped 6 previous similar messages [ 4372.462991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4372.466145] Lustre: Skipped 16 previous similar messages [ 4377.778135] LustreError: 228812:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4377.780530] LustreError: 228812:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4377.842486] Lustre: server umount lustre-OST0000 complete [ 4385.806158] Lustre: server umount lustre-MDT0000 complete [ 4387.845616] LustreError: 222695:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764110230 with bad export cookie 17556536426928524200 [ 4387.848351] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4387.850399] LustreError: 222695:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4402.145074] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4402.268917] Lustre: server umount lustre-MDT0001 complete [ 4404.961649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4406.941316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4408.831038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4410.570046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4413.636887] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 17:37:35 (1764110255) [ 4416.714386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4416.872701] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4416.891369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4418.861428] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4423.115379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4423.191682] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4423.207569] Lustre: 230707:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4425.574259] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4428.075379] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4430.132776] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4434.341630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4434.472273] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4438.291784] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4441.540594] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4452.689135] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4452.739311] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4456.438559] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4459.702667] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4460.595651] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4462.629763] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4462.765079] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4464.837445] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4464.952315] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4466.691527] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92e0104dd800.ost_server_uuid 50 [ 4467.406716] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92e0104dd800.ost_server_uuid in FULL state after 0 sec [ 4467.925964] Lustre: Permanently deactivating lustre-OST0001 [ 4467.936852] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4469.710575] Lustre: Permanently reactivating lustre-OST0001 [ 4477.612001] Lustre: server umount lustre-OST0001 complete [ 4479.461725] LustreError: 232332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4479.471127] LustreError: 232332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [ 4494.304183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4494.429977] Lustre: server umount lustre-OST0000 complete [ 4511.200143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4511.344622] Lustre: server umount lustre-MDT0000 complete [ 4513.618564] LustreError: 230699:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764110355 with bad export cookie 17556536426928525474 [ 4513.622639] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4513.623631] LustreError: 230699:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4513.814536] Lustre: server umount lustre-MDT0001 complete [ 4517.580914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4520.211688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4522.878201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4525.599674] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4529.264250] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 17:39:30 (1764110370) [ 4529.802412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4533.359258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4533.525555] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4533.545418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4533.550298] Lustre: Skipped 5 previous similar messages [ 4535.811309] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4541.250928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4541.317280] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4541.341361] Lustre: 236863:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4543.833632] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4546.850761] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4548.710358] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4552.671205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4552.718348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4552.844387] Lustre: Permanently deactivating lustre-OST0000 [ 4552.857396] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4556.371721] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4559.655464] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4560.386939] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4563.444662] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4566.593316] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4569.198210] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4570.169507] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4575.171815] Lustre: Permanently reactivating lustre-OST0000 [ 4575.179746] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 4575.182734] Lustre: Skipped 2 previous similar messages [ 4578.893846] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4578.897739] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4578.904794] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4578.922913] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4578.929980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4593.353578] Lustre: server umount lustre-OST0001 complete [ 4601.201906] Lustre: server umount lustre-OST0000 complete [ 4609.666914] Lustre: server umount lustre-MDT0000 complete [ 4611.635423] LustreError: 239409:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764110453 with bad export cookie 17556536426928527112 [ 4611.640510] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4611.641155] LustreError: 239409:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4611.826735] Lustre: server umount lustre-MDT0001 complete [ 4616.745569] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 4618.137942] Key type lgssc unregistered [ 4618.316471] LNet: 241277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4618.320605] LNetError: 241277:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4618.328614] LNet: Removed LNI 192.168.201.142@tcp [ 4618.761851] Key type .llcrypt unregistered [ 4618.764189] Key type ._llcrypt unregistered [ 4630.016442] Key type ._llcrypt registered [ 4630.017964] Key type .llcrypt registered [ 4630.073862] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 4630.487978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4630.567372] alg: No test for adler32 (adler32-zlib) [ 4631.476668] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 4631.586979] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 4633.192331] Key type lgssc registered [ 4633.711663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4635.379334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4638.939944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4638.949174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4640.218328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4642.114591] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4646.500930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4646.543966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4646.612871] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4646.631740] Lustre: 242693:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4646.638371] Lustre: Permanently deactivating lustre-MDT0001 [ 4646.648493] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4646.813851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4647.735211] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4647.739186] LustreError: 242734:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4647.744083] LustreError: 242734:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4647.749685] LustreError: 242734:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 1, failed: rc = -108 [ 4648.761321] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4651.291988] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4652.948168] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4656.127722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4656.281314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4658.763467] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4661.029464] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4663.984270] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4664.095869] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4665.134560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4665.135854] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4666.698184] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4669.025455] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4670.774613] Lustre: Permanently reactivating lustre-MDT0001 [ 4670.783192] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4670.786492] Lustre: Skipped 2 previous similar messages [ 4671.274134] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4671.275506] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4680.448925] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4681.162238] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4683.132715] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4683.228517] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4684.882453] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4684.979877] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4686.118387] Lustre: Permanently deactivating lustre-MDT0001 [ 4686.125527] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4686.128279] Lustre: Skipped 2 previous similar messages [ 4696.070201] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4703.201427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4703.206382] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4703.207104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4703.216388] Lustre: Skipped 2 previous similar messages [ 4708.323067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4708.329337] Lustre: Skipped 6 previous similar messages [ 4709.007299] LustreError: 246819:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4709.058529] Lustre: server umount lustre-MDT0000 complete [ 4711.047272] LustreError: 242684:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764110553 with bad export cookie 6650010534014911063 [ 4711.051299] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4711.052717] LustreError: 242684:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4711.138436] LustreError: 247020:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 4711.141038] LustreError: 247020:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4711.213975] Lustre: server umount lustre-MDT0001 complete [ 4719.224254] LustreError: 247221:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4719.226953] LustreError: 247221:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4719.254889] Lustre: server umount lustre-OST0000 complete [ 4724.288701] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 17:42:46 (1764110566) [ 4728.877603] LustreError: 248226:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4728.880312] LustreError: 248226:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4728.933479] Lustre: server umount lustre-OST0001 complete [ 4731.465276] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 4735.458329] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 4739.605996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4742.330033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4744.940717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4747.677149] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4750.681995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4750.715446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4750.819261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4750.823560] Lustre: Skipped 2 previous similar messages [ 4750.839818] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4750.885965] Lustre: lustre-MDT0000: new disk, initializing [ 4750.927031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4750.936374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4752.699427] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4758.243417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4758.283809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4758.333959] Lustre: 250407:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4758.363282] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4758.365927] Lustre: Skipped 1 previous similar message [ 4758.407212] Lustre: lustre-MDT0001: new disk, initializing [ 4758.483184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4758.491954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4760.290365] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4764.243631] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4765.844334] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4769.087267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4769.127887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4769.257573] Lustre: lustre-OST0000: new disk, initializing [ 4769.260501] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4769.302727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4769.306571] Lustre: Skipped 1 previous similar message [ 4771.921706] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4775.654871] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4775.949046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4775.954070] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4776.004265] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4781.594990] LustreError: 251805:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4786.214855] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4786.243798] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4786.314428] Lustre: lustre-OST0001: new disk, initializing [ 4786.317137] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4786.362771] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4788.666561] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4791.688398] LustreError: 251805:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4791.703263] LustreError: 251805:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4792.458091] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4792.852490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4792.857108] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4792.904562] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4796.210483] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 4801.792197] LustreError: 251805:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4803.044919] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4803.047685] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4803.049316] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4803.053683] Lustre: Skipped 2 previous similar messages [ 4805.493190] Lustre: lustre-OST0001: Not available for connect from 192.168.201.42@tcp (stopping) [ 4805.496855] Lustre: Skipped 1 previous similar message [ 4808.624200] LustreError: 253903:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4808.626870] LustreError: 253903:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4808.654533] Lustre: server umount lustre-OST0001 complete [ 4813.285979] LustreError: 252760:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4813.292154] LustreError: 252760:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4818.403363] LustreError: 252135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4818.411159] LustreError: 252135:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4821.985383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4821.985698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4821.989266] LustreError: Skipped 2 previous similar messages [ 4821.995801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4821.998980] Lustre: Skipped 3 previous similar messages [ 4823.530444] LustreError: 252760:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4823.537458] LustreError: 252760:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4827.257229] LustreError: 254128:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4827.266021] LustreError: 254128:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4827.330214] Lustre: server umount lustre-OST0000 complete [ 4830.180928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4830.185994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4830.193375] Lustre: Skipped 1 previous similar message [ 4830.197228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4830.199925] Lustre: Skipped 2 previous similar messages [ 4835.853354] Lustre: server umount lustre-MDT0000 complete [ 4838.001755] LustreError: 253159:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764110680 with bad export cookie 6650010534014913835 [ 4838.002075] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4838.007721] LustreError: 253159:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4844.169533] LustreError: 254530:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4844.173075] LustreError: 254530:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 4844.266865] Lustre: server umount lustre-MDT0001 complete [ 4849.078738] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 4850.322457] Key type lgssc unregistered [ 4850.490132] LNet: 255062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4850.493518] LNetError: 255062:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4850.502471] LNet: Removed LNI 192.168.201.142@tcp [ 4850.930882] Key type .llcrypt unregistered [ 4850.932786] Key type ._llcrypt unregistered [ 4866.069537] Key type ._llcrypt registered [ 4866.071144] Key type .llcrypt registered [ 4866.117792] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 4873.724906] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 4874.262271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4874.278652] alg: No test for adler32 (adler32-zlib) [ 4875.168098] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 4875.284483] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 4876.880242] Key type lgssc registered [ 4877.374848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4881.132101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4883.922638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4886.833669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4889.724652] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4892.767147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4892.797740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4892.807479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4893.921433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4893.939798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4893.984817] Lustre: lustre-MDT0000: new disk, initializing [ 4894.025055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4894.034362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4895.542354] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4900.597576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4900.620822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4900.660281] Lustre: 258984:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4900.688919] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4900.691758] Lustre: Skipped 1 previous similar message [ 4900.730864] Lustre: lustre-MDT0001: new disk, initializing [ 4900.772780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4900.788087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4900.791810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4902.403099] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4906.001737] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4907.509438] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4910.414930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4910.442466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4910.553280] Lustre: lustre-OST0000: new disk, initializing [ 4910.555994] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4910.595912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4912.636175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4912.640644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4912.683569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4912.916449] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4916.496669] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4918.385498] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4918.490923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4920.221166] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4920.309317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4922.850562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4922.851835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4922.853055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4922.854067] LustreError: Skipped 1 previous similar message [ 4922.861275] Lustre: Skipped 1 previous similar message [ 4927.150293] LustreError: 261922:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4927.201772] Lustre: server umount lustre-OST0000 complete [ 4931.553231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4931.557050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4931.563140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4931.565499] Lustre: Skipped 1 previous similar message [ 4933.089325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4933.089617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4938.210509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4938.213318] Lustre: Skipped 2 previous similar messages [ 4943.328672] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4943.330272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4943.336130] Lustre: Skipped 1 previous similar message [ 4943.377820] LustreError: 262123:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4943.380520] LustreError: 262123:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4943.426878] Lustre: server umount lustre-MDT0000 complete [ 4945.050920] LustreError: 258976:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764110787 with bad export cookie 1703063582342834731 [ 4945.054765] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4945.056869] LustreError: 258976:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4945.145827] LustreError: 262326:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4945.148942] LustreError: 262326:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4945.232924] Lustre: server umount lustre-MDT0001 complete [ 4948.664398] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 17:46:30 (1764110790) [ 4951.379532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4951.635912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4953.250911] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4956.640990] LustreError: 262996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4956.647286] LustreError: 262996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4956.743304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4956.882496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4958.392876] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4960.614562] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4962.060202] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4964.732396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4967.214904] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 4969.441960] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5022.178303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5022.179140] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5022.180614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5022.182414] Lustre: Skipped 1 previous similar message [ 5025.839145] LustreError: 265340:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5025.842277] LustreError: 265340:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5025.896768] Lustre: server umount lustre-OST0000 complete [ 5027.481952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5032.088305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5032.217565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5032.220531] Lustre: Skipped 1 previous similar message [ 5034.458883] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5036.606910] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5042.469085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5042.475484] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5042.475573] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5042.483208] Lustre: Skipped 1 previous similar message [ 5073.377691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5073.378799] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5073.380690] LustreError: Skipped 1 previous similar message [ 5073.381343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5073.381347] Lustre: Skipped 1 previous similar message [ 5073.385913] Lustre: Skipped 1 previous similar message [ 5076.717181] LustreError: 267003:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5076.719405] LustreError: 267003:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5076.768791] Lustre: server umount lustre-OST0000 complete [ 5084.253877] Lustre: server umount lustre-MDT0000 complete [ 5085.589062] LustreError: 262976:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764110927 with bad export cookie 1703063582342835599 [ 5085.591861] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5085.593238] LustreError: 262976:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5085.664024] LustreError: 267407:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5085.667031] LustreError: 267407:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5085.733939] Lustre: server umount lustre-MDT0001 complete [ 5089.512312] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 5090.661971] Key type lgssc unregistered [ 5090.794633] LNet: 267938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5090.798507] LNetError: 267938:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5090.809574] LNet: Removed LNI 192.168.201.142@tcp [ 5091.123286] Key type .llcrypt unregistered [ 5091.125152] Key type ._llcrypt unregistered [ 5100.579674] Key type ._llcrypt registered [ 5100.580701] Key type .llcrypt registered [ 5100.618175] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 5100.965326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5101.003084] alg: No test for adler32 (adler32-zlib) [ 5101.878094] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 5101.971889] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 5103.560293] Key type lgssc registered [ 5103.951293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5106.776099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5106.782733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5107.955321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5109.464207] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5112.758965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5112.898079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5114.368167] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5116.359060] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5117.643290] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5120.129101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5120.267900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5122.516129] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5124.875491] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5134.640888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5149.153127] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5149.157333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5149.163703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5150.178624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5150.180067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5150.180332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5150.191989] Lustre: Skipped 1 previous similar message [ 5152.496303] LustreError: 272350:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5152.552491] Lustre: server umount lustre-OST0000 complete [ 5155.298839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5155.300091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5155.304163] Lustre: Skipped 1 previous similar message [ 5155.309072] Lustre: Skipped 1 previous similar message [ 5160.261532] LustreError: 272553:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5160.264345] LustreError: 272553:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5160.296733] Lustre: server umount lustre-MDT0000 complete [ 5160.417680] LustreError: 269313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5160.424042] LustreError: 269313:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5161.759571] LustreError: 270031:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764111003 with bad export cookie 11732664520016001378 [ 5161.762716] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5161.764056] LustreError: 270031:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5161.832911] LustreError: 272755:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5161.835550] LustreError: 272755:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5161.894939] Lustre: server umount lustre-MDT0001 complete [ 5165.862316] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 5167.086075] Key type lgssc unregistered [ 5167.228555] LNet: 273288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5167.231952] LNetError: 273288:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5167.242451] LNet: Removed LNI 192.168.201.142@tcp [ 5167.577126] Key type .llcrypt unregistered [ 5167.578797] Key type ._llcrypt unregistered [ 5175.109344] Key type ._llcrypt registered [ 5175.110951] Key type .llcrypt registered [ 5175.162919] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 5175.627861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5175.634917] alg: No test for adler32 (adler32-zlib) [ 5176.516149] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 5176.607545] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 5178.192230] Key type lgssc registered [ 5178.610224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5181.713092] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5181.721099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5182.904410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5184.346514] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5187.584806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5187.709200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5189.123032] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5191.009137] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5192.280801] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5194.811731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5194.987775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5197.309666] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5199.359864] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5208.294535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5213.667453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5213.667800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5213.667940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5213.673022] Lustre: Skipped 1 previous similar message [ 5218.789313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5218.791378] Lustre: Skipped 2 previous similar messages [ 5219.698183] LustreError: 276877:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5219.744900] Lustre: server umount lustre-OST0000 complete [ 5223.905586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5223.905801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5223.913217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5223.914764] Lustre: Skipped 1 previous similar message [ 5227.264384] LustreError: 277079:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5227.267313] LustreError: 277079:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5227.301587] Lustre: server umount lustre-MDT0000 complete [ 5228.791085] LustreError: 274447:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764111070 with bad export cookie 7246594306181189028 [ 5228.794768] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5228.795984] LustreError: 274447:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5228.862665] LustreError: 277281:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5228.864618] LustreError: 277281:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5228.918787] Lustre: server umount lustre-MDT0001 complete [ 5232.847697] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 5233.963151] Key type lgssc unregistered [ 5234.097489] LNet: 277812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5234.101128] LNetError: 277812:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5234.111451] LNet: Removed LNI 192.168.201.142@tcp [ 5234.444866] Key type .llcrypt unregistered [ 5234.446409] Key type ._llcrypt unregistered [ 5243.846215] Key type ._llcrypt registered [ 5243.847130] Key type .llcrypt registered [ 5243.881894] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 5244.254556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5244.277995] alg: No test for adler32 (adler32-zlib) [ 5245.153950] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 5245.271772] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 5246.872172] Key type lgssc registered [ 5247.288627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5250.165088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5250.172902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5251.335299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5252.730761] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5255.994957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5256.140483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5257.561601] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5259.631665] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5261.004510] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5263.495199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5263.625656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5265.689838] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5267.612244] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5275.943600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5287.392835] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5287.395102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5287.400611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5291.490549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5291.492094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5291.493403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5292.592115] LustreError: 282226:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5292.637211] Lustre: server umount lustre-OST0000 complete [ 5296.609702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5296.614546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5296.616192] Lustre: Skipped 2 previous similar messages [ 5301.729101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5301.732124] Lustre: Skipped 1 previous similar message [ 5306.849660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5306.851861] Lustre: Skipped 1 previous similar message [ 5308.384127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5308.421228] LustreError: 282428:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5308.423828] LustreError: 282428:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5308.451121] Lustre: server umount lustre-MDT0000 complete [ 5309.752151] LustreError: 279167:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764111151 with bad export cookie 13676867048832846019 [ 5309.755218] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5309.756543] LustreError: 279167:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5309.813622] LustreError: 282631:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5309.815943] LustreError: 282631:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5309.871972] Lustre: server umount lustre-MDT0001 complete [ 5313.494911] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 5314.563322] Key type lgssc unregistered [ 5314.697392] LNet: 283162:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5314.700297] LNetError: 283162:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5314.708260] LNet: Removed LNI 192.168.201.142@tcp [ 5315.010320] Key type .llcrypt unregistered [ 5315.011415] Key type ._llcrypt unregistered [ 5322.078114] Key type ._llcrypt registered [ 5322.079933] Key type .llcrypt registered [ 5322.126366] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 5322.576080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5322.583989] alg: No test for adler32 (adler32-zlib) [ 5323.442309] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 5323.532158] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 5325.120173] Key type lgssc registered [ 5325.529991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5328.257139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5328.265923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5329.432078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5330.802082] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5333.866661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5334.002674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5335.337079] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5337.076666] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5338.295529] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5340.651059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5340.766863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5342.697013] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5344.511236] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5354.087162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 5359.585970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5359.588751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5359.590177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5359.593542] Lustre: Skipped 1 previous similar message [ 5360.495119] LustreError: 286787:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5360.545662] Lustre: server umount lustre-OST0000 complete [ 5364.707163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5364.707212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5364.707900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5364.707905] Lustre: Skipped 1 previous similar message [ 5364.709236] LustreError: Skipped 1 previous similar message [ 5364.716621] Lustre: Skipped 1 previous similar message [ 5367.995723] LustreError: 286987:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5367.997629] LustreError: 286987:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5368.030813] Lustre: server umount lustre-MDT0000 complete [ 5369.249343] LustreError: 284320:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764111211 with bad export cookie 16367187743735409688 [ 5369.252032] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5369.254949] LustreError: 284320:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5369.325650] LustreError: 287190:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5369.327105] LustreError: 287190:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5369.388331] Lustre: server umount lustre-MDT0001 complete [ 5372.666778] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 5373.615391] Key type lgssc unregistered [ 5373.747483] LNet: 287721:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5373.750133] LNetError: 287721:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5373.756446] LNet: Removed LNI 192.168.201.142@tcp [ 5374.035436] Key type .llcrypt unregistered [ 5374.037208] Key type ._llcrypt unregistered [ 5377.578543] Key type ._llcrypt registered [ 5377.579778] Key type .llcrypt registered [ 5377.621887] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5383.062599] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 5389.058983] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 5389.486206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5389.492179] alg: No test for adler32 (adler32-zlib) [ 5390.341893] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 5390.419067] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 5392.000183] Key type lgssc registered [ 5392.366065] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5395.283518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5397.378198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5399.544883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5401.589255] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5403.898140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5403.918527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5403.924919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5405.003878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5405.014127] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5405.046995] Lustre: lustre-MDT0000: new disk, initializing [ 5405.071037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5405.077509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5406.399472] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5410.514790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5410.537669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5410.570823] Lustre: 291203:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5410.595139] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5410.597691] Lustre: Skipped 1 previous similar message [ 5410.635398] Lustre: lustre-MDT0001: new disk, initializing [ 5410.667672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5410.681485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5410.685270] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5412.009034] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5415.118448] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5416.313885] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5418.620518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5418.645066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5418.746177] Lustre: lustre-OST0000: new disk, initializing [ 5418.748163] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5418.776372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5420.345105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5420.348532] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5420.382565] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5420.623599] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5423.741486] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5425.103771] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5425.177241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5426.426547] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5426.497826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5429.165112] LustreError: 294141:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5429.204587] Lustre: server umount lustre-OST0000 complete [ 5430.753789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5430.754372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5430.757472] Lustre: Skipped 1 previous similar message [ 5435.873175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5435.875398] Lustre: Skipped 2 previous similar messages [ 5439.971955] LustreError: 294342:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5439.974101] LustreError: 294342:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5439.996225] Lustre: server umount lustre-MDT0000 complete [ 5441.220146] LustreError: 291194:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764111283 with bad export cookie 11799088919119390001 [ 5441.222429] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5441.225128] LustreError: 291194:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5451.748203] LustreError: 294544:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5451.750066] LustreError: 294544:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5451.795454] Lustre: server umount lustre-MDT0001 complete [ 5454.941688] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 17:54:56 (1764111296) [ 5457.280623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5457.435213] LustreError: 295215:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5457.441678] LustreError: 295215:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5457.457975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5458.740403] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5461.639265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5461.755972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5463.029243] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5464.833797] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5466.109797] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5468.413038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5470.451367] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5472.328462] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5487.073993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5487.074127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5487.076807] LustreError: Skipped 1 previous similar message [ 5487.081146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5487.085375] Lustre: Skipped 1 previous similar message [ 5491.820116] LustreError: 297484:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5491.822883] LustreError: 297484:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5491.860277] Lustre: server umount lustre-OST0000 complete [ 5495.779074] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5495.781486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5495.785034] Lustre: Skipped 1 previous similar message [ 5495.786965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5499.261202] LustreError: 297685:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5499.263213] LustreError: 297685:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5499.288252] Lustre: server umount lustre-MDT0000 complete [ 5500.621604] LustreError: 295195:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764111342 with bad export cookie 11799088919119390869 [ 5500.624653] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5500.625051] LustreError: 295195:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5500.630208] LustreError: 295938:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 5500.762966] Lustre: server umount lustre-MDT0001 complete [ 5504.316232] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 5505.370955] Key type lgssc unregistered [ 5505.496447] LNet: 298417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5505.499287] LNetError: 298417:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5505.505310] LNet: Removed LNI 192.168.201.142@tcp [ 5505.795085] Key type .llcrypt unregistered [ 5505.796323] Key type ._llcrypt unregistered [ 5514.518936] Key type ._llcrypt registered [ 5514.520256] Key type .llcrypt registered [ 5514.557643] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 5520.809095] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 5521.162221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5521.210926] alg: No test for adler32 (adler32-zlib) [ 5522.060132] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 5522.145367] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 5523.728158] Key type lgssc registered [ 5524.108810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5527.030158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5529.177286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5531.302268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5533.704487] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5535.726176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5537.736285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5540.394267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5540.415234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5540.421439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5541.516365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5541.531462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5541.571765] Lustre: lustre-MDT0000: new disk, initializing [ 5541.598308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5541.606461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5542.999215] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5547.369496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5547.396335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5547.438474] Lustre: 302194:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5547.467266] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5547.469783] Lustre: Skipped 1 previous similar message [ 5547.508757] Lustre: lustre-MDT0001: new disk, initializing [ 5547.545566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5547.560481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5547.564402] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5548.941607] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5552.201473] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5553.576821] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5555.836689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5555.859373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5555.961757] Lustre: lustre-OST2710: new disk, initializing [ 5555.964882] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5555.996621] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5557.941666] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5559.223370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5559.228430] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5559.261985] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5561.006959] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 5563.287519] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5563.310972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5563.365215] Lustre: lustre-OST03e8: new disk, initializing [ 5563.367157] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5563.397961] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5565.362147] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5566.465188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5566.469910] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5566.498231] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5567.260684] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 5569.943687] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5570.019941] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5571.338524] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5571.417973] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5572.750381] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5572.820729] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5574.149474] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5574.225452] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5576.675267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5576.676405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5576.678492] Lustre: Skipped 3 previous similar messages [ 5576.681050] Lustre: Skipped 3 previous similar messages [ 5581.437694] LustreError: 306540:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5581.466890] Lustre: server umount lustre-MDT0000 complete [ 5582.712314] LustreError: 302186:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764111424 with bad export cookie 2638497679322245521 [ 5582.717674] LustreError: 302186:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5582.717758] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5582.783690] LustreError: 306741:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5582.785515] LustreError: 306741:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5582.836077] Lustre: server umount lustre-MDT0001 complete [ 5594.107087] LustreError: 306941:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5594.109020] LustreError: 306941:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5594.126607] Lustre: server umount lustre-OST2710 complete [ 5605.553806] LustreError: 307143:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5605.555991] LustreError: 307143:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5605.592575] Lustre: server umount lustre-OST03e8 complete [ 5611.675018] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 5615.232398] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 5618.741603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5620.836167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5622.856299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5624.966959] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5627.189467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5627.211276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5627.286608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5627.298566] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5627.333319] Lustre: lustre-MDT0000: new disk, initializing [ 5627.355437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5627.361591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5628.650204] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5632.767453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5632.789523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5632.820045] Lustre: 309934:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5632.919262] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5634.204426] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5637.307799] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5638.599821] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5640.907577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5640.931454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5641.019503] Lustre: lustre-OST0000: new disk, initializing [ 5641.021260] Lustre: Skipped 1 previous similar message [ 5641.023657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5641.026404] Lustre: Skipped 2 previous similar messages [ 5641.051703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5641.054581] Lustre: Skipped 1 previous similar message [ 5642.359070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5642.362651] Lustre: Skipped 1 previous similar message [ 5642.364193] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5642.394507] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5642.926366] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5645.989262] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5647.328547] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5647.394406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5648.658722] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5648.731221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5652.961532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5652.962536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5652.963925] LustreError: Skipped 1 previous similar message [ 5652.968833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5652.971008] Lustre: Skipped 1 previous similar message [ 5655.406107] LustreError: 312870:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5655.408972] LustreError: 312870:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5655.446812] Lustre: server umount lustre-OST0000 complete [ 5658.081896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5658.082455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5658.086316] Lustre: Skipped 2 previous similar messages [ 5662.745827] Lustre: server umount lustre-MDT0000 complete [ 5663.201127] LustreError: 309945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5663.205550] LustreError: 309945:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 5664.073644] LustreError: 309924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764111506 with bad export cookie 2638497679322247187 [ 5664.078709] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5664.144225] LustreError: 313273:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5664.147065] LustreError: 313273:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5664.209664] Lustre: server umount lustre-MDT0001 complete [ 5667.010960] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 17:58:28 (1764111508) [ 5667.504201] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5668.132470] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 17:58:30 (1764111510) [ 5669.122266] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 5671.882745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5673.234514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5674.546084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5675.850836] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5676.227644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5678.544296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5678.635422] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5678.642879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5678.705468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5680.103996] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5683.091033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5683.133441] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5683.143531] Lustre: 315397:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5684.710474] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5686.714521] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5687.988074] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5690.194506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5690.223349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5690.285340] Lustre: Denying initial registration attempt from nid 192.168.201.142@tcp, specified as failover [ 5690.288392] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5690.291316] LustreError: 317013:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 5690.293458] LustreError: 317013:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 5690.295712] LustreError: 317013:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 5690.315622] Lustre: server umount lustre-OST0000 complete [ 5690.317042] LustreError: 317013:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5693.408748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5693.411282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5693.415100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5693.417607] Lustre: Skipped 1 previous similar message [ 5698.622942] LustreError: 317380:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 5698.624815] LustreError: 317380:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5698.649163] Lustre: server umount lustre-MDT0000 complete [ 5699.824987] LustreError: 315389:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764111542 with bad export cookie 2638497679322248055 [ 5699.828244] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5699.829120] LustreError: 315389:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5699.934602] Lustre: server umount lustre-MDT0001 complete [ 5703.539817] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 5704.532819] Key type lgssc unregistered [ 5704.664354] LNet: 318116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5704.667569] LNetError: 318116:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5704.676287] LNet: Removed LNI 192.168.201.142@tcp [ 5704.955952] Key type .llcrypt unregistered [ 5704.956907] Key type ._llcrypt unregistered [ 5708.718072] Key type ._llcrypt registered [ 5708.719470] Key type .llcrypt registered [ 5708.757618] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 5709.075279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5709.122110] alg: No test for adler32 (adler32-zlib) [ 5709.971203] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 5710.041735] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 5711.616147] Key type lgssc registered [ 5711.948476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5714.188398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5715.418568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5716.607913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5717.819060] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5718.147996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5724.714381] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 5727.562152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5727.569668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5728.653866] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5728.662057] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5728.713470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5729.993522] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5733.090917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5733.129566] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5733.143740] Lustre: 320848:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5733.255812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5734.731107] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5736.584927] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5737.771273] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5740.126737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5740.155584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5740.227382] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5740.293122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5742.296689] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5744.172204] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5747.683224] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5747.685410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5747.686201] LustreError: Skipped 1 previous similar message [ 5747.693518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5747.696253] Lustre: Skipped 1 previous similar message [ 5750.958105] LustreError: 323133:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5750.999784] Lustre: server umount lustre-OST0000 complete [ 5752.801465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5752.802148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5752.806168] Lustre: Skipped 2 previous similar messages [ 5757.921209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5757.924725] Lustre: Skipped 2 previous similar messages [ 5758.403778] LustreError: 323334:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5758.407194] LustreError: 323334:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5758.437054] Lustre: server umount lustre-MDT0000 complete [ 5759.737516] LustreError: 320838:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764111601 with bad export cookie 13294591618017852170 [ 5759.740706] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5759.744040] LustreError: 320838:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5759.814677] LustreError: 323536:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5759.816742] LustreError: 323536:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5759.866531] Lustre: server umount lustre-MDT0001 complete [ 5763.609347] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 5764.688488] Key type lgssc unregistered [ 5764.820564] LNet: 324066:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5764.822948] LNetError: 324066:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5764.833321] LNet: Removed LNI 192.168.201.142@tcp [ 5765.141237] Key type .llcrypt unregistered [ 5765.142318] Key type ._llcrypt unregistered [ 5774.680833] Key type ._llcrypt registered [ 5774.682461] Key type .llcrypt registered [ 5774.727543] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 5775.072846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5775.089059] alg: No test for adler32 (adler32-zlib) [ 5775.958699] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 5776.047379] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 5777.640198] Key type lgssc registered [ 5778.045694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5780.743202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5780.751067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5781.917934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5783.285586] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5786.346494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5786.476627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5787.907386] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5789.783765] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5791.118604] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5793.572541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5793.694826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5795.711915] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5797.737282] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5812.194738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5812.194749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5812.195761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5812.199958] Lustre: Skipped 1 previous similar message [ 5813.612392] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.42@tcp (stopping) [ 5813.615416] Lustre: Skipped 2 previous similar messages [ 5816.633896] LustreError: 327717:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5816.660835] Lustre: server umount lustre-MDT0000 complete [ 5817.313317] LustreError: 326161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5817.318810] LustreError: 326161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5817.850083] LustreError: 325430:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764111660 with bad export cookie 3811164310457446432 [ 5817.853180] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5817.853186] LustreError: 325430:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5818.725715] LustreError: 325444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5818.734077] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.42@tcp (stopping) [ 5822.433143] LustreError: 327524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5822.433349] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5822.444101] Lustre: Skipped 1 previous similar message [ 5822.446784] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5827.552757] LustreError: 327524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5827.553077] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5827.559707] LustreError: 327524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5827.562105] Lustre: Skipped 1 previous similar message [ 5832.160160] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5832.195254] LustreError: 327920:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5832.197597] LustreError: 327920:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5832.243993] Lustre: server umount lustre-MDT0001 complete [ 5834.447370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5837.236352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5842.912640] LustreError: 324758:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007d5de868 x1849805103980928/t0(0) o250->MGC192.168.201.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5843.027728] LustreError: 327057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.034311] LustreError: 327057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5843.054859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5843.080970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 5844.434114] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5847.480626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5847.609111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5849.003559] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5850.964394] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5858.793561] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5859.665278] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec [ 5860.863337] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5861.423436] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5862.881683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5862.883903] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5862.887543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5862.889745] Lustre: Skipped 2 previous similar messages [ 5863.906153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5877.216301] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5877.240088] LustreError: 330234:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5877.241901] LustreError: 330234:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5877.280705] Lustre: server umount lustre-OST0000 complete [ 5879.265862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5879.266827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5879.272459] Lustre: Skipped 2 previous similar messages [ 5879.274328] Lustre: Skipped 6 previous similar messages [ 5884.668463] LustreError: 330437:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5884.670187] LustreError: 330437:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5884.698438] Lustre: server umount lustre-MDT0000 complete [ 5886.030383] LustreError: 325473:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764111728 with bad export cookie 3811164310457453551 [ 5886.035966] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5886.036507] LustreError: 325473:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5886.163817] Lustre: server umount lustre-MDT0001 complete [ 5889.966311] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 5891.029729] Key type lgssc unregistered [ 5891.194564] LNet: 331170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5891.197442] LNetError: 331170:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5891.206524] LNet: Removed LNI 192.168.201.142@tcp [ 5891.527702] Key type .llcrypt unregistered [ 5891.529554] Key type ._llcrypt unregistered [ 5901.161154] Key type ._llcrypt registered [ 5901.163090] Key type .llcrypt registered [ 5901.204429] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 5901.548420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5901.585547] alg: No test for adler32 (adler32-zlib) [ 5902.454606] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 5902.560898] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 5904.152162] Key type lgssc registered [ 5904.567246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5907.409593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5907.417910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5908.590497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5909.973140] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5912.952032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5913.080145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5914.443581] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5916.353479] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5917.701734] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5922.914191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5922.914829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5922.919235] Lustre: Skipped 1 previous similar message [ 5922.924068] Lustre: Skipped 1 previous similar message [ 5924.925323] LustreError: 333938:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5924.951783] Lustre: server umount lustre-MDT0000 complete [ 5926.186093] LustreError: 333941:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764111768 with bad export cookie 15776867566601685392 [ 5926.189623] LustreError: 333941:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5926.189992] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5926.257870] LustreError: 334142:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5926.259656] LustreError: 334142:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5926.320212] Lustre: server umount lustre-MDT0001 complete [ 5931.337610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5931.440476] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5931.452922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5931.534276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5932.873792] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5935.822426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5935.854056] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5935.863768] Lustre: 334910:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5935.957554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5937.251224] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5939.169803] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5940.454455] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5943.231555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5943.310741] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5943.315458] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5943.319405] LustreError: 336569:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 5943.323049] LustreError: 336569:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 5943.325559] LustreError: 336569:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 5943.352346] Lustre: server umount lustre-OST0000 complete [ 5943.353803] LustreError: 336569:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5946.381331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5946.450814] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5946.518614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5948.574209] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5950.473197] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5953.266747] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5953.291656] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5953.319398] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5953.349259] Lustre: lustre-OST0001: new disk, initializing [ 5953.351664] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5955.476382] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 5955.607721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5955.614417] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5955.632489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 5955.659120] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5957.483316] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5960.673721] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5960.674032] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5960.676630] LustreError: Skipped 1 previous similar message [ 5960.677921] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5960.681083] Lustre: Skipped 1 previous similar message [ 5960.685756] Lustre: Skipped 1 previous similar message [ 5964.530638] LustreError: 338683:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5964.532681] LustreError: 338683:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5964.553839] Lustre: server umount lustre-OST0001 complete [ 5965.794435] LustreError: 338361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5965.794560] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5965.795304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5965.796397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5965.799668] LustreError: 338361:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5965.801870] LustreError: Skipped 1 previous similar message [ 5970.915737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5970.915797] LustreError: 337038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5970.917805] Lustre: Skipped 2 previous similar messages [ 5970.923638] LustreError: 337038:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5976.035680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5976.035792] LustreError: 337036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5976.037936] Lustre: Skipped 1 previous similar message [ 5976.042026] LustreError: 337036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5980.128141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5980.150080] LustreError: 338884:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5980.151759] LustreError: 338884:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5980.194530] Lustre: server umount lustre-OST0000 complete [ 5982.176925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5982.179858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5982.183256] Lustre: Skipped 1 previous similar message [ 5986.273042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5986.273686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5986.274930] Lustre: Skipped 1 previous similar message [ 5987.584534] LustreError: 339088:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5987.587127] LustreError: 339088:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5987.618461] Lustre: server umount lustre-MDT0000 complete [ 5988.962263] LustreError: 337053:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764111831 with bad export cookie 15776867566601685931 [ 5988.965196] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5988.966899] LustreError: 337053:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5989.102842] Lustre: server umount lustre-MDT0001 complete [ 5992.887213] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 5993.990657] Key type lgssc unregistered [ 5994.129454] LNet: 339819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5994.131920] LNetError: 339819:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5994.140323] LNet: Removed LNI 192.168.201.142@tcp [ 5994.454232] Key type .llcrypt unregistered [ 5994.456308] Key type ._llcrypt unregistered [ 6006.241434] Key type ._llcrypt registered [ 6006.243063] Key type .llcrypt registered [ 6006.281922] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 6012.581666] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 6012.870281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6012.889705] alg: No test for adler32 (adler32-zlib) [ 6013.748040] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 6013.840500] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 6015.424117] Key type lgssc registered [ 6015.809858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6018.926262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6021.264214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6023.518211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6025.774365] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6028.244661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6028.268582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6028.273228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6029.355380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6029.366815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6029.408984] Lustre: lustre-MDT0000: new disk, initializing [ 6029.435511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6029.443908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6030.818499] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6035.066944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6035.092927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6035.130434] Lustre: 343741:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6035.156606] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6035.158441] Lustre: Skipped 1 previous similar message [ 6035.194064] Lustre: lustre-MDT0001: new disk, initializing [ 6035.227258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6035.240560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6035.245314] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6036.728614] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6039.937185] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6041.280965] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6043.735143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6043.754686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6043.849375] Lustre: lustre-OST0000: new disk, initializing [ 6043.851761] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6043.881482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6045.440232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6045.444418] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6045.483766] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6045.904460] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6049.081948] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6050.469391] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6050.545388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6051.956316] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6052.035651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6054.704101] LustreError: 346678:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6054.755994] Lustre: server umount lustre-OST0000 complete [ 6061.026220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6061.026260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6061.027053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6061.030678] Lustre: Skipped 1 previous similar message [ 6066.145415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6066.147968] Lustre: Skipped 2 previous similar messages [ 6066.148869] LustreError: 346880:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6066.152121] LustreError: 346880:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6066.179187] Lustre: server umount lustre-MDT0000 complete [ 6067.378130] LustreError: 344520:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764111909 with bad export cookie 3157463928114801955 [ 6067.380704] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6067.381735] LustreError: 344520:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6076.900725] LustreError: 347083:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6076.902846] LustreError: 347083:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6076.949328] Lustre: server umount lustre-MDT0001 complete [ 6079.902415] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 18:05:21 (1764111921) [ 6081.856618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6083.740482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6091.099829] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 6094.187820] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 6097.425257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6099.501059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6101.493656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6103.593832] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6105.730710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6105.750380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6105.819272] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6105.829659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6105.862049] Lustre: lustre-MDT0000: new disk, initializing [ 6105.882570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6105.890020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6107.059500] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6111.036452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6111.060622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6111.086881] Lustre: 350851:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6111.104921] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6111.106591] Lustre: Skipped 1 previous similar message [ 6111.140662] Lustre: lustre-MDT0001: new disk, initializing [ 6111.169159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6111.179545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6111.182829] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6112.359613] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6115.315490] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6116.462389] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6118.630824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6118.652806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6120.514761] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6123.558727] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6124.926800] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6125.562055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6125.565771] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6125.600415] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6126.017657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6127.288977] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6127.358095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6130.659123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6130.659312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6130.661302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6130.661369] LustreError: Skipped 1 previous similar message [ 6130.664715] Lustre: Skipped 1 previous similar message [ 6134.061098] LustreError: 353794:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6134.063377] LustreError: 353794:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6134.104517] Lustre: server umount lustre-OST0000 complete [ 6135.777227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6135.777944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6135.781068] Lustre: Skipped 1 previous similar message [ 6135.784288] Lustre: Skipped 2 previous similar messages [ 6140.897271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6140.900421] Lustre: Skipped 1 previous similar message [ 6141.507623] LustreError: 353995:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6141.509250] LustreError: 353995:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6141.540276] Lustre: server umount lustre-MDT0000 complete [ 6142.861444] LustreError: 350841:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764111985 with bad export cookie 3157463928114802823 [ 6142.866270] LustreError: 350841:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6142.866692] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6143.014567] Lustre: server umount lustre-MDT0001 complete [ 6145.968351] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 18:06:27 (1764111987) [ 6148.479293] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 18:06:30 (1764111990) [ 6151.433686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6151.618643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6151.621503] Lustre: Skipped 1 previous similar message [ 6153.079259] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6156.056806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6157.579668] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6159.446782] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6160.795523] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6163.305381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6165.465498] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6167.454375] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6181.628316] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6182.198669] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6182.728222] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6183.325500] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6184.000101] Lustre: Failing over lustre-MDT0000 [ 6184.224151] LustreError: 357669:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6184.226014] LustreError: 357669:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6184.260248] Lustre: server umount lustre-MDT0000 complete [ 6186.978615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6186.980807] LustreError: 355205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6186.982933] Lustre: Skipped 2 previous similar messages [ 6186.989752] LustreError: 355205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6186.991684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6188.395403] LustreError: 355206:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6192.098126] LustreError: 355210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6192.103771] LustreError: 355210:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6197.217519] LustreError: 355210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6197.223522] LustreError: 355210:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6197.313250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6197.351812] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6197.440131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6197.442553] Lustre: Skipped 2 previous similar messages [ 6197.463098] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6198.629799] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6198.783676] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6202.860346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6202.866168] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6202.883057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6204.158148] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6204.772733] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6206.682169] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 6207.375113] Lustre: Failing over lustre-MDT0000 [ 6207.460879] LustreError: 358749:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6207.462647] LustreError: 358749:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6207.492472] Lustre: server umount lustre-MDT0000 complete [ 6207.969680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6207.969995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6207.971055] LustreError: 355205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6207.973685] Lustre: Skipped 2 previous similar messages [ 6218.211739] LustreError: 356515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6218.230972] LustreError: 356515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6223.231536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6223.300538] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6223.542089] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6224.239467] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6226.028537] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6228.964704] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6228.980569] Lustre: Skipped 2 previous similar messages [ 6229.000709] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6229.038332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6231.862271] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6233.168518] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6239.205346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6239.207245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6239.210080] Lustre: Skipped 1 previous similar message [ 6239.210203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6243.456555] LustreError: 359779:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6243.458789] LustreError: 359779:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6243.487738] Lustre: server umount lustre-MDT0000 complete [ 6244.321122] LustreError: 355210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6244.324700] LustreError: 355210:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6244.649145] LustreError: 355235:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764112086 with bad export cookie 3157463928114805490 [ 6244.652099] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6244.654048] LustreError: 355235:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6244.754235] Lustre: server umount lustre-MDT0001 complete [ 6256.221380] Lustre: server umount lustre-OST0000 complete [ 6261.515351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6261.682166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6261.684578] Lustre: Skipped 1 previous similar message [ 6262.967369] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6265.955475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6267.503162] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6269.294072] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6270.481193] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6272.930321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6274.860738] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6276.760403] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6286.371442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6291.425931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6291.426835] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6291.427678] LustreError: Skipped 1 previous similar message [ 6291.431403] Lustre: Skipped 1 previous similar message [ 6291.433174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6291.435833] Lustre: Skipped 3 previous similar messages [ 6294.543552] Lustre: server umount lustre-OST0000 complete [ 6310.368124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6310.401615] LustreError: 363479:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6310.403448] LustreError: 363479:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 6310.427890] Lustre: server umount lustre-MDT0000 complete [ 6310.881161] LustreError: 362341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6310.886142] LustreError: 362341:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6311.626995] LustreError: 360990:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764112153 with bad export cookie 3157463928114806225 [ 6311.629639] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6311.630805] LustreError: 360990:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6311.738251] Lustre: server umount lustre-MDT0001 complete [ 6315.290512] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 6316.394806] Key type lgssc unregistered [ 6316.531360] LNet: 364213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6316.534398] LNetError: 364213:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6316.546446] LNet: Removed LNI 192.168.201.142@tcp [ 6316.889131] Key type .llcrypt unregistered [ 6316.890732] Key type ._llcrypt unregistered [ 6326.426251] Key type ._llcrypt registered [ 6326.427510] Key type .llcrypt registered [ 6326.465443] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 6326.865124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6326.874216] alg: No test for adler32 (adler32-zlib) [ 6327.724563] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 6327.804397] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 6329.384144] Key type lgssc registered [ 6329.687111] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6332.342378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6332.350571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6333.515801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6334.866742] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6337.885224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6338.020726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6339.430647] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6341.281371] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6342.591073] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6344.904169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6345.011522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6346.808286] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6348.564542] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6358.311040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6363.618259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6363.619458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6363.620225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6363.622216] Lustre: Skipped 1 previous similar message [ 6368.702808] LustreError: 367956:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6368.729564] Lustre: server umount lustre-MDT0000 complete [ 6368.737515] LustreError: 365688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6368.744450] LustreError: 365688:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6369.872131] LustreError: 365668:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764112212 with bad export cookie 5439525909419541627 [ 6369.874786] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6369.877680] LustreError: 365668:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6369.935155] LustreError: 368159:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6369.937224] LustreError: 368159:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6369.981176] Lustre: server umount lustre-MDT0001 complete [ 6381.427191] LustreError: 368360:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6381.430507] LustreError: 368360:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6381.474597] Lustre: server umount lustre-OST0000 complete [ 6386.244928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6386.408811] LustreError: 369089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6386.437585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6387.796949] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6390.876054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6391.010450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6392.459461] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6394.332185] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6395.551937] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6397.755197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6399.658026] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6401.416514] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6411.174907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6421.473933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6421.476588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6421.478268] Lustre: Skipped 3 previous similar messages [ 6421.480408] Lustre: Skipped 3 previous similar messages [ 6424.898943] LustreError: 371459:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6424.901610] LustreError: 371459:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6424.931059] Lustre: server umount lustre-MDT0000 complete [ 6426.256276] LustreError: 369069:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764112268 with bad export cookie 5439525909419542726 [ 6426.259855] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6426.260757] LustreError: 369069:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6426.371753] Lustre: server umount lustre-MDT0001 complete [ 6436.978091] LustreError: 371862:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6436.980149] LustreError: 371862:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6437.023294] Lustre: server umount lustre-OST0000 complete [ 6442.550918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6442.709727] LustreError: 372690:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6442.734743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6442.737966] Lustre: Skipped 1 previous similar message [ 6444.124482] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6447.129246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6448.740658] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6450.682169] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6452.014480] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6454.514557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6456.703841] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6458.625973] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6466.919535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6472.165228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6472.167055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6472.168088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6472.170635] Lustre: Skipped 1 previous similar message [ 6472.173104] Lustre: Skipped 1 previous similar message [ 6476.334086] LustreError: 374955:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6476.337348] LustreError: 374955:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6476.375906] Lustre: server umount lustre-OST0000 complete [ 6478.304890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6478.309108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6478.315767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6478.317798] Lustre: Skipped 1 previous similar message [ 6482.401311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6482.401764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6483.806846] Lustre: server umount lustre-MDT0000 complete [ 6485.147554] LustreError: 373408:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764112327 with bad export cookie 5439525909419543734 [ 6485.150967] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6485.152765] LustreError: 373408:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6485.275685] Lustre: server umount lustre-MDT0001 complete [ 6489.206919] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 6490.291619] Key type lgssc unregistered [ 6490.426381] LNet: 375890:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6490.429046] LNetError: 375890:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6490.438393] LNet: Removed LNI 192.168.201.142@tcp [ 6490.764065] Key type .llcrypt unregistered [ 6490.765252] Key type ._llcrypt unregistered [ 6500.635322] Key type ._llcrypt registered [ 6500.637225] Key type .llcrypt registered [ 6500.685090] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 6501.145569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6501.182955] alg: No test for adler32 (adler32-zlib) [ 6502.043448] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 6502.123301] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 6503.704175] Key type lgssc registered [ 6504.067141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6506.605813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6506.611775] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6506.615060] LustreError: 377288:0:(osd_handler.c:8577:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6506.625104] LustreError: 377288:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6506.627059] LustreError: 377288:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6506.629825] LustreError: 377288:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6506.632722] LustreError: 377288:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6509.304499] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6509.307672] LustreError: 377685:0:(osd_handler.c:8577:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6509.316338] LustreError: 377685:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6509.319164] LustreError: 377685:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6509.322176] LustreError: 377685:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6509.325488] LustreError: 377685:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6518.385709] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 6519.806542] Key type lgssc unregistered [ 6519.976952] LNet: 378669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6519.981095] LNetError: 378669:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6519.990607] LNet: Removed LNI 192.168.201.142@tcp [ 6520.387605] Key type .llcrypt unregistered [ 6520.389154] Key type ._llcrypt unregistered [ 6529.150814] Key type ._llcrypt registered [ 6529.152217] Key type .llcrypt registered [ 6529.193932] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 6535.158209] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 6535.450199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6535.477483] alg: No test for adler32 (adler32-zlib) [ 6536.319596] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 6536.398049] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 6537.976159] Key type lgssc registered [ 6538.315448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6541.149097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6543.261637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6545.352850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6547.646271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6549.981543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6550.000934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6550.006385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6551.092817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6551.106585] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6551.147484] Lustre: lustre-MDT0000: new disk, initializing [ 6551.179659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6551.187434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6552.621060] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6557.733173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6557.760071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6557.805352] Lustre: 381772:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6557.840143] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6557.843055] Lustre: Skipped 1 previous similar message [ 6557.883409] Lustre: lustre-MDT0001: new disk, initializing [ 6557.929407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6557.946403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6557.951836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6559.961095] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6564.639454] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6566.625674] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6569.971433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6570.007903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6570.128225] Lustre: lustre-OST0000: new disk, initializing [ 6570.130977] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6570.173923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6573.018373] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6575.136954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6575.145546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6575.203487] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6577.558838] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6579.227394] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6579.319382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6581.009877] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6581.114265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6585.316816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6585.317079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6585.317898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6585.321925] Lustre: Skipped 1 previous similar message [ 6585.332158] LustreError: Skipped 1 previous similar message [ 6588.014081] LustreError: 384705:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6588.051495] Lustre: server umount lustre-OST0000 complete [ 6590.433673] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6590.434326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6590.437629] Lustre: Skipped 1 previous similar message [ 6590.439345] Lustre: Skipped 1 previous similar message [ 6595.454651] LustreError: 384906:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6595.456947] LustreError: 384906:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6595.486399] Lustre: server umount lustre-MDT0000 complete [ 6596.845071] LustreError: 382559:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764112439 with bad export cookie 6283895213228669380 [ 6596.848053] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6596.848571] LustreError: 382559:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6596.915458] LustreError: 385108:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6596.918629] LustreError: 385108:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6596.963454] Lustre: server umount lustre-MDT0001 complete [ 6599.894579] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6600.513475] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 18:14:02 (1764112442) [ 6602.931427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6603.096500] LustreError: 385827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6603.105031] LustreError: 385827:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6603.127301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6604.564281] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6607.599447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6607.741809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6609.192466] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6611.086987] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6612.292336] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6614.690188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6616.830780] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6618.736522] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6621.237510] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6621.262799] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6621.321300] Lustre: lustre-OST0001: new disk, initializing [ 6621.323478] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6621.355359] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6621.358286] Lustre: Skipped 1 previous similar message [ 6623.413697] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6625.799484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6625.803435] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6625.838380] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6626.669092] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6630.881402] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6630.881625] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6630.883604] LustreError: Skipped 1 previous similar message [ 6630.887669] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6630.891304] Lustre: Skipped 2 previous similar messages [ 6632.931882] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6632.933818] Lustre: Skipped 1 previous similar message [ 6634.543107] LustreError: 389025:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6634.545349] LustreError: 389025:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6634.565329] Lustre: server umount lustre-OST0001 complete [ 6638.052140] LustreError: 388858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6638.058610] LustreError: 388858:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6643.176657] LustreError: 387423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6643.182747] LustreError: 387423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6647.777035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6647.780181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6647.784686] Lustre: Skipped 1 previous similar message [ 6647.786893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6647.789469] Lustre: Skipped 1 previous similar message [ 6648.292274] LustreError: 387778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6648.298174] LustreError: 387778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6652.718097] LustreError: 389249:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6652.721025] LustreError: 389249:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6652.760674] Lustre: server umount lustre-OST0000 complete [ 6658.529711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6658.530274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6658.533310] Lustre: Skipped 2 previous similar messages [ 6658.534712] Lustre: Skipped 2 previous similar messages [ 6660.313243] Lustre: server umount lustre-MDT0000 complete [ 6661.710132] LustreError: 389150:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764112503 with bad export cookie 6283895213228670262 [ 6661.713093] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6661.714304] LustreError: 389150:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6667.847083] LustreError: 389652:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6667.850046] LustreError: 389652:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6667.910625] Lustre: server umount lustre-MDT0001 complete [ 6671.877338] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 6672.933573] Key type lgssc unregistered [ 6673.054437] LNet: 390184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6673.057731] LNetError: 390184:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6673.066350] LNet: Removed LNI 192.168.201.142@tcp [ 6673.375604] Key type .llcrypt unregistered [ 6673.376933] Key type ._llcrypt unregistered [ 6685.432848] Key type ._llcrypt registered [ 6685.434638] Key type .llcrypt registered [ 6685.484966] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 6693.085466] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 6693.444386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6693.515546] alg: No test for adler32 (adler32-zlib) [ 6694.374401] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 6694.464707] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 6696.056159] Key type lgssc registered [ 6696.491050] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6699.937460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6702.534353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6705.168936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6707.784858] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6710.510135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6710.537547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6710.545160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6711.652475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6711.667292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6711.708394] Lustre: lustre-MDT0000: new disk, initializing [ 6711.737517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6711.747343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6713.220235] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6718.008280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6718.030209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6718.069655] Lustre: 394107:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6718.096664] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6718.099226] Lustre: Skipped 1 previous similar message [ 6718.137236] Lustre: lustre-MDT0001: new disk, initializing [ 6718.193917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6718.212726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6718.218051] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6719.686781] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6723.113142] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6724.451170] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6726.835526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6726.863176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6726.966796] Lustre: lustre-OST0000: new disk, initializing [ 6726.968736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6726.998293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6728.578868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6728.582092] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6728.620112] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6728.999126] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6732.177721] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6733.634862] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6733.722712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6735.137629] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6735.220214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6738.914133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6738.915464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6738.917395] LustreError: Skipped 1 previous similar message [ 6738.917898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6738.921577] Lustre: Skipped 1 previous similar message [ 6744.033909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6744.036246] Lustre: Skipped 2 previous similar messages [ 6749.155624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6749.157800] Lustre: Skipped 1 previous similar message [ 6750.176128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6750.193784] LustreError: 397046:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6750.232895] Lustre: server umount lustre-OST0000 complete [ 6754.273975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6754.274017] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6754.274717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6754.278037] Lustre: Skipped 1 previous similar message [ 6757.565421] LustreError: 397249:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6757.568052] LustreError: 397249:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6757.594787] Lustre: server umount lustre-MDT0000 complete [ 6758.859257] LustreError: 394098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764112601 with bad export cookie 3892342490231890151 [ 6758.862540] LustreError: 394098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6758.862548] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6758.928915] LustreError: 397453:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6758.930917] LustreError: 397453:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6758.989805] Lustre: server umount lustre-MDT0001 complete [ 6761.867149] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 18:16:43 (1764112603) [ 6764.973190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6767.913584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6768.058285] Lustre: lustre-MDT0000: new disk, initializing [ 6768.092124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6769.509578] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6772.624645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6772.756703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6774.110260] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6776.056816] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6777.509284] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6782.434313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6782.435414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6782.439334] Lustre: Skipped 1 previous similar message [ 6782.443931] Lustre: Skipped 2 previous similar messages [ 6784.444051] LustreError: 400060:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6784.446045] LustreError: 400060:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6784.470378] Lustre: server umount lustre-MDT0000 complete [ 6785.728322] LustreError: 400062:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764112627 with bad export cookie 3892342490231891019 [ 6785.732098] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6785.732851] LustreError: 400062:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6785.865880] Lustre: server umount lustre-MDT0001 complete [ 6789.127142] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 18:17:11 (1764112631) [ 6791.435325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6791.592268] LustreError: 400984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6791.600525] LustreError: 400984:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6791.617944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6792.999858] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6795.758141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6797.257947] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6799.157546] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6800.479872] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6802.772815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6804.869168] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6806.736478] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6819.961012] Lustre: Permanently deactivating lustre-OST0000 [ 6819.966862] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6829.554395] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6829.962245] LustreError: 403793:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6829.967650] LustreError: 403793:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 6829.971444] LustreError: 403793:0:(mgs_handler.c:1167:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6830.634960] LustreError: 403841:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 6830.637759] LustreError: 403841:0:(mgs_handler.c:1167:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6837.359102] LustreError: 403938:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6837.361300] LustreError: 403938:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6837.396432] Lustre: server umount lustre-OST0000 complete [ 6838.340518] LustreError: 404091:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 6838.342991] LustreError: 404091:0:(mgs_handler.c:1167:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6841.825758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6841.825768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6841.826474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6841.830496] Lustre: Skipped 1 previous similar message [ 6848.996100] LustreError: 404188:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6848.998678] LustreError: 404188:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6849.025800] Lustre: server umount lustre-MDT0000 complete [ 6850.365314] LustreError: 402600:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764112692 with bad export cookie 3892342490231891558 [ 6850.369254] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6850.370203] LustreError: 402600:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6860.821271] Lustre: server umount lustre-MDT0001 complete [ 6864.238636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6865.852146] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6866.960172] LustreError: 405231:0:(mgs_handler.c:1167:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6869.465937] Lustre: 405569:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.142@tcp [ 6870.240632] LustreError: 405667:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6870.242525] LustreError: 405667:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6870.269362] Lustre: server umount MGS complete [ 6874.483764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6874.638357] LustreError: 406294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6874.643352] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6874.648322] Lustre: Skipped 1 previous similar message [ 6874.668542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6874.671182] Lustre: Skipped 2 previous similar messages [ 6875.973715] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6878.893975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6878.997060] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6880.396486] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6882.371206] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6883.690075] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6884.386637] Lustre: Permanently reactivating lustre-OST0000 [ 6884.392389] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6884.395184] Lustre: Skipped 2 previous similar messages [ 6910.945081] LustreError: 407896:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6917.824568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6918.080552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6918.083210] Lustre: Skipped 1 previous similar message [ 6922.720237] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 6927.556721] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6931.429136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6931.440719] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6931.446126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6931.446134] Lustre: Skipped 3 previous similar messages [ 6933.240100] LustreError: 409561:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6933.246590] LustreError: 409561:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6933.373471] Lustre: server umount lustre-OST0000 complete [ 6940.642373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6940.645607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6940.684686] Lustre: Skipped 1 previous similar message [ 6942.955194] Lustre: server umount lustre-MDT0000 complete [ 6946.795713] LustreError: 407008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6946.803339] LustreError: 407008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6947.065061] LustreError: 406274:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764112789 with bad export cookie 3892342490231892503 [ 6947.070574] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6947.110873] LustreError: 406274:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6947.665588] Lustre: server umount lustre-MDT0001 complete [ 6960.793179] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 6964.022961] Key type lgssc unregistered [ 6964.416976] LNet: 410500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6964.427620] LNetError: 410500:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6964.447551] LNet: Removed LNI 192.168.201.142@tcp [ 6965.208871] Key type .llcrypt unregistered [ 6965.210877] Key type ._llcrypt unregistered [ 6978.020291] Key type ._llcrypt registered [ 6978.021879] Key type .llcrypt registered [ 6978.079144] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 6987.790085] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 6988.342475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6988.420276] alg: No test for adler32 (adler32-zlib) [ 6989.378177] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 6989.524616] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 6991.144217] Key type lgssc registered [ 6991.823502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6996.798712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7000.447675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7004.005731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7007.910706] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7011.758306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7011.798699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7011.807307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7012.949895] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7012.976114] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7013.021761] Lustre: lustre-MDT0000: new disk, initializing [ 7013.078535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7013.092731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7015.315463] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7021.824652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7021.869229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7021.925306] Lustre: 413608:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7021.976923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7021.980238] Lustre: Skipped 1 previous similar message [ 7022.024165] Lustre: lustre-MDT0001: new disk, initializing [ 7022.093817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7022.120416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7022.125884] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7024.423785] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7029.117695] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7031.349346] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7035.233782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7035.292730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7035.456383] Lustre: lustre-OST0000: new disk, initializing [ 7035.461178] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7035.518387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7038.835911] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7043.331512] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7045.647970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7045.652356] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7045.747308] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7045.761380] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7045.895199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7048.092145] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7048.231053] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7050.722690] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7050.727549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7050.728245] LustreError: Skipped 1 previous similar message [ 7050.728836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7050.740412] Lustre: Skipped 1 previous similar message [ 7055.408285] LustreError: 416546:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7055.460049] Lustre: server umount lustre-OST0000 complete [ 7057.890533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7057.893457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7057.899430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7057.902092] Lustre: Skipped 1 previous similar message [ 7060.962937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7060.965891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7060.969988] Lustre: Skipped 1 previous similar message [ 7063.387596] LustreError: 416747:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7063.394704] LustreError: 416747:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7063.467952] Lustre: server umount lustre-MDT0000 complete [ 7065.337909] LustreError: 414390:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764112907 with bad export cookie 8195753119840956618 [ 7065.343050] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7065.344540] LustreError: 414390:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7065.429216] LustreError: 416948:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7065.431414] LustreError: 416948:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7065.515257] Lustre: server umount lustre-MDT0001 complete [ 7069.467527] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7070.298054] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 18:21:52 (1764112912) [ 7073.709788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7074.011918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7075.820549] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7079.395678] LustreError: 417667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7079.406530] LustreError: 417667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7080.088140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7080.279404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7082.609447] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7085.897856] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7088.658230] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7092.852662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7093.070987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7096.005506] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7099.149851] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7100.362238] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7102.398944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7110.627725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7110.630240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7110.634163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7110.641773] Lustre: Skipped 1 previous similar message [ 7112.317101] LustreError: 420106:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7112.322823] LustreError: 420106:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7112.400664] Lustre: server umount lustre-OST0000 complete [ 7115.746061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7115.748929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7115.753829] Lustre: Skipped 2 previous similar messages [ 7121.246673] LustreError: 420307:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7121.251061] LustreError: 420307:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7121.305541] Lustre: server umount lustre-MDT0000 complete [ 7123.338040] LustreError: 420308:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764112965 with bad export cookie 8195753119840957486 [ 7123.341036] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7123.343212] LustreError: 420308:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7123.529320] Lustre: server umount lustre-MDT0001 complete [ 7128.717421] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 7130.051725] Key type lgssc unregistered [ 7130.197472] LNet: 421041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7130.202370] LNetError: 421041:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7130.214533] LNet: Removed LNI 192.168.201.142@tcp [ 7130.615463] Key type .llcrypt unregistered [ 7130.617904] Key type ._llcrypt unregistered [ 7139.452079] Key type ._llcrypt registered [ 7139.453367] Key type .llcrypt registered [ 7139.490777] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 7139.603371] Key type .llcrypt unregistered [ 7139.604674] Key type ._llcrypt unregistered [ 7146.314984] Key type ._llcrypt registered [ 7146.316072] Key type .llcrypt registered [ 7146.351428] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 7146.692579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7146.733304] alg: No test for adler32 (adler32-zlib) [ 7147.577545] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 7147.658628] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 7149.240157] Key type lgssc registered [ 7149.608092] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7152.261649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7152.267386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7153.414852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7154.760635] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7157.646649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7157.758340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7159.646252] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7161.426914] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7162.849119] LustreError: 423208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7162.854372] LustreError: 423208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7163.708031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7163.829897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7164.841739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7165.092710] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7168.993193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7168.996578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7169.001590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7170.019659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7170.019913] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7170.019927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7170.022374] Lustre: Skipped 1 previous similar message [ 7173.999437] LustreError: 425243:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 7174.033966] Lustre: server umount lustre-OST0000 complete [ 7178.722339] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7178.724676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7178.730611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7181.437084] LustreError: 425445:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7181.438850] LustreError: 425445:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7181.468208] Lustre: server umount lustre-MDT0000 complete [ 7182.686116] LustreError: 423170:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764113024 with bad export cookie 3105711386977002230 [ 7182.688860] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7182.689366] LustreError: 423170:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7182.763603] LustreError: 425647:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7182.766786] LustreError: 425647:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7182.819738] Lustre: server umount lustre-MDT0001 complete [ 7186.107072] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 7187.174963] Key type lgssc unregistered [ 7187.332414] LNet: 426178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7187.335647] LNetError: 426178:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7187.344384] LNet: Removed LNI 192.168.201.142@tcp [ 7187.660657] Key type .llcrypt unregistered [ 7187.662544] Key type ._llcrypt unregistered [ 7196.714842] Key type ._llcrypt registered [ 7196.715949] Key type .llcrypt registered [ 7196.750818] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 7197.046431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7197.077067] alg: No test for adler32 (adler32-zlib) [ 7197.921683] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 7197.998478] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 7199.576109] Key type lgssc registered [ 7199.904718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7202.410333] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7202.417784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7217.824359] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7222.944168] LustreError: 427525:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.142@tcp: failed processing log, type 4: rc = -110 [ 7249.568171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7251.481928] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7253.347363] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7255.685892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7255.858375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7255.882634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7257.242782] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7260.350109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7260.486414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7261.930255] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7263.907707] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7265.054676] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7269.859843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7269.860750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7269.863608] Lustre: Skipped 1 previous similar message [ 7269.867391] Lustre: Skipped 1 previous similar message [ 7273.519213] LustreError: 429807:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 7273.565323] Lustre: server umount lustre-OST0000 complete [ 7274.977274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7274.978034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7274.981415] Lustre: Skipped 1 previous similar message [ 7280.097153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7280.099318] Lustre: Skipped 2 previous similar messages [ 7281.024053] LustreError: 430010:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 7281.027119] LustreError: 430010:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7281.054079] Lustre: server umount lustre-MDT0000 complete [ 7282.339361] LustreError: 430011:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764113124 with bad export cookie 1070940508975750477 [ 7282.342894] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7282.346180] LustreError: 430011:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7282.415084] LustreError: 430214:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7282.418541] LustreError: 430214:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7282.474769] Lustre: server umount lustre-MDT0001 complete [ 7286.228347] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 7287.305676] Key type lgssc unregistered [ 7287.428471] LNet: 430745:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7287.432407] LNetError: 430745:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7287.442378] LNet: Removed LNI 192.168.201.142@tcp [ 7287.742322] Key type .llcrypt unregistered [ 7287.744118] Key type ._llcrypt unregistered [ 7297.154436] Key type ._llcrypt registered [ 7297.155455] Key type .llcrypt registered [ 7297.190055] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 7297.511206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7297.576776] alg: No test for adler32 (adler32-zlib) [ 7298.435757] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 7298.518429] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 7300.104131] Key type lgssc registered [ 7300.460481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7303.134309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7303.142732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7304.316456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7305.609989] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7308.517666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7308.647859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7310.021602] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7311.842160] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7313.206647] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7315.493342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7315.604702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7317.600798] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7319.587459] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7329.955948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7334.369238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7334.372517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7334.377758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7335.276943] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.42@tcp (stopping) [ 7335.395955] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7335.402346] Lustre: Skipped 1 previous similar message [ 7337.216606] LustreError: 434382:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7337.250226] Lustre: server umount lustre-MDT0000 complete [ 7338.107609] LustreError: 432099:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.42@tcp arrived at 1764113180 with bad export cookie 17400674652668854567 [ 7338.114579] LustreError: 432099:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7338.806019] LustreError: 432099:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764113181 with bad export cookie 17400674652668853916 [ 7338.809676] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7338.809868] LustreError: 432099:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7339.488865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7339.492386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7339.496532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7339.498782] Lustre: Skipped 3 previous similar messages [ 7340.512943] LustreError: 432119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7340.517791] LustreError: 432119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7343.520925] LustreError: 432119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7343.521131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7343.528467] LustreError: 432119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7343.530401] Lustre: Skipped 1 previous similar message [ 7344.879098] LustreError: 434586:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7344.881434] LustreError: 434586:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7344.917074] Lustre: server umount lustre-OST0000 complete [ 7347.318597] LustreError: 434940:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7347.320548] LustreError: 434940:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7347.387120] Lustre: server umount lustre-MDT0001 complete [ 7351.193082] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 7352.310757] Key type lgssc unregistered [ 7352.448451] LNet: 435471:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7352.451601] LNetError: 435471:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7352.459316] LNet: Removed LNI 192.168.201.142@tcp [ 7352.792426] Key type .llcrypt unregistered [ 7352.793484] Key type ._llcrypt unregistered [ 7363.022264] Key type ._llcrypt registered [ 7363.024039] Key type .llcrypt registered [ 7363.095133] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 7363.657530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7363.692770] alg: No test for adler32 (adler32-zlib) [ 7364.745620] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 7364.907076] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 7366.528197] Key type lgssc registered [ 7367.435251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7371.534415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7371.543645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7372.790172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7374.794547] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7379.837807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7380.028467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7382.069590] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7384.117172] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7385.334700] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7387.518714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7387.675224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7389.609388] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7391.455572] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7396.908991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7406.960329] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.42@tcp (stopping) [ 7407.077638] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7412.070120] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.42@tcp (stopping) [ 7412.074227] Lustre: Skipped 2 previous similar messages [ 7417.190312] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.42@tcp (stopping) [ 7417.192369] Lustre: Skipped 2 previous similar messages [ 7417.312112] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7417.351618] LustreError: 439113:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7417.373877] Lustre: server umount lustre-MDT0001 complete [ 7422.435189] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7422.435213] LustreError: 436847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7422.438084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7422.444903] LustreError: 436847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7422.449419] Lustre: Skipped 1 previous similar message [ 7422.450142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7422.455886] Lustre: Skipped 2 previous similar messages [ 7425.198138] LustreError: 439319:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7425.201293] LustreError: 439319:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7425.241148] Lustre: server umount lustre-OST0000 complete [ 7426.628107] LustreError: 439520:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7426.630247] LustreError: 439520:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7426.694139] Lustre: server umount lustre-MDT0000 complete [ 7431.633837] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 7432.753892] Key type lgssc unregistered [ 7432.882549] LNet: 440203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7432.885939] LNetError: 440203:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7432.895502] LNet: Removed LNI 192.168.201.142@tcp [ 7433.224061] Key type .llcrypt unregistered [ 7433.225135] Key type ._llcrypt unregistered [ 7441.827477] Key type ._llcrypt registered [ 7441.829264] Key type .llcrypt registered [ 7441.868229] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 7448.072161] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 7448.397499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7448.411060] alg: No test for adler32 (adler32-zlib) [ 7449.288902] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 7449.381866] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 7450.968175] Key type lgssc registered [ 7451.402501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7454.271380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7456.477195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7458.526289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7460.618973] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7463.772487] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 7467.027050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7469.323321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7469.347735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7469.352747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7470.438953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7470.449166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7470.486629] Lustre: lustre-MDT0000: new disk, initializing [ 7470.509784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7470.516155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7471.818877] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7475.407851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7477.840779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7477.865820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7477.961571] Lustre: lustre-OST0000: new disk, initializing [ 7477.963513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7477.965180] Lustre: Skipped 1 previous similar message [ 7477.987112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7478.342058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7478.345996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7478.359685] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7480.053845] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7483.233977] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7487.673168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7490.049424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7490.077555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7490.114890] Lustre: 444149:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7490.141057] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7490.178136] Lustre: lustre-MDT0001: new disk, initializing [ 7490.225833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7490.243233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 7490.247290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 7491.603062] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7495.280070] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7495.362168] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7496.768504] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7498.884593] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 2 sec [ 7501.164880] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.42@tcp (stopping) [ 7503.329871] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7503.332692] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7503.334347] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7503.337515] Lustre: Skipped 1 previous similar message [ 7506.278490] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.42@tcp (stopping) [ 7506.281837] Lustre: Skipped 1 previous similar message [ 7506.691052] LustreError: 447583:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7506.719261] Lustre: server umount lustre-MDT0001 complete [ 7513.572230] LustreError: 444154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7513.579745] LustreError: 444154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7518.691128] LustreError: 444154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7518.696212] LustreError: 444154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7519.200828] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7519.203112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7519.207321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7523.809297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7523.809927] LustreError: 446387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7525.232110] LustreError: 447855:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 7525.233906] LustreError: 447855:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7525.272122] Lustre: server umount lustre-OST0000 complete [ 7532.735659] LustreError: 448056:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 7532.738103] LustreError: 448056:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7532.800352] Lustre: server umount lustre-MDT0000 complete [ 7537.608342] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 7538.702685] Key type lgssc unregistered [ 7538.836386] LNet: 448741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7538.838868] LNetError: 448741:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7538.846287] LNet: Removed LNI 192.168.201.142@tcp [ 7539.169531] Key type .llcrypt unregistered [ 7539.171040] Key type ._llcrypt unregistered [ 7552.731389] Key type ._llcrypt registered [ 7552.732628] Key type .llcrypt registered [ 7552.772116] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 7552.881229] Key type .llcrypt unregistered [ 7552.882247] Key type ._llcrypt unregistered [ 7559.798367] Key type ._llcrypt registered [ 7559.799704] Key type .llcrypt registered [ 7559.839726] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 7560.241452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7560.277370] alg: No test for adler32 (adler32-zlib) [ 7561.160540] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 7561.257763] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 7562.840225] Key type lgssc registered [ 7563.241774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7565.942589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7568.180520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7570.277692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7572.630270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7572.651257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7572.655815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7573.747332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7573.760899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7573.806421] Lustre: lustre-MDT0000: new disk, initializing [ 7573.838295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7573.845561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7575.193195] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7579.401334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7579.424894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7579.454812] Lustre: 452059:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7579.478317] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7579.480578] Lustre: Skipped 1 previous similar message [ 7579.515727] Lustre: lustre-MDT0001: new disk, initializing [ 7579.548390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7579.560727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7579.563831] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7580.974502] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7584.101513] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7585.358388] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7587.658906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7587.682721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7587.783516] Lustre: lustre-OST0000: new disk, initializing [ 7587.785289] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7587.815753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7589.795416] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7590.589643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7590.592701] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7590.623426] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7592.983099] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7596.003080] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7596.003871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7596.006489] Lustre: Skipped 2 previous similar messages [ 7596.009349] Lustre: Skipped 2 previous similar messages [ 7601.121626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7601.125044] Lustre: Skipped 2 previous similar messages [ 7601.406668] LustreError: 454499:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7601.440019] Lustre: server umount lustre-MDT0000 complete [ 7602.722806] LustreError: 452051:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764113444 with bad export cookie 10074542803817916901 [ 7602.728939] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7602.734107] LustreError: 452051:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7602.806374] LustreError: 454701:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7602.808103] LustreError: 454701:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7602.862902] Lustre: server umount lustre-MDT0001 complete [ 7614.387648] LustreError: 454901:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7614.389760] LustreError: 454901:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7614.428958] Lustre: server umount lustre-OST0000 complete [ 7617.930325] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 18:30:59 (1764113459) [ 7618.285070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7620.531702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7620.740509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7621.989575] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7624.837994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7624.954461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7626.266315] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7628.125296] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7629.382594] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7631.678456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7631.704185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7631.768932] Lustre: Found index 0 for lustre-OST0000, updating log [ 7633.722659] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7635.474985] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7645.160217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7650.279855] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7650.280490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7650.287769] LustreError: Skipped 1 previous similar message [ 7650.296707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7650.301850] Lustre: Skipped 1 previous similar message [ 7653.422288] LustreError: 458007:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7653.424805] LustreError: 458007:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7653.462161] Lustre: server umount lustre-OST0000 complete [ 7654.412617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7655.904966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7655.909294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7655.916139] Lustre: Skipped 1 previous similar message [ 7655.919163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7655.922340] Lustre: Skipped 2 previous similar messages [ 7661.189202] LustreError: 458261:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7661.191218] LustreError: 458261:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7661.222288] Lustre: server umount lustre-MDT0000 complete [ 7662.557468] LustreError: 457343:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764113504 with bad export cookie 10074542803817918280 [ 7662.560360] LustreError: 457343:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7662.561065] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7662.694391] Lustre: server umount lustre-MDT0001 complete [ 7665.715242] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 18:31:47 (1764113507) [ 7666.075087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7666.444017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7666.792151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7667.127425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7667.476026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7667.872643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7668.230672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7668.595133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7668.978687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7669.325990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7669.664035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7670.021893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7670.366595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7670.705447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7671.058720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7671.425605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7671.770839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7672.156452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7672.525223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7672.875743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7673.247657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7673.643231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7673.993343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7674.354553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7674.731537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7675.084312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7675.417721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7675.757639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7676.105266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7676.437296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7676.765516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7677.094522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7677.416263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7677.755078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7678.081872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7678.417439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7678.741600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7679.082300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7679.419351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7679.751318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7680.088610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7680.418624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7680.764244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7681.102671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7681.467647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7681.821219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7682.187411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7682.513075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7682.854495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7683.207611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7683.563039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7686.274627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7686.434770] LustreError: 461846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7686.440648] LustreError: 461846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7686.459591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7686.462354] Lustre: Skipped 1 previous similar message [ 7687.856688] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7690.947621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7692.434909] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7694.373065] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7695.687693] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7697.441176] LNet: 1 local NIs in recovery (showing 1): 192.168.201.142@tcp [ 7698.206172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7698.248247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7698.351156] Lustre: Found index 0 for lustre-OST0000, updating log [ 7700.511049] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7702.354873] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7706.595669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7706.595791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7706.600171] Lustre: Skipped 1 previous similar message [ 7706.601811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7706.603645] Lustre: Skipped 4 previous similar messages [ 7709.228426] LustreError: 464161:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7709.231279] LustreError: 464161:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7709.270343] Lustre: server umount lustre-OST0000 complete [ 7711.585877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7711.590697] Lustre: Skipped 1 previous similar message [ 7714.657386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7714.659204] Lustre: Skipped 7 previous similar messages [ 7716.693713] Lustre: server umount lustre-MDT0000 complete [ 7716.705618] LustreError: 461845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7717.729436] LustreError: 461845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7717.734241] LustreError: 461845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7717.919143] LustreError: 463522:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764113560 with bad export cookie 10074542803817919351 [ 7717.922182] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7717.923732] LustreError: 463522:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7718.043723] Lustre: server umount lustre-MDT0001 complete [ 7719.067685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7724.363258] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 7727.910426] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 7731.648497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7733.903553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7736.010227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7738.349183] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7740.885554] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 18:33:02 (1764113582) [ 7756.832640] LNet: 1 local NIs in recovery (showing 1): 192.168.201.142@tcp [ 7768.101414] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 7769.625785] Key type lgssc unregistered [ 7769.834480] LNet: 468147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7769.839740] LNetError: 468147:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7898.855532] LNet: Removed LNI 192.168.201.142@tcp [ 7899.204483] Key type .llcrypt unregistered [ 7899.206382] Key type ._llcrypt unregistered [ 7906.188629] Key type ._llcrypt registered [ 7906.189604] Key type .llcrypt registered [ 7906.228242] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 7906.602515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7906.656031] alg: No test for adler32 (adler32-zlib) [ 7907.511503] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 7907.605511] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 7909.192215] Key type lgssc registered [ 7909.708852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7911.749450] LNet: Added LNI 192.168.201.142@tcp100 [8/256/0/180] [ 7912.169725] LNet: Added LNI 192.168.201.142@tcp101 [8/256/0/180] [ 7912.565832] LNet: Added LNI 192.168.201.142@tcp102 [8/256/0/180] [ 7912.997302] LNet: Added LNI 192.168.201.142@tcp103 [8/256/0/180] [ 7913.512586] LNet: Added LNI 192.168.201.142@tcp104 [8/256/0/180] [ 7913.943246] LNet: Added LNI 192.168.201.142@tcp105 [8/256/0/180] [ 7914.374868] LNet: Added LNI 192.168.201.142@tcp106 [8/256/0/180] [ 7914.762976] LNet: Added LNI 192.168.201.142@tcp107 [8/256/0/180] [ 7915.136337] LNet: Added LNI 192.168.201.142@tcp108 [8/256/0/180] [ 7915.522398] LNet: Added LNI 192.168.201.142@tcp109 [8/256/0/180] [ 7915.904990] LNet: Added LNI 192.168.201.142@tcp110 [8/256/0/180] [ 7916.249308] LNet: Added LNI 192.168.201.142@tcp111 [8/256/0/180] [ 7916.582400] LNet: Added LNI 192.168.201.142@tcp112 [8/256/0/180] [ 7916.903645] LNet: Added LNI 192.168.201.142@tcp113 [8/256/0/180] [ 7917.231236] LNet: Added LNI 192.168.201.142@tcp114 [8/256/0/180] [ 7917.537467] LNet: Added LNI 192.168.201.142@tcp115 [8/256/0/180] [ 7917.841648] LNet: Added LNI 192.168.201.142@tcp116 [8/256/0/180] [ 7918.155448] LNet: Added LNI 192.168.201.142@tcp117 [8/256/0/180] [ 7918.492734] LNet: Added LNI 192.168.201.142@tcp118 [8/256/0/180] [ 7918.846168] LNet: Added LNI 192.168.201.142@tcp119 [8/256/0/180] [ 7919.186946] LNet: Added LNI 192.168.201.142@tcp120 [8/256/0/180] [ 7919.517762] LNet: Added LNI 192.168.201.142@tcp121 [8/256/0/180] [ 7919.856988] LNet: Added LNI 192.168.201.142@tcp122 [8/256/0/180] [ 7920.227803] LNet: Added LNI 192.168.201.142@tcp123 [8/256/0/180] [ 7920.605799] LNet: Added LNI 192.168.201.142@tcp124 [8/256/0/180] [ 7920.997918] LNet: Added LNI 192.168.201.142@tcp125 [8/256/0/180] [ 7921.381582] LNet: Added LNI 192.168.201.142@tcp126 [8/256/0/180] [ 7921.776482] LNet: Added LNI 192.168.201.142@tcp127 [8/256/0/180] [ 7922.170063] LNet: Added LNI 192.168.201.142@tcp128 [8/256/0/180] [ 7922.559617] LNet: Added LNI 192.168.201.142@tcp129 [8/256/0/180] [ 7922.970554] LNet: Added LNI 192.168.201.142@tcp130 [8/256/0/180] [ 7923.373551] LNet: Added LNI 192.168.201.142@tcp131 [8/256/0/180] [ 7923.743501] LNet: Added LNI 192.168.201.142@tcp132 [8/256/0/180] [ 7924.164142] LNet: Added LNI 192.168.201.142@tcp133 [8/256/0/180] [ 7924.645562] LNet: Added LNI 192.168.201.142@tcp134 [8/256/0/180] [ 7925.107336] LNet: Added LNI 192.168.201.142@tcp135 [8/256/0/180] [ 7928.791745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7928.820547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7928.830892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7929.892277] Lustre: 471136:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.142@tcp: can't add NID 192.168.201.142@tcp130: rc = -75 [ 7929.968708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7929.988996] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7930.032218] Lustre: lustre-MDT0000: new disk, initializing [ 7930.080253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7930.090371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7931.926068] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7937.675982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7937.716864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7937.790993] Lustre: 471153:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7937.823057] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7937.827142] Lustre: Skipped 1 previous similar message [ 7937.880330] Lustre: lustre-MDT0001: new disk, initializing [ 7937.909875] Lustre: 471939:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.142@tcp: can't add NID 192.168.201.142@tcp130: rc = -75 [ 7937.914830] Lustre: 471939:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7937.989806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7938.024296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7938.030522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7940.113455] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7944.449863] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7946.214048] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7949.006617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7949.037149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7949.162487] Lustre: lustre-OST0000: new disk, initializing [ 7949.165112] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7949.212690] Lustre: 472911:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.142@tcp: can't add NID 192.168.201.142@tcp130: rc = -75 [ 7949.216225] Lustre: 472911:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [ 7949.242388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7951.510534] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 7955.510104] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7958.543606] Lustre: 473708:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.142@tcp: can't add NID 192.168.201.142@tcp130: rc = -75 [ 7958.550465] Lustre: 473708:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7958.582164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7958.587788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7958.641727] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7963.624374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7963.624783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7963.630499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7963.635207] Lustre: Skipped 1 previous similar message [ 7964.915142] LustreError: 473745:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7964.968706] Lustre: server umount lustre-OST0000 complete [ 7968.737611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7968.737789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7968.743946] Lustre: Skipped 1 previous similar message [ 7968.745765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7968.747817] Lustre: Skipped 1 previous similar message [ 7972.414162] LustreError: 473946:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7972.416031] LustreError: 473946:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7972.444489] Lustre: server umount lustre-MDT0000 complete [ 7973.738709] LustreError: 471145:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764113815 with bad export cookie 13319911518603808962 [ 7973.741108] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7973.742782] LustreError: 471145:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7973.814591] LustreError: 474148:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7973.817473] LustreError: 474148:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7973.880934] Lustre: server umount lustre-MDT0001 complete [ 7977.401484] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 7978.397232] Key type lgssc unregistered [ 7978.564793] LNet: 474687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7978.567821] LNetError: 474687:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7978.569563] LNet: Removed LNI 192.168.201.142@tcp [ 7979.616117] LNet: Removed LNI 192.168.201.142@tcp100 [ 7979.617447] LNetError: 474687:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7979.619470] LNetError: 474687:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7979.622212] LNet: Removed LNI 192.168.201.142@tcp101 [ 7980.640120] LNet: Removed LNI 192.168.201.142@tcp102 [ 7980.641567] LNetError: 474687:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7980.643742] LNetError: 474687:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7980.645744] LNet: Removed LNI 192.168.201.142@tcp103 [ 7981.664108] LNet: Removed LNI 192.168.201.142@tcp104 [ 7981.665396] LNet: Removed LNI 192.168.201.142@tcp105 [ 7981.666501] LNet: Removed LNI 192.168.201.142@tcp106 [ 7981.667689] LNet: Removed LNI 192.168.201.142@tcp107 [ 7981.668934] LNet: Removed LNI 192.168.201.142@tcp108 [ 7981.670068] LNet: Removed LNI 192.168.201.142@tcp109 [ 7981.671124] LNet: Removed LNI 192.168.201.142@tcp110 [ 7981.672667] LNet: Removed LNI 192.168.201.142@tcp111 [ 7981.673650] LNet: Removed LNI 192.168.201.142@tcp112 [ 7981.675278] LNet: Removed LNI 192.168.201.142@tcp113 [ 7981.676696] LNet: Removed LNI 192.168.201.142@tcp114 [ 7981.678275] LNet: Removed LNI 192.168.201.142@tcp115 [ 7981.680123] LNet: Removed LNI 192.168.201.142@tcp116 [ 7981.681425] LNet: Removed LNI 192.168.201.142@tcp117 [ 7981.682456] LNet: Removed LNI 192.168.201.142@tcp118 [ 7981.683428] LNet: Removed LNI 192.168.201.142@tcp119 [ 7981.684737] LNet: Removed LNI 192.168.201.142@tcp120 [ 7981.686792] LNet: Removed LNI 192.168.201.142@tcp121 [ 7981.688393] LNet: Removed LNI 192.168.201.142@tcp122 [ 7981.689373] LNet: Removed LNI 192.168.201.142@tcp123 [ 7981.690362] LNet: Removed LNI 192.168.201.142@tcp124 [ 7981.691314] LNet: Removed LNI 192.168.201.142@tcp125 [ 7981.692263] LNet: Removed LNI 192.168.201.142@tcp126 [ 7981.693186] LNet: Removed LNI 192.168.201.142@tcp127 [ 7981.694228] LNet: Removed LNI 192.168.201.142@tcp128 [ 7981.695369] LNet: Removed LNI 192.168.201.142@tcp129 [ 7981.696562] LNet: Removed LNI 192.168.201.142@tcp130 [ 7981.697549] LNet: Removed LNI 192.168.201.142@tcp131 [ 7981.698515] LNet: Removed LNI 192.168.201.142@tcp132 [ 7981.699590] LNet: Removed LNI 192.168.201.142@tcp133 [ 7981.700603] LNet: Removed LNI 192.168.201.142@tcp134 [ 7981.707339] LNet: Removed LNI 192.168.201.142@tcp135 [ 7981.961820] Key type .llcrypt unregistered [ 7981.962702] Key type ._llcrypt unregistered [ 7989.822927] Key type ._llcrypt registered [ 7989.824202] Key type .llcrypt registered [ 7989.866751] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 7996.247876] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 7996.728257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7996.742951] alg: No test for adler32 (adler32-zlib) [ 7997.599990] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 7997.691459] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 7999.280174] Key type lgssc registered [ 7999.639797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8002.401785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8004.467680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8006.587308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8008.806110] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8011.174728] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 18:37:33 (1764113853) [ 8017.028938] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 8018.061359] Key type lgssc unregistered [ 8018.188404] LNet: 478698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8018.190538] LNetError: 478698:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8018.197289] LNet: Removed LNI 192.168.201.142@tcp [ 8018.499705] Key type .llcrypt unregistered [ 8018.500709] Key type ._llcrypt unregistered [ 8025.188460] Key type ._llcrypt registered [ 8025.189684] Key type .llcrypt registered [ 8025.233119] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 8025.785763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8025.796652] alg: No test for adler32 (adler32-zlib) [ 8026.659201] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 8026.756808] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 8028.344172] Key type lgssc registered [ 8028.795380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8032.145927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8032.173661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8032.180459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8033.289943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8033.306721] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8033.364883] Lustre: lustre-MDT0000: new disk, initializing [ 8033.395628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8033.405095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8034.755309] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 8039.599683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8039.624471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8039.655192] Lustre: 479866:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8039.682975] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8039.684424] Lustre: Skipped 1 previous similar message [ 8039.722320] Lustre: lustre-MDT0001: new disk, initializing [ 8039.759570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8039.774835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8039.778027] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8041.164346] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 8044.333898] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8045.700692] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8048.219272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8048.249693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8048.347866] Lustre: lustre-OST0000: new disk, initializing [ 8048.349640] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8048.380882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8050.343170] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 8053.563517] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8054.271735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8054.276572] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8054.319846] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8054.863803] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing lnet_if_list [ 8056.635203] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 8056.646710] LNet: Added LNI 192.168.201.142@tcp10 [8/256/0/180] [ 8058.636839] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 8058.650357] LNet: Added LNI 192.168.201.142@tcp11 [8/256/0/180] [ 8060.110411] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 8060.122665] LNet: Added LNI 192.168.201.142@tcp12 [8/256/0/180] [ 8061.682986] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 8061.698455] LNet: Added LNI 192.168.201.142@tcp13 [8/256/0/180] [ 8063.203928] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 8063.217509] LNet: Added LNI 192.168.201.142@tcp14 [8/256/0/180] [ 8064.837557] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 8064.850872] LNet: Added LNI 192.168.201.142@tcp15 [8/256/0/180] [ 8066.457298] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 8066.471610] LNet: Added LNI 192.168.201.142@tcp16 [8/256/0/180] [ 8068.103733] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 8068.118673] LNet: Added LNI 192.168.201.142@tcp17 [8/256/0/180] [ 8069.695070] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 8069.706635] LNet: Added LNI 192.168.201.142@tcp18 [8/256/0/180] [ 8071.370331] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 8071.382803] LNet: Added LNI 192.168.201.142@tcp19 [8/256/0/180] [ 8073.004667] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 8073.016656] LNet: Added LNI 192.168.201.142@tcp20 [8/256/0/180] [ 8074.608310] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 8074.618986] LNet: Added LNI 192.168.201.142@tcp21 [8/256/0/180] [ 8076.388240] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 8076.401029] LNet: Added LNI 192.168.201.142@tcp22 [8/256/0/180] [ 8078.139964] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8078.151705] LNet: Added LNI 192.168.201.142@tcp23 [8/256/0/180] [ 8079.818205] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8079.828363] LNet: Added LNI 192.168.201.142@tcp24 [8/256/0/180] [ 8081.658888] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8081.671471] LNet: Added LNI 192.168.201.142@tcp25 [8/256/0/180] [ 8083.554897] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8083.565570] LNet: Added LNI 192.168.201.142@tcp26 [8/256/0/180] [ 8085.443684] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8085.455494] LNet: Added LNI 192.168.201.142@tcp27 [8/256/0/180] [ 8085.461448] LNetError: 479131:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.142 [ 8085.464054] LNetError: Protocol error connecting to 192.168.201.142@tcp26 at host 192.168.201.142:988: is it running a compatible version of Lustre? [ 8087.588439] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8087.600134] LNet: Added LNI 192.168.201.142@tcp28 [8/256/0/180] [ 8087.605770] LNetError: 479130:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.142 [ 8087.609224] LNetError: Protocol error connecting to 192.168.201.142@tcp26 at host 192.168.201.142:988: is it running a compatible version of Lustre? [ 8087.612671] LNetError: Skipped 1 previous similar message [ 8089.695051] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8089.710247] LNet: Added LNI 192.168.201.142@tcp29 [8/256/0/180] [ 8089.715446] LNetError: 479131:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.142 [ 8089.720144] LNetError: 479131:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [ 8089.722747] LNetError: Protocol error connecting to 192.168.201.142@tcp27 at host 192.168.201.142:988: is it running a compatible version of Lustre? [ 8091.958412] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8091.971247] LNet: Added LNI 192.168.201.142@tcp30 [8/256/0/180] [ 8091.976853] LNetError: 479128:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.142 [ 8091.980948] LNetError: Protocol error connecting to 192.168.201.142@tcp29 at host 192.168.201.142:988: is it running a compatible version of Lustre? [ 8092.320292] LNet: 1 local NIs in recovery (showing 1): 192.168.201.142@tcp29 [ 8094.119393] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8094.130363] LNet: Added LNI 192.168.201.142@tcp31 [8/256/0/180] [ 8096.146417] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8096.158088] LNet: Added LNI 192.168.201.142@tcp32 [8/256/0/180] [ 8096.165109] LNetError: 479129:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.142 [ 8096.168868] LNetError: 479129:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 4 previous similar messages [ 8096.171152] LNetError: Protocol error connecting to 192.168.201.142@tcp31 at host 192.168.201.142:988: is it running a compatible version of Lustre? [ 8096.173506] LNetError: Skipped 4 previous similar messages [ 8098.425294] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8098.436360] LNet: Added LNI 192.168.201.142@tcp33 [8/256/0/180] [ 8100.767330] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8100.779548] LNet: Added LNI 192.168.201.142@tcp34 [8/256/0/180] [ 8103.065264] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8103.077544] LNet: Added LNI 192.168.201.142@tcp35 [8/256/0/180] [ 8104.609405] LNetError: 479131:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.142 [ 8104.613316] LNetError: 479131:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 8 previous similar messages [ 8104.615727] LNetError: Protocol error connecting to 192.168.201.142@tcp33 at host 192.168.201.142:988: is it running a compatible version of Lustre? [ 8104.619111] LNetError: Skipped 8 previous similar messages [ 8105.325167] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8105.336472] LNet: Added LNI 192.168.201.142@tcp36 [8/256/0/180] [ 8107.599319] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8107.611025] LNet: Added LNI 192.168.201.142@tcp37 [8/256/0/180] [ 8109.811775] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8109.827253] LNet: Added LNI 192.168.201.142@tcp38 [8/256/0/180] [ 8110.058866] LNetError: 479138:0:(peer.c:1877:lnet_peer_add_nid()) Peer 192.168.201.142@tcp NID 192.168.201.142@tcp is already locked with peer 192.168.201.142@tcp [ 8110.064541] LNetError: 479138:0:(lib-move.c:2321:lnet_handle_find_routed_path()) peer 192.168.201.142@tcp has no available nets [ 8112.145671] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8112.160771] LNet: Added LNI 192.168.201.142@tcp39 [8/256/0/180] [ 8112.162330] LNetError: 479138:0:(lib-move.c:2321:lnet_handle_find_routed_path()) peer 192.168.201.142@tcp has no available nets [ 8114.385937] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8114.397538] LNet: Added LNI 192.168.201.142@tcp40 [8/256/0/180] [ 8114.399341] LNetError: 479138:0:(lib-move.c:2321:lnet_handle_find_routed_path()) peer 192.168.201.142@tcp has no available nets [ 8114.402224] LNetError: 479138:0:(lib-move.c:2321:lnet_handle_find_routed_path()) Skipped 1 previous similar message [ 8116.648708] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 8116.658773] LNet: Added LNI 192.168.201.142@tcp41 [8/256/0/180] [ 8116.660322] LNetError: 479138:0:(lib-move.c:2321:lnet_handle_find_routed_path()) peer 192.168.201.142@tcp has no available nets [ 8116.663222] LNetError: 479138:0:(lib-move.c:2321:lnet_handle_find_routed_path()) Skipped 1 previous similar message [ 8118.932250] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8118.941833] LNet: Added LNI 192.168.201.142@tcp42 [8/256/0/180] [ 8121.219194] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 8121.230939] LNet: Added LNI 192.168.201.142@tcp43 [8/256/0/180] [ 8122.020100] LNetError: 479131:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.142 [ 8122.023478] LNetError: 479131:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 33 previous similar messages [ 8122.025862] LNetError: Protocol error connecting to 192.168.201.142@tcp39 at host 192.168.201.142:988: is it running a compatible version of Lustre? [ 8122.028758] LNetError: Skipped 33 previous similar messages [ 8123.607677] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 8123.618746] LNet: Added LNI 192.168.201.142@tcp44 [8/256/0/180] [ 8126.230900] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 8126.242094] LNet: Added LNI 192.168.201.142@tcp45 [8/256/0/180] [ 8128.924846] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 8128.935324] LNet: Added LNI 192.168.201.142@tcp46 [8/256/0/180] [ 8131.395057] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 8131.405030] LNet: Added LNI 192.168.201.142@tcp47 [8/256/0/180] [ 8134.056502] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 8134.068909] LNet: Added LNI 192.168.201.142@tcp48 [8/256/0/180] [ 8136.585281] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 8136.596755] LNet: Added LNI 192.168.201.142@tcp49 [8/256/0/180] [ 8139.296193] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 8139.308701] LNet: Added LNI 192.168.201.142@tcp50 [8/256/0/180] [ 8151.525825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8151.525923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8151.532673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8151.535117] LustreError: Skipped 1 previous similar message [ 8151.540255] Lustre: Skipped 1 previous similar message [ 8152.736368] LNet: 7 local NIs in recovery (showing 7): 192.168.201.142@tcp46, 192.168.201.142@tcp45, 192.168.201.142@tcp47, 192.168.201.142@tcp44, 192.168.201.142@tcp43, 192.168.201.142@tcp35, 192.168.201.142@tcp42 [ 8152.948202] LustreError: 504052:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8153.009507] Lustre: server umount lustre-OST0000 complete [ 8154.787869] LNetError: 479131:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.142 [ 8154.793121] LNetError: 479131:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 43 previous similar messages [ 8154.797300] LNetError: Protocol error connecting to 192.168.201.142@tcp42 at host 192.168.201.142:988: is it running a compatible version of Lustre? [ 8154.802125] LNetError: Skipped 43 previous similar messages [ 8156.642614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8156.643522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8156.647180] Lustre: Skipped 2 previous similar messages [ 8160.578333] LustreError: 504253:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8160.580371] LustreError: 504253:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8160.617344] Lustre: server umount lustre-MDT0000 complete [ 8161.916230] LustreError: 479861:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764114004 with bad export cookie 5516523209410035124 [ 8161.920914] LustreError: 479861:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8161.921339] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8162.017626] LustreError: 504455:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8162.019771] LustreError: 504455:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8162.092290] Lustre: server umount lustre-MDT0001 complete [ 8166.053860] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 8167.237086] Key type lgssc unregistered [ 8167.404752] LNet: 504995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8167.408374] LNetError: 504995:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8167.410701] LNet: Removed LNI 192.168.201.142@tcp [ 8168.416116] LNet: Removed LNI 192.168.201.142@tcp10 [ 8168.417645] LNetError: 504995:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8168.420194] LNetError: 504995:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8168.422770] LNet: Removed LNI 192.168.201.142@tcp11 [ 8169.440332] LNet: Removed LNI 192.168.201.142@tcp12 [ 8169.442827] LNetError: 504995:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8169.447906] LNetError: 504995:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8169.453545] LNet: Removed LNI 192.168.201.142@tcp13 [ 8170.464136] LNet: Removed LNI 192.168.201.142@tcp14 [ 8171.488156] LNet: Removed LNI 192.168.201.142@tcp15 [ 8171.490216] LNetError: 504995:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8171.493308] LNetError: 504995:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 8171.497691] LNet: Removed LNI 192.168.201.142@tcp16 [ 8172.512132] LNet: Removed LNI 192.168.201.142@tcp17 [ 8173.536175] LNet: Removed LNI 192.168.201.142@tcp18 [ 8174.560122] LNet: Removed LNI 192.168.201.142@tcp19 [ 8175.584107] LNet: Removed LNI 192.168.201.142@tcp20 [ 8175.585466] LNetError: 504995:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8175.587495] LNetError: 504995:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 8175.589791] LNet: Removed LNI 192.168.201.142@tcp21 [ 8176.608167] LNet: Removed LNI 192.168.201.142@tcp22 [ 8177.632180] LNet: Removed LNI 192.168.201.142@tcp23 [ 8178.656466] LNet: Removed LNI 192.168.201.142@tcp24 [ 8179.680092] LNet: Removed LNI 192.168.201.142@tcp25 [ 8180.704184] LNet: Removed LNI 192.168.201.142@tcp26 [ 8181.728191] LNet: Removed LNI 192.168.201.142@tcp27 [ 8182.752294] LNet: Removed LNI 192.168.201.142@tcp28 [ 8183.776167] LNet: Removed LNI 192.168.201.142@tcp29 [ 8183.778131] LNetError: 504995:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8183.781331] LNetError: 504995:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 8183.784422] LNet: Removed LNI 192.168.201.142@tcp30 [ 8184.800432] LNet: Removed LNI 192.168.201.142@tcp31 [ 8185.824160] LNet: Removed LNI 192.168.201.142@tcp32 [ 8186.848233] LNet: Removed LNI 192.168.201.142@tcp33 [ 8187.872112] LNet: Removed LNI 192.168.201.142@tcp34 [ 8187.873557] LNet: Removed LNI 192.168.201.142@tcp35 [ 8188.896207] LNet: Removed LNI 192.168.201.142@tcp36 [ 8189.920276] LNet: Removed LNI 192.168.201.142@tcp37 [ 8190.944280] LNet: Removed LNI 192.168.201.142@tcp38 [ 8191.968295] LNet: Removed LNI 192.168.201.142@tcp39 [ 8192.992328] LNet: Removed LNI 192.168.201.142@tcp40 [ 8194.016124] LNet: Removed LNI 192.168.201.142@tcp41 [ 8194.017893] LNet: Removed LNI 192.168.201.142@tcp42 [ 8194.019077] LNet: Removed LNI 192.168.201.142@tcp43 [ 8194.020163] LNet: Removed LNI 192.168.201.142@tcp44 [ 8195.040242] LNet: Removed LNI 192.168.201.142@tcp45 [ 8195.042248] LNet: Removed LNI 192.168.201.142@tcp46 [ 8195.044085] LNet: Removed LNI 192.168.201.142@tcp47 [ 8196.064228] LNet: Removed LNI 192.168.201.142@tcp48 [ 8196.066807] LNet: Removed LNI 192.168.201.142@tcp49 [ 8196.082737] LNet: Removed LNI 192.168.201.142@tcp50 [ 8196.420583] LNetError: 505058:0:(module.c:692:libcfs_exit()) Portals memory leaked: 376 bytes [ 8196.446079] Key type .llcrypt unregistered [ 8196.447294] Key type ._llcrypt unregistered [ 8203.925673] Key type ._llcrypt registered [ 8203.927056] Key type .llcrypt registered [ 8203.975769] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 8204.418629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8204.458928] alg: No test for adler32 (adler32-zlib) [ 8205.382075] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 8205.482461] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 8207.072243] Key type lgssc registered [ 8207.497074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8210.273201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8210.281644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8211.483554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8212.947808] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 8216.680370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8216.867694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8218.340150] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 8220.100287] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8221.281406] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8223.716332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8223.840388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8225.940571] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 8227.814740] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8242.036077] loop0: detected capacity change from 0 to 204800000 [ 8242.157427] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8242.161531] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8242.166374] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8242.197118] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8244.604534] loop0: detected capacity change from 0 to 204800000 [ 8244.669196] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8244.672555] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8244.675793] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8244.737487] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8246.504105] loop0: detected capacity change from 0 to 204800000 [ 8248.435160] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8248.468405] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8248.524096] Lustre: fs15246-OST0000: new disk, initializing [ 8248.526187] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8248.544937] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8250.582970] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 8254.656195] loop1: detected capacity change from 0 to 204800000 [ 8256.646035] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8256.673163] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8256.707325] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8256.728175] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8256.772505] Lustre: fs15246-MDT0000: new disk, initializing [ 8256.822336] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8256.831451] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8256.871172] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8257.888747] Lustre: 510615:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000003da15728 x1849807650978560/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1764114116 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8258.172636] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 8266.954323] LustreError: 510615:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 8266.961439] LustreError: 510615:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 8266.968940] LustreError: 510615:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 8270.821027] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8270.825385] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8273.106864] LustreError: 511738:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 8273.145232] Lustre: server umount fs15246-MDT0000 complete [ 8275.945299] LustreError: 506243:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8275.950964] LustreError: 506243:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8276.988113] LustreError: 512241:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 8276.990082] LustreError: 512241:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8277.007306] Lustre: server umount fs15246-OST0000 complete [ 8281.061558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8281.069683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8281.072909] Lustre: Skipped 1 previous similar message [ 8286.179995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8286.184214] Lustre: Skipped 1 previous similar message [ 8291.302123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8291.305651] Lustre: Skipped 1 previous similar message [ 8295.392162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8295.414713] LustreError: 512742:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8295.417390] LustreError: 512742:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 8295.463543] Lustre: server umount lustre-OST0000 complete [ 8298.977322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8298.979272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8298.981978] Lustre: Skipped 1 previous similar message [ 8298.983568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8301.537815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8311.264143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8311.305202] LustreError: 512946:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8311.307153] LustreError: 512946:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8311.335651] Lustre: server umount lustre-MDT0000 complete [ 8311.777063] LustreError: 507430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8311.781413] LustreError: 507430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8312.542156] LustreError: 510609:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764114154 with bad export cookie 750934692994154936 [ 8312.544892] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8312.545440] LustreError: 510609:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8312.661142] Lustre: server umount lustre-MDT0001 complete [ 8316.472241] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 8317.476699] Key type lgssc unregistered [ 8317.613371] LNet: 513678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8317.617202] LNetError: 513678:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8317.623305] LNet: Removed LNI 192.168.201.142@tcp [ 8317.939769] Key type .llcrypt unregistered [ 8317.941294] Key type ._llcrypt unregistered [ 8325.321418] Key type ._llcrypt registered [ 8325.323277] Key type .llcrypt registered [ 8325.367328] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 8325.694277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8325.720616] alg: No test for adler32 (adler32-zlib) [ 8326.581114] Lustre: Lustre: Build Version: 2.16.61_56_gd832583 [ 8326.669950] LNet: Added LNI 192.168.201.142@tcp [8/256/0/180] [ 8328.248280] Key type lgssc registered [ 8328.720638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8331.319772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8333.340177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8335.132867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8337.068724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8343.630678] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 8346.878998] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 8350.132455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8352.227562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8354.437158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8356.664384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8358.920746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8358.938015] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8358.942556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8360.031307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8360.045844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8360.080783] Lustre: lustre-MDT0000: new disk, initializing [ 8360.107334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8360.115551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8361.362392] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 8365.469422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8365.495090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8365.528961] Lustre: 518080:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8365.562776] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8365.563309] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8365.565958] Lustre: Skipped 1 previous similar message [ 8365.606964] Lustre: lustre-MDT0001: new disk, initializing [ 8365.643759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8365.655971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8365.658694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8366.945272] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 8369.961216] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8371.087707] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8373.306776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8373.336995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8373.430654] Lustre: lustre-OST0000: new disk, initializing [ 8373.432360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8373.461241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8375.627207] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_default_debug -1 all [ 8378.888886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8378.892402] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8378.935583] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8379.144884] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8380.622120] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8380.707403] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8382.390376] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8382.460696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8383.971229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8383.971801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8383.973396] LustreError: Skipped 1 previous similar message [ 8383.977023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8383.978916] Lustre: Skipped 1 previous similar message [ 8385.263081] LustreError: 521019:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8385.306386] Lustre: server umount lustre-OST0000 complete [ 8389.089653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8389.090581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8389.094865] Lustre: Skipped 2 previous similar messages [ 8392.831882] LustreError: 521220:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8392.834619] LustreError: 521220:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8392.863310] Lustre: server umount lustre-MDT0000 complete [ 8394.312683] LustreError: 518072:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764114236 with bad export cookie 12577452740120798176 [ 8394.316467] LustreError: MGC192.168.201.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8394.321782] LustreError: 518072:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8394.389230] LustreError: 521421:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8394.391140] LustreError: 521421:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8394.449396] Lustre: server umount lustre-MDT0001 complete [ 8403.548731] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing set_hostid [ 8406.959541] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing load_modules_local [ 8410.332379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8413.394948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8415.898756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8418.737897] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8419.454056] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8255 sec ========== 18:44:21 (1764114261) [ 8420.146659] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:44:21 (1764114261) === [ 8421.365321] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:44:23 (1764114263) === [ 8436.231910] Lustre: DEBUG MARKER: oleg142-server.virtnet: executing unload_modules_local [ 8437.292574] Key type lgssc unregistered [ 8437.431389] LNet: 526269:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8437.433890] LNetError: 526269:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8437.445456] LNet: Removed LNI 192.168.201.142@tcp [ 8437.800215] Key type .llcrypt unregistered [ 8437.801220] Key type ._llcrypt unregistered