[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 524521281 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002235] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.006602] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009102] LSM: Security Framework initializing [ 0.010056] Yama: becoming mindful. [ 0.011041] SELinux: Initializing. [ 0.012084] *** VALIDATE selinux *** [ 0.020495] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025031] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027008] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028099] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029086] *** VALIDATE tmpfs *** [ 0.031007] *** VALIDATE proc *** [ 0.032257] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.035117] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036162] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038028] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042122] debug: unmapping init [mem 0xffffffffb0059000-0xffffffffb0060fff] [ 0.044155] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045684] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047015] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051014] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.053298] rcu: Hierarchical SRCU implementation. [ 0.055199] smp: Bringing up secondary CPUs ... [ 0.056406] x86: Booting SMP configuration: [ 0.057020] .... node #0, CPUs: #1 #2 #3 [ 0.061083] smp: Brought up 1 node, 4 CPUs [ 0.063013] smpboot: Max logical packages: 1 [ 0.064019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.166029] node 0 deferred pages initialised in 100ms [ 0.171166] devtmpfs: initialized [ 0.172257] x86/mm: Memory block size: 128MB [ 0.174363] gcov: version magic: 0x41383552 [ 0.177476] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.182177] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.184352] pinctrl core: initialized pinctrl subsystem [ 0.188189] [ 0.188734] ************************************************************* [ 0.191012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.194014] ** ** [ 0.197013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.199010] ** ** [ 0.201012] ** This means that this kernel is built to expose internal ** [ 0.204012] ** IOMMU data structures, which may compromise security on ** [ 0.206012] ** your system. ** [ 0.208013] ** ** [ 0.211011] ** If you see this message and you are not debugging the ** [ 0.213012] ** kernel, report this immediately to your vendor! ** [ 0.215011] ** ** [ 0.218011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.220010] ************************************************************* [ 0.222681] NET: Registered protocol family 16 [ 0.225430] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.227073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.230061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.234103] cpuidle: using governor menu [ 0.236865] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.239619] PCI: Using configuration type 1 for base access [ 0.242157] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.252067] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.255038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.258113] cryptd: max_cpu_qlen set to 1000 [ 0.262323] ACPI: Added _OSI(Module Device) [ 0.263018] ACPI: Added _OSI(Processor Device) [ 0.264019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.266018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.271560] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.279522] ACPI: Interpreter enabled [ 0.281087] ACPI: PM: (supports S0 S3 S4 S5) [ 0.283024] ACPI: Using IOAPIC for interrupt routing [ 0.285216] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.289426] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.299015] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.301061] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.304021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.307084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.312541] acpiphp: Slot [2] registered [ 0.314104] acpiphp: Slot [5] registered [ 0.316113] acpiphp: Slot [6] registered [ 0.317124] acpiphp: Slot [7] registered [ 0.319120] acpiphp: Slot [8] registered [ 0.320120] acpiphp: Slot [9] registered [ 0.322237] acpiphp: Slot [10] registered [ 0.324229] acpiphp: Slot [3] registered [ 0.326171] acpiphp: Slot [4] registered [ 0.328163] acpiphp: Slot [11] registered [ 0.330180] acpiphp: Slot [12] registered [ 0.332172] acpiphp: Slot [13] registered [ 0.334248] acpiphp: Slot [14] registered [ 0.336409] acpiphp: Slot [15] registered [ 0.338169] acpiphp: Slot [16] registered [ 0.340207] acpiphp: Slot [17] registered [ 0.342156] acpiphp: Slot [18] registered [ 0.344166] acpiphp: Slot [19] registered [ 0.346245] acpiphp: Slot [20] registered [ 0.348166] acpiphp: Slot [21] registered [ 0.349226] acpiphp: Slot [22] registered [ 0.351230] acpiphp: Slot [23] registered [ 0.353114] acpiphp: Slot [24] registered [ 0.355156] acpiphp: Slot [25] registered [ 0.356148] acpiphp: Slot [26] registered [ 0.358180] acpiphp: Slot [27] registered [ 0.360091] acpiphp: Slot [28] registered [ 0.361130] acpiphp: Slot [29] registered [ 0.363150] acpiphp: Slot [30] registered [ 0.365140] acpiphp: Slot [31] registered [ 0.367074] PCI host bridge to bus 0000:00 [ 0.368024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.371027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.374026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.377028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.380023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.383029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.385217] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.388678] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.392503] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.403024] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.408496] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.411034] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.413025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.416027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.419696] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.422797] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.425058] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.429619] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.436018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.449020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.455016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.462721] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.472027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.480027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.498027] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.512000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.520018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.531020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.548021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.561522] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.571022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.579028] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.597031] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.610074] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.620018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.629020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.644022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.654011] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.662023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.669025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.684025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.696610] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.713021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.722013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.739018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.749540] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.753501] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.755386] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.758457] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.761333] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.767138] iommu: Default domain type: Passthrough [ 0.769526] SCSI subsystem initialized [ 0.770200] ACPI: bus type USB registered [ 0.772139] usbcore: registered new interface driver usbfs [ 0.773121] usbcore: registered new interface driver hub [ 0.776174] usbcore: registered new device driver usb [ 0.777178] pps_core: LinuxPPS API ver. 1 registered [ 0.779015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.782116] PTP clock support registered [ 0.784173] EDAC MC: Ver: 3.0.0 [ 0.786150] PCI: Using ACPI for IRQ routing [ 0.789369] NetLabel: Initializing [ 0.791015] NetLabel: domain hash size = 128 [ 0.793013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.794102] NetLabel: unlabeled traffic allowed by default [ 0.797112] vgaarb: loaded [ 0.798301] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.800022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.810000] clocksource: Switched to clocksource kvm-clock [ 0.932695] VFS: Disk quotas dquot_6.6.0 [ 0.934480] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.937255] *** VALIDATE ramfs *** [ 0.938612] *** VALIDATE hugetlbfs *** [ 0.940445] pnp: PnP ACPI init [ 0.943237] pnp: PnP ACPI: found 6 devices [ 0.972159] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.975721] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.978433] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.981417] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.983528] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.985933] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.990922] NET: Registered protocol family 2 [ 0.993306] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.999174] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.002320] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.007556] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.012075] TCP: Hash tables configured (established 65536 bind 65536) [ 1.015462] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.018786] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.022459] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.025685] NET: Registered protocol family 1 [ 1.029473] RPC: Registered named UNIX socket transport module. [ 1.031493] RPC: Registered udp transport module. [ 1.033268] RPC: Registered tcp transport module. [ 1.034924] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.037348] NET: Registered protocol family 44 [ 1.039148] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.041410] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.043355] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.045889] PCI: CLS 0 bytes, default 64 [ 1.047498] Unpacking initramfs... [ 2.478522] debug: unmapping init [mem 0xffff8d99bcc54000-0xffff8d99bffbffff] [ 2.482323] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.484419] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.487188] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.979390] Initialise system trusted keyrings [ 2.981259] Key type blacklist registered [ 2.983306] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.994060] zbud: loaded [ 2.997449] *** VALIDATE nfs *** [ 2.998884] *** VALIDATE nfs4 *** [ 3.001067] pstore: using deflate compression [ 3.016801] Platform Keyring initialized [ 3.143242] NET: Registered protocol family 38 [ 3.145322] Key type asymmetric registered [ 3.147109] Asymmetric key parser 'x509' registered [ 3.149234] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.152757] io scheduler mq-deadline registered [ 3.154489] io scheduler kyber registered [ 3.156610] io scheduler bfq registered [ 3.158671] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.162126] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.165358] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.168626] ACPI: Power Button [PWRF] [ 3.174686] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.182206] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.196939] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.205301] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.221746] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.251689] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.280820] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.285948] Non-volatile memory driver v1.3 [ 3.287682] Linux agpgart interface v0.103 [ 3.324740] virtio_blk virtio1: [vda] 145808 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.327763] vda: detected capacity change from 0 to 74653696 [ 3.342637] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.345704] vdb: detected capacity change from 0 to 1073741824 [ 3.363832] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.367324] vdc: detected capacity change from 0 to 2621440000 [ 3.385316] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.388316] vdd: detected capacity change from 0 to 2621440000 [ 3.412343] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.415634] vde: detected capacity change from 0 to 4294967296 [ 3.430951] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.434040] vdf: detected capacity change from 0 to 4294967296 [ 3.440539] libphy: Fixed MDIO Bus: probed [ 3.447206] usbcore: registered new interface driver usbserial_generic [ 3.449579] usbserial: USB Serial support registered for generic [ 3.451266] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.454093] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.455632] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.457944] mousedev: PS/2 mouse device common for all mice [ 3.460170] rtc_cmos 00:05: RTC can wake from S4 [ 3.462954] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.465981] rtc_cmos 00:05: registered as rtc0 [ 3.467755] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.471687] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.474612] intel_pstate: CPU model not supported [ 3.476280] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.480461] hid: raw HID events driver (C) Jiri Kosina [ 3.482555] usbcore: registered new interface driver usbhid [ 3.484549] usbhid: USB HID core driver [ 3.486387] drop_monitor: Initializing network drop monitor service [ 3.488953] Initializing XFRM netlink socket [ 3.490877] NET: Registered protocol family 10 [ 3.493729] Segment Routing with IPv6 [ 3.495106] NET: Registered protocol family 17 [ 3.497905] mpls_gso: MPLS GSO support [ 3.503752] RAS: Correctable Errors collector initialized. [ 3.506627] AVX version of gcm_enc/dec engaged. [ 3.508178] AES CTR mode by8 optimization enabled [ 3.576674] sched_clock: Marking stable (3576647921, 0)->(4423316756, -846668835) [ 3.581041] registered taskstats version 1 [ 3.583108] Loading compiled-in X.509 certificates [ 3.584830] zswap: loaded using pool lzo/zbud [ 3.615124] Key type big_key registered [ 3.625865] Key type encrypted registered [ 3.627634] ima: No TPM chip found, activating TPM-bypass! [ 3.629867] ima: Allocated hash algorithm: sha1 [ 3.631767] ima: No architecture policies found [ 3.633757] evm: Initialising EVM extended attributes: [ 3.635650] evm: security.selinux [ 3.636844] evm: security.ima [ 3.637838] evm: security.capability [ 3.639119] evm: HMAC attrs: 0x1 [ 3.641629] rtc_cmos 00:05: setting system clock to 2026-07-20 21:48:10 UTC (1784584090) [ 3.647924] debug: unmapping init [mem 0xffffffffb1003000-0xffffffffb11fffff] [ 3.651098] debug: unmapping init [mem 0xffffffffafd82000-0xffffffffb0058fff] [ 3.660050] Write protecting the kernel read-only data: 28672k [ 3.662442] debug: unmapping init [mem 0xffffffffae403000-0xffffffffae5fffff] [ 3.664607] debug: unmapping init [mem 0xffffffffaed14000-0xffffffffaedfffff] [ 3.698038] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.706648] systemd[1]: Detected virtualization kvm. [ 3.708519] systemd[1]: Detected architecture x86-64. [ 3.710150] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.733211] systemd[1]: No hostname configured. [ 3.735391] systemd[1]: Set hostname to . [ 3.737676] random: systemd: uninitialized urandom read (16 bytes read) [ 3.740090] systemd[1]: Initializing machine ID from random generator. [ 3.778489] random: ln: uninitialized urandom read (6 bytes read) [ 3.852212] random: systemd: uninitialized urandom read (16 bytes read) [ 3.855131] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.859918] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.864323] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.474818] device-mapper: uevent: version 1.0.3 [ 4.476643] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] [ 5.055926] random: fast init done Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.299553] virtio_net virtio0 ens2: renamed from eth0 [ 5.316968] scsi host0: ata_piix [ 5.359053] scsi host1: ata_piix [ 5.361931] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.364381] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.372738] dracut-initqueue[596]: RTNETLINK answers: File exists [ 10.145777] random: crng init done [ 10.147192] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.731039] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.994900] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.291317] SELinux: Disabled at runtime. [ 12.353688] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.363167] systemd[1]: Detected virtualization kvm. [ 12.365083] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.928653] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.932079] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.937100] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.940821] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.943438] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.952139] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.959202] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ 13.119571] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.469470] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.805701] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.853223] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.006191] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.017601] EDAC sbridge: Ver: 1.1.2 [ 15.613105] Key type dns_resolver registered [ 15.914309] NFS: Registering the id_resolver key type [ 15.916360] Key type id_resolver registered [ 15.918065] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg137-server login: [ 42.340703] libcfs: loading out-of-tree module taints kernel. [ 42.361304] Key type ._llcrypt registered [ 42.362848] Key type .llcrypt registered [ 42.421130] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 60.266460] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 62.020554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.042135] alg: No test for adler32 (adler32-zlib) [ 63.602462] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 64.783497] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 66.527198] Key type lgssc registered [ 68.104114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 87.040985] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 97.844116] hrtimer: interrupt took 4104236 ns [ 129.210297] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 142.538064] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 142.591806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 144.042149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 144.111296] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 144.197978] Lustre: lustre-MDT0000: new disk, initializing [ 144.272133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 144.293912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 147.931166] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 162.555412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 162.684700] Lustre: 6523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 162.717868] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 162.726322] Lustre: Skipped 1 previous similar message [ 162.843980] Lustre: lustre-MDT0001: new disk, initializing [ 162.905822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 162.927967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 162.941933] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 167.538660] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.426497] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 184.242940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 184.714463] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 184.723245] Lustre: lustre-OST0000: new disk, initializing [ 184.744383] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 184.750859] Lustre: 8463:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 184.811806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 189.948229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 189.955814] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 189.986628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 191.030618] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.571741] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 206.726799] Lustre: lustre-OST0001: new disk, initializing [ 206.730813] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 206.735424] Lustre: 9534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 206.815848] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.484928] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.097307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 215.109187] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 215.190976] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 226.109712] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 233.332647] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 239.953222] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing check_logdir /tmp/testlogs/ [ 246.528694] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing yml_node [ 250.362860] Lustre: DEBUG MARKER: Client: 2.17.54.183 [ 253.151718] Lustre: DEBUG MARKER: MDS: 2.17.54.183 [ 256.012081] Lustre: DEBUG MARKER: OSS: 2.17.54.183 [ 257.960917] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jul 20 17:52:22 EDT 2026 [ 278.145808] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 279.874696] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 297.440867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.443394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 297.454955] Lustre: Skipped 1 previous similar message [ 297.475817] Lustre: Skipped 3 previous similar messages [ 302.564184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 302.583060] Lustre: Skipped 3 previous similar messages [ 303.121801] Lustre: server umount lustre-MDT0000 complete [ 306.974430] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784584393 with bad export cookie 1602814892389207754 [ 306.991932] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 306.994296] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 307.681063] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.696191] Lustre: Skipped 2 previous similar messages [ 307.705378] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 307.708175] Lustre: Skipped 1 previous similar message [ 311.713893] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 311.723877] Lustre: Skipped 1 previous similar message [ 313.712794] Lustre: server umount lustre-MDT0001 complete [ 318.484263] Lustre: server umount lustre-OST0000 complete [ 323.655835] Lustre: server umount lustre-OST0001 complete [ 331.947823] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 343.507368] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 383.687790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 383.990468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 384.037161] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 384.172361] Lustre: lustre-MDT0000: new disk, initializing [ 384.268584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 384.294111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 388.863626] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 399.510936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 399.636981] Lustre: 15090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 399.707945] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 399.712094] Lustre: Skipped 1 previous similar message [ 399.781865] Lustre: lustre-MDT0001: new disk, initializing [ 399.894102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 399.944712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 399.964903] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 404.481172] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 414.172781] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 420.919661] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 427.992050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 428.364612] Lustre: lustre-OST0000: new disk, initializing [ 428.370337] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 428.384803] Lustre: 17058:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 428.492157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 430.415615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 430.424893] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 430.548617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 435.019930] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 445.067547] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 449.575276] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 449.894648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 454.620459] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 454.840567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 461.281826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 461.292955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 461.305173] Lustre: Skipped 1 previous similar message [ 461.316880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 461.334132] Lustre: Skipped 1 previous similar message [ 463.129723] Lustre: server umount lustre-OST0000 complete [ 471.528109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 471.537802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 471.543039] Lustre: Skipped 1 previous similar message [ 471.560032] LustreError: Skipped 1 previous similar message [ 473.752396] Lustre: server umount lustre-MDT0000 complete [ 476.640369] LustreError: 15892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 476.661049] LustreError: 15892:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 477.982644] LustreError: 15083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784584564 with bad export cookie 1602814892389209287 [ 477.984084] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 477.993380] LustreError: 15083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 478.563537] Lustre: server umount lustre-MDT0001 complete [ 483.197194] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 17:56:08 (1784584568) [ 486.086256] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 488.397099] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 17:56:13 (1784584573) [ 494.998584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.427877] LustreError: 19467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.496083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 499.614251] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 500.705462] LustreError: 19468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 505.827726] LustreError: 19467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.072131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 513.524643] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 521.945760] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 528.405736] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 535.906081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 543.010677] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 551.781644] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 564.195888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 564.197663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 564.213869] Lustre: Skipped 1 previous similar message [ 564.217806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 564.240523] Lustre: Skipped 3 previous similar messages [ 570.371861] Lustre: server umount lustre-OST0000 complete [ 574.442119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 574.465922] Lustre: Skipped 2 previous similar messages [ 588.255142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 588.689797] Lustre: server umount lustre-MDT0000 complete [ 589.801194] LustreError: 19467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 589.835920] LustreError: 19467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 592.480579] LustreError: 19447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784584679 with bad export cookie 1602814892389210169 [ 592.496960] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 592.750965] Lustre: server umount lustre-MDT0001 complete [ 604.196699] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 606.936393] Key type lgssc unregistered [ 607.247073] LNet: 23015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 607.260875] LNetError: 23015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 607.291935] LNet: Removed LNI 192.168.201.137@tcp [ 608.110151] Key type .llcrypt unregistered [ 608.112460] Key type ._llcrypt unregistered [ 631.923948] Key type ._llcrypt registered [ 631.925669] Key type .llcrypt registered [ 632.026529] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 633.444714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 633.480654] alg: No test for adler32 (adler32-zlib) [ 634.437889] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 634.621498] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 636.264624] Key type lgssc registered [ 637.601694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 645.507406] loop: module loaded [ 645.557317] loop0: detected capacity change from 0 to 204800000 [ 645.720176] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 645.733985] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 646.030194] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 651.469366] loop0: detected capacity change from 0 to 204800000 [ 651.623795] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 651.647427] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 651.654914] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 656.224173] loop0: detected capacity change from 0 to 204800000 [ 661.335943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 661.353659] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 662.659584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 662.715856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 662.816072] Lustre: lustre-MDT0000: new disk, initializing [ 662.887641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 662.905083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 667.473348] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 676.798808] loop1: detected capacity change from 0 to 204800000 [ 683.418990] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 683.735465] Lustre: lustre-OST07c6: new disk, initializing [ 683.738371] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 683.741547] Lustre: Skipped 1 previous similar message [ 683.747612] Lustre: 26205:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 683.794858] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 691.313700] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 693.882874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 693.897407] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 693.942733] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 697.456655] Lustre: 25187:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 697.463028] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 697.468091] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 697.472921] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 697.481329] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 697.488919] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 714.110489] Lustre: 25189:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 714.122458] Lustre: 25189:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 714.130519] Lustre: 25189:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 714.141629] Lustre: 25189:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 714.153973] Lustre: 25189:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 714.159530] Lustre: 25189:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 719.118199] Lustre: 25189:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 719.141606] Lustre: 25189:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 719.158148] Lustre: 25189:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 719.160847] Lustre: 25189:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.176987] Lustre: 25189:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 719.185055] Lustre: 25189:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.192696] Lustre: 25189:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 719.197888] Lustre: 25189:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.210742] Lustre: 25189:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 719.224237] Lustre: 25189:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.238268] Lustre: 25189:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 719.249261] Lustre: 25189:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.541057] Lustre: 25188:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 722.558145] Lustre: 25188:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 722.573475] Lustre: 25188:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 722.586578] Lustre: 25188:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.594393] Lustre: 25188:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 722.605718] Lustre: 25188:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.613315] Lustre: 25188:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 722.625130] Lustre: 25188:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.631741] Lustre: 25188:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 722.650937] Lustre: 25188:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.665179] Lustre: 25188:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 722.678273] Lustre: 25188:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.734784] Lustre: 25187:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 727.741240] Lustre: 25187:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 727.747650] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 727.752155] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.757684] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 727.762298] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.766987] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 727.772080] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.779020] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 727.782672] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.787163] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 727.792473] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 735.791362] Lustre: 26779:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 735.803196] Lustre: 26779:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 735.815594] Lustre: 26779:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 735.825440] Lustre: 26779:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 735.833387] Lustre: 26779:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 735.845607] Lustre: 26779:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 735.854772] Lustre: 26779:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 735.862144] Lustre: 26779:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 735.873539] Lustre: 26779:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 735.885933] Lustre: 26779:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 735.903499] Lustre: 26779:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 735.913230] Lustre: 26779:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 791.015241] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 791.018198] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 791.035107] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 794.163708] Lustre: server umount lustre-OST07c6 complete [ 804.382255] Lustre: server umount lustre-MDT0000 complete [ 821.128981] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 824.244367] Key type lgssc unregistered [ 824.681426] LNet: 28776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.701497] LNetError: 28776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 824.744866] LNet: Removed LNI 192.168.201.137@tcp [ 825.773153] Key type .llcrypt unregistered [ 825.776713] Key type ._llcrypt unregistered [ 848.807255] Key type ._llcrypt registered [ 848.809405] Key type .llcrypt registered [ 848.945185] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 850.196763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 850.305135] alg: No test for adler32 (adler32-zlib) [ 851.377384] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 851.748314] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 853.607776] Key type lgssc registered [ 855.070709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 864.448993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 864.471759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 866.168025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 871.194872] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 875.493324] LustreError: 30047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 875.506550] LustreError: 30047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 880.361458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 880.626145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 885.823754] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 894.164369] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 899.718381] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 906.344531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 906.637501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 910.776905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 913.248428] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 920.106655] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 941.549921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 941.550319] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 941.572580] LustreError: Skipped 1 previous similar message [ 941.580366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 941.611554] Lustre: Skipped 1 previous similar message [ 945.950784] Lustre: server umount lustre-OST0000 complete [ 951.778962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 951.780911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 951.794361] Lustre: Skipped 1 previous similar message [ 951.816459] Lustre: Skipped 2 previous similar messages [ 956.898122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 956.903974] Lustre: Skipped 1 previous similar message [ 957.808603] Lustre: server umount lustre-MDT0000 complete [ 961.907667] LustreError: 30027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784585048 with bad export cookie 9362073716746002597 [ 961.918677] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 961.929514] LustreError: 30027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 962.332307] Lustre: server umount lustre-MDT0001 complete [ 975.246950] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 978.319237] Key type lgssc unregistered [ 978.577607] LNet: 33785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.582285] LNetError: 33785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.622227] LNet: Removed LNI 192.168.201.137@tcp [ 980.445475] Key type .llcrypt unregistered [ 980.447904] Key type ._llcrypt unregistered [ 1000.219860] Key type ._llcrypt registered [ 1000.224150] Key type .llcrypt registered [ 1000.415691] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1001.836174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1001.999124] alg: No test for adler32 (adler32-zlib) [ 1003.230763] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 1003.485966] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1005.226916] Key type lgssc registered [ 1006.483794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1014.909319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1014.923022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1016.397242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1021.617877] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1026.535522] LustreError: 35065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1026.572236] LustreError: 35065:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1031.651428] LustreError: 35066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1032.283733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1032.556433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1037.560890] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1045.819763] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1053.002835] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1059.797890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1060.055212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1062.156961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1066.538944] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1075.451193] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1093.089051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1093.091762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.092302] LustreError: Skipped 1 previous similar message [ 1093.093476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1093.099830] Lustre: Skipped 1 previous similar message [ 1093.145527] Lustre: Skipped 1 previous similar message [ 1096.132422] Lustre: server umount lustre-OST0000 complete [ 1103.328591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.343691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1103.346748] Lustre: Skipped 1 previous similar message [ 1103.365084] Lustre: Skipped 1 previous similar message [ 1106.769480] Lustre: server umount lustre-MDT0000 complete [ 1108.448535] LustreError: 35061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1108.471224] LustreError: 35061:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1110.378269] LustreError: 35045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784585197 with bad export cookie 3162133136572973273 [ 1110.392375] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1110.400202] LustreError: 35045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1110.769918] Lustre: server umount lustre-MDT0001 complete [ 1124.736680] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1129.221860] Key type lgssc unregistered [ 1129.759736] LNet: 38620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1129.780328] LNetError: 38620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1129.818789] LNet: Removed LNI 192.168.201.137@tcp [ 1131.325182] Key type .llcrypt unregistered [ 1131.329947] Key type ._llcrypt unregistered [ 1158.671979] Key type ._llcrypt registered [ 1158.676760] Key type .llcrypt registered [ 1158.788337] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1159.826558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1159.882951] alg: No test for adler32 (adler32-zlib) [ 1161.181685] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 1161.495809] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1163.207214] Key type lgssc registered [ 1164.980664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1174.001376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1174.020679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1175.686908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1181.175808] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1185.259980] LustreError: 40042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1185.284835] LustreError: 40042:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1190.372356] LustreError: 40041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1193.099197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1193.616920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1199.235876] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1208.025700] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1213.991794] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1222.983895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1223.699278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1231.840492] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1241.335682] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1245.239276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1253.334107] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 1254.883158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1254.895270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1254.916793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1255.390498] Lustre: server umount lustre-MDT0000 complete [ 1255.413431] LustreError: 40042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1255.414052] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1255.447952] Lustre: Skipped 1 previous similar message [ 1258.423669] LustreError: 40037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1258.438382] LustreError: 40037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1259.844742] LustreError: 40023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784585346 with bad export cookie 3667724208596556919 [ 1259.870473] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1260.514056] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1260.532782] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1263.541519] LustreError: 40038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1263.572580] LustreError: 40038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1263.596456] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [ 1268.660368] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [ 1268.667899] Lustre: Skipped 1 previous similar message [ 1273.787183] LustreError: 40037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.806228] LustreError: 40037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1274.335532] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1274.595582] Lustre: server umount lustre-MDT0001 complete [ 1289.767564] Lustre: server umount lustre-OST0000 complete [ 1306.437487] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1309.032615] Key type lgssc unregistered [ 1309.316317] LNet: 43901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1309.326852] LNetError: 43901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1310.377724] LNet: Removed LNI 192.168.201.137@tcp [ 1311.140294] Key type .llcrypt unregistered [ 1311.143162] Key type ._llcrypt unregistered [ 1339.589926] Key type ._llcrypt registered [ 1339.600088] Key type .llcrypt registered [ 1339.776750] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1340.882453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1340.932996] alg: No test for adler32 (adler32-zlib) [ 1342.174760] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 1342.582966] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1344.255682] Key type lgssc registered [ 1346.065952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1355.866726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1355.879600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1357.409438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1361.602575] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1367.521878] LustreError: 45318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1367.538494] LustreError: 45318:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1370.588904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1371.111204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1376.515635] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1383.605994] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1390.781500] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1398.147576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1398.475290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1402.610064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1405.004979] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1414.466848] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1423.328635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1423.338961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1423.372102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1426.404143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1426.405490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1426.420411] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1426.448537] Lustre: Skipped 1 previous similar message [ 1431.537197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1431.564600] Lustre: Skipped 2 previous similar messages [ 1436.127347] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1436.355307] Lustre: server umount lustre-OST0000 complete [ 1455.827231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1455.846958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1455.850135] Lustre: Skipped 1 previous similar message [ 1455.871575] Lustre: Skipped 2 previous similar messages [ 1462.264324] LustreError: 45318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1462.543340] Lustre: server umount lustre-MDT0000 complete [ 1467.360962] LustreError: 45319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1467.398276] LustreError: 45319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1467.465263] LustreError: 45299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784585554 with bad export cookie 3871686305638618360 [ 1467.472783] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1467.482319] LustreError: 45299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1474.127228] Lustre: server umount lustre-MDT0001 complete [ 1489.477147] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1494.567743] Key type lgssc unregistered [ 1495.064748] LNet: 49035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1495.079211] LNetError: 49035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1495.103848] LNet: Removed LNI 192.168.201.137@tcp [ 1496.641993] Key type .llcrypt unregistered [ 1496.644066] Key type ._llcrypt unregistered [ 1521.871591] Key type ._llcrypt registered [ 1521.877494] Key type .llcrypt registered [ 1522.036553] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1523.220833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1523.297286] alg: No test for adler32 (adler32-zlib) [ 1524.364415] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 1524.575918] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1526.240289] Key type lgssc registered [ 1527.642813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1536.880656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1536.908524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1538.722357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1543.684483] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1548.267643] LustreError: 50453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1548.290954] LustreError: 50453:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1553.378674] LustreError: 50454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1554.417087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1554.911995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1559.700140] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1568.326472] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1574.322938] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1582.006467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1582.437128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1583.484560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1589.098713] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1598.130168] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1607.542237] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1612.137069] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1625.658139] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-2018h15m08s [ 1627.864792] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1631.202799] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 1631.715246] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1631.717772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1631.750754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1631.771379] Lustre: Skipped 2 previous similar messages [ 1632.802201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1632.820285] Lustre: Skipped 3 previous similar messages [ 1636.543383] Lustre: server umount lustre-MDT0000 complete [ 1638.338638] LustreError: 50450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1640.451540] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1644.477863] LustreError: 50449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1644.500451] LustreError: 50449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1647.551981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1647.708807] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1647.991035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1648.095444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1649.007153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1652.859383] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1656.316648] Lustre: DEBUG MARKER: Wait for df (36693) ... [ 1658.307922] Lustre: DEBUG MARKER: done [ 1663.462165] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1663.477807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1663.501250] Lustre: Skipped 2 previous similar messages [ 1663.514422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1664.500448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1669.453294] Lustre: server umount lustre-OST0000 complete [ 1674.721623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1674.741358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1674.744391] Lustre: Skipped 2 previous similar messages [ 1679.900896] Lustre: server umount lustre-MDT0000 complete [ 1681.893115] LustreError: 51182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1681.909177] LustreError: 51182:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1683.428347] LustreError: 50435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784585770 with bad export cookie 14880805083289144814 [ 1683.432827] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1683.438522] LustreError: 50435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1683.852664] Lustre: server umount lustre-MDT0001 complete [ 1696.213826] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1699.103409] Key type lgssc unregistered [ 1699.490829] LNet: 55378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1699.499080] LNetError: 55378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1699.539675] LNet: Removed LNI 192.168.201.137@tcp [ 1700.408702] Key type .llcrypt unregistered [ 1700.410875] Key type ._llcrypt unregistered [ 1744.417155] Key type ._llcrypt registered [ 1744.418925] Key type .llcrypt registered [ 1744.518818] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1746.071654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1746.096672] alg: No test for adler32 (adler32-zlib) [ 1747.091220] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 1747.412966] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1749.071203] Key type lgssc registered [ 1750.169411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1757.690488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1757.704470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1759.016856] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1759.049786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1759.233228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1764.045903] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1773.649344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1773.824196] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1773.876972] Lustre: 57587:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1774.263662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1779.630786] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1788.278515] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1793.280198] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1802.708770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1802.852714] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1803.125625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1810.180339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1810.418478] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1818.935827] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1825.879876] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-2018h18m30s [ 1828.086685] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1837.886309] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1841.638700] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1841.981706] LustreError: 58940:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1862.121648] Lustre: lustre-MDT0000: Client 0f9029b3-389d-45cc-a786-7cb40cd3cf17 (at 192.168.201.37@tcp) reconnecting [ 1882.015196] LustreError: 58940:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1884.872101] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1887.931487] Lustre: DEBUG MARKER: done [ 1895.405122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1895.421709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1895.427715] Lustre: Skipped 1 previous similar message [ 1900.522773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1900.534139] Lustre: Skipped 1 previous similar message [ 1901.160316] Lustre: server umount lustre-OST0000 complete [ 1905.637622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1905.640320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1905.669628] Lustre: Skipped 2 previous similar messages [ 1905.684669] Lustre: Skipped 1 previous similar message [ 1911.290048] Lustre: server umount lustre-MDT0000 complete [ 1915.209952] LustreError: 57580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784586002 with bad export cookie 11038023409957641546 [ 1915.213519] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1915.220952] LustreError: 57580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1915.574436] Lustre: server umount lustre-MDT0001 complete [ 1926.097761] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1928.944291] Key type lgssc unregistered [ 1929.290689] LNet: 61926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1929.299051] LNetError: 61926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1929.316952] LNet: Removed LNI 192.168.201.137@tcp [ 1930.107138] Key type .llcrypt unregistered [ 1930.108441] Key type ._llcrypt unregistered [ 1975.756764] Key type ._llcrypt registered [ 1975.762688] Key type .llcrypt registered [ 1975.902502] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1977.786566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1977.837062] alg: No test for adler32 (adler32-zlib) [ 1978.884799] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 1979.213573] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1980.911294] Key type lgssc registered [ 1982.704769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1986.221048] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1986.244368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1987.503287] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1987.534042] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1987.755926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1988.650981] Lustre: Failing over lustre-MDT0000 [ 1989.012877] Lustre: server umount lustre-MDT0000 complete [ 1996.657350] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 18:21:21 (1784586081) [ 2003.240979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2003.593283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2008.238582] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2018.651562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2018.761625] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2018.809743] Lustre: 64723:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2019.107568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2024.909346] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2034.556646] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2041.096786] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2048.055307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2048.277785] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2048.484428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2053.874374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2055.412320] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2063.973867] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2070.149531] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2079.730931] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2079.741029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2079.752720] Lustre: Skipped 2 previous similar messages [ 2079.773899] Lustre: Skipped 2 previous similar messages [ 2083.412738] Lustre: server umount lustre-MDT0000 complete [ 2087.864070] LustreError: 64717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784586174 with bad export cookie 1100662085951852682 [ 2087.873279] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2088.221890] Lustre: server umount lustre-MDT0001 complete [ 2092.876603] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2098.512928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2105.311230] Lustre: 63646:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784586176/real 1784586176] req@00000000b1f1fb4f x1871274118783360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784586192 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2105.351797] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2108.384114] Lustre: 63647:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784586179/real 1784586179] req@00000000da9bfefc x1871274118783488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784586195 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2108.839527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2110.431378] Lustre: 63648:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784586181/real 1784586181] req@000000006af3038e x1871274118783616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784586197 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2112.545657] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf465655eae6996d [ 2112.563371] Lustre: MGC192.168.201.137@tcp: Connection restored to 0@lo (at 0@lo) [ 2113.011670] LustreError: 66573:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2113.027170] LustreError: 66573:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2113.147286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2113.277656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2114.145757] LustreError: 66574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2114.172160] Lustre: 63648:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784586184/real 1784586184] req@000000000457e845 x1871274118783872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784586200 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2115.169163] LustreError: 66573:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2115.197101] LustreError: 66573:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2118.825309] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2120.163209] Lustre: 63647:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784586190/real 1784586190] req@00000000acb6b7c2 x1871274118784000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784586206 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2120.674907] LustreError: 66574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2120.698098] LustreError: 66574:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2125.794020] LustreError: 68499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2125.805541] LustreError: 68499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2127.183052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2127.646070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2132.905616] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2132.978741] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2135.116604] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2141.671877] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2147.271783] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2155.498446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2155.505409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2155.509080] Lustre: Skipped 2 previous similar messages [ 2155.530673] Lustre: Skipped 2 previous similar messages [ 2160.613656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2160.621297] Lustre: Skipped 2 previous similar messages [ 2165.731705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2165.752220] Lustre: Skipped 2 previous similar messages [ 2169.311113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2169.695754] Lustre: server umount lustre-MDT0000 complete [ 2170.851509] LustreError: 68498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2170.874633] LustreError: 68498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2173.941956] LustreError: 64716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784586260 with bad export cookie 1100662085951854957 [ 2173.945246] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2173.953052] LustreError: 64716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2174.405338] Lustre: server umount lustre-MDT0001 complete [ 2181.604743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2190.893821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2191.329248] Lustre: 63649:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784586262/real 1784586262] req@000000000d2d1ea3 x1871274118814208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784586278 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2191.395813] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.082594] LustreError: 69488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2199.157626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2199.216380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2200.183740] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2205.149890] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2216.271449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2221.476389] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2221.556735] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2229.813180] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2236.039953] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2247.141547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2247.157294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2247.158564] Lustre: Skipped 1 previous similar message [ 2247.162447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2247.188987] Lustre: Skipped 2 previous similar messages [ 2249.830862] Lustre: server umount lustre-MDT0000 complete [ 2252.286594] LustreError: 66573:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2252.321929] LustreError: 66573:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 2254.722030] LustreError: 64716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784586341 with bad export cookie 1100662085951856336 [ 2254.725714] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2254.737139] LustreError: 64716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2255.197787] Lustre: server umount lustre-MDT0001 complete [ 2262.310608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2266.272189] Lustre: DEBUG MARKER: files compared the same [ 2277.696716] Lustre: server umount lustre-OST0000 complete [ 2295.161435] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2297.696543] Key type lgssc unregistered [ 2297.883516] LNet: 74460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2297.897234] LNetError: 74460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2297.917662] LNet: Removed LNI 192.168.201.137@tcp [ 2298.740573] Key type .llcrypt unregistered [ 2298.744560] Key type ._llcrypt unregistered [ 2322.139244] Key type ._llcrypt registered [ 2322.142801] Key type .llcrypt registered [ 2322.256401] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2323.279359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2323.363521] alg: No test for adler32 (adler32-zlib) [ 2324.399706] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 2324.589201] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2326.239889] Key type lgssc registered [ 2327.130742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2334.602635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2334.617512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2336.009196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2340.526825] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug +malloc all [ 2345.445145] LustreError: 75875:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2345.472779] LustreError: 75875:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2349.021471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2349.402910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2354.631461] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug +malloc all [ 2363.108671] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2370.306561] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2377.299032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2377.635453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2380.711981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2385.068637] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug +malloc all [ 2394.241207] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2405.346340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2405.353642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2405.392091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2405.401782] Lustre: Skipped 1 previous similar message [ 2410.466781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2411.409320] Lustre: server umount lustre-OST0000 complete [ 2415.586227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.594914] Lustre: Skipped 1 previous similar message [ 2415.598451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2415.601369] Lustre: Skipped 1 previous similar message [ 2420.705885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2420.717430] Lustre: Skipped 1 previous similar message [ 2421.577852] Lustre: server umount lustre-MDT0000 complete [ 2425.824327] LustreError: 75871:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.842849] LustreError: 75871:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2425.880030] LustreError: 75855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784586512 with bad export cookie 17865599170468495101 [ 2425.889680] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2425.890046] LustreError: 75855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2426.277096] Lustre: server umount lustre-MDT0001 complete [ 2439.945622] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2443.578539] Key type lgssc unregistered [ 2444.004596] LNet: 79410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2444.012769] LNetError: 79410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2444.032534] LNet: Removed LNI 192.168.201.137@tcp [ 2445.030438] Key type .llcrypt unregistered [ 2445.038318] Key type ._llcrypt unregistered [ 2475.295834] Key type ._llcrypt registered [ 2475.297729] Key type .llcrypt registered [ 2475.413810] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2476.777788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2476.826754] alg: No test for adler32 (adler32-zlib) [ 2478.045916] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 2478.423293] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2480.167218] Key type lgssc registered [ 2481.338992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2489.598942] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2489.620508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2505.247494] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2510.367398] LustreError: 80803:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.137@tcp: failed processing log, type 4: rc = -110 [ 2537.119236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2543.669651] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2552.251346] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2560.551196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2560.763920] Lustre: *** cfs_fail_loc=706, val=0*** [ 2561.118835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2561.209037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2566.115907] LustreError: 80829:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2566.138416] LustreError: 80829:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2566.236266] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2571.233613] LustreError: 80828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2572.267614] LustreError: 81888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2572.282574] LustreError: 81888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2575.519292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2575.902557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2581.442622] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2589.980222] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2596.265662] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2601.455088] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2601.458094] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2601.474068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2601.483124] Lustre: Skipped 1 previous similar message [ 2601.494174] Lustre: Skipped 1 previous similar message [ 2606.570317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2606.576216] Lustre: Skipped 1 previous similar message [ 2611.682493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2611.697917] Lustre: Skipped 1 previous similar message [ 2613.215181] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2613.527665] Lustre: server umount lustre-OST0000 complete [ 2621.926780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2621.937396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2621.952023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2624.465166] Lustre: server umount lustre-MDT0000 complete [ 2627.041228] LustreError: 81890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.065268] LustreError: 81890:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2628.563937] LustreError: 80810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784586715 with bad export cookie 1831331659260447985 [ 2628.571584] LustreError: 80810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2628.577345] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2628.940232] Lustre: server umount lustre-MDT0001 complete [ 2642.754526] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2645.821398] Key type lgssc unregistered [ 2646.233752] LNet: 84427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2646.258164] LNetError: 84427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2646.269390] LNet: Removed LNI 192.168.201.137@tcp [ 2647.143324] Key type .llcrypt unregistered [ 2647.145456] Key type ._llcrypt unregistered [ 2676.272767] Key type ._llcrypt registered [ 2676.279725] Key type .llcrypt registered [ 2676.403378] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2677.607595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2677.739454] alg: No test for adler32 (adler32-zlib) [ 2678.734690] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 2678.886403] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2680.535302] Key type lgssc registered [ 2681.979403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2689.503650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2689.529752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2694.779529] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2702.913667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2703.517188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2707.983135] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2708.969732] LustreError: 86530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2708.979855] LustreError: 86530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2714.083467] LustreError: 86534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2714.114804] LustreError: 86534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2718.029931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2718.355289] LustreError: 86530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2718.387429] LustreError: 86530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2718.421897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2723.811701] LustreError: 86533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2723.833620] LustreError: 86533:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2725.643402] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2728.931674] LustreError: 87276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2728.955342] LustreError: 87276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2735.509761] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2735.519191] Lustre: MGS: Client 4092f243-06a5-4aa1-8bf5-64ad033653a5 (at 0@lo) reconnecting [ 2735.544788] LustreError: 86544:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2735.689820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2735.726387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2740.350746] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2756.080630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2756.092151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2756.115514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2756.122329] Lustre: Skipped 1 previous similar message [ 2760.403846] Lustre: server umount lustre-OST0000 complete [ 2766.305110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2766.316666] Lustre: Skipped 2 previous similar messages [ 2766.317083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2771.427838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2771.454024] Lustre: Skipped 2 previous similar messages [ 2771.569958] Lustre: server umount lustre-MDT0000 complete [ 2776.552464] LustreError: 86534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2776.567765] LustreError: 86534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2777.100256] Lustre: server umount lustre-MDT0001 complete [ 2781.594986] Lustre: server umount MGS complete [ 2794.099752] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 18:34:38 (1784586878) [ 2826.708692] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 2838.125777] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2884.613054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2889.626515] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2900.331899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2900.758435] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2900.869880] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2901.051105] Lustre: lustre-MDT0001: new disk, initializing [ 2901.146570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2901.265830] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2901.276284] Lustre: 93707:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2901.300986] LustreError: 93707:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2906.666966] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2918.997063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2919.314925] Lustre: lustre-OST0000: new disk, initializing [ 2919.319472] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2919.331242] Lustre: 94474:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2919.419491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2928.209370] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2930.273190] Lustre: 94868:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000003de02f97 x1871274851738112/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1784587033 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2939.206966] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2946.552692] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2946.563602] Lustre: MGS: Client fbc79481-e459-4861-9174-a9e215c4ad8f (at 0@lo) reconnecting [ 2946.602544] Lustre: 95520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2946.624773] LustreError: 95520:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2946.633376] LustreError: 95520:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2946.651030] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2946.760386] Lustre: lustre-MDT0000: new disk, initializing [ 2946.955244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2946.975735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2947.014953] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2948.071523] Lustre: 95545:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000f6e7790c x1871274851745024/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1784587050 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2951.805497] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2953.125791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2953.135596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2953.361755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2977.762046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2977.767989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2977.773828] Lustre: Skipped 1 previous similar message [ 2977.794064] Lustre: Skipped 1 previous similar message [ 2982.846401] Lustre: server umount lustre-OST0000 complete [ 2988.009519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2988.011462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2988.032603] Lustre: Skipped 1 previous similar message [ 2992.801764] Lustre: server umount lustre-MDT0000 complete [ 2996.201544] LustreError: 93679:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3003.860211] Lustre: server umount lustre-MDT0001 complete [ 3007.744730] Lustre: server umount MGS complete [ 3022.088211] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 18:38:26 (1784587106) [ 3043.185477] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 3046.130300] Key type lgssc unregistered [ 3046.418135] LNet: 98043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3046.429423] LNetError: 98043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3046.451739] LNet: Removed LNI 192.168.201.137@tcp [ 3047.330495] Key type .llcrypt unregistered [ 3047.332301] Key type ._llcrypt unregistered [ 3066.299926] Key type ._llcrypt registered [ 3066.302126] Key type .llcrypt registered [ 3066.411463] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3067.854592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3067.937292] alg: No test for adler32 (adler32-zlib) [ 3069.069807] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 3069.365079] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 3071.095187] Key type lgssc registered [ 3072.360566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3080.539749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3080.558204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3082.042241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3086.496468] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3091.510847] LustreError: 99265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3091.534339] LustreError: 99265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3095.063199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3095.322648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3100.411301] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3111.116495] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3121.276452] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3127.786156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3128.180743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3134.265682] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3142.531265] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3151.739596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3166.178110] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3166.193048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3166.219625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3167.203157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3167.206783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3167.226782] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3167.276213] Lustre: Skipped 1 previous similar message [ 3170.093368] Lustre: server umount lustre-OST0000 complete [ 3177.441545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3177.460566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3177.471562] Lustre: Skipped 1 previous similar message [ 3181.085301] Lustre: server umount lustre-MDT0000 complete [ 3182.560493] LustreError: 99265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3182.583362] LustreError: 99265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3186.604938] LustreError: 99246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784587273 with bad export cookie 17744767623701108688 [ 3186.611475] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3186.627979] LustreError: 99246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3187.294765] Lustre: server umount lustre-MDT0001 complete [ 3200.330895] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 3204.324229] Key type lgssc unregistered [ 3204.749734] LNet: 102825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3204.760064] LNetError: 102825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3205.800347] LNet: Removed LNI 192.168.201.137@tcp [ 3207.067516] Key type .llcrypt unregistered [ 3207.071548] Key type ._llcrypt unregistered [ 3230.221359] Key type ._llcrypt registered [ 3230.228654] Key type .llcrypt registered [ 3230.327412] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3231.744487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3231.766819] alg: No test for adler32 (adler32-zlib) [ 3232.771683] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 3232.977823] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 3234.671362] Key type lgssc registered [ 3236.264169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3242.806095] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3242.814842] LustreError: 103904:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3242.853285] LustreError: 103911:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3242.862048] LustreError: 103904:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4963 [ 3242.871646] LustreError: 103904:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3242.885518] LustreError: 103904:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3242.895549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3244.532419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3253.036068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3253.456774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3258.224843] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3265.283641] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing lsmod [ 3268.809092] LustreError: 105112:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3268.863393] LustreError: 105117:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3268.872225] LustreError: 105112:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4949 [ 3268.884318] LustreError: 105112:0:(obd_mount.c:234:lustre_start_simple()) lustre-OST0000-osd attach error -17 [ 3268.900335] LustreError: 105112:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -17 [ 3268.911415] LustreError: 105112:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 3268.914484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3269.414951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3270.479635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3274.211481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3274.219221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3274.243029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3275.750631] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3275.751526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3275.769138] Lustre: Skipped 1 previous similar message [ 3275.796425] Lustre: Skipped 2 previous similar messages [ 3278.886488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3278.903015] Lustre: Skipped 2 previous similar messages [ 3279.133810] Lustre: server umount lustre-MDT0000 complete [ 3283.511324] LustreError: 103925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784587370 with bad export cookie 412430219000897989 [ 3283.525783] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3283.535412] Lustre: 103940:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3283.944677] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3283.949748] LustreError: 103945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3283.962079] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3283.990915] LustreError: 103945:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3285.087422] LustreError: 104380:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008b17aa0e x1871275432629248/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3285.380812] Lustre: server umount lustre-MDT0001 complete [ 3289.728386] Lustre: server umount lustre-OST0000 complete [ 3298.834508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3299.158842] LustreError: 106181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3299.248656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3303.486461] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3304.416740] LustreError: 106182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3309.540086] LustreError: 106181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3312.766433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3313.061811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3318.050833] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3326.362573] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3331.922741] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3338.438217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3338.772702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3344.957281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3345.336987] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3353.693482] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3366.885494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3366.896039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3366.916486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3368.353552] Lustre: server umount lustre-OST0000 complete [ 3374.563067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3374.572215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3374.577865] Lustre: Skipped 1 previous similar message [ 3377.643382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3377.652159] Lustre: Skipped 3 previous similar messages [ 3379.432427] Lustre: server umount lustre-MDT0000 complete [ 3382.752570] LustreError: 108569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3382.769108] LustreError: 108569:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3382.983187] LustreError: 106161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784587469 with bad export cookie 412430219000898920 [ 3382.985035] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3382.996051] LustreError: 106161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3383.403510] Lustre: server umount lustre-MDT0001 complete [ 3395.119508] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 3398.572926] Key type lgssc unregistered [ 3398.903711] LNet: 109728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3398.910601] LNetError: 109728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3398.938779] LNet: Removed LNI 192.168.201.137@tcp [ 3400.014188] Key type .llcrypt unregistered [ 3400.016095] Key type ._llcrypt unregistered [ 3428.172989] Key type ._llcrypt registered [ 3428.176212] Key type .llcrypt registered [ 3428.276313] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3429.173648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3429.222552] alg: No test for adler32 (adler32-zlib) [ 3430.318508] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 3430.532464] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 3432.224343] Key type lgssc registered [ 3433.664402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3441.010573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3441.037364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3442.426888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3446.836446] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3452.907204] LustreError: 111149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3452.936721] LustreError: 111149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3455.851809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3456.192119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3460.679610] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3469.394576] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3475.513293] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3482.140727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3482.412907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3487.544793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3488.564321] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3496.346968] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3505.534475] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3509.713276] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3516.410628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3516.424105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3516.440589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3520.700451] Lustre: server umount lustre-OST0000 complete [ 3526.625575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3526.636939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3526.662357] Lustre: Skipped 2 previous similar messages [ 3526.669644] Lustre: Skipped 2 previous similar messages [ 3531.069710] Lustre: server umount lustre-MDT0000 complete [ 3531.748013] LustreError: 112391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3531.766381] LustreError: 112391:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3534.251892] LustreError: 111130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784587621 with bad export cookie 16362966849960003364 [ 3534.253229] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3534.265405] LustreError: 111130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3534.639942] Lustre: server umount lustre-MDT0001 complete [ 3548.226485] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 3551.429835] Key type lgssc unregistered [ 3551.742920] LNet: 114789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3551.751558] LNetError: 114789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3551.768347] LNet: Removed LNI 192.168.201.137@tcp [ 3552.600873] Key type .llcrypt unregistered [ 3552.606841] Key type ._llcrypt unregistered [ 3572.866762] Key type ._llcrypt registered [ 3572.872452] Key type .llcrypt registered [ 3573.027949] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3574.569484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3574.641993] alg: No test for adler32 (adler32-zlib) [ 3575.718529] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 3575.958613] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 3577.616286] Key type lgssc registered [ 3579.308707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3589.544545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3589.566539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3591.276801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3595.794055] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3601.897664] LustreError: 116011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3601.928181] LustreError: 116011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3604.469794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3604.888597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3610.367194] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3618.686775] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3624.555227] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3631.074371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3631.581314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3636.730232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3637.709215] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3645.805297] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3660.260299] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3660.263315] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3660.281362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3660.299132] Lustre: Skipped 1 previous similar message [ 3665.301836] Lustre: server umount lustre-OST0000 complete [ 3670.496299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3670.504286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3670.518036] Lustre: Skipped 2 previous similar messages [ 3670.540544] Lustre: Skipped 1 previous similar message [ 3675.482241] Lustre: server umount lustre-MDT0000 complete [ 3675.618064] LustreError: 116007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3675.642547] LustreError: 116007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3678.860479] LustreError: 115993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784587765 with bad export cookie 3901563507060692271 [ 3678.863978] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3678.867325] LustreError: 115993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3679.200475] Lustre: server umount lustre-MDT0001 complete [ 3691.906314] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 3694.570946] Key type lgssc unregistered [ 3694.833826] LNet: 119572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3694.843228] LNetError: 119572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3694.863810] LNet: Removed LNI 192.168.201.137@tcp [ 3695.697227] Key type .llcrypt unregistered [ 3695.704820] Key type ._llcrypt unregistered [ 3719.102778] Key type ._llcrypt registered [ 3719.105481] Key type .llcrypt registered [ 3719.277441] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3720.493927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3720.540746] alg: No test for adler32 (adler32-zlib) [ 3721.683653] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 3721.993223] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 3723.655158] Key type lgssc registered [ 3724.806848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3731.991242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3732.011890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3733.401073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3738.060875] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3743.716499] LustreError: 120991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3743.731730] LustreError: 120991:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3746.048042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3746.314583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3750.578454] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3758.275541] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3763.191521] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3769.269950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3769.651210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3775.817069] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3778.866345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3783.801691] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3790.208760] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3794.151918] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3794.163885] Lustre: Skipped 2 previous similar messages [ 3798.691869] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3798.737498] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3799.014862] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3799.025160] Lustre: Skipped 2 previous similar messages [ 3806.814536] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3806.824442] Lustre: Skipped 1 previous similar message [ 3812.579396] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.37@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3812.583571] Lustre: Skipped 2 previous similar messages [ 3813.566935] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.37@tcp 192.168.0.[2,10]@tcp [ 3823.673872] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.37/24@tcp in log lustre-MDT0000 [ 3823.692660] Lustre: Skipped 2 previous similar messages [ 3829.964124] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.37/24@tcp [ 3829.972563] Lustre: Skipped 1 previous similar message [ 3840.495854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3840.513680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3840.524875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3840.541834] Lustre: Skipped 1 previous similar message [ 3840.576934] Lustre: Skipped 1 previous similar message [ 3844.777525] Lustre: server umount lustre-OST0000 complete [ 3850.723717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.726230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3850.748320] Lustre: Skipped 1 previous similar message [ 3855.447365] Lustre: server umount lustre-MDT0000 complete [ 3855.845357] LustreError: 120992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3855.860306] LustreError: 120992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3859.713594] LustreError: 120971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784587946 with bad export cookie 10397043807634155359 [ 3859.723148] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3859.723921] LustreError: 120971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3860.168380] Lustre: server umount lustre-MDT0001 complete [ 3871.166446] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 3874.847446] Key type lgssc unregistered [ 3875.086606] LNet: 125886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3875.094150] LNetError: 125886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3875.105699] LNet: Removed LNI 192.168.201.137@tcp [ 3876.258418] Key type .llcrypt unregistered [ 3876.260169] Key type ._llcrypt unregistered [ 3907.792985] Key type ._llcrypt registered [ 3907.795502] Key type .llcrypt registered [ 3907.885255] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3908.896451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3908.934722] alg: No test for adler32 (adler32-zlib) [ 3910.040941] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 3910.318936] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 3912.015705] Key type lgssc registered [ 3913.302810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3922.560495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3922.583589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3924.176346] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3924.229482] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.37/24@tcp [ 3924.236587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3929.647236] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3934.698066] LustreError: 127452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3934.748727] LustreError: 127452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3939.809822] LustreError: 127451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3940.247126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3940.851355] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3940.885419] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.37/24@tcp [ 3941.012557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3947.214454] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3956.840328] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3963.240935] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3970.027841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3970.349052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3976.968946] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3985.277268] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3994.993463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4010.464260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4010.469656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4010.476054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4010.480311] Lustre: Skipped 1 previous similar message [ 4014.323200] Lustre: server umount lustre-OST0000 complete [ 4020.705577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4020.728359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4020.736510] Lustre: Skipped 2 previous similar messages [ 4025.073553] Lustre: server umount lustre-MDT0000 complete [ 4025.835290] LustreError: 127451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4025.859627] LustreError: 127451:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4029.866273] LustreError: 127432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784588116 with bad export cookie 15255379899959243988 [ 4029.884107] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4029.886315] LustreError: 127432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4030.481410] Lustre: server umount lustre-MDT0001 complete [ 4042.625423] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4045.033241] Key type lgssc unregistered [ 4045.360468] LNet: 131061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4045.364496] LNetError: 131061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4045.384165] LNet: Removed LNI 192.168.201.137@tcp [ 4046.239372] Key type .llcrypt unregistered [ 4046.242083] Key type ._llcrypt unregistered [ 4071.983111] Key type ._llcrypt registered [ 4071.985432] Key type .llcrypt registered [ 4072.081233] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 4084.862756] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4085.846229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4086.035911] alg: No test for adler32 (adler32-zlib) [ 4087.114689] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 4087.374484] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4089.087894] Key type lgssc registered [ 4090.291371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4127.611479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4127.640870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4128.870383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4128.929443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4129.052351] Lustre: lustre-MDT0000: new disk, initializing [ 4129.149028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4129.184224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4133.715669] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4143.447753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4143.569389] Lustre: 134444:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4143.622851] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4143.626873] Lustre: Skipped 1 previous similar message [ 4143.690324] Lustre: lustre-MDT0001: new disk, initializing [ 4143.766230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4143.800347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4143.810731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4148.091327] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4159.024554] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4164.949711] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4171.284087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4171.560467] Lustre: lustre-OST0000: new disk, initializing [ 4171.563317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4171.567850] Lustre: 136413:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4171.652169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4177.355625] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4177.976771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4177.984912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4178.081511] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4186.532530] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4191.192909] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4191.380693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4195.601699] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4195.800848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4198.888552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4198.902264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4198.907497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4198.913582] Lustre: Skipped 1 previous similar message [ 4204.009852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4204.015197] Lustre: Skipped 2 previous similar messages [ 4207.075630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4207.086086] Lustre: Skipped 1 previous similar message [ 4212.191283] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4212.195697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4212.212473] Lustre: Skipped 1 previous similar message [ 4212.372491] Lustre: server umount lustre-OST0000 complete [ 4217.315282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4217.318697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4217.341228] Lustre: Skipped 1 previous similar message [ 4217.368735] Lustre: Skipped 1 previous similar message [ 4222.590667] Lustre: server umount lustre-MDT0000 complete [ 4226.324405] LustreError: 134437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784588313 with bad export cookie 11460708989473649305 [ 4226.337487] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4226.348699] LustreError: 134437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4226.808858] Lustre: server umount lustre-MDT0001 complete [ 4235.847828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4236.325600] LustreError: 138632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4236.343806] LustreError: 138632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4236.390339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4240.839467] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4241.892148] LustreError: 138633:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4247.015733] LustreError: 138632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4249.885755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4250.335894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4254.395977] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4261.934392] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4268.110975] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4276.278119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4276.687614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4283.020156] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4289.637960] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4296.010653] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4296.335405] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4301.025254] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4301.259893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4310.258051] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4310.439477] Lustre: lustre-OST0001: new disk, initializing [ 4310.443767] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4310.450529] Lustre: 142001:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4310.539503] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4316.799342] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4317.794885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4317.807902] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4317.877768] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4324.917533] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4325.102810] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4329.250328] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4329.448544] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4334.467772] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ea510775000.ost_server_uuid 50 [ 4335.988801] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9ea510775000.ost_server_uuid in FULL state after 0 sec [ 4343.778059] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4343.786400] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4343.824501] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4343.827832] Lustre: Skipped 1 previous similar message [ 4348.866688] Lustre: server umount lustre-OST0001 complete [ 4348.929108] LustreError: 140452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4348.968309] LustreError: 140452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4353.504744] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4353.516166] LustreError: Skipped 1 previous similar message [ 4353.526571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4353.547683] Lustre: Skipped 1 previous similar message [ 4354.020748] LustreError: 140452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4354.054438] LustreError: 140452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4364.263736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4364.272240] LustreError: 143592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4364.274154] Lustre: Skipped 6 previous similar messages [ 4364.300538] LustreError: 143592:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4367.327135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4367.611552] Lustre: server umount lustre-OST0000 complete [ 4373.472726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4373.490619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4373.506587] Lustre: Skipped 1 previous similar message [ 4377.470159] Lustre: server umount lustre-MDT0000 complete [ 4380.826050] LustreError: 138612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784588467 with bad export cookie 11460708989473650187 [ 4380.834089] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4380.839206] LustreError: 138612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4381.125966] Lustre: server umount lustre-MDT0001 complete [ 4401.001905] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4403.519877] Key type lgssc unregistered [ 4403.812118] LNet: 145018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4403.816777] LNetError: 145018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4404.840369] LNet: Removed LNI 192.168.201.137@tcp [ 4405.728407] Key type .llcrypt unregistered [ 4405.730373] Key type ._llcrypt unregistered [ 4445.233672] Key type ._llcrypt registered [ 4445.236333] Key type .llcrypt registered [ 4445.336336] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 4460.674665] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4461.773916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4461.951392] alg: No test for adler32 (adler32-zlib) [ 4463.093157] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 4463.299380] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4464.951222] Key type lgssc registered [ 4466.346569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4502.974241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4503.006762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4504.441656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4504.507495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4504.658613] Lustre: lustre-MDT0000: new disk, initializing [ 4504.776180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4504.804616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4511.161836] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4524.320689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4524.465235] Lustre: 149172:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4524.524512] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4524.528435] Lustre: Skipped 1 previous similar message [ 4524.596181] Lustre: lustre-MDT0001: new disk, initializing [ 4524.781347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4524.866442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4524.878693] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4530.340038] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4541.531064] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4547.882675] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4554.764755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4555.197468] Lustre: lustre-OST0000: new disk, initializing [ 4555.207159] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4555.223773] Lustre: 151145:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4555.364719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4561.362692] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4564.139497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4564.157862] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4564.355278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4570.959922] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4576.213601] Lustre: Failing over lustre-OST0000 [ 4576.498467] Lustre: server umount lustre-OST0000 complete [ 4593.186144] Lustre: 147399:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784588664/real 1784588664] req@00000000298d9b43 x1871276723438848/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1784588680 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4593.217812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4595.679192] Lustre: 147401:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784588666/real 1784588666] req@000000008aff6d0f x1871276723439104/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1784588682 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4595.718613] Lustre: 147401:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4595.742403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4595.957539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4596.331843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4596.350429] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4600.415261] Lustre: 147398:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784588671/real 1784588671] req@00000000050c24f3 x1871276723440896/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784588687 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4600.464774] Lustre: 147398:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4603.494662] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4603.635104] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4605.535167] Lustre: 147399:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784588676/real 1784588676] req@00000000faaeb605 x1871276723442048/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784588692 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4605.579310] Lustre: 147399:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4605.581881] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4605.594229] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4607.621498] Lustre: Failing over lustre-MDT0000 [ 4607.912524] Lustre: server umount lustre-MDT0000 complete [ 4608.545946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4608.569155] Lustre: Skipped 1 previous similar message [ 4609.574523] Lustre: 147399:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784588680/real 1784588680] req@000000004f60fc4c x1871276723443072/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784588696 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4613.605104] LustreError: 150592:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4613.633732] LustreError: 150592:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4615.609594] LustreError: 151437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4618.725679] LustreError: 149185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4618.753568] LustreError: 149185:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4620.727269] LustreError: 153059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4624.864179] Lustre: 147400:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784588695/real 1784588695] req@00000000531c5229 x1871276723454592/t0(0) o400->MGC192.168.201.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784588711 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4624.912843] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4624.928043] LustreError: 149185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4624.945986] LustreError: 149185:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4626.974726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4635.104054] LustreError: 147397:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007f23c74b x1871276723460096/t0(0) o250->MGC192.168.201.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4635.163629] LustreError: 149185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4635.207931] LustreError: 149185:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4635.666797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4635.745561] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4636.089481] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4640.751090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4640.765113] Lustre: Skipped 1 previous similar message [ 4640.795170] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4640.848101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4641.019743] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4650.985263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4651.002415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4651.013752] Lustre: Skipped 2 previous similar messages [ 4656.085454] Lustre: server umount lustre-OST0000 complete [ 4661.234095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4661.240958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4661.241310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4661.263602] Lustre: Skipped 1 previous similar message [ 4661.278081] Lustre: Skipped 2 previous similar messages [ 4666.235877] Lustre: server umount lustre-MDT0000 complete [ 4666.342177] LustreError: 149185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4666.366546] LustreError: 149185:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4670.527588] LustreError: 149166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784588757 with bad export cookie 4418532129966546806 [ 4670.529026] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4670.538115] LustreError: 149166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4671.046800] Lustre: server umount lustre-MDT0001 complete [ 4684.342335] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4688.014355] Key type lgssc unregistered [ 4688.417594] LNet: 154805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4688.434474] LNetError: 154805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4688.449726] LNet: Removed LNI 192.168.201.137@tcp [ 4689.575724] Key type .llcrypt unregistered [ 4689.579210] Key type ._llcrypt unregistered [ 4727.577945] Key type ._llcrypt registered [ 4727.579925] Key type .llcrypt registered [ 4727.662887] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4728.563566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4728.630547] alg: No test for adler32 (adler32-zlib) [ 4729.825262] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 4730.113284] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4731.783216] Key type lgssc registered [ 4733.092111] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4742.289504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4742.322204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4743.734651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4748.576413] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4753.908598] LustreError: 156325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4753.936385] LustreError: 156325:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4758.631383] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4759.014477] LustreError: 156326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4762.761930] LustreError: 156355:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4762.781407] LustreError: 156355:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4762.795860] LustreError: 156355:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4763.271848] Lustre: server umount lustre-MDT0000 complete [ 4778.220928] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 4788.309924] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4830.984074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4831.241879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4831.275389] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4831.357357] Lustre: lustre-MDT0000: new disk, initializing [ 4831.427889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4831.443501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4836.115820] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4847.180863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4847.316422] Lustre: 159660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4847.388545] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4847.396290] Lustre: Skipped 1 previous similar message [ 4847.523326] Lustre: lustre-MDT0001: new disk, initializing [ 4847.611276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4847.679343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4847.696567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4852.036673] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4862.028174] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4868.235851] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4874.543784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4874.788712] Lustre: lustre-OST0000: new disk, initializing [ 4874.792733] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4874.799062] Lustre: 161629:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4874.886705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4880.005152] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4883.050141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4883.059727] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4883.198584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4890.464265] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4895.302768] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4896.822942] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 19:09:42 (1784588982) [ 5103.584960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5103.584971] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5103.588621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5103.615738] Lustre: Skipped 2 previous similar messages [ 5105.807773] Lustre: server umount lustre-MDT0000 complete [ 5108.708658] LustreError: 160573:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5108.728861] LustreError: 160573:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5109.531030] LustreError: 159652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784589196 with bad export cookie 7329172882695429033 [ 5109.539166] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5109.557964] LustreError: 159652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5113.823982] LustreError: 162349:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5113.834414] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5113.881966] Lustre: Skipped 2 previous similar messages [ 5113.887225] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5115.984720] Lustre: server umount lustre-MDT0001 complete [ 5126.472150] Lustre: server umount lustre-OST0000 complete [ 5137.120521] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 5147.283510] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5183.521247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5183.830571] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5183.858648] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5183.954386] Lustre: lustre-MDT0000: new disk, initializing [ 5184.092536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5184.127746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5188.775885] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5199.968522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5200.122811] Lustre: 166285:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5200.192053] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5200.205105] Lustre: Skipped 1 previous similar message [ 5200.366760] Lustre: lustre-MDT0001: new disk, initializing [ 5200.533838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5200.592749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5200.612236] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5205.370511] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5217.294379] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5224.447749] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5232.899850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5233.309753] Lustre: lustre-OST0000: new disk, initializing [ 5233.313507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5233.321822] Lustre: 168253:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5233.460192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5239.843551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5239.851327] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5240.031781] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5240.203208] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5249.669456] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5254.530810] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5254.787896] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5259.735594] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5260.007317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5265.377066] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5265.392652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5265.426384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5265.441773] Lustre: Skipped 1 previous similar message [ 5268.161347] Lustre: server umount lustre-OST0000 complete [ 5276.130804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5276.145058] Lustre: Skipped 1 previous similar message [ 5276.149598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5276.166752] Lustre: Skipped 1 previous similar message [ 5278.610918] Lustre: server umount lustre-MDT0000 complete [ 5281.259311] LustreError: 166291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5281.292597] LustreError: 166291:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5283.355456] LustreError: 166275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784589370 with bad export cookie 7329172882695587863 [ 5283.359801] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5283.372346] LustreError: 166275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5283.918574] Lustre: server umount lustre-MDT0001 complete [ 5293.819097] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:16:18 (1784589378) [ 5310.212518] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 5319.388475] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5355.323660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5355.655638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5355.688491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5355.779651] Lustre: lustre-MDT0000: new disk, initializing [ 5355.876762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5355.899564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5360.224634] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5371.787115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5372.022399] Lustre: 173011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5372.369581] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5377.654441] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5388.506507] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5394.373638] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5401.819487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5402.092556] Lustre: lustre-OST0000: new disk, initializing [ 5402.097784] Lustre: Skipped 1 previous similar message [ 5402.101413] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5402.104282] Lustre: Skipped 2 previous similar messages [ 5402.109595] Lustre: 174989:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5408.344195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5408.355682] Lustre: Skipped 1 previous similar message [ 5408.359735] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5408.496246] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5409.199647] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5418.584662] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5429.220516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5429.221056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5429.223036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5429.234990] LustreError: Skipped 2 previous similar messages [ 5429.270987] Lustre: Skipped 2 previous similar messages [ 5432.157956] Lustre: server umount lustre-OST0000 complete [ 5438.944373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5438.953478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5438.968305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5438.970847] Lustre: Skipped 1 previous similar message [ 5442.872282] Lustre: server umount lustre-MDT0000 complete [ 5444.589158] LustreError: 173022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5444.604433] LustreError: 173022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5447.130094] LustreError: 173810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784589533 with bad export cookie 7329172882695588738 [ 5447.132730] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5447.144818] LustreError: 173810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5447.914840] Lustre: server umount lustre-MDT0001 complete [ 5456.558855] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:19:01 (1784589541) [ 5473.964669] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 5484.885462] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5522.733894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5523.009404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5523.044244] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5523.138595] Lustre: lustre-MDT0000: new disk, initializing [ 5523.242776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5523.254565] Lustre: Skipped 2 previous similar messages [ 5523.271721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5528.703941] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5539.030768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5539.114389] Lustre: 179313:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5539.305248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5543.624614] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5552.555136] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5558.130754] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5564.579166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5564.870148] Lustre: 181288:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5570.702749] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5571.145568] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5580.619375] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5591.010590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5591.011954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5591.018803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5591.018812] Lustre: Skipped 2 previous similar messages [ 5591.033840] Lustre: Skipped 2 previous similar messages [ 5594.431981] Lustre: server umount lustre-OST0000 complete [ 5600.739579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5604.798490] Lustre: server umount lustre-MDT0000 complete [ 5606.371334] LustreError: 179325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5608.305159] LustreError: 179304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784589695 with bad export cookie 7329172882695589830 [ 5608.311965] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5608.324878] LustreError: 179304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5608.866725] Lustre: server umount lustre-MDT0001 complete [ 5622.767840] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 5626.472391] Key type lgssc unregistered [ 5627.095807] LNet: 183148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5627.103147] LNetError: 183148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5627.117135] LNet: Removed LNI 192.168.201.137@tcp [ 5628.480293] Key type .llcrypt unregistered [ 5628.485273] Key type ._llcrypt unregistered [ 5653.085181] Key type ._llcrypt registered [ 5653.089974] Key type .llcrypt registered [ 5653.293218] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5654.441871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5654.579739] alg: No test for adler32 (adler32-zlib) [ 5655.741974] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 5656.028957] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 5657.737237] Key type lgssc registered [ 5658.979789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5667.160772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5667.189561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5668.623910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5672.636469] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5679.074727] LustreError: 184562:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5679.121074] LustreError: 184562:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5680.110360] LustreError: 184563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5681.005696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5681.293152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5686.043549] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5693.073440] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5698.458542] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5705.515231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5705.779406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5712.216755] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5715.013286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5720.282620] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5742.561152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5742.569925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5742.591986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5745.639035] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5745.645738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5745.655017] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5745.669043] Lustre: Skipped 1 previous similar message [ 5747.216205] Lustre: server umount lustre-OST0000 complete [ 5752.808765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5752.824978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5752.867237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5755.876172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5755.879225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5757.842398] Lustre: server umount lustre-MDT0000 complete [ 5760.996852] LustreError: 184563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5761.011462] LustreError: 184563:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5763.104690] LustreError: 184543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784589849 with bad export cookie 8756163716176778979 [ 5763.109652] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5763.115210] LustreError: 184543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5763.646898] Lustre: server umount lustre-MDT0001 complete [ 5776.587268] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 5780.019797] Key type lgssc unregistered [ 5780.405617] LNet: 188116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5780.419209] LNetError: 188116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5780.449236] LNet: Removed LNI 192.168.201.137@tcp [ 5781.369109] Key type .llcrypt unregistered [ 5781.372813] Key type ._llcrypt unregistered [ 5811.077427] Key type ._llcrypt registered [ 5811.079640] Key type .llcrypt registered [ 5811.236707] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5813.043281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5813.068053] alg: No test for adler32 (adler32-zlib) [ 5814.021066] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 5814.201182] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 5815.847631] Key type lgssc registered [ 5816.859557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5826.740442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5826.758907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5828.177253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5834.149784] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5837.284417] LustreError: 189538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5837.326802] LustreError: 189538:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5842.401233] LustreError: 189539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5845.449444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5845.834723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5850.281145] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5856.819430] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5863.122862] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5869.886104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5870.256087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5873.363214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5877.606978] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5885.537320] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5893.593653] Lustre: lustre-OST0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 5894.112309] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5894.120792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5894.137183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5894.150922] Lustre: Skipped 1 previous similar message [ 5894.172944] Lustre: Skipped 1 previous similar message [ 5898.609178] Lustre: server umount lustre-OST0000 complete [ 5906.656092] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9ea506a73800.ost_server_uuid 50 [ 5908.606989] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5924.840737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5924.846151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5924.870513] Lustre: Skipped 1 previous similar message [ 5924.908500] Lustre: Skipped 1 previous similar message [ 5929.496357] Lustre: server umount lustre-MDT0000 complete [ 5929.952707] LustreError: 189538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5929.981158] LustreError: 189538:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5933.340865] LustreError: 189518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784590020 with bad export cookie 7505549672763925453 [ 5933.348262] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5933.348766] LustreError: 189518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5933.683632] Lustre: server umount lustre-MDT0001 complete [ 5943.545001] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 19:27:08 (1784590028) [ 5949.959505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5950.252253] LustreError: 193453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5950.307081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5954.662939] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5955.558305] LustreError: 193454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5963.612629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5964.044202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5968.184855] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5975.709455] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5981.124128] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5988.728256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5989.086874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5995.259990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5995.414639] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6003.023312] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6010.024326] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6010.332891] Lustre: lustre-OST0001: new disk, initializing [ 6010.335454] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6010.343410] Lustre: 196294:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6010.428196] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6016.399782] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6017.106634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6017.119569] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6017.232568] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6026.062482] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6029.282930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6029.296667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6029.326424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6031.328377] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6031.338846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6036.937291] Lustre: lustre-OST0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 6036.954622] Lustre: Skipped 4 previous similar messages [ 6043.104661] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6043.395955] Lustre: server umount lustre-OST0000 complete [ 6047.156758] LustreError: 195623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6047.177570] LustreError: 195623:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6053.015681] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6053.179434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6060.012440] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6060.340202] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6063.589117] LustreError: 195276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6063.611471] LustreError: 195276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6078.951911] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6078.955683] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6078.974507] Lustre: Skipped 1 previous similar message [ 6078.993395] Lustre: Skipped 6 previous similar messages [ 6081.780658] Lustre: server umount lustre-OST0001 complete [ 6087.137030] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6087.153682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6091.949450] Lustre: server umount lustre-MDT0000 complete [ 6095.413759] LustreError: 193436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784590182 with bad export cookie 7505549672763926524 [ 6095.421288] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6095.436353] LustreError: 193436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6095.916691] Lustre: server umount lustre-MDT0001 complete [ 6122.502781] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 19:30:07 (1784590207) [ 6130.146338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6130.342115] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6130.360217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6130.489129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6134.480610] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6143.425545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6143.547821] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6143.595930] Lustre: 199735:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6149.390375] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6157.794359] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6163.652712] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6171.209049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6171.402786] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6178.399775] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6178.963946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6186.339610] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6193.717408] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6193.826582] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6200.476168] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6207.762970] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6213.628575] Lustre: lustre-OST0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 6213.634225] Lustre: Skipped 3 previous similar messages [ 6214.159489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6214.183240] Lustre: Skipped 2 previous similar messages [ 6216.798285] Lustre: server umount lustre-OST0000 complete [ 6218.690658] LustreError: 201589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6218.722938] LustreError: 201589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6239.202184] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6241.129418] Lustre: server umount lustre-OST0001 complete [ 6246.376381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6246.388851] Lustre: Skipped 2 previous similar messages [ 6246.398191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6246.406427] Lustre: Skipped 4 previous similar messages [ 6251.496404] Lustre: server umount lustre-MDT0000 complete [ 6255.988227] LustreError: 199727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784590342 with bad export cookie 7505549672763927931 [ 6255.993801] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6256.002322] LustreError: 199727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6256.318965] Lustre: server umount lustre-MDT0001 complete [ 6281.225471] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 19:32:46 (1784590366) [ 6300.539495] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 6313.396919] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6353.094327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6353.366635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6353.417851] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6353.489926] Lustre: lustre-MDT0000: new disk, initializing [ 6353.590217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6353.604984] Lustre: Skipped 3 previous similar messages [ 6353.625560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6358.777883] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6370.564583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6370.741431] Lustre: 207834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6370.793312] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6370.798362] Lustre: Skipped 1 previous similar message [ 6370.873818] Lustre: lustre-MDT0001: new disk, initializing [ 6371.029468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6371.037975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6376.239764] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6386.573557] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6392.556899] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6400.982903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6401.466855] Lustre: lustre-OST0000: new disk, initializing [ 6401.472782] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6401.499783] Lustre: 209800:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6408.390275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6408.412557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6408.578211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6409.675161] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6421.212908] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6426.321382] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6426.502881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6432.695671] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6433.014606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6439.392941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6439.394286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6439.415393] LustreError: Skipped 3 previous similar messages [ 6439.420947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6439.420957] Lustre: Skipped 2 previous similar messages [ 6439.458089] Lustre: Skipped 2 previous similar messages [ 6442.520048] Lustre: server umount lustre-OST0000 complete [ 6453.249454] Lustre: server umount lustre-MDT0000 complete [ 6454.757966] LustreError: 207845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6454.789698] LustreError: 207845:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 6457.429575] LustreError: 207825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784590544 with bad export cookie 7505549672763929492 [ 6457.435921] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6457.446920] LustreError: 207825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6457.917870] Lustre: server umount lustre-MDT0001 complete [ 6467.221021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6472.657322] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6483.167647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6483.701587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6483.704654] Lustre: Skipped 3 previous similar messages [ 6488.395950] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6497.839685] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6505.027858] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6513.801844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6521.743460] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6530.819491] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6535.833337] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6539.437974] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6544.594494] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6544.964374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6553.659970] Lustre: server umount lustre-OST0000 complete [ 6562.045648] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6567.724712] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6598.184827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6606.554707] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6614.742714] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6615.082516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6615.127649] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6615.134933] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6615.147085] Lustre: Skipped 1 previous similar message [ 6616.767940] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6625.251758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6625.260523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6625.265158] Lustre: Skipped 5 previous similar messages [ 6625.276880] Lustre: Skipped 7 previous similar messages [ 6628.566808] Lustre: server umount lustre-OST0000 complete [ 6638.982900] Lustre: server umount lustre-MDT0000 complete [ 6640.609360] LustreError: 215767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6640.651384] LustreError: 215767:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6642.751327] LustreError: 212001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784590729 with bad export cookie 7505549672763930374 [ 6642.752210] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6642.777951] LustreError: 212001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6643.184279] Lustre: server umount lustre-MDT0001 complete [ 6653.631986] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 19:38:58 (1784590738) [ 6661.804630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6666.830148] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6675.398461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6682.763208] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6692.029604] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6698.342620] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6706.599377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6713.303386] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6722.013055] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6726.750503] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6726.996605] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6732.101684] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6732.305247] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6739.406035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6739.711494] Lustre: lustre-OST0001: new disk, initializing [ 6739.717749] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6739.728177] Lustre: 221377:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6739.826039] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6739.831694] Lustre: Skipped 5 previous similar messages [ 6746.565650] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6747.801183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6747.824810] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6747.966362] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6757.868176] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6763.217672] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6763.452861] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6767.911553] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6768.209499] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6773.734918] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6773.750523] LustreError: Skipped 2 previous similar messages [ 6776.816501] Lustre: server umount lustre-OST0001 complete [ 6784.562845] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6784.802824] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6790.045144] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6790.281290] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6818.936952] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6824.429178] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6824.445901] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6824.453161] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6825.380756] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6833.086470] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6834.587409] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6839.784108] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6842.748806] Lustre: server umount lustre-OST0001 complete [ 6866.309368] Lustre: server umount lustre-OST0000 complete [ 6885.346102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6885.628773] Lustre: server umount lustre-MDT0000 complete [ 6889.256107] LustreError: 217989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784590976 with bad export cookie 7505549672763931340 [ 6889.267908] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6889.273066] LustreError: 217989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6895.956476] Lustre: server umount lustre-MDT0001 complete [ 6923.572845] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 19:43:28 (1784591008) [ 6932.659829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6933.346214] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6933.423354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6938.799956] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6948.606495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6948.670599] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6948.700185] Lustre: 226627:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6954.070363] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6962.428992] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6969.263038] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6977.258153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6977.453236] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6986.173175] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6995.365915] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7006.617369] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7006.778492] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7016.829267] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7025.962530] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7027.915502] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7034.310452] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7034.558618] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7040.984628] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7041.361556] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7049.196983] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ea504709000.ost_server_uuid 50 [ 7051.108583] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9ea504709000.ost_server_uuid in IDLE state after 0 sec [ 7052.382849] Lustre: Permanently deactivating lustre-OST0001 [ 7052.403814] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7056.850934] Lustre: Permanently reactivating lustre-OST0001 [ 7056.880943] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7056.883691] Lustre: Skipped 2 previous similar messages [ 7062.524221] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7062.542308] LustreError: Skipped 1 previous similar message [ 7062.551995] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7062.580917] Lustre: Skipped 10 previous similar messages [ 7062.612726] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7062.630613] Lustre: Skipped 16 previous similar messages [ 7076.319433] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7076.500814] Lustre: server umount lustre-OST0001 complete [ 7078.883487] LustreError: 228475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7078.924602] LustreError: 228475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [ 7095.263299] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7095.567822] Lustre: server umount lustre-OST0000 complete [ 7106.239803] Lustre: server umount lustre-MDT0000 complete [ 7110.783966] LustreError: 226621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784591197 with bad export cookie 7505549672763932628 [ 7110.788985] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7110.799712] LustreError: 226621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7111.145479] Lustre: server umount lustre-MDT0001 complete [ 7137.797753] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 19:47:02 (1784591222) [ 7145.629682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7145.919330] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7145.965844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7145.975392] Lustre: Skipped 2 previous similar messages [ 7150.800996] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7160.074393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7160.179674] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7160.205985] Lustre: 233329:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7165.070772] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7172.301889] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7178.288656] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7185.824644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7186.196395] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7186.248508] Lustre: Permanently deactivating lustre-OST0000 [ 7186.263556] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7193.015839] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7195.195224] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7200.091289] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7207.076592] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7207.142097] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7213.316464] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7214.048968] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7222.654563] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7224.330930] Lustre: Permanently reactivating lustre-OST0000 [ 7224.352970] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7224.356338] Lustre: Skipped 2 previous similar messages [ 7231.058928] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7231.067134] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 35 seconds. I think it's dead, and I am evicting it. exp 0000000059baf4c4, cur 1784591317 deadline 1784591312 last 1784591282 [ 7231.073085] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7231.101632] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7231.116144] Lustre: Skipped 1 previous similar message [ 7231.170045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7239.145074] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7239.150994] LustreError: Skipped 4 previous similar messages [ 7244.707889] Lustre: server umount lustre-OST0001 complete [ 7255.694298] Lustre: server umount lustre-OST0000 complete [ 7266.337924] Lustre: server umount lustre-MDT0000 complete [ 7270.499700] LustreError: 233322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784591357 with bad export cookie 7505549672763934294 [ 7270.512548] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7270.518526] LustreError: 233322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7271.241110] Lustre: server umount lustre-MDT0001 complete [ 7283.127131] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7285.758654] Key type lgssc unregistered [ 7285.989544] LNet: 238210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7285.998743] LNetError: 238210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7286.029414] LNet: Removed LNI 192.168.201.137@tcp [ 7287.002472] Key type .llcrypt unregistered [ 7287.004398] Key type ._llcrypt unregistered [ 7310.459697] Key type ._llcrypt registered [ 7310.462440] Key type .llcrypt registered [ 7310.554215] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7311.450126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7311.498256] alg: No test for adler32 (adler32-zlib) [ 7312.604154] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 7312.862128] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 7314.511270] Key type lgssc registered [ 7315.419183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7325.525761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7325.556760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7327.147159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7330.971682] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7336.423681] LustreError: 239692:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7336.446861] LustreError: 239692:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7340.787500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7340.907800] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7340.952132] Lustre: 239679:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7340.964552] Lustre: Permanently deactivating lustre-MDT0001 [ 7340.982963] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7341.528074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7346.728294] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7347.127833] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7356.546663] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7362.277397] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7370.116982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7370.484331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7371.499024] LustreError: 241587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7371.521138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7376.904477] LustreError: 241587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7377.323771] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7381.986599] LustreError: 241535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7386.092192] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7387.118934] LustreError: 241587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7388.155061] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 7395.032460] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7395.404884] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7400.440785] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7400.456479] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7403.107644] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7411.723896] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7414.767624] Lustre: Permanently reactivating lustre-MDT0001 [ 7414.787825] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7414.794508] Lustre: Skipped 2 previous similar messages [ 7418.343176] Lustre: lustre-MDT0001: Client 45813bd1-5562-4cef-bbba-136fea3600ab (at 192.168.201.37@tcp) reconnecting [ 7423.558329] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7423.566041] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7423.574839] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7423.586676] LustreError: 243406:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7423.594789] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7423.595714] LustreError: 243406:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7423.595887] LustreError: 243406:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.137@tcp: cfg command failed: rc = -108 [ 7423.596040] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7423.596040] [ 7423.597549] LustreError: 239684:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7424.927956] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7426.676684] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7432.437409] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7432.686916] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7438.413560] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7438.654423] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7441.269399] Lustre: Permanently deactivating lustre-MDT0001 [ 7441.286374] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7441.293258] Lustre: Skipped 2 previous similar messages [ 7449.677668] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7464.419506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7464.427726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7464.440730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7464.942222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7464.950443] Lustre: Skipped 2 previous similar messages [ 7466.547362] Lustre: server umount lustre-MDT0000 complete [ 7470.059499] LustreError: 242254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7470.098154] LustreError: 242254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 7470.574642] LustreError: 239673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784591557 with bad export cookie 2707246530232483027 [ 7470.579801] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7470.590625] LustreError: 239673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7470.829538] Lustre: server umount lustre-MDT0001 complete [ 7477.054290] Lustre: server umount lustre-OST0000 complete [ 7487.853879] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 19:52:53 (1784591573) [ 7499.184752] Lustre: server umount lustre-OST0001 complete [ 7506.227325] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 7517.233702] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7553.820682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7554.052840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7554.062416] Lustre: Skipped 2 previous similar messages [ 7554.079532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7554.143901] Lustre: lustre-MDT0000: new disk, initializing [ 7554.228229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7554.286970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7558.804658] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7570.009890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7570.184258] Lustre: 248003:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7570.241330] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7570.245388] Lustre: Skipped 1 previous similar message [ 7570.320721] Lustre: lustre-MDT0001: new disk, initializing [ 7570.428220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7570.471086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7570.491719] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7575.743095] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7586.960548] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7593.509346] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7601.182269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7601.557623] Lustre: lustre-OST0000: new disk, initializing [ 7601.563649] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7601.574613] Lustre: 249970:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7601.682329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7608.571842] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7611.478418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7611.496635] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7611.600113] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7618.058766] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7620.011345] LustreError: 249977:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7628.682650] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7628.842894] Lustre: lustre-OST0001: new disk, initializing [ 7628.846322] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7628.852403] Lustre: 251085:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7630.040213] LustreError: 249977:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7630.089374] LustreError: 248009:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7635.022497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7635.033509] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7635.168406] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7635.322707] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7640.145600] LustreError: 248009:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7644.422884] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7646.207458] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7650.787126] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7650.789360] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7650.792048] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7650.799750] Lustre: Skipped 4 previous similar messages [ 7650.818021] LustreError: Skipped 1 previous similar message [ 7654.707883] Lustre: server umount lustre-OST0001 complete [ 7655.923894] LustreError: 249954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7655.945504] LustreError: 249954:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7673.313490] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7673.323396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7673.347583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7673.351746] Lustre: Skipped 1 previous similar message [ 7678.198571] Lustre: server umount lustre-OST0000 complete [ 7683.040956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7683.047503] LustreError: Skipped 1 previous similar message [ 7683.058059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7683.077876] Lustre: Skipped 1 previous similar message [ 7683.086686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7683.094028] Lustre: Skipped 2 previous similar messages [ 7691.746445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7691.756260] Lustre: Skipped 3 previous similar messages [ 7696.352788] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7696.866842] LustreError: 248014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7696.894745] LustreError: 248014:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 7696.912101] Lustre: server umount lustre-MDT0000 complete [ 7701.159218] LustreError: 248796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784591788 with bad export cookie 2707246530232486072 [ 7701.174430] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7701.179481] LustreError: 248796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7707.836109] Lustre: server umount lustre-MDT0001 complete [ 7718.813621] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7721.362822] Key type lgssc unregistered [ 7721.656263] LNet: 253118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7721.674210] LNetError: 253118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7721.687493] LNet: Removed LNI 192.168.201.137@tcp [ 7722.670554] Key type .llcrypt unregistered [ 7722.672234] Key type ._llcrypt unregistered [ 7762.347843] Key type ._llcrypt registered [ 7762.351891] Key type .llcrypt registered [ 7762.438545] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7763.837812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7763.866625] alg: No test for adler32 (adler32-zlib) [ 7764.833420] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 7765.031470] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 7766.695177] Key type lgssc registered [ 7767.738071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7775.486000] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7775.497584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7776.753612] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7776.765730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7776.870673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7781.394606] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7789.702989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7789.943350] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7789.970159] Lustre: 255327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7790.441352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7795.204523] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7802.414393] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7808.496875] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7815.489506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7815.661939] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7815.805476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7820.768287] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7825.472328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7827.034410] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7881.702853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7881.704533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7881.712117] LustreError: Skipped 1 previous similar message [ 7881.712923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7881.743107] Lustre: Skipped 1 previous similar message [ 7883.043823] Lustre: server umount lustre-OST0000 complete [ 7887.049399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7898.382229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7898.707936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7901.808739] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7901.818376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7901.824368] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7905.303610] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7912.144198] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7953.385316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7953.386265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7953.397840] LustreError: Skipped 1 previous similar message [ 7953.400576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7953.413689] Lustre: Skipped 1 previous similar message [ 7953.420667] Lustre: Skipped 1 previous similar message [ 7956.610760] Lustre: server umount lustre-OST0000 complete [ 7963.623161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7963.626732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7963.635399] Lustre: Skipped 1 previous similar message [ 7963.637922] Lustre: Skipped 1 previous similar message [ 7965.878751] Lustre: server umount lustre-MDT0000 complete [ 7968.894038] LustreError: 257190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784592055 with bad export cookie 14770607853074789785 [ 7968.900681] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7968.909908] LustreError: 257190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7969.169140] Lustre: server umount lustre-MDT0001 complete [ 7979.125189] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7981.615959] Key type lgssc unregistered [ 7981.888482] LNet: 260764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7981.896946] LNetError: 260764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7981.931886] LNet: Removed LNI 192.168.201.137@tcp [ 7982.741542] Key type .llcrypt unregistered [ 7982.744310] Key type ._llcrypt unregistered [ 8002.054834] Key type ._llcrypt registered [ 8002.057494] Key type .llcrypt registered [ 8002.146347] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 8003.057718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8003.117884] alg: No test for adler32 (adler32-zlib) [ 8004.132879] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 8004.315848] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 8005.983423] Key type lgssc registered [ 8006.949667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8013.591674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8013.614864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8015.020531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8018.946419] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8024.548030] LustreError: 262163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8024.567958] LustreError: 262163:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8026.926392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8027.299477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8031.305887] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8037.703523] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8043.531599] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8049.028317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8049.299688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8054.995112] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8059.565688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8061.670374] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8083.426292] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8083.429533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8083.443282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8085.486507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8085.493486] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8085.493914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8088.124705] Lustre: server umount lustre-OST0000 complete [ 8093.665308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8093.675156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8093.690249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8093.700060] Lustre: Skipped 1 previous similar message [ 8095.712321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8095.713078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8095.736778] Lustre: Skipped 1 previous similar message [ 8097.143921] Lustre: server umount lustre-MDT0000 complete [ 8100.834667] LustreError: 262162:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8100.852963] LustreError: 262162:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8100.936504] LustreError: 265511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784592187 with bad export cookie 815339566648540132 [ 8100.941796] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8100.950810] LustreError: 265511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8101.246750] Lustre: server umount lustre-MDT0001 complete [ 8110.624652] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 8112.980117] Key type lgssc unregistered [ 8113.267656] LNet: 266464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8113.275981] LNetError: 266464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8113.302972] LNet: Removed LNI 192.168.201.137@tcp [ 8114.007529] Key type .llcrypt unregistered [ 8114.010724] Key type ._llcrypt unregistered [ 8127.065341] Key type ._llcrypt registered [ 8127.068666] Key type .llcrypt registered [ 8127.143325] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 8127.954628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8127.988358] alg: No test for adler32 (adler32-zlib) [ 8128.994597] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 8129.174891] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 8130.831809] Key type lgssc registered [ 8131.659130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8136.606562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8136.624992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8138.021282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8141.381224] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8147.489726] LustreError: 267662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8147.505509] LustreError: 267662:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8147.982651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8148.200418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8151.638971] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8157.512283] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8161.754846] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8166.760669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8167.040123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8171.538639] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8177.032128] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8182.383577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8188.896678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8188.903737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8188.921714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8192.995309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8193.001178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8193.007420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8194.796725] Lustre: server umount lustre-OST0000 complete [ 8198.118354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8198.127068] Lustre: Skipped 1 previous similar message [ 8198.129375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8198.143312] Lustre: Skipped 2 previous similar messages [ 8203.233309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8203.240216] Lustre: Skipped 1 previous similar message [ 8203.679961] Lustre: server umount lustre-MDT0000 complete [ 8206.308079] LustreError: 269505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784592293 with bad export cookie 2489485617118411686 [ 8206.310119] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8206.317219] LustreError: 269505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8206.640518] Lustre: server umount lustre-MDT0001 complete [ 8214.773182] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 8216.741202] Key type lgssc unregistered [ 8216.965592] LNet: 271337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8216.972091] LNetError: 271337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8218.025988] LNet: Removed LNI 192.168.201.137@tcp [ 8218.656994] Key type .llcrypt unregistered [ 8218.659125] Key type ._llcrypt unregistered [ 8235.798465] Key type ._llcrypt registered [ 8235.801589] Key type .llcrypt registered [ 8235.883235] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 8236.549394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8236.583617] alg: No test for adler32 (adler32-zlib) [ 8237.587759] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 8237.776672] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 8239.415232] Key type lgssc registered [ 8240.164729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8245.456677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8245.478583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8246.750517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8250.219947] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8256.203531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8256.454699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8259.387861] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8264.387927] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8268.227417] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8272.628498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8272.849723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8276.634261] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8278.003752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8281.071791] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8297.442239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8297.449641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8297.454795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8297.459209] Lustre: Skipped 1 previous similar message [ 8302.566138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8302.570684] Lustre: Skipped 2 previous similar messages [ 8303.008816] Lustre: server umount lustre-OST0000 complete [ 8307.681795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8307.682340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8307.684959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8307.689874] Lustre: Skipped 1 previous similar message [ 8312.287240] Lustre: server umount lustre-MDT0000 complete [ 8312.800405] LustreError: 276083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8312.816428] LustreError: 276083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8315.001275] LustreError: 272714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784592401 with bad export cookie 43452802867869108 [ 8315.015709] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8315.016884] LustreError: 272714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8315.203783] Lustre: server umount lustre-MDT0001 complete [ 8320.749361] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 8322.289092] Key type lgssc unregistered [ 8322.485368] LNet: 277036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8322.491242] LNetError: 277036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8322.505755] LNet: Removed LNI 192.168.201.137@tcp [ 8323.058669] Key type .llcrypt unregistered [ 8323.062042] Key type ._llcrypt unregistered [ 8335.836189] Key type ._llcrypt registered [ 8335.839387] Key type .llcrypt registered [ 8335.951531] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 8336.675120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8336.689764] alg: No test for adler32 (adler32-zlib) [ 8337.654306] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 8337.837154] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 8339.471140] Key type lgssc registered [ 8340.209774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8345.396275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8345.413757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8346.780140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8349.562457] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8355.833870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8356.062985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8358.812611] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8364.279085] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8368.174853] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8372.604130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8372.906638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8376.352902] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8381.195467] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8391.344599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8401.888325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8401.891528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8401.897426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8401.898415] Lustre: Skipped 1 previous similar message [ 8404.250650] Lustre: server umount lustre-OST0000 complete [ 8407.010358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8407.019934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8407.026086] Lustre: Skipped 2 previous similar messages [ 8412.132129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8412.137773] Lustre: Skipped 1 previous similar message [ 8417.253544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8417.257652] Lustre: Skipped 1 previous similar message [ 8421.343608] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8421.345611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8421.360028] Lustre: Skipped 1 previous similar message [ 8421.502473] Lustre: server umount lustre-MDT0000 complete [ 8424.262112] LustreError: 278217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784592511 with bad export cookie 12419589106463575766 [ 8424.271026] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8424.273853] LustreError: 278217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8424.491589] Lustre: server umount lustre-MDT0001 complete [ 8431.485295] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 8433.378949] Key type lgssc unregistered [ 8433.573497] LNet: 281970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8433.587575] LNetError: 281970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8433.608094] LNet: Removed LNI 192.168.201.137@tcp [ 8434.244986] Key type .llcrypt unregistered [ 8434.247237] Key type ._llcrypt unregistered [ 8440.754721] Key type ._llcrypt registered [ 8440.757256] Key type .llcrypt registered [ 8440.853684] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8450.881380] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 8460.240388] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 8460.827788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8460.883575] alg: No test for adler32 (adler32-zlib) [ 8461.837706] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 8462.007944] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 8463.631537] Key type lgssc registered [ 8464.388580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8487.363539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8487.374311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8488.487955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8488.500900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8488.550871] Lustre: lustre-MDT0000: new disk, initializing [ 8488.594639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8488.606307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8491.331450] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8498.382921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8498.482309] Lustre: 285497:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8498.524752] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8498.528853] Lustre: Skipped 1 previous similar message [ 8498.575136] Lustre: lustre-MDT0001: new disk, initializing [ 8498.628417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8498.649418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8498.657492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8501.443542] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8507.573234] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8511.196428] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8515.698681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8515.889922] Lustre: lustre-OST0000: new disk, initializing [ 8515.893396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8515.898299] Lustre: 287467:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8515.955355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8519.810311] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8522.796618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8522.802976] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8522.902895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8527.097606] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8530.298023] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8530.476558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8533.179981] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8533.330678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8538.592042] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8538.594822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8538.600737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8538.610033] Lustre: Skipped 1 previous similar message [ 8538.619505] Lustre: Skipped 1 previous similar message [ 8540.840303] Lustre: server umount lustre-OST0000 complete [ 8543.713565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8543.722849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8543.725532] Lustre: Skipped 1 previous similar message [ 8543.732428] Lustre: Skipped 1 previous similar message [ 8548.837254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8548.844223] Lustre: Skipped 1 previous similar message [ 8549.623651] Lustre: server umount lustre-MDT0000 complete [ 8552.589420] LustreError: 286302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784592639 with bad export cookie 7459638959742550618 [ 8552.591833] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8552.604658] LustreError: 286302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8552.882857] Lustre: server umount lustre-MDT0001 complete [ 8559.187525] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 20:10:44 (1784592644) [ 8564.282353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8564.586563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8567.064459] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8569.827127] LustreError: 289879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8569.843780] LustreError: 289879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8572.312146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8572.527177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8574.869352] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8580.387816] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8584.321953] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8589.555000] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8589.799136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8593.966544] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8598.958825] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8606.177171] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8606.180319] LustreError: Skipped 1 previous similar message [ 8606.182384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8606.188997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8608.228117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8608.228798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8611.973119] Lustre: server umount lustre-OST0000 complete [ 8614.494887] Lustre: server umount lustre-MDT0000 complete [ 8616.815246] LustreError: 290612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784592703 with bad export cookie 7459638959742551500 [ 8616.822015] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8616.975392] Lustre: server umount lustre-MDT0001 complete [ 8623.821954] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 8625.425726] Key type lgssc unregistered [ 8625.625883] LNet: 293408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8625.629948] LNetError: 293408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8625.648159] LNet: Removed LNI 192.168.201.137@tcp [ 8626.141710] Key type .llcrypt unregistered [ 8626.143961] Key type ._llcrypt unregistered [ 8642.724364] Key type ._llcrypt registered [ 8642.726752] Key type .llcrypt registered [ 8642.795151] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 8651.671132] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 8652.241746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8652.264256] alg: No test for adler32 (adler32-zlib) [ 8653.187953] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 8653.336172] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 8654.975169] Key type lgssc registered [ 8655.675745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8680.655576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8680.676552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8681.874705] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8681.896589] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8681.959472] Lustre: lustre-MDT0000: new disk, initializing [ 8682.005688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8682.015956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8684.609372] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8691.362571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8691.416296] Lustre: 297215:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8691.459570] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8691.462707] Lustre: Skipped 1 previous similar message [ 8691.526784] Lustre: lustre-MDT0001: new disk, initializing [ 8691.584351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8691.613629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8691.620814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8694.267854] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8700.674436] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8704.551738] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8708.926591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8709.094903] Lustre: lustre-OST2710: new disk, initializing [ 8709.097898] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8709.102816] Lustre: 299181:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8709.187650] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8712.577553] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8715.796139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8715.800767] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8715.867538] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8718.607637] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8722.937865] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8723.057835] Lustre: lustre-OST03e8: new disk, initializing [ 8723.062801] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8723.068309] Lustre: 300246:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8723.148181] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8726.866289] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8728.598952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8728.603643] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8728.691420] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8731.341693] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8736.776509] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8736.922587] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8739.830903] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8739.973496] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8742.564189] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8742.646222] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8745.189784] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8745.341475] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8748.001097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8748.004559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8748.021522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8749.027881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8749.033241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8749.038296] Lustre: Skipped 3 previous similar messages [ 8753.565477] Lustre: server umount lustre-MDT0000 complete [ 8755.942145] LustreError: 297208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784592842 with bad export cookie 14732911567186281994 [ 8755.951591] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8755.956187] LustreError: 297208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8756.223651] Lustre: server umount lustre-MDT0001 complete [ 8768.566838] Lustre: server umount lustre-OST2710 complete [ 8781.832056] Lustre: server umount lustre-OST03e8 complete [ 8792.372167] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 8797.635205] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 8818.841030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8819.034378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8819.049695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8819.099200] Lustre: lustre-MDT0000: new disk, initializing [ 8819.147971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8819.157925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8821.391787] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8827.385684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8827.436078] Lustre: 305490:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8827.464413] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8827.466878] Lustre: Skipped 1 previous similar message [ 8827.506987] Lustre: lustre-MDT0001: new disk, initializing [ 8827.552416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8827.574142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8827.579169] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8829.828884] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8835.472468] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8838.882419] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8843.016200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8843.179210] Lustre: 307457:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8846.963105] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8851.031905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8851.039787] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8851.103602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8852.751443] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8855.594944] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8855.739573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8858.134216] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8858.266123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8861.668062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8861.668418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8861.672365] LustreError: Skipped 1 previous similar message [ 8861.681071] Lustre: Skipped 2 previous similar messages [ 8861.688936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8865.401618] Lustre: server umount lustre-OST0000 complete [ 8868.832194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8868.837885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8868.851924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8868.857102] Lustre: Skipped 1 previous similar message [ 8873.729739] Lustre: server umount lustre-MDT0000 complete [ 8876.067483] LustreError: 305481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784592962 with bad export cookie 14732911567186283674 [ 8876.075602] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8876.274792] Lustre: server umount lustre-MDT0001 complete [ 8881.155295] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 20:16:06 (1784592966) [ 8881.921510] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8882.846937] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 20:16:08 (1784592968) [ 8885.060685] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 8901.476142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8901.605930] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8901.617555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8901.704757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8901.707982] Lustre: Skipped 1 previous similar message [ 8904.172157] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8908.870775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8908.943974] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8908.956519] Lustre: 311312:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8911.181374] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 8928.809038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8928.939563] Lustre: Denying initial registration attempt from nid 192.168.201.137@tcp, specified as failover [ 8928.945901] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8928.951357] LustreError: 312761:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8928.955547] LustreError: 312761:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8928.958839] LustreError: 312761:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8929.000501] Lustre: server umount lustre-OST0000 complete [ 8929.003435] LustreError: 312761:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8934.880786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8934.888286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8934.893946] Lustre: Skipped 1 previous similar message [ 8934.897290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8934.901033] Lustre: Skipped 2 previous similar messages [ 8938.768734] Lustre: server umount lustre-MDT0000 complete [ 8941.140459] LustreError: 311303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784593027 with bad export cookie 14732911567186284549 [ 8941.149961] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8941.151538] LustreError: 311303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8941.368306] Lustre: server umount lustre-MDT0001 complete [ 8947.591141] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 8949.176459] Key type lgssc unregistered [ 8949.339754] LNet: 313882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8949.343872] LNetError: 313882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8949.353095] LNet: Removed LNI 192.168.201.137@tcp [ 8949.750209] Key type .llcrypt unregistered [ 8949.751712] Key type ._llcrypt unregistered [ 8956.973480] Key type ._llcrypt registered [ 8956.975898] Key type .llcrypt registered [ 8957.047498] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 8957.865808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8957.883249] alg: No test for adler32 (adler32-zlib) [ 8958.870850] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 8959.037233] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 8960.663171] Key type lgssc registered [ 8961.340604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8992.397750] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 8997.468960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8997.481246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8998.634744] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8998.646992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8998.728368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9001.558520] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9006.558857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9006.620503] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9006.636534] Lustre: 316620:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9006.763393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9008.855093] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9012.503612] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9015.148875] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9031.601972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9031.781379] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9031.913366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9035.726345] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9040.083135] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9042.403957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9042.404077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9042.407319] LustreError: Skipped 1 previous similar message [ 9042.413311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9042.417831] Lustre: Skipped 1 previous similar message [ 9043.461631] Lustre: server umount lustre-OST0000 complete [ 9047.521029] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9047.523480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9047.529203] Lustre: Skipped 2 previous similar messages [ 9052.643084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9052.651479] Lustre: Skipped 2 previous similar messages [ 9057.763825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9057.768286] Lustre: Skipped 1 previous similar message [ 9060.319929] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9060.453537] Lustre: server umount lustre-MDT0000 complete [ 9062.699665] LustreError: 316613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784593149 with bad export cookie 14513976826574859253 [ 9062.700543] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9062.705981] LustreError: 316613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9062.911819] Lustre: server umount lustre-MDT0001 complete [ 9068.867188] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 9070.418606] Key type lgssc unregistered [ 9070.653633] LNet: 320191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9070.656600] LNetError: 320191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9070.665903] LNet: Removed LNI 192.168.201.137@tcp [ 9071.147831] Key type .llcrypt unregistered [ 9071.149186] Key type ._llcrypt unregistered [ 9084.827161] Key type ._llcrypt registered [ 9084.833834] Key type .llcrypt registered [ 9084.900861] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 9085.506094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9085.594133] alg: No test for adler32 (adler32-zlib) [ 9086.620575] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 9086.801620] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 9088.401097] Key type lgssc registered [ 9089.058803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9094.180108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9094.191735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9095.424364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9098.121633] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9103.161304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9103.388532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9106.130281] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9110.801417] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9113.743267] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9119.591968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9119.771205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9122.948429] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9126.445258] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9139.139368] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 9139.168174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9139.170360] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9139.176899] Lustre: Skipped 2 previous similar messages [ 9142.970761] Lustre: server umount lustre-MDT0000 complete [ 9144.246262] LustreError: 321584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9144.255540] LustreError: 321584:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9144.740928] LustreError: 321567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784593231 with bad export cookie 16604653644094398836 [ 9144.744698] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9144.747490] LustreError: 321567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9149.364492] LustreError: 321585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9149.381168] LustreError: 321585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9149.386572] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [ 9149.392993] Lustre: Skipped 3 previous similar messages [ 9149.408631] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9154.483833] LustreError: 321583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9154.483853] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [ 9154.498127] LustreError: 321583:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9154.501374] Lustre: Skipped 1 previous similar message [ 9159.135145] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9159.235930] Lustre: server umount lustre-MDT0001 complete [ 9162.375785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9166.809704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9169.888093] LustreError: 320896:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008905eb98 x1871281570999936/t0(0) o250->MGC192.168.201.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9170.102340] LustreError: 323417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9170.114475] LustreError: 323417:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9170.154854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9170.189266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9172.552383] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9175.520553] LustreError: 323418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9177.795163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9177.997994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9180.073759] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9180.174812] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9184.167322] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9184.418287] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9186.484568] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 9189.855311] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9191.018701] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9194.471488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9194.474099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9194.484943] Lustre: Skipped 1 previous similar message [ 9194.490085] Lustre: Skipped 1 previous similar message [ 9199.589124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9199.594573] Lustre: Skipped 2 previous similar messages [ 9208.287142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9208.399723] Lustre: server umount lustre-OST0000 complete [ 9213.923648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9213.927873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9213.938242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9213.940894] Lustre: Skipped 2 previous similar messages [ 9216.623222] Lustre: server umount lustre-MDT0000 complete [ 9218.656230] LustreError: 321568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784593305 with bad export cookie 16604653644094405976 [ 9218.659353] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9218.663264] LustreError: 321568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9218.849252] Lustre: server umount lustre-MDT0001 complete [ 9225.154349] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 9226.874291] Key type lgssc unregistered [ 9227.044502] LNet: 327852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9227.051241] LNetError: 327852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9227.069165] LNet: Removed LNI 192.168.201.137@tcp [ 9227.595113] Key type .llcrypt unregistered [ 9227.596671] Key type ._llcrypt unregistered [ 9242.187831] Key type ._llcrypt registered [ 9242.189578] Key type .llcrypt registered [ 9242.257185] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 9242.841137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9242.911311] alg: No test for adler32 (adler32-zlib) [ 9243.877151] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 9244.028849] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 9245.663158] Key type lgssc registered [ 9246.374164] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9251.448916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9251.477213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9252.755743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9255.605597] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9261.127551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9261.337545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9263.895069] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9268.628846] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9272.396334] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9276.901253] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9276.903829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9276.916522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9276.928457] Lustre: Skipped 1 previous similar message [ 9281.234571] Lustre: server umount lustre-MDT0000 complete [ 9282.020803] LustreError: 330863:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9282.037763] LustreError: 330863:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9283.423725] LustreError: 329229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784593370 with bad export cookie 13727794057580581893 [ 9283.426235] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9283.430904] LustreError: 329229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9283.643537] Lustre: server umount lustre-MDT0001 complete [ 9292.698692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9292.863218] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9292.884116] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9293.002544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9295.774070] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9301.320974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9301.398877] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9301.431454] Lustre: 331833:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9301.620957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9304.677937] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9309.787419] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9314.066911] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9321.001959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9321.145993] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9321.151763] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9321.156816] LustreError: 333715:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9321.161348] LustreError: 333715:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9321.164962] LustreError: 333715:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9321.198518] Lustre: server umount lustre-OST0000 complete [ 9321.200327] LustreError: 333715:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9328.624490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9328.767086] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9328.908594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9332.606199] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9335.349489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9336.629718] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9341.525384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9341.582749] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9341.643000] Lustre: lustre-OST0001: new disk, initializing [ 9341.647240] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9341.652909] Lustre: 335255:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9341.714183] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9342.958361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9342.963322] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9343.019715] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9345.380165] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9349.555398] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9353.189085] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9353.190332] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9353.194352] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9353.201285] Lustre: Skipped 2 previous similar messages [ 9353.223387] LustreError: Skipped 1 previous similar message [ 9357.472515] Lustre: server umount lustre-OST0001 complete [ 9358.308259] LustreError: 334181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9358.321954] LustreError: 334181:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9360.864635] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9360.864933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9360.867927] LustreError: Skipped 1 previous similar message [ 9360.878377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9360.887545] Lustre: Skipped 2 previous similar messages [ 9363.432830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9363.434467] LustreError: 334180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9363.436430] Lustre: Skipped 1 previous similar message [ 9363.454424] LustreError: 334180:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9368.556316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9368.559706] LustreError: 336261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9368.561775] Lustre: Skipped 1 previous similar message [ 9368.575892] LustreError: 336261:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9373.668431] LustreError: 334180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9373.683193] LustreError: 334180:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9374.176288] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9374.308607] Lustre: server umount lustre-OST0000 complete [ 9378.272429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9378.280406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9378.285801] Lustre: Skipped 1 previous similar message [ 9378.288109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9378.293859] Lustre: Skipped 2 previous similar messages [ 9391.071268] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9391.196225] Lustre: server umount lustre-MDT0000 complete [ 9394.122093] LustreError: 331825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784593480 with bad export cookie 13727794057580582432 [ 9394.123256] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9394.128480] LustreError: 331825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9394.431541] Lustre: server umount lustre-MDT0001 complete [ 9401.382465] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 9403.283444] Key type lgssc unregistered [ 9403.518233] LNet: 337215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9403.529125] LNetError: 337215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9404.583399] LNet: Removed LNI 192.168.201.137@tcp [ 9405.160868] Key type .llcrypt unregistered [ 9405.163098] Key type ._llcrypt unregistered [ 9449.005801] Key type ._llcrypt registered [ 9449.007254] Key type .llcrypt registered [ 9449.070701] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 9457.742670] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 9458.550481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9458.566058] alg: No test for adler32 (adler32-zlib) [ 9459.483965] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 9459.676835] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 9461.287176] Key type lgssc registered [ 9461.886574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9480.931698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9480.944856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9482.076368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9482.091317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9482.151890] Lustre: lustre-MDT0000: new disk, initializing [ 9482.219647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9482.234320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9484.639834] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9491.165868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9491.249490] Lustre: 342122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9491.295055] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9491.300996] Lustre: Skipped 1 previous similar message [ 9491.351414] Lustre: lustre-MDT0001: new disk, initializing [ 9491.422238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9491.446215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9491.451233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9494.593073] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9500.440346] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9503.912069] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9507.718324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9507.882841] Lustre: lustre-OST0000: new disk, initializing [ 9507.885782] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9507.891734] Lustre: 344088:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9507.956939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9511.476945] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9513.490283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9513.495388] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9513.582704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9517.531705] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9520.008576] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9520.133914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9522.668801] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9522.843601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9524.191912] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9524.199582] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9524.213963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9528.803337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9528.804878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9528.810536] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9528.817116] Lustre: Skipped 1 previous similar message [ 9530.234630] Lustre: server umount lustre-OST0000 complete [ 9533.921211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9533.924254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9533.931912] Lustre: Skipped 1 previous similar message [ 9533.945636] Lustre: Skipped 1 previous similar message [ 9538.873209] Lustre: server umount lustre-MDT0000 complete [ 9541.161192] LustreError: 342112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784593628 with bad export cookie 9766771972782900971 [ 9541.164234] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9541.172567] LustreError: 342112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9541.439506] Lustre: server umount lustre-MDT0001 complete [ 9547.425520] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 20:27:12 (1784593632) [ 9552.167498] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 20:27:17 (1784593637) [ 9556.457456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9556.701495] LustreError: 346844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9556.716698] LustreError: 346844:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9556.747474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9559.331661] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9562.085236] LustreError: 346845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9563.933071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9564.134402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9566.213185] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9570.110386] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9573.284927] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9577.314258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9577.556980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9581.389364] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9585.715075] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9595.820357] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9596.739747] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9597.692813] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9598.690702] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9599.625728] Lustre: Failing over lustre-MDT0000 [ 9599.816535] Lustre: server umount lustre-MDT0000 complete [ 9599.968544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9599.971280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9599.975391] LustreError: 346845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9603.526969] LustreError: 346841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9604.583079] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9604.597181] Lustre: Skipped 1 previous similar message [ 9608.628222] LustreError: 346840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9608.639206] LustreError: 346840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9615.730391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9615.837164] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9615.992373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9616.046869] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9618.880100] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9619.151780] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9621.482116] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9621.496093] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9621.522849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9625.444142] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9626.464425] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9628.914877] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9630.047750] Lustre: Failing over lustre-MDT0000 [ 9630.230606] Lustre: server umount lustre-MDT0000 complete [ 9631.713107] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9631.713735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9631.718067] LustreError: 346839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9631.718078] LustreError: 346839:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 9631.722270] Lustre: Skipped 2 previous similar messages [ 9646.141783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9646.233127] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9646.430576] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9649.085963] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9649.591354] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9651.693771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9651.700593] Lustre: Skipped 2 previous similar messages [ 9651.715411] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 9651.745279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9655.144775] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9656.182763] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9661.922159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9661.922612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9661.926619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9661.949020] Lustre: Skipped 1 previous similar message [ 9667.041929] LustreError: 346840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9667.056931] LustreError: 346840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 9667.706810] Lustre: server umount lustre-MDT0000 complete [ 9669.813280] LustreError: 346825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784593756 with bad export cookie 9766771972782903666 [ 9669.821666] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9669.822953] LustreError: 346825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9670.034447] Lustre: server umount lustre-MDT0001 complete [ 9682.557640] Lustre: server umount lustre-OST0000 complete [ 9690.220758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9690.467419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9690.470462] Lustre: Skipped 1 previous similar message [ 9692.886516] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9698.255525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9700.781349] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9705.037583] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9708.102775] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9711.774953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9715.473520] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9716.144748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9719.476412] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9726.432387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9726.436817] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9726.442852] Lustre: Skipped 1 previous similar message [ 9726.446186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9726.448683] Lustre: Skipped 2 previous similar messages [ 9732.261506] Lustre: server umount lustre-OST0000 complete [ 9734.628609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9734.631448] Lustre: Skipped 2 previous similar messages [ 9740.218304] Lustre: server umount lustre-MDT0000 complete [ 9741.945457] LustreError: 353152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784593828 with bad export cookie 9766771972782904415 [ 9741.949350] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9741.955610] LustreError: 353152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9742.113629] Lustre: server umount lustre-MDT0001 complete [ 9747.430302] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 9749.105154] Key type lgssc unregistered [ 9749.305084] LNet: 356698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9749.308820] LNetError: 356698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9749.317711] LNet: Removed LNI 192.168.201.137@tcp [ 9749.851778] Key type .llcrypt unregistered [ 9749.853790] Key type ._llcrypt unregistered [ 9764.347789] Key type ._llcrypt registered [ 9764.349393] Key type .llcrypt registered [ 9764.423896] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 9764.880593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9764.887613] alg: No test for adler32 (adler32-zlib) [ 9765.855714] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 9766.010730] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 9767.631227] Key type lgssc registered [ 9768.202052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9772.356817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9772.369106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9773.596917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9775.675642] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9779.685218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9779.865840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9781.762722] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9784.894042] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9787.219684] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9790.426092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9790.621300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9793.604950] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9797.480259] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9798.826276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9815.524845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9815.527771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9815.529773] Lustre: Skipped 2 previous similar messages [ 9815.536232] Lustre: Skipped 1 previous similar message [ 9818.811965] Lustre: server umount lustre-MDT0000 complete [ 9820.643231] LustreError: 358189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9820.653963] LustreError: 358189:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9820.869630] LustreError: 360792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784593907 with bad export cookie 13073403961589735005 [ 9820.869961] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9820.877606] LustreError: 360792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9821.043973] Lustre: server umount lustre-MDT0001 complete [ 9832.246618] Lustre: server umount lustre-OST0000 complete [ 9839.662992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9839.884416] LustreError: 361922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9839.917067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9842.298086] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9845.220709] LustreError: 361923:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9847.013558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9847.244239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9849.318101] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9853.127166] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9856.158708] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9859.110267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9859.260649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9861.821051] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9864.431705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9865.151675] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9872.864881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9872.868343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9872.874794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9873.892805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9873.893668] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9873.899080] Lustre: Skipped 1 previous similar message [ 9873.909864] Lustre: Skipped 1 previous similar message [ 9878.644573] Lustre: server umount lustre-MDT0000 complete [ 9879.012391] LustreError: 361923:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9879.021569] LustreError: 361923:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9880.635324] LustreError: 361902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784593967 with bad export cookie 13073403961589736097 [ 9880.637744] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9880.640611] LustreError: 361902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9882.079300] LustreError: 362650:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007813232b x1871282283003136/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9882.150294] Lustre: server umount lustre-MDT0001 complete [ 9894.136503] Lustre: server umount lustre-OST0000 complete [ 9902.508690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9902.780930] LustreError: 365828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9902.819250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9904.905876] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9909.414794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9911.832709] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9915.942865] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9919.090452] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9922.983471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9926.219599] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 9927.274138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9929.656268] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9932.772350] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9932.772685] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9932.776877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9932.781703] Lustre: Skipped 1 previous similar message [ 9933.366775] Lustre: server umount lustre-OST0000 complete [ 9935.327926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9937.369468] Lustre: server umount lustre-MDT0000 complete [ 9939.440373] LustreError: 365810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784594026 with bad export cookie 13073403961589737105 [ 9939.446498] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9941.640631] Lustre: server umount lustre-MDT0001 complete [ 9945.996363] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 9947.288713] Key type lgssc unregistered [ 9947.430446] LNet: 369353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9947.433575] LNetError: 369353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9947.441370] LNet: Removed LNI 192.168.201.137@tcp [ 9947.900330] Key type .llcrypt unregistered [ 9947.905248] Key type ._llcrypt unregistered [ 9959.371587] Key type ._llcrypt registered [ 9959.372804] Key type .llcrypt registered [ 9959.425643] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 9959.813542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9959.888073] alg: No test for adler32 (adler32-zlib) [ 9960.798464] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 9960.940769] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 9962.543155] Key type lgssc registered [ 9962.999321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9966.600776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9966.613308] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9966.622705] LustreError: 370770:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9966.638717] LustreError: 370770:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9966.642649] LustreError: 370770:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9966.645961] LustreError: 370770:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9966.649452] LustreError: 370770:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9969.963058] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9969.966142] LustreError: 371169:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9969.975632] LustreError: 371169:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9969.978966] LustreError: 371169:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9969.981809] LustreError: 371169:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9969.985257] LustreError: 371169:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9979.220190] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 9980.530374] Key type lgssc unregistered [ 9980.698261] LNet: 372170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9980.704480] LNetError: 372170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9980.717673] LNet: Removed LNI 192.168.201.137@tcp [ 9981.080142] Key type .llcrypt unregistered [ 9981.081673] Key type ._llcrypt unregistered [ 9989.379194] Key type ._llcrypt registered [ 9989.380575] Key type .llcrypt registered [ 9989.431363] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 9997.072988] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 9997.567919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9997.600848] alg: No test for adler32 (adler32-zlib) [ 9998.494733] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 9998.612101] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [10000.207130] Key type lgssc registered [10000.717956] Lustre: Echo OBD driver; http://www.lustre.org/ [10017.615281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10017.627097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10018.754747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10018.770972] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10018.826614] Lustre: lustre-MDT0000: new disk, initializing [10018.881615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10018.892242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10021.042930] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10026.934160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10026.986114] Lustre: 375301:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10027.017798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10027.020771] Lustre: Skipped 1 previous similar message [10027.059211] Lustre: lustre-MDT0001: new disk, initializing [10027.111693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10027.138565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10027.142187] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10028.898728] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10033.715704] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10036.172459] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10039.058711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10039.172809] Lustre: lustre-OST0000: new disk, initializing [10039.174738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10039.177849] Lustre: 377266:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10039.213853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10041.639600] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10045.915701] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10047.615984] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10048.513108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10048.525403] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10048.560483] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10048.746700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [10050.558907] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10050.648944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10053.603069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10053.604022] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10053.611630] LustreError: Skipped 1 previous similar message [10053.613554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10053.618571] Lustre: Skipped 1 previous similar message [10057.677471] Lustre: server umount lustre-OST0000 complete [10062.815972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10062.819482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10062.824975] Lustre: Skipped 1 previous similar message [10062.827453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10063.840340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10063.840747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10065.731247] Lustre: server umount lustre-MDT0000 complete [10067.379435] LustreError: 375294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784594154 with bad export cookie 11761920166233847142 [10067.384746] LustreError: 375294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10067.386936] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10067.533921] Lustre: server umount lustre-MDT0001 complete [10071.031785] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10071.741040] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 20:35:57 (1784594157) [10075.087493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10075.334650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10077.184428] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10080.647803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10080.736558] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10080.739875] Lustre: Skipped 1 previous similar message [10080.803789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10082.921901] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10087.158234] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10090.051881] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10093.034720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10093.195474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10095.916271] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10098.944082] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10101.852066] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10101.921581] Lustre: lustre-OST0001: new disk, initializing [10101.923749] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10101.927039] Lustre: 382565:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10102.535771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10102.539189] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10102.574097] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10104.339330] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10108.452790] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10110.947340] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10110.948875] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10110.956766] Lustre: Skipped 1 previous similar message [10116.485747] Lustre: server umount lustre-OST0001 complete [10121.188831] LustreError: 381549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10121.195174] LustreError: 381549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10126.308045] LustreError: 382502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10126.315355] LustreError: 382502:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10131.426391] LustreError: 381549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10131.426554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10131.436767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10131.438432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10131.438438] Lustre: Skipped 4 previous similar messages [10131.439274] LustreError: 381549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10136.548533] LustreError: 381549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10136.555833] LustreError: 381549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10141.666708] LustreError: 381552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10141.675791] LustreError: 381552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10143.199132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10143.260745] Lustre: server umount lustre-OST0000 complete [10146.784962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10146.790986] Lustre: Skipped 1 previous similar message [10151.068756] Lustre: server umount lustre-MDT0000 complete [10151.905142] LustreError: 383375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10151.911447] LustreError: 383375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10152.554102] LustreError: 379714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784594239 with bad export cookie 11761920166233848010 [10152.556024] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10152.560920] LustreError: 379714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10158.725153] Lustre: server umount lustre-MDT0001 complete [10163.382599] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [10164.767982] Key type lgssc unregistered [10164.932411] LNet: 384557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10164.935657] LNetError: 384557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10164.945254] LNet: Removed LNI 192.168.201.137@tcp [10165.381526] Key type .llcrypt unregistered [10165.382694] Key type ._llcrypt unregistered [10188.171583] Key type ._llcrypt registered [10188.173197] Key type .llcrypt registered [10188.230578] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [10188.664071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10188.676152] alg: No test for adler32 (adler32-zlib) [10189.566183] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [10189.687694] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [10191.297184] Key type lgssc registered [10191.796210] Lustre: Echo OBD driver; http://www.lustre.org/ [10195.076125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10195.083455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10196.206163] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10196.215362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10196.281211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10197.934903] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10201.556231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10201.612378] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10201.627548] Lustre: 387329:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10201.742204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10203.387063] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10206.578184] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10209.137073] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10210.784519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10210.785497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10210.789734] Lustre: Skipped 1 previous similar message [10215.904254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10215.906867] Lustre: Skipped 2 previous similar messages [10216.163891] Lustre: server umount lustre-MDT0000 complete [10217.837928] LustreError: 387322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784594304 with bad export cookie 12458881380693140730 [10217.840619] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10217.843153] LustreError: 387322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10217.975347] Lustre: server umount lustre-MDT0001 complete [10221.863803] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 20:38:27 (1784594307) [10224.635120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10224.845591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10226.416137] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10230.242430] LustreError: 389841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10230.248866] LustreError: 389841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10230.403111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10230.606089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10232.343346] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10235.254360] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10237.606139] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10240.715755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10240.804328] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10240.877589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10243.204409] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10245.925496] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10255.247517] Lustre: Permanently deactivating lustre-OST0000 [10255.258160] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10258.106513] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10258.714716] LustreError: 392726:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10258.718963] LustreError: 392726:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [10258.721083] LustreError: 392726:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10259.401537] LustreError: 392774:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [10259.404438] LustreError: 392774:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10266.202401] Lustre: server umount lustre-OST0000 complete [10267.273839] LustreError: 393023:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [10267.276962] LustreError: 393023:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10270.689320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10270.689898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10270.693845] Lustre: Skipped 1 previous similar message [10275.807989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10275.810276] Lustre: Skipped 2 previous similar messages [10277.880349] Lustre: server umount lustre-MDT0000 complete [10279.256812] LustreError: 390572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784594366 with bad export cookie 12458881380693141297 [10279.257838] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10279.261413] LustreError: 390572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10289.687428] Lustre: server umount lustre-MDT0001 complete [10292.757724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10294.316337] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10295.248792] LustreError: 394182:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10297.585516] Lustre: 394520:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.137@tcp [10298.338117] Lustre: server umount MGS complete [10302.533025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10302.690184] LustreError: 395242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10302.695329] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10302.700385] Lustre: Skipped 1 previous similar message [10302.722657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10304.186378] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10307.266944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10307.388377] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10309.111640] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10312.278372] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10314.496209] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10315.292839] Lustre: Permanently reactivating lustre-OST0000 [10315.298607] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10315.302465] Lustre: Skipped 2 previous similar messages [10342.367124] LustreError: 397071:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10346.000439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10346.156514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10346.160973] Lustre: Skipped 1 previous similar message [10348.569727] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10351.481669] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10366.944046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10366.947195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10366.952439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10367.971266] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10370.129308] Lustre: server umount lustre-OST0000 complete [10373.088773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10373.094914] Lustre: Skipped 2 previous similar messages [10377.629415] Lustre: server umount lustre-MDT0000 complete [10378.208070] LustreError: 397120:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10378.215313] LustreError: 397120:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10379.057442] LustreError: 396816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784594465 with bad export cookie 12458881380693142333 [10379.061278] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10379.061849] LustreError: 396816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10379.203572] Lustre: server umount lustre-MDT0001 complete [10383.423843] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [10384.620523] Key type lgssc unregistered [10384.757498] LNet: 399910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10384.761329] LNetError: 399910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10384.771465] LNet: Removed LNI 192.168.201.137@tcp [10385.117083] Key type .llcrypt unregistered [10385.118631] Key type ._llcrypt unregistered [10392.459596] Key type ._llcrypt registered [10392.461086] Key type .llcrypt registered [10392.502192] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [10399.405249] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [10399.842481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10399.850815] alg: No test for adler32 (adler32-zlib) [10400.731503] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [10400.835729] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [10402.439152] Key type lgssc registered [10402.930442] Lustre: Echo OBD driver; http://www.lustre.org/ [10417.697418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10417.713126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10418.824136] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10418.843794] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10418.886457] Lustre: lustre-MDT0000: new disk, initializing [10418.931332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10418.939816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10420.694123] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10425.716455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10425.768024] Lustre: 403041:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10425.800240] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10425.803275] Lustre: Skipped 1 previous similar message [10425.842256] Lustre: lustre-MDT0001: new disk, initializing [10425.895535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10425.915720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10425.924213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10427.571899] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10432.207481] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10434.468617] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10437.348246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10437.510488] Lustre: lustre-OST0000: new disk, initializing [10437.515348] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10437.520598] Lustre: 405007:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10437.569901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10439.958138] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10443.816873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10443.822696] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10443.876193] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10444.448405] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10446.222368] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10446.299725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10447.927024] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10448.011617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10448.866761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10448.867100] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10448.873693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10448.874877] LustreError: Skipped 1 previous similar message [10448.877134] Lustre: Skipped 1 previous similar message [10450.904838] Lustre: server umount lustre-OST0000 complete [10453.985593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10453.987098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10453.990271] Lustre: Skipped 2 previous similar messages [10453.994893] Lustre: Skipped 1 previous similar message [10458.464719] Lustre: server umount lustre-MDT0000 complete [10460.094234] LustreError: 403833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784594546 with bad export cookie 8304947723415742721 [10460.097231] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10460.098420] LustreError: 403833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10460.236343] Lustre: server umount lustre-MDT0001 complete [10463.328754] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10464.039217] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 20:42:29 (1784594549) [10466.989292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10467.215550] LustreError: 407462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10467.222648] LustreError: 407462:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10467.244358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10468.905623] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10472.420509] LustreError: 407463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10472.668642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10472.847537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10474.718587] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10477.783962] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10480.237303] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10483.318450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10483.540771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10486.447924] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10490.089644] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10490.905225] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10498.035523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10503.136968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10503.137632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10503.140626] LustreError: Skipped 1 previous similar message [10503.141848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10503.145700] Lustre: Skipped 1 previous similar message [10505.760138] Lustre: server umount lustre-OST0000 complete [10508.257698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10508.265431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10508.268844] Lustre: Skipped 1 previous similar message [10508.271022] Lustre: Skipped 1 previous similar message [10513.376589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10513.380304] Lustre: Skipped 1 previous similar message [10513.436324] Lustre: server umount lustre-MDT0000 complete [10514.898459] LustreError: 409304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784594601 with bad export cookie 8304947723415743589 [10514.901501] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10514.908503] LustreError: 409304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10515.022289] Lustre: server umount lustre-MDT0001 complete [10519.187373] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [10520.341065] Key type lgssc unregistered [10520.506519] LNet: 411185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10520.509625] LNetError: 411185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10520.520362] LNet: Removed LNI 192.168.201.137@tcp [10520.884102] Key type .llcrypt unregistered [10520.885504] Key type ._llcrypt unregistered [10529.940560] Key type ._llcrypt registered [10529.942462] Key type .llcrypt registered [10529.988060] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [10530.097302] Key type .llcrypt unregistered [10530.098437] Key type ._llcrypt unregistered [10538.220848] Key type ._llcrypt registered [10538.223385] Key type .llcrypt registered [10538.298799] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [10538.795967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10538.819844] alg: No test for adler32 (adler32-zlib) [10539.706339] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [10539.816175] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [10541.399169] Key type lgssc registered [10541.887103] Lustre: Echo OBD driver; http://www.lustre.org/ [10545.562299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10545.572093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10546.768758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10548.499901] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10552.214166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10552.361262] LustreError: 413369:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10552.366037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10552.369465] LustreError: 413369:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10554.952063] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10556.449415] LustreError: 413374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10556.456357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10556.457866] LustreError: 413374:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10557.899576] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10560.673335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10560.849596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10562.549244] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10571.237963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10571.239091] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10571.240546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10571.242354] Lustre: Skipped 1 previous similar message [10576.353635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10576.357779] Lustre: Skipped 2 previous similar messages [10576.882355] Lustre: server umount lustre-OST0000 complete [10580.963321] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10580.966262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10580.971831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10586.592406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10586.597924] Lustre: Skipped 3 previous similar messages [10591.712559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10591.714535] Lustre: Skipped 1 previous similar message [10593.247300] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10593.325361] Lustre: server umount lustre-MDT0000 complete [10594.741489] LustreError: 413353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784594681 with bad export cookie 3082474582446118361 [10594.744338] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10594.750206] LustreError: 413353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10594.878706] Lustre: server umount lustre-MDT0001 complete [10599.179850] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [10600.401977] Key type lgssc unregistered [10600.539699] LNet: 416520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10600.543071] LNetError: 416520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10600.552421] LNet: Removed LNI 192.168.201.137@tcp [10600.940728] Key type .llcrypt unregistered [10600.942718] Key type ._llcrypt unregistered [10611.006980] Key type ._llcrypt registered [10611.008387] Key type .llcrypt registered [10611.058493] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [10611.443778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10611.452822] alg: No test for adler32 (adler32-zlib) [10612.320621] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [10612.430218] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [10614.031280] Key type lgssc registered [10614.423209] Lustre: Echo OBD driver; http://www.lustre.org/ [10617.342756] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10617.349344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10632.799412] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10637.919321] LustreError: 417886:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.137@tcp: failed processing log, type 4: rc = -110 [10664.543324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10666.975988] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10670.114273] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10672.735029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10672.946405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10672.972629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10674.587414] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10678.213851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10678.241095] LustreError: 417912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10678.248077] LustreError: 417912:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10678.376851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10679.891494] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10682.846456] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10684.922546] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10688.480346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10688.483637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10688.488465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10688.490709] Lustre: Skipped 1 previous similar message [10690.018264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10690.021059] Lustre: Skipped 1 previous similar message [10693.666241] Lustre: server umount lustre-OST0000 complete [10695.136689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10695.137361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10695.141900] Lustre: Skipped 2 previous similar messages [10700.256269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10700.260921] Lustre: Skipped 2 previous similar messages [10701.205171] Lustre: server umount lustre-MDT0000 complete [10702.826382] LustreError: 417892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784594789 with bad export cookie 15377282668707346734 [10702.828534] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10702.831768] LustreError: 417892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10702.985188] Lustre: server umount lustre-MDT0001 complete [10706.930808] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [10708.078597] Key type lgssc unregistered [10708.223331] LNet: 421454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10708.225745] LNetError: 421454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10708.236451] LNet: Removed LNI 192.168.201.137@tcp [10708.589656] Key type .llcrypt unregistered [10708.591375] Key type ._llcrypt unregistered [10718.578890] Key type ._llcrypt registered [10718.580090] Key type .llcrypt registered [10718.617406] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [10718.930109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10718.955149] alg: No test for adler32 (adler32-zlib) [10719.816122] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [10719.915697] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [10721.503171] Key type lgssc registered [10721.887016] Lustre: Echo OBD driver; http://www.lustre.org/ [10724.459946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10724.465184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10725.642938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10727.185132] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10730.261899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10730.381670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10731.798945] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10734.267561] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10736.061935] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10738.431155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10738.552159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10740.632982] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10743.320425] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10750.828255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10756.067036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10756.067628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10756.068498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10756.078376] Lustre: Skipped 2 previous similar messages [10759.095932] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [10759.097762] Lustre: Skipped 2 previous similar messages [10761.185090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10761.187670] Lustre: Skipped 2 previous similar messages [10761.237016] Lustre: server umount lustre-MDT0000 complete [10762.045079] LustreError: 422827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.37@tcp arrived at 1784594848 with bad export cookie 6369551363421598450 [10762.053110] LustreError: 422827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10762.736196] LustreError: 423581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784594849 with bad export cookie 6369551363421597792 [10762.740466] LustreError: 423581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10762.740655] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10766.304036] LustreError: 424280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10766.304513] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10766.304574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10766.305343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10766.309569] LustreError: 424280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10767.329889] LustreError: 423579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10767.336037] LustreError: 423579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10768.862832] Lustre: server umount lustre-OST0000 complete [10771.729042] Lustre: server umount lustre-MDT0001 complete [10776.148893] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [10777.433574] Key type lgssc unregistered [10777.588692] LNet: 426550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10777.593686] LNetError: 426550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10777.604330] LNet: Removed LNI 192.168.201.137@tcp [10777.992984] Key type .llcrypt unregistered [10777.994515] Key type ._llcrypt unregistered [10788.394450] Key type ._llcrypt registered [10788.395936] Key type .llcrypt registered [10788.437451] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [10788.862221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10788.869805] alg: No test for adler32 (adler32-zlib) [10789.747559] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [10789.873264] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [10791.471150] Key type lgssc registered [10791.922127] Lustre: Echo OBD driver; http://www.lustre.org/ [10795.183872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10795.192234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10796.383721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10798.041700] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10801.578209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10801.727308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10803.317771] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10806.285919] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10808.611757] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10811.344228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10811.529585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10813.767900] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10816.464530] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10822.825696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10828.257710] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10828.262756] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10828.265059] Lustre: Skipped 1 previous similar message [10832.313606] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [10833.378216] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10833.380791] Lustre: Skipped 1 previous similar message [10834.259556] Lustre: server umount lustre-MDT0001 complete [10838.496452] LustreError: 428672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10838.497077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10838.500654] LustreError: 428672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10838.503767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10838.516315] Lustre: Skipped 1 previous similar message [10838.518743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10842.195397] Lustre: server umount lustre-OST0000 complete [10843.616431] LustreError: 427944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10843.929581] Lustre: server umount lustre-MDT0000 complete [10849.012452] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [10850.075690] Key type lgssc unregistered [10850.209464] LNet: 431646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10850.212196] LNetError: 431646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10850.227338] LNet: Removed LNI 192.168.201.137@tcp [10850.557703] Key type .llcrypt unregistered [10850.559100] Key type ._llcrypt unregistered [10859.632415] Key type ._llcrypt registered [10859.633646] Key type .llcrypt registered [10859.670869] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [10866.316140] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [10866.774230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10866.785420] alg: No test for adler32 (adler32-zlib) [10867.655143] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [10867.760469] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [10869.359115] Key type lgssc registered [10869.706370] Lustre: Echo OBD driver; http://www.lustre.org/ [10882.627234] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [10887.797710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10887.806666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10888.896095] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10888.910129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10888.947095] Lustre: lustre-MDT0000: new disk, initializing [10888.971826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10888.977793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10890.341329] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10896.310047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10896.410883] Lustre: lustre-OST0000: new disk, initializing [10896.413099] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10896.415164] Lustre: Skipped 1 previous similar message [10896.417704] Lustre: 436650:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10896.443304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10898.045305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10898.048550] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10898.057550] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10898.285928] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10902.075941] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10907.930757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10907.974328] Lustre: 435631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10908.004259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10908.038777] Lustre: lustre-MDT0001: new disk, initializing [10908.079544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10908.093208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [10908.096384] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [10909.456914] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [10913.073061] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [10913.146041] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [10914.522023] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [10914.595969] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [10916.793560] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [10919.393624] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10919.393755] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10919.397681] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10919.402161] Lustre: Skipped 1 previous similar message [10919.405111] Lustre: Skipped 1 previous similar message [10921.906641] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [10922.257924] Lustre: server umount lustre-MDT0001 complete [10929.632991] LustreError: 435638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10929.638729] LustreError: 435638:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10934.752041] LustreError: 437990:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10939.359714] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10939.362343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10939.367399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10939.872394] LustreError: 435643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10940.620972] Lustre: server umount lustre-OST0000 complete [10956.255131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10956.339700] Lustre: server umount lustre-MDT0000 complete [10960.879497] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [10961.873384] Key type lgssc unregistered [10962.007379] LNet: 440398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10962.009644] LNetError: 440398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10962.015290] LNet: Removed LNI 192.168.201.137@tcp [10962.325517] Key type .llcrypt unregistered [10962.326641] Key type ._llcrypt unregistered [10975.874986] Key type ._llcrypt registered [10975.876198] Key type .llcrypt registered [10975.925947] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [10976.036421] Key type .llcrypt unregistered [10976.038234] Key type ._llcrypt unregistered [10983.187122] Key type ._llcrypt registered [10983.188160] Key type .llcrypt registered [10983.229829] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [10983.633457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10983.675862] alg: No test for adler32 (adler32-zlib) [10984.538828] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [10984.630255] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [10986.215172] Key type lgssc registered [10986.634736] Lustre: Echo OBD driver; http://www.lustre.org/ [10996.419695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10996.428722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10997.519489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10997.533085] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10997.567978] Lustre: lustre-MDT0000: new disk, initializing [10997.590341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10997.597321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10999.122828] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11003.746159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11003.804921] Lustre: 443747:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11003.832654] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11003.834942] Lustre: Skipped 1 previous similar message [11003.871914] Lustre: lustre-MDT0001: new disk, initializing [11003.912821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11003.930526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11003.935514] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11005.547535] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11009.562033] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11011.488812] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11013.908534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11014.017443] Lustre: lustre-OST0000: new disk, initializing [11014.020067] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11014.024433] Lustre: 445713:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11014.055662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11016.072917] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11019.765546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11019.768721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11019.804200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11019.844816] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11024.351917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11024.354679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11024.359835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11024.865827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11024.866656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11024.871298] Lustre: Skipped 1 previous similar message [11028.310555] Lustre: server umount lustre-MDT0000 complete [11029.679167] LustreError: 445711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784595116 with bad export cookie 10841191859038182685 [11029.682637] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11029.683882] LustreError: 445711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11029.802918] Lustre: server umount lustre-MDT0001 complete [11040.283359] Lustre: server umount lustre-OST0000 complete [11043.903483] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 20:52:09 (1784595129) [11046.673795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11046.887599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11048.315190] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11051.209766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11051.335045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11052.672733] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11055.115742] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11056.898752] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11072.290512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11072.382424] Lustre: Found index 0 for lustre-OST0000, updating log [11072.452807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11074.475836] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11076.999584] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11092.966905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11100.947515] Lustre: server umount lustre-OST0000 complete [11118.047964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11118.050864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11118.054829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11118.056473] Lustre: Skipped 1 previous similar message [11119.055156] Lustre: 442283:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784595189/real 1784595189] req@0000000040b83155 x1871283561227776/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1784595205 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11119.583193] Lustre: 442284:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784595190/real 1784595190] req@0000000092920753 x1871283561228928/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784595206 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11123.191791] Lustre: server umount lustre-MDT0000 complete [11124.192039] LustreError: 450238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11124.198061] LustreError: 450238:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11124.394094] LustreError: 449579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784595211 with bad export cookie 10841191859038184064 [11124.397492] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11124.398396] LustreError: 449579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11124.515848] Lustre: server umount lustre-MDT0001 complete [11127.332095] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 20:53:33 (1784595213) [11132.127128] LNet: 1 local NIs in recovery (showing 1): 192.168.201.137@tcp [11719.307611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11719.468301] LustreError: 454178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11719.491279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11720.820387] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11723.725221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11723.896202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11725.250535] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11727.629762] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11729.547636] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11744.866146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11744.946925] Lustre: Found index 0 for lustre-OST0000, updating log [11745.040342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11746.847920] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11748.135079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [11749.353202] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11751.009897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11751.010144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11751.011670] Lustre: Skipped 3 previous similar messages [11751.016273] Lustre: Skipped 2 previous similar messages [11756.107710] Lustre: server umount lustre-OST0000 complete [11758.368498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11758.369665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11758.372412] Lustre: Skipped 1 previous similar message [11758.375705] Lustre: Skipped 2 previous similar messages [11759.392421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11759.394685] Lustre: Skipped 1 previous similar message [11761.440119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11761.442253] Lustre: Skipped 3 previous similar messages [11763.475837] Lustre: server umount lustre-MDT0000 complete [11763.489225] LustreError: 454178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11763.495157] LustreError: 454178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11764.624252] LustreError: 454160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784595851 with bad export cookie 10841191859038185135 [11764.627514] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11764.628906] LustreError: 454160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11764.729761] Lustre: server umount lustre-MDT0001 complete [11772.127116] LNet: 1 local NIs in recovery (showing 1): 192.168.201.137@tcp [11781.770432] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [11784.754171] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [11796.510706] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 21:04:42 (1784595882) [11802.340345] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [11803.371951] Key type lgssc unregistered [11807.583405] LNet: 460830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11807.586032] LNetError: 460830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11876.197441] LNet: Removed LNI 192.168.201.137@tcp [11876.509187] Key type .llcrypt unregistered [11876.510719] Key type ._llcrypt unregistered [11883.241209] Key type ._llcrypt registered [11883.242904] Key type .llcrypt registered [11883.286970] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [11883.686653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11883.702596] alg: No test for adler32 (adler32-zlib) [11884.560989] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [11884.653484] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [11886.231131] Key type lgssc registered [11886.603230] Lustre: Echo OBD driver; http://www.lustre.org/ [11888.010414] LNet: Added LNI 192.168.201.137@tcp100 [8/256/0/180] [11888.286613] LNet: Added LNI 192.168.201.137@tcp101 [8/256/0/180] [11888.562662] LNet: Added LNI 192.168.201.137@tcp102 [8/256/0/180] [11888.833229] LNet: Added LNI 192.168.201.137@tcp103 [8/256/0/180] [11889.104321] LNet: Added LNI 192.168.201.137@tcp104 [8/256/0/180] [11889.385759] LNet: Added LNI 192.168.201.137@tcp105 [8/256/0/180] [11889.662247] LNet: Added LNI 192.168.201.137@tcp106 [8/256/0/180] [11889.938656] LNet: Added LNI 192.168.201.137@tcp107 [8/256/0/180] [11890.242823] LNet: Added LNI 192.168.201.137@tcp108 [8/256/0/180] [11890.544361] LNet: Added LNI 192.168.201.137@tcp109 [8/256/0/180] [11890.847954] LNet: Added LNI 192.168.201.137@tcp110 [8/256/0/180] [11891.154700] LNet: Added LNI 192.168.201.137@tcp111 [8/256/0/180] [11891.465457] LNet: Added LNI 192.168.201.137@tcp112 [8/256/0/180] [11891.766704] LNet: Added LNI 192.168.201.137@tcp113 [8/256/0/180] [11892.072333] LNet: Added LNI 192.168.201.137@tcp114 [8/256/0/180] [11892.368632] LNet: Added LNI 192.168.201.137@tcp115 [8/256/0/180] [11892.662972] LNet: Added LNI 192.168.201.137@tcp116 [8/256/0/180] [11892.949076] LNet: Added LNI 192.168.201.137@tcp117 [8/256/0/180] [11893.240118] LNet: Added LNI 192.168.201.137@tcp118 [8/256/0/180] [11893.534579] LNet: Added LNI 192.168.201.137@tcp119 [8/256/0/180] [11893.829978] LNet: Added LNI 192.168.201.137@tcp120 [8/256/0/180] [11894.121720] LNet: Added LNI 192.168.201.137@tcp121 [8/256/0/180] [11894.399736] LNet: Added LNI 192.168.201.137@tcp122 [8/256/0/180] [11894.677577] LNet: Added LNI 192.168.201.137@tcp123 [8/256/0/180] [11894.954273] LNet: Added LNI 192.168.201.137@tcp124 [8/256/0/180] [11895.245034] LNet: Added LNI 192.168.201.137@tcp125 [8/256/0/180] [11895.513567] LNet: Added LNI 192.168.201.137@tcp126 [8/256/0/180] [11895.795337] LNet: Added LNI 192.168.201.137@tcp127 [8/256/0/180] [11896.074762] LNet: Added LNI 192.168.201.137@tcp128 [8/256/0/180] [11896.343649] LNet: Added LNI 192.168.201.137@tcp129 [8/256/0/180] [11896.624226] LNet: Added LNI 192.168.201.137@tcp130 [8/256/0/180] [11896.906865] LNet: Added LNI 192.168.201.137@tcp131 [8/256/0/180] [11897.199701] LNet: Added LNI 192.168.201.137@tcp132 [8/256/0/180] [11897.485284] LNet: Added LNI 192.168.201.137@tcp133 [8/256/0/180] [11897.781354] LNet: Added LNI 192.168.201.137@tcp134 [8/256/0/180] [11898.086206] LNet: Added LNI 192.168.201.137@tcp135 [8/256/0/180] [11900.341726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11900.347510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11901.388597] Lustre: 463828:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.137@tcp: can't add NID 192.168.201.137@tcp130: rc = -75 [11901.417443] LNetError: 461290:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.137 [11901.435374] LNetError: Protocol error connecting to 192.168.201.137@tcp122 at host 192.168.201.137:988: is it running a compatible version of Lustre? [11901.439616] LNetError: Skipped 1 previous similar message [11901.470406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11901.480628] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11901.515271] Lustre: lustre-MDT0000: new disk, initializing [11901.539211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11901.546367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11902.047202] LNet: 1 local NIs in recovery (showing 1): 192.168.201.137@tcp122 [11902.048330] LNetError: 463850:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.137 [11902.053534] LNetError: 463850:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [11902.056635] LNetError: Protocol error connecting to 192.168.201.137@tcp at host 192.168.201.137:988: is it running a compatible version of Lustre? [11902.876573] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11904.096174] LNetError: 463850:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.137 [11904.099488] LNetError: 463850:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [11904.101877] LNetError: Protocol error connecting to 192.168.201.137@tcp122 at host 192.168.201.137:988: is it running a compatible version of Lustre? [11904.105500] LNetError: Skipped 8 previous similar messages [11907.143536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11907.198248] Lustre: 463846:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11907.226705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11907.229322] Lustre: Skipped 1 previous similar message [11907.265625] Lustre: lustre-MDT0001: new disk, initializing [11907.278088] Lustre: 464648:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.137@tcp: can't add NID 192.168.201.137@tcp130: rc = -75 [11907.281679] Lustre: 464648:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11907.323717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11907.336996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11907.341700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11908.192416] LNetError: 463853:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.137 [11908.196358] LNetError: Protocol error connecting to 192.168.201.137@tcp122 at host 192.168.201.137:988: is it running a compatible version of Lustre? [11908.704236] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11910.258250] Lustre: 465012:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.137@tcp: can't add NID 192.168.201.137@tcp130: rc = -75 [11910.262357] Lustre: 465012:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11912.412887] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11914.219819] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11916.384174] LNetError: 463863:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.137 [11916.388340] LNetError: Protocol error connecting to 192.168.201.137@tcp122 at host 192.168.201.137:988: is it running a compatible version of Lustre? [11916.475382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11916.598779] Lustre: lustre-OST0000: new disk, initializing [11916.600559] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11916.603155] Lustre: 465822:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11916.644250] Lustre: 465827:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.137@tcp: can't add NID 192.168.201.137@tcp130: rc = -75 [11916.647217] Lustre: 465827:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [11916.666643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11918.522357] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [11918.914242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11918.918725] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11918.954132] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11922.119183] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11929.056105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11929.058938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11929.064112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11929.570189] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11929.574070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11929.580408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11930.581317] Lustre: server umount lustre-OST0000 complete [11932.768755] LNetError: 463849:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.137 [11932.773865] LNetError: Protocol error connecting to 192.168.201.137@tcp122 at host 192.168.201.137:988: is it running a compatible version of Lustre? [11933.152032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11933.155673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11933.161507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11938.074584] Lustre: server umount lustre-MDT0000 complete [11939.315128] LustreError: 463837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784596026 with bad export cookie 2944487933900726924 [11939.318065] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11939.319503] LustreError: 463837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11939.420633] Lustre: server umount lustre-MDT0001 complete [11942.954037] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [11944.010474] Key type lgssc unregistered [11944.157893] LNet: 467731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11944.161860] LNetError: 467731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11944.164553] LNet: Removed LNI 192.168.201.137@tcp [11945.183129] LNet: Removed LNI 192.168.201.137@tcp100 [11945.184544] LNetError: 467731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11945.187041] LNetError: 467731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11945.189739] LNet: Removed LNI 192.168.201.137@tcp101 [11946.207131] LNet: Removed LNI 192.168.201.137@tcp102 [11946.208843] LNetError: 467731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11946.211385] LNetError: 467731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11946.213926] LNet: Removed LNI 192.168.201.137@tcp103 [11947.231158] LNet: Removed LNI 192.168.201.137@tcp104 [11948.255156] LNet: Removed LNI 192.168.201.137@tcp105 [11948.256618] LNetError: 467731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11948.259195] LNetError: 467731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [11948.261926] LNet: Removed LNI 192.168.201.137@tcp106 [11949.279168] LNet: Removed LNI 192.168.201.137@tcp107 [11950.303133] LNet: Removed LNI 192.168.201.137@tcp108 [11951.327158] LNet: Removed LNI 192.168.201.137@tcp109 [11952.351167] LNet: Removed LNI 192.168.201.137@tcp110 [11952.352629] LNetError: 467731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11952.355070] LNetError: 467731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [11952.358123] LNet: Removed LNI 192.168.201.137@tcp111 [11953.375144] LNet: Removed LNI 192.168.201.137@tcp112 [11954.399165] LNet: Removed LNI 192.168.201.137@tcp113 [11955.423159] LNet: Removed LNI 192.168.201.137@tcp114 [11956.447151] LNet: Removed LNI 192.168.201.137@tcp115 [11957.471165] LNet: Removed LNI 192.168.201.137@tcp116 [11958.495158] LNet: Removed LNI 192.168.201.137@tcp117 [11959.519152] LNet: Removed LNI 192.168.201.137@tcp118 [11960.543157] LNet: Removed LNI 192.168.201.137@tcp119 [11960.544557] LNetError: 467731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11960.547588] LNetError: 467731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [11960.551162] LNet: Removed LNI 192.168.201.137@tcp120 [11961.567208] LNet: Removed LNI 192.168.201.137@tcp121 [11961.569235] LNet: Removed LNI 192.168.201.137@tcp122 [11962.591155] LNet: Removed LNI 192.168.201.137@tcp123 [11963.615157] LNet: Removed LNI 192.168.201.137@tcp124 [11964.639139] LNet: Removed LNI 192.168.201.137@tcp125 [11965.663140] LNet: Removed LNI 192.168.201.137@tcp126 [11966.687088] LNet: Removed LNI 192.168.201.137@tcp127 [11967.711089] LNet: Removed LNI 192.168.201.137@tcp128 [11968.735135] LNet: Removed LNI 192.168.201.137@tcp129 [11968.736459] LNet: Removed LNI 192.168.201.137@tcp130 [11968.738361] LNet: Removed LNI 192.168.201.137@tcp131 [11968.739521] LNet: Removed LNI 192.168.201.137@tcp132 [11968.740616] LNet: Removed LNI 192.168.201.137@tcp133 [11968.741724] LNet: Removed LNI 192.168.201.137@tcp134 [11968.748462] LNet: Removed LNI 192.168.201.137@tcp135 [11969.008202] Key type .llcrypt unregistered [11969.009275] Key type ._llcrypt unregistered [11976.170169] Key type ._llcrypt registered [11976.171812] Key type .llcrypt registered [11976.219100] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [11982.167100] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [11982.474971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11982.528585] alg: No test for adler32 (adler32-zlib) [11983.371347] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [11983.450847] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [11985.031150] Key type lgssc registered [11985.374585] Lustre: Echo OBD driver; http://www.lustre.org/ [11995.843435] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 21:08:01 (1784596081) [12001.513673] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [12002.496755] Key type lgssc unregistered [12002.618341] LNet: 471789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12002.620720] LNetError: 471789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12002.631297] LNet: Removed LNI 192.168.201.137@tcp [12002.930332] Key type .llcrypt unregistered [12002.931533] Key type ._llcrypt unregistered [12009.139476] Key type ._llcrypt registered [12009.140451] Key type .llcrypt registered [12009.176486] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [12009.530698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12009.571507] alg: No test for adler32 (adler32-zlib) [12010.416975] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [12010.498552] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [12012.079154] Key type lgssc registered [12012.418271] Lustre: Echo OBD driver; http://www.lustre.org/ [12015.326722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12015.333990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12016.412865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12016.424699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12016.456597] Lustre: lustre-MDT0000: new disk, initializing [12016.477498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12016.484915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12017.813661] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12021.741377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12021.780988] Lustre: 472979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12021.807188] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12021.809847] Lustre: Skipped 1 previous similar message [12021.848748] Lustre: lustre-MDT0001: new disk, initializing [12021.878761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12021.891034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12021.894649] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12023.246290] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12026.925201] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12028.682552] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12030.767758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12030.857051] Lustre: lustre-OST0000: new disk, initializing [12030.858698] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12030.861586] Lustre: 474949:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12030.889668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12032.439877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12032.443671] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12032.481029] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12032.665670] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12036.163602] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12037.215580] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing lnet_if_list [12038.717592] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [12038.728496] LNet: Added LNI 192.168.201.137@tcp10 [8/256/0/180] [12040.442279] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [12040.453936] LNet: Added LNI 192.168.201.137@tcp11 [8/256/0/180] [12041.797665] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [12041.807935] LNet: Added LNI 192.168.201.137@tcp12 [8/256/0/180] [12043.182304] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [12043.194541] LNet: Added LNI 192.168.201.137@tcp13 [8/256/0/180] [12044.564420] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [12044.575910] LNet: Added LNI 192.168.201.137@tcp14 [8/256/0/180] [12045.955286] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [12045.968732] LNet: Added LNI 192.168.201.137@tcp15 [8/256/0/180] [12047.384135] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [12047.397378] LNet: Added LNI 192.168.201.137@tcp16 [8/256/0/180] [12048.976242] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [12048.987505] LNet: Added LNI 192.168.201.137@tcp17 [8/256/0/180] [12050.546655] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [12050.559961] LNet: Added LNI 192.168.201.137@tcp18 [8/256/0/180] [12052.001295] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [12052.010996] LNet: Added LNI 192.168.201.137@tcp19 [8/256/0/180] [12053.511970] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [12053.523327] LNet: Added LNI 192.168.201.137@tcp20 [8/256/0/180] [12055.026596] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [12055.036996] LNet: Added LNI 192.168.201.137@tcp21 [8/256/0/180] [12056.569701] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [12056.580902] LNet: Added LNI 192.168.201.137@tcp22 [8/256/0/180] [12058.123834] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [12058.134762] LNet: Added LNI 192.168.201.137@tcp23 [8/256/0/180] [12059.737491] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [12059.747902] LNet: Added LNI 192.168.201.137@tcp24 [8/256/0/180] [12061.352949] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [12061.362439] LNet: Added LNI 192.168.201.137@tcp25 [8/256/0/180] [12063.040721] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [12063.051613] LNet: Added LNI 192.168.201.137@tcp26 [8/256/0/180] [12064.848060] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [12064.857834] LNet: Added LNI 192.168.201.137@tcp27 [8/256/0/180] [12066.721634] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [12066.733934] LNet: Added LNI 192.168.201.137@tcp28 [8/256/0/180] [12068.646694] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [12068.660558] LNet: Added LNI 192.168.201.137@tcp29 [8/256/0/180] [12070.625112] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [12070.639255] LNet: Added LNI 192.168.201.137@tcp30 [8/256/0/180] [12072.491091] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [12072.504304] LNet: Added LNI 192.168.201.137@tcp31 [8/256/0/180] [12074.475111] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [12074.485910] LNet: Added LNI 192.168.201.137@tcp32 [8/256/0/180] [12076.410629] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [12076.421412] LNet: Added LNI 192.168.201.137@tcp33 [8/256/0/180] [12078.381768] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [12078.394327] LNet: Added LNI 192.168.201.137@tcp34 [8/256/0/180] [12080.356600] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [12080.367498] LNet: Added LNI 192.168.201.137@tcp35 [8/256/0/180] [12082.410747] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [12082.420894] LNet: Added LNI 192.168.201.137@tcp36 [8/256/0/180] [12084.380865] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [12084.391429] LNet: Added LNI 192.168.201.137@tcp37 [8/256/0/180] [12086.475377] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [12086.486876] LNet: Added LNI 192.168.201.137@tcp38 [8/256/0/180] [12088.662742] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [12088.674576] LNet: Added LNI 192.168.201.137@tcp39 [8/256/0/180] [12090.897399] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [12090.907184] LNet: Added LNI 192.168.201.137@tcp40 [8/256/0/180] [12093.230832] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [12093.241687] LNet: Added LNI 192.168.201.137@tcp41 [8/256/0/180] [12095.591252] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [12095.602256] LNet: Added LNI 192.168.201.137@tcp42 [8/256/0/180] [12097.887346] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [12097.898550] LNet: Added LNI 192.168.201.137@tcp43 [8/256/0/180] [12100.206911] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [12100.217832] LNet: Added LNI 192.168.201.137@tcp44 [8/256/0/180] [12102.546726] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [12102.556434] LNet: Added LNI 192.168.201.137@tcp45 [8/256/0/180] [12104.830850] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [12104.841419] LNet: Added LNI 192.168.201.137@tcp46 [8/256/0/180] [12107.148493] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [12107.161101] LNet: Added LNI 192.168.201.137@tcp47 [8/256/0/180] [12109.473474] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [12109.483850] LNet: Added LNI 192.168.201.137@tcp48 [8/256/0/180] [12111.882186] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [12111.894978] LNet: Added LNI 192.168.201.137@tcp49 [8/256/0/180] [12114.270435] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [12114.282125] LNet: Added LNI 192.168.201.137@tcp50 [8/256/0/180] [12124.641197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12124.642110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12124.648026] Lustre: Skipped 2 previous similar messages [12124.649970] Lustre: Skipped 1 previous similar message [12124.652782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12127.191340] Lustre: server umount lustre-MDT0000 complete [12128.377186] LustreError: 472971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784596215 with bad export cookie 3854332452287325525 [12128.379963] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12128.381516] LustreError: 472971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12128.486049] Lustre: server umount lustre-MDT0001 complete [12135.538893] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [12138.178025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12138.911384] LustreError: 472250:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000043ec72c5 x1871284637087232/t0(0) o250->MGC192.168.201.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12139.003135] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12139.006278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12140.206667] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12142.840595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12144.203272] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12144.994861] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12146.015140] Lustre: 472252:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784596216/real 1784596216] req@00000000d711eccf x1871284637086848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784596232 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12146.474280] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12148.148388] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12149.215168] Lustre: 472253:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784596220/real 1784596220] req@00000000ef9d0592 x1871284637086976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784596236 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12150.239148] Lustre: 472254:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784596221/real 1784596221] req@00000000d6b8f505 x1871284637087104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784596237 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12154.335131] Lustre: 472251:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784596225/real 1784596225] req@00000000a9f02b94 x1871284637087360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784596241 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12155.361530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12155.367440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12155.369006] Lustre: Skipped 2 previous similar messages [12160.481259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12160.484029] Lustre: Skipped 1 previous similar message [12161.228620] Lustre: server umount lustre-OST0000 complete [12163.551797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12163.553436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12163.556586] Lustre: Skipped 1 previous similar message [12163.557959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12168.540059] Lustre: server umount lustre-MDT0000 complete [12169.845242] LustreError: 472972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784596256 with bad export cookie 3854332452287327534 [12169.848277] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12169.851531] LustreError: 472972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12169.969103] Lustre: server umount lustre-MDT0001 complete [12173.452859] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [12174.390544] Key type lgssc unregistered [12174.547276] LNet: 501905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12174.550686] LNetError: 501905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12174.553053] LNet: Removed LNI 192.168.201.137@tcp [12174.554663] LNet: Removed LNI 192.168.201.137@tcp10 [12174.555987] LNet: Removed LNI 192.168.201.137@tcp11 [12174.557063] LNet: Removed LNI 192.168.201.137@tcp12 [12174.558058] LNet: Removed LNI 192.168.201.137@tcp13 [12174.559592] LNet: Removed LNI 192.168.201.137@tcp14 [12174.561390] LNet: Removed LNI 192.168.201.137@tcp15 [12174.563217] LNet: Removed LNI 192.168.201.137@tcp16 [12174.564670] LNet: Removed LNI 192.168.201.137@tcp17 [12174.566510] LNet: Removed LNI 192.168.201.137@tcp18 [12174.568452] LNet: Removed LNI 192.168.201.137@tcp19 [12174.570365] LNet: Removed LNI 192.168.201.137@tcp20 [12174.572227] LNet: Removed LNI 192.168.201.137@tcp21 [12174.573349] LNet: Removed LNI 192.168.201.137@tcp22 [12174.574722] LNet: Removed LNI 192.168.201.137@tcp23 [12174.575745] LNet: Removed LNI 192.168.201.137@tcp24 [12174.577245] LNet: Removed LNI 192.168.201.137@tcp25 [12174.579026] LNet: Removed LNI 192.168.201.137@tcp26 [12174.580868] LNet: Removed LNI 192.168.201.137@tcp27 [12174.582733] LNet: Removed LNI 192.168.201.137@tcp28 [12174.584618] LNet: Removed LNI 192.168.201.137@tcp29 [12174.586442] LNet: Removed LNI 192.168.201.137@tcp30 [12174.588224] LNet: Removed LNI 192.168.201.137@tcp31 [12174.590088] LNet: Removed LNI 192.168.201.137@tcp32 [12174.591851] LNet: Removed LNI 192.168.201.137@tcp33 [12174.593569] LNet: Removed LNI 192.168.201.137@tcp34 [12174.595370] LNet: Removed LNI 192.168.201.137@tcp35 [12174.597150] LNet: Removed LNI 192.168.201.137@tcp36 [12174.599041] LNet: Removed LNI 192.168.201.137@tcp37 [12174.600874] LNet: Removed LNI 192.168.201.137@tcp38 [12174.602542] LNet: Removed LNI 192.168.201.137@tcp39 [12174.603565] LNet: Removed LNI 192.168.201.137@tcp40 [12174.604713] LNet: Removed LNI 192.168.201.137@tcp41 [12174.606035] LNet: Removed LNI 192.168.201.137@tcp42 [12174.607205] LNet: Removed LNI 192.168.201.137@tcp43 [12174.608498] LNet: Removed LNI 192.168.201.137@tcp44 [12174.609703] LNet: Removed LNI 192.168.201.137@tcp45 [12174.610916] LNet: Removed LNI 192.168.201.137@tcp46 [12174.612219] LNet: Removed LNI 192.168.201.137@tcp47 [12174.613279] LNet: Removed LNI 192.168.201.137@tcp48 [12174.614421] LNet: Removed LNI 192.168.201.137@tcp49 [12174.619285] LNet: Removed LNI 192.168.201.137@tcp50 [12174.865398] Key type .llcrypt unregistered [12174.866979] Key type ._llcrypt unregistered [12180.852014] Key type ._llcrypt registered [12180.852857] Key type .llcrypt registered [12180.884138] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [12186.437733] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [12186.852123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12186.867234] alg: No test for adler32 (adler32-zlib) [12187.706544] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [12187.783892] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [12189.359130] Key type lgssc registered [12189.677971] Lustre: Echo OBD driver; http://www.lustre.org/ [12200.057905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12200.062603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12201.137699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12201.146635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12201.181440] Lustre: lustre-MDT0000: new disk, initializing [12201.201340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12201.206191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12202.492271] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12206.528174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12206.567371] Lustre: 505088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12206.589585] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12206.591147] Lustre: Skipped 1 previous similar message [12206.624768] Lustre: lustre-MDT0001: new disk, initializing [12206.656673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12206.668770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12206.671565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12207.911988] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12211.426075] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12213.136070] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12215.550465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12215.623364] Lustre: *** cfs_fail_loc=90f, val=0*** [12215.661975] Lustre: lustre-OST0000: new disk, initializing [12215.664423] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12215.668239] Lustre: 507099:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12215.708589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12217.599047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12217.602827] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12217.649338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12217.772173] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12221.534459] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12228.066122] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12228.066927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12228.067838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12228.073333] Lustre: Skipped 1 previous similar message [12230.231424] Lustre: server umount lustre-OST0000 complete [12232.160025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12232.163638] LustreError: Skipped 1 previous similar message [12232.164936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12232.168809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12232.170578] Lustre: Skipped 1 previous similar message [12233.184637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12233.184948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12237.662489] Lustre: server umount lustre-MDT0000 complete [12238.867083] LustreError: 505079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784596325 with bad export cookie 5049486669927692178 [12238.870227] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12238.870257] LustreError: 505079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12238.994262] Lustre: server umount lustre-MDT0001 complete [12242.765073] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [12243.829168] Key type lgssc unregistered [12243.945431] LNet: 509003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12243.947948] LNetError: 509003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12243.956270] LNet: Removed LNI 192.168.201.137@tcp [12244.281955] Key type .llcrypt unregistered [12244.283054] Key type ._llcrypt unregistered [12252.123915] Key type ._llcrypt registered [12252.125072] Key type .llcrypt registered [12252.162425] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [12252.615070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12252.627305] alg: No test for adler32 (adler32-zlib) [12253.485825] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [12253.573437] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [12255.151151] Key type lgssc registered [12255.523072] Lustre: Echo OBD driver; http://www.lustre.org/ [12258.196992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12258.203128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12259.357129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12260.591945] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12263.151395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12263.262956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12264.510452] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12266.765071] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12268.548749] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12270.624576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12270.724203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12272.483164] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12274.708072] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12289.016624] loop0: detected capacity change from 0 to 204800000 [12289.128337] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12289.131730] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12289.135686] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12291.075701] loop0: detected capacity change from 0 to 204800000 [12291.135324] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12291.139627] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12291.145303] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12292.889295] loop0: detected capacity change from 0 to 204800000 [12294.751199] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12294.803917] Lustre: fs15246-OST0000: new disk, initializing [12294.806881] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [12294.811263] Lustre: 513862:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12294.830046] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12296.685756] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12300.539027] loop1: detected capacity change from 0 to 204800000 [12302.377162] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12302.419438] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [12302.438730] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [12302.474419] Lustre: fs15246-MDT0000: new disk, initializing [12302.522956] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12302.531638] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12302.574100] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12303.583991] Lustre: 514985:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000050c1e55d x1871284891844608/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1784596406 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [12303.859906] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12312.671254] LustreError: 514985:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [12312.674575] LustreError: 514985:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [12312.677164] LustreError: 514985:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [12315.108505] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12315.112676] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12317.194380] Lustre: server umount fs15246-MDT0000 complete [12320.228645] LustreError: 510275:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12320.234138] LustreError: 510275:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12320.273921] Lustre: server umount fs15246-OST0000 complete [12325.345681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12325.347422] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12325.349898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12325.353568] Lustre: Skipped 1 previous similar message [12329.811512] Lustre: server umount lustre-OST0000 complete [12335.071845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12335.074496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12335.078018] Lustre: Skipped 1 previous similar message [12335.079675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12340.704272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12340.707486] Lustre: Skipped 2 previous similar messages [12345.311129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12345.377729] Lustre: server umount lustre-MDT0000 complete [12345.824144] LustreError: 510268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12345.829623] LustreError: 510268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12346.626130] LustreError: 510231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784596433 with bad export cookie 398066805981633191 [12346.629987] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12346.630474] LustreError: 510231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12346.749796] Lustre: server umount lustre-MDT0001 complete [12350.666916] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [12351.719090] Key type lgssc unregistered [12351.851458] LNet: 518076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12351.855549] LNetError: 518076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12351.864414] LNet: Removed LNI 192.168.201.137@tcp [12352.206882] Key type .llcrypt unregistered [12352.208048] Key type ._llcrypt unregistered [12359.557767] Key type ._llcrypt registered [12359.558783] Key type .llcrypt registered [12359.591325] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [12359.955553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12359.960880] alg: No test for adler32 (adler32-zlib) [12360.798674] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [12360.875018] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [12362.447142] Key type lgssc registered [12362.833652] Lustre: Echo OBD driver; http://www.lustre.org/ [12375.215033] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [12378.520783] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [12390.240476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12390.247417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12391.339633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12391.349847] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12391.386818] Lustre: lustre-MDT0000: new disk, initializing [12391.410422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12391.416752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12392.839528] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12396.898131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12396.946076] Lustre: 522509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12396.974466] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12396.976136] Lustre: Skipped 1 previous similar message [12397.015386] Lustre: lustre-MDT0001: new disk, initializing [12397.073497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12397.089272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12397.094049] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12398.431333] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12402.115688] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12403.913820] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12406.254607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12406.359262] Lustre: lustre-OST0000: new disk, initializing [12406.361510] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12406.366053] Lustre: 524477:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12406.402094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12408.410851] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [12409.399657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12409.404694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12409.435282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12412.169165] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12413.683357] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12413.756945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12414.969172] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12415.029504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12419.555988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12419.556661] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12419.557557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12419.560661] Lustre: Skipped 1 previous similar message [12419.562956] LustreError: Skipped 1 previous similar message [12421.716210] Lustre: server umount lustre-OST0000 complete [12424.673122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12424.673654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12424.679525] Lustre: Skipped 1 previous similar message [12424.681385] Lustre: Skipped 1 previous similar message [12429.079206] Lustre: server umount lustre-MDT0000 complete [12430.325540] LustreError: 522504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784596517 with bad export cookie 4197759832702032380 [12430.330389] LustreError: 522504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12430.330962] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12430.449311] Lustre: server umount lustre-MDT0001 complete [12438.307372] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [12441.650206] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [12452.246344] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12194 sec ========= 21:15:38 (1784596538) [12452.831724] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:15:38 (1784596538) === [12453.983764] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:15:39 (1784596539) === [12467.353645] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [12468.411558] Key type lgssc unregistered [12468.551378] LNet: 531107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12468.553485] LNetError: 531107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12468.562371] LNet: Removed LNI 192.168.201.137@tcp [12468.907935] Key type .llcrypt unregistered [12468.909035] Key type ._llcrypt unregistered