[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 530816954 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003374] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.008720] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.012047] LSM: Security Framework initializing [ 0.013047] Yama: becoming mindful. [ 0.015014] SELinux: Initializing. [ 0.016077] *** VALIDATE selinux *** [ 0.023179] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027723] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028111] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030119] *** VALIDATE tmpfs *** [ 0.031314] *** VALIDATE proc *** [ 0.032208] *** VALIDATE cgroup *** [ 0.033007] *** VALIDATE cgroup2 *** [ 0.034253] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035135] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037024] Spectre V2 : User space: Vulnerable [ 0.038005] Speculative Store Bypass: Vulnerable [ 0.041801] debug: unmapping init [mem 0xffffffff9fa59000-0xffffffff9fa60fff] [ 0.043228] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044803] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045036] ... version: 2 [ 0.047009] ... bit width: 48 [ 0.048009] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050007] ... max period: 00007fffffffffff [ 0.051583] ... fixed-purpose events: 3 [ 0.052007] ... event mask: 000000070000000f [ 0.053301] rcu: Hierarchical SRCU implementation. [ 0.056073] smp: Bringing up secondary CPUs ... [ 0.057611] x86: Booting SMP configuration: [ 0.058035] .... node #0, CPUs: #1 #2 #3 [ 0.070012] smp: Brought up 1 node, 4 CPUs [ 0.072008] smpboot: Max logical packages: 1 [ 0.073012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.105752] node 0 deferred pages initialised in 31ms [ 0.109300] devtmpfs: initialized [ 0.110210] x86/mm: Memory block size: 128MB [ 0.113660] gcov: version magic: 0x41383552 [ 0.115243] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.116066] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.117621] pinctrl core: initialized pinctrl subsystem [ 0.118173] [ 0.119007] ************************************************************* [ 0.120011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.121021] ** ** [ 0.122011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.123013] ** ** [ 0.124009] ** This means that this kernel is built to expose internal ** [ 0.125009] ** IOMMU data structures, which may compromise security on ** [ 0.126008] ** your system. ** [ 0.127014] ** ** [ 0.128008] ** If you see this message and you are not debugging the ** [ 0.129008] ** kernel, report this immediately to your vendor! ** [ 0.130012] ** ** [ 0.131011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.132010] ************************************************************* [ 0.134558] NET: Registered protocol family 16 [ 0.135573] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.136040] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.137057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.139137] cpuidle: using governor menu [ 0.141909] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.147254] PCI: Using configuration type 1 for base access [ 0.152751] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.175131] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.176013] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.179072] cryptd: max_cpu_qlen set to 1000 [ 0.181066] ACPI: Added _OSI(Module Device) [ 0.182009] ACPI: Added _OSI(Processor Device) [ 0.183007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.184007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.188480] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.190940] ACPI: Interpreter enabled [ 0.191040] ACPI: PM: (supports S0 S3 S4 S5) [ 0.192007] ACPI: Using IOAPIC for interrupt routing [ 0.193058] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.194348] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.205361] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.206021] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.207011] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.208074] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.210432] acpiphp: Slot [2] registered [ 0.211065] acpiphp: Slot [5] registered [ 0.212087] acpiphp: Slot [6] registered [ 0.213085] acpiphp: Slot [7] registered [ 0.214078] acpiphp: Slot [8] registered [ 0.215133] acpiphp: Slot [9] registered [ 0.216088] acpiphp: Slot [10] registered [ 0.217072] acpiphp: Slot [3] registered [ 0.218064] acpiphp: Slot [4] registered [ 0.219079] acpiphp: Slot [11] registered [ 0.220252] acpiphp: Slot [12] registered [ 0.221064] acpiphp: Slot [13] registered [ 0.222065] acpiphp: Slot [14] registered [ 0.223058] acpiphp: Slot [15] registered [ 0.224141] acpiphp: Slot [16] registered [ 0.225064] acpiphp: Slot [17] registered [ 0.226077] acpiphp: Slot [18] registered [ 0.227070] acpiphp: Slot [19] registered [ 0.228064] acpiphp: Slot [20] registered [ 0.229065] acpiphp: Slot [21] registered [ 0.230056] acpiphp: Slot [22] registered [ 0.231113] acpiphp: Slot [23] registered [ 0.232093] acpiphp: Slot [24] registered [ 0.233077] acpiphp: Slot [25] registered [ 0.234237] acpiphp: Slot [26] registered [ 0.235069] acpiphp: Slot [27] registered [ 0.236094] acpiphp: Slot [28] registered [ 0.237086] acpiphp: Slot [29] registered [ 0.238055] acpiphp: Slot [30] registered [ 0.239107] acpiphp: Slot [31] registered [ 0.240059] PCI host bridge to bus 0000:00 [ 0.241011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.242008] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.243009] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.244077] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.245015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.246019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.247150] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.249152] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.251352] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.257011] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.260664] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.261014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.262016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.263078] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.265322] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.266831] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.267036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.268969] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.271012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.280014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.283013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.286820] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.301014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.315028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.343023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.358512] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.372018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.386024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.410016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.426302] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.445000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.461036] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.515021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.538861] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.553013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.571015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.604015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.614069] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.622015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.637021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.658020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.677077] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.686014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.692019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.712015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.727121] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.729289] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.732398] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.735369] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.737170] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.741153] iommu: Default domain type: Passthrough [ 0.742372] SCSI subsystem initialized [ 0.743125] ACPI: bus type USB registered [ 0.744000] usbcore: registered new interface driver usbfs [ 0.748148] usbcore: registered new interface driver hub [ 0.750162] usbcore: registered new device driver usb [ 0.752232] pps_core: LinuxPPS API ver. 1 registered [ 0.754008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.758048] PTP clock support registered [ 0.760120] EDAC MC: Ver: 3.0.0 [ 0.762172] PCI: Using ACPI for IRQ routing [ 0.763821] NetLabel: Initializing [ 0.765010] NetLabel: domain hash size = 128 [ 0.767007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.769166] NetLabel: unlabeled traffic allowed by default [ 0.770097] vgaarb: loaded [ 0.771187] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.773019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.794948] clocksource: Switched to clocksource kvm-clock [ 0.897329] VFS: Disk quotas dquot_6.6.0 [ 0.898652] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.900880] *** VALIDATE ramfs *** [ 0.901970] *** VALIDATE hugetlbfs *** [ 0.903139] pnp: PnP ACPI init [ 0.905607] pnp: PnP ACPI: found 6 devices [ 0.921644] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.924573] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.926499] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.928252] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.930194] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.931884] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.934067] NET: Registered protocol family 2 [ 0.936177] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.940893] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.943584] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.947955] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.950904] TCP: Hash tables configured (established 65536 bind 65536) [ 0.953061] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.955092] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.957555] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.959778] NET: Registered protocol family 1 [ 0.962340] RPC: Registered named UNIX socket transport module. [ 0.963805] RPC: Registered udp transport module. [ 0.965247] RPC: Registered tcp transport module. [ 0.967355] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.969046] NET: Registered protocol family 44 [ 0.970158] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.971618] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.973348] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.975466] PCI: CLS 0 bytes, default 64 [ 0.976920] Unpacking initramfs... [ 2.821590] debug: unmapping init [mem 0xffff900cbcc54000-0xffff900cbffbffff] [ 2.826214] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.828463] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.831769] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.525850] Initialise system trusted keyrings [ 3.531187] Key type blacklist registered [ 3.534781] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.545848] zbud: loaded [ 3.548931] *** VALIDATE nfs *** [ 3.550186] *** VALIDATE nfs4 *** [ 3.553286] pstore: using deflate compression [ 3.557501] Platform Keyring initialized [ 3.762682] NET: Registered protocol family 38 [ 3.765729] Key type asymmetric registered [ 3.768867] Asymmetric key parser 'x509' registered [ 3.771215] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.776633] io scheduler mq-deadline registered [ 3.778166] io scheduler kyber registered [ 3.782495] io scheduler bfq registered [ 3.784812] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.790676] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.793556] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.796794] ACPI: Power Button [PWRF] [ 3.804700] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.819630] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.851819] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.864104] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.882106] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.912562] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.942631] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.948253] Non-volatile memory driver v1.3 [ 3.950096] Linux agpgart interface v0.103 [ 3.983607] virtio_blk virtio1: [vda] 146672 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.987776] vda: detected capacity change from 0 to 75096064 [ 4.007174] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.010228] vdb: detected capacity change from 0 to 1073741824 [ 4.040098] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.045643] vdc: detected capacity change from 0 to 2621440000 [ 4.062760] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.066314] vdd: detected capacity change from 0 to 2621440000 [ 4.081734] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.084893] vde: detected capacity change from 0 to 4294967296 [ 4.103800] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.107584] vdf: detected capacity change from 0 to 4294967296 [ 4.114140] libphy: Fixed MDIO Bus: probed [ 4.123467] usbcore: registered new interface driver usbserial_generic [ 4.125975] usbserial: USB Serial support registered for generic [ 4.128568] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.133532] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.135500] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.138178] mousedev: PS/2 mouse device common for all mice [ 4.140866] rtc_cmos 00:05: RTC can wake from S4 [ 4.143889] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.146683] rtc_cmos 00:05: registered as rtc0 [ 4.150831] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.151831] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.157585] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.159250] intel_pstate: CPU model not supported [ 4.165068] hid: raw HID events driver (C) Jiri Kosina [ 4.167279] usbcore: registered new interface driver usbhid [ 4.169536] usbhid: USB HID core driver [ 4.171458] drop_monitor: Initializing network drop monitor service [ 4.174526] Initializing XFRM netlink socket [ 4.176965] NET: Registered protocol family 10 [ 4.180791] Segment Routing with IPv6 [ 4.182509] NET: Registered protocol family 17 [ 4.185121] mpls_gso: MPLS GSO support [ 4.192347] RAS: Correctable Errors collector initialized. [ 4.195068] AVX version of gcm_enc/dec engaged. [ 4.197181] AES CTR mode by8 optimization enabled [ 4.315559] sched_clock: Marking stable (4315475579, 0)->(5693584508, -1378108929) [ 4.320487] registered taskstats version 1 [ 4.322861] Loading compiled-in X.509 certificates [ 4.325477] zswap: loaded using pool lzo/zbud [ 4.359393] Key type big_key registered [ 4.379855] Key type encrypted registered [ 4.381939] ima: No TPM chip found, activating TPM-bypass! [ 4.384209] ima: Allocated hash algorithm: sha1 [ 4.386565] ima: No architecture policies found [ 4.389256] evm: Initialising EVM extended attributes: [ 4.391669] evm: security.selinux [ 4.393557] evm: security.ima [ 4.395174] evm: security.capability [ 4.396672] evm: HMAC attrs: 0x1 [ 4.399833] rtc_cmos 00:05: setting system clock to 2026-09-08 15:28:16 UTC (1788881296) [ 4.407470] debug: unmapping init [mem 0xffffffffa0a03000-0xffffffffa0bfffff] [ 4.410445] debug: unmapping init [mem 0xffffffff9f782000-0xffffffff9fa58fff] [ 4.419172] Write protecting the kernel read-only data: 28672k [ 4.423570] debug: unmapping init [mem 0xffffffff9de03000-0xffffffff9dffffff] [ 4.426839] debug: unmapping init [mem 0xffffffff9e714000-0xffffffff9e7fffff] [ 4.468750] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.474531] systemd[1]: Detected virtualization kvm. [ 4.475859] systemd[1]: Detected architecture x86-64. [ 4.477198] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.500906] systemd[1]: No hostname configured. [ 4.503266] systemd[1]: Set hostname to . [ 4.505540] random: systemd: uninitialized urandom read (16 bytes read) [ 4.508687] systemd[1]: Initializing machine ID from random generator. [ 4.687130] random: ln: uninitialized urandom read (6 bytes read) [ 4.955140] random: systemd: uninitialized urandom read (16 bytes read) [ 4.964230] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.968409] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.977619] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.470630] device-mapper: uevent: version 1.0.3 [ 6.473161] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.413551] random: fast init done [ 8.497165] scsi host0: ata_piix [ 8.501761] scsi host1: ata_piix [ 8.503217] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.505832] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.600681] virtio_net virtio0 ens2: renamed from eth0 [ 13.078843] random: crng init done [ 13.083347] random: 7 urandom warning(s) missed due to ratelimiting [ 15.760307] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 18.072863] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. [ 18.252612] hrtimer: interrupt took 8019118 ns Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.232781] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.655599] SELinux: Disabled at runtime. [ 21.744592] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.773536] systemd[1]: Detected virtualization kvm. [ 21.775153] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.227584] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.232466] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.243614] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.251808] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.256919] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.345261] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.374107] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ 23.795641] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 26.592115] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 30.478890] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.529866] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 33.162471] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [* ] A start job is running for Configur…only root support (10s / no limit)[ 33.811482] EDAC sbridge: Ver: 1.1.2 [** ] A start job is running for Configur…only root support (11s / no limit) [*** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ *] A start job is running for Configur…only root support (13s / no limit)[ 37.273628] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (14s / no limit)[ 37.740157] NFS: Registering the id_resolver key type [ 37.742438] Key type id_resolver registered [ 37.744313] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (15s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit) [ *** ] A start job is running for Configur…only root support (16s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg260-server login: [ 113.570192] libcfs: loading out-of-tree module taints kernel. [ 113.642602] Key type ._llcrypt registered [ 113.645319] Key type .llcrypt registered [ 113.723523] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 125.409758] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 126.454498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 126.462902] alg: No test for adler32 (adler32-zlib) [ 127.645218] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 128.261680] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 129.969883] Key type lgssc registered [ 131.078757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 142.435938] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 168.541310] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 176.921896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 176.957173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.105645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 178.125648] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 178.185801] Lustre: lustre-MDT0000: new disk, initializing [ 178.268333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 178.285096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 181.059234] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.872375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.930208] Lustre: 6505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 189.959848] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 189.963240] Lustre: Skipped 1 previous similar message [ 190.023786] Lustre: lustre-MDT0001: new disk, initializing [ 190.072775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 190.098227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 190.109822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 192.786768] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.017555] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 201.621485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 201.770937] Lustre: lustre-OST0000: new disk, initializing [ 201.778764] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 201.783173] Lustre: 8440:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 201.841576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.878111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 202.889637] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 202.944162] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 205.939587] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.039662] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 216.151153] Lustre: lustre-OST0001: new disk, initializing [ 216.156593] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 216.160986] Lustre: 9513:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 216.229490] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 220.188942] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.258735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 224.265290] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 224.299738] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 228.814345] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 232.744120] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 237.656600] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing check_logdir /tmp/testlogs/ [ 241.869444] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing yml_node [ 245.285770] Lustre: DEBUG MARKER: Client: 2.17.57.103 [ 247.261655] Lustre: DEBUG MARKER: MDS: 2.17.57.103 [ 249.195699] Lustre: DEBUG MARKER: OSS: 2.17.57.103 [ 250.323823] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 11:32:21 EDT 2026 [ 262.388857] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 263.389616] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 277.474088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 277.481610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 277.494730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 280.545338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 280.547466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 280.554476] Lustre: Skipped 1 previous similar message [ 280.563215] Lustre: Skipped 3 previous similar messages [ 282.959872] Lustre: server umount lustre-MDT0000 complete [ 285.195811] LustreError: 7451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788881577 with bad export cookie 17414364242019241799 [ 285.200858] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 285.202038] LustreError: 7451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 285.666305] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 285.672494] Lustre: Skipped 1 previous similar message [ 285.675226] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 285.678349] Lustre: Skipped 1 previous similar message [ 290.785038] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 290.785077] LustreError: 6511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 290.800137] LustreError: 6511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 291.396904] Lustre: server umount lustre-MDT0001 complete [ 293.554339] Lustre: server umount lustre-OST0000 complete [ 296.065886] Lustre: server umount lustre-OST0001 complete [ 299.805168] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 305.639268] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 327.256990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 327.419121] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 327.442152] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 327.491696] Lustre: lustre-MDT0000: new disk, initializing [ 327.535102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 327.543317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 329.965472] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 335.862194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 335.917837] Lustre: 15053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 335.957413] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 335.959749] Lustre: Skipped 1 previous similar message [ 336.005642] Lustre: lustre-MDT0001: new disk, initializing [ 336.052860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 336.073368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 336.077764] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 337.713403] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 342.711110] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 345.648171] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 349.396026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 349.538238] Lustre: 17023:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 351.586216] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 351.644034] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 352.681675] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 358.253455] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 360.366417] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 360.467568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 362.451389] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 362.564165] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 367.076278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.077563] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 367.078041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 367.078045] Lustre: Skipped 1 previous similar message [ 367.084460] Lustre: Skipped 2 previous similar messages [ 367.088336] LustreError: Skipped 1 previous similar message [ 369.627944] Lustre: server umount lustre-OST0000 complete [ 372.195312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 372.195444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 372.203352] Lustre: Skipped 1 previous similar message [ 377.241793] Lustre: server umount lustre-MDT0000 complete [ 377.313057] LustreError: 15063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 377.318047] LustreError: 15063:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 378.559174] LustreError: 15045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788881670 with bad export cookie 17414364242019243332 [ 378.561603] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 378.562947] LustreError: 15045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 378.688479] Lustre: server umount lustre-MDT0001 complete [ 380.087273] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 11:34:32 (1788881672) [ 380.895152] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 381.429862] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 11:34:33 (1788881673) [ 383.454754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 383.593046] LustreError: 19430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 383.615066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 383.617306] Lustre: Skipped 1 previous similar message [ 384.900849] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 387.690218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 389.135474] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 391.702266] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 393.749592] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 396.700161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 399.109702] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 402.181572] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 418.786386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 418.786975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 418.788830] LustreError: Skipped 1 previous similar message [ 418.793667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 418.797600] Lustre: Skipped 4 previous similar messages [ 422.320627] Lustre: server umount lustre-OST0000 complete [ 429.032617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 430.454264] Lustre: server umount lustre-MDT0000 complete [ 431.862141] LustreError: 19412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788881723 with bad export cookie 17414364242019244214 [ 431.865682] LustreError: 19412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 431.865864] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 431.984679] Lustre: server umount lustre-MDT0001 complete [ 436.056276] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 437.201122] Key type lgssc unregistered [ 437.345568] LNet: 22981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 437.349215] LNetError: 22981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 437.358411] LNet: Removed LNI 192.168.202.160@tcp [ 437.765163] Key type .llcrypt unregistered [ 437.766543] Key type ._llcrypt unregistered [ 446.422501] Key type ._llcrypt registered [ 446.423960] Key type .llcrypt registered [ 446.463789] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 446.863075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 446.870672] alg: No test for adler32 (adler32-zlib) [ 447.742665] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 447.844465] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 449.432165] Key type lgssc registered [ 449.796884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 452.424664] loop: module loaded [ 452.442437] loop0: detected capacity change from 0 to 204800000 [ 452.543432] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 452.547825] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 452.657262] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 454.450845] loop0: detected capacity change from 0 to 204800000 [ 454.502871] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 454.507907] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 454.511409] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 456.214542] loop0: detected capacity change from 0 to 204800000 [ 458.028935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 458.034302] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 459.113891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 459.129134] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 459.166251] Lustre: lustre-MDT0000: new disk, initializing [ 459.185119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 459.191457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 460.593534] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 464.132673] loop1: detected capacity change from 0 to 204800000 [ 465.985716] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 466.080740] Lustre: lustre-OST07c6: new disk, initializing [ 466.082320] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 466.084406] Lustre: Skipped 1 previous similar message [ 466.086483] Lustre: 26139:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 466.109187] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 466.210312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 466.213203] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 466.221945] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 468.029587] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 470.151866] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 470.155806] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 470.158831] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 470.161149] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 470.163754] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 470.165544] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 488.597851] Lustre: 25126:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 488.600694] Lustre: 25126:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 488.602817] Lustre: 25126:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 488.606293] Lustre: 25126:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 488.609979] Lustre: 25126:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 488.613233] Lustre: 25126:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 492.111971] Lustre: 25124:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 492.115012] Lustre: 25124:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 492.117245] Lustre: 25124:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 492.120086] Lustre: 25124:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 492.122262] Lustre: 25124:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 492.124205] Lustre: 25124:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 492.126640] Lustre: 25124:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 492.130034] Lustre: 25124:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 492.132194] Lustre: 25124:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 492.134732] Lustre: 25124:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 492.138214] Lustre: 25124:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 492.141354] Lustre: 25124:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 495.199863] Lustre: 26654:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 495.202418] Lustre: 26654:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 495.204126] Lustre: 26654:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 495.206159] Lustre: 26654:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 495.208266] Lustre: 26654:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 495.210453] Lustre: 26654:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 495.212910] Lustre: 26654:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 495.215325] Lustre: 26654:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 495.218802] Lustre: 26654:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 495.220617] Lustre: 26654:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 495.222932] Lustre: 26654:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 495.224907] Lustre: 26654:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 501.761461] Lustre: 25126:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 501.764086] Lustre: 25126:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 501.766082] Lustre: 25126:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 501.767997] Lustre: 25126:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 501.770531] Lustre: 25126:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 501.773013] Lustre: 25126:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 501.775682] Lustre: 25126:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 501.778071] Lustre: 25126:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 501.780192] Lustre: 25126:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 501.782719] Lustre: 25126:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 501.785314] Lustre: 25126:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 501.787456] Lustre: 25126:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 511.656735] Lustre: 25126:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 511.659852] Lustre: 25126:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 511.661842] Lustre: 25126:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 511.664234] Lustre: 25126:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 511.666529] Lustre: 25126:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 511.668571] Lustre: 25126:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 511.671108] Lustre: 25126:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 511.673179] Lustre: 25126:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 511.675404] Lustre: 25126:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 511.678231] Lustre: 25126:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 511.681619] Lustre: 25126:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 511.683885] Lustre: 25126:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 552.504971] Lustre: server umount lustre-OST07c6 complete [ 565.270416] Lustre: server umount lustre-MDT0000 complete [ 570.647296] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 571.749764] Key type lgssc unregistered [ 571.883351] LNet: 28710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 571.885839] LNetError: 28710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 571.894348] LNet: Removed LNI 192.168.202.160@tcp [ 572.237063] Key type .llcrypt unregistered [ 572.238514] Key type ._llcrypt unregistered [ 579.225569] Key type ._llcrypt registered [ 579.226579] Key type .llcrypt registered [ 579.257795] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 579.569964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 579.611437] alg: No test for adler32 (adler32-zlib) [ 580.448039] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 580.527647] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 582.104192] Key type lgssc registered [ 582.424930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 584.743161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 584.748471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.908153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 587.316578] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 590.080815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 590.202191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 591.495421] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 593.801987] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 595.655633] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 597.886280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 598.018243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 599.972645] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 602.289455] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 610.340407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 620.517873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.518952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 620.520522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 620.522728] Lustre: Skipped 1 previous similar message [ 620.526512] LustreError: Skipped 2 previous similar messages [ 623.381568] Lustre: server umount lustre-OST0000 complete [ 625.633946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 625.640904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 625.643428] Lustre: Skipped 2 previous similar messages [ 630.753604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 630.756450] Lustre: Skipped 1 previous similar message [ 630.879625] Lustre: server umount lustre-MDT0000 complete [ 632.143476] LustreError: 29935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788881924 with bad export cookie 12192891467032311173 [ 632.146836] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 632.147784] LustreError: 29935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 632.273322] Lustre: server umount lustre-MDT0001 complete [ 635.918737] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 636.926895] Key type lgssc unregistered [ 637.073412] LNet: 33708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 637.076552] LNetError: 33708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 637.085260] LNet: Removed LNI 192.168.202.160@tcp [ 637.398573] Key type .llcrypt unregistered [ 637.399733] Key type ._llcrypt unregistered [ 644.696894] Key type ._llcrypt registered [ 644.697923] Key type .llcrypt registered [ 644.731697] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 645.062485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 645.096199] alg: No test for adler32 (adler32-zlib) [ 645.946956] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 646.032041] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 647.608104] Key type lgssc registered [ 647.972564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 650.558900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 650.565802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.725964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 653.090419] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 655.981390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 656.109776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 657.490831] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 659.949189] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 661.872135] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 664.208388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 664.329148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 666.275228] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 668.822391] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 676.648314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 687.074439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 687.075364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 687.077019] LustreError: Skipped 1 previous similar message [ 687.080742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 687.083244] Lustre: Skipped 1 previous similar message [ 689.357619] Lustre: server umount lustre-OST0000 complete [ 692.194466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 692.194991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 692.195775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 692.197614] Lustre: Skipped 2 previous similar messages [ 696.668356] Lustre: server umount lustre-MDT0000 complete [ 697.313099] LustreError: 36385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 697.317540] LustreError: 36385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 697.975520] LustreError: 35686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788881990 with bad export cookie 15975059312448984676 [ 697.980252] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 697.980504] LustreError: 35686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 698.105086] Lustre: server umount lustre-MDT0001 complete [ 701.622198] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 702.646275] Key type lgssc unregistered [ 702.789473] LNet: 38500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 702.792503] LNetError: 38500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 702.799340] LNet: Removed LNI 192.168.202.160@tcp [ 703.121720] Key type .llcrypt unregistered [ 703.123421] Key type ._llcrypt unregistered [ 712.082795] Key type ._llcrypt registered [ 712.083953] Key type .llcrypt registered [ 712.123498] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 712.547603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 712.558617] alg: No test for adler32 (adler32-zlib) [ 713.408255] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 713.496830] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 715.080099] Key type lgssc registered [ 715.435937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 717.977510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 717.984518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 719.152508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 720.538547] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 723.416268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 723.539077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 724.887719] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 727.229274] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 729.021095] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 731.320859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 731.435039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 733.328744] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 735.682947] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 744.748429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 749.025220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 749.028747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 749.032564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 750.050192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 750.050253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 750.053949] Lustre: Skipped 1 previous similar message [ 750.057173] Lustre: Skipped 2 previous similar messages [ 751.135727] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.60@tcp (stopping) [ 754.463817] Lustre: server umount lustre-MDT0000 complete [ 755.169109] LustreError: 40627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 755.175944] LustreError: 40627:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 755.727039] LustreError: 41729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882047 with bad export cookie 4760403820134535945 [ 755.730599] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 755.731168] LustreError: 41729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 756.248254] LustreError: 39886:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 756.257664] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.60@tcp (stopping) [ 760.289255] LustreError: 39885:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 760.289416] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 760.298855] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 765.409152] LustreError: 39884:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 765.414069] LustreError: 39884:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 770.016130] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 770.109567] Lustre: server umount lustre-MDT0001 complete [ 796.128172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 796.178780] Lustre: server umount lustre-OST0000 complete [ 801.843198] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 802.925669] Key type lgssc unregistered [ 803.046349] LNet: 43766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.048822] LNetError: 43766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 803.057269] LNet: Removed LNI 192.168.202.160@tcp [ 803.362403] Key type .llcrypt unregistered [ 803.363860] Key type ._llcrypt unregistered [ 812.453467] Key type ._llcrypt registered [ 812.454733] Key type .llcrypt registered [ 812.493491] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 812.808328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 812.855100] alg: No test for adler32 (adler32-zlib) [ 813.715208] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 813.805390] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 815.392188] Key type lgssc registered [ 815.749179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 818.243224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 818.248927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 819.397816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 820.731613] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 823.517931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 823.653966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 825.043323] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 827.362255] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 829.120581] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 831.336275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 831.455186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 833.344298] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 835.703517] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 845.801851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 848.865124] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 848.868806] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 848.872560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 850.914822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 850.915112] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 850.915784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 850.917381] Lustre: Skipped 1 previous similar message [ 854.672577] Lustre: server umount lustre-OST0000 complete [ 867.318274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 867.319761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 867.321659] Lustre: Skipped 1 previous similar message [ 867.324734] Lustre: Skipped 2 previous similar messages [ 872.417246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 872.420156] Lustre: Skipped 1 previous similar message [ 873.565897] Lustre: server umount lustre-MDT0000 complete [ 874.751502] LustreError: 45135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882166 with bad export cookie 9511772691296537275 [ 874.755303] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 874.755711] LustreError: 45135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 889.312110] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 889.394699] Lustre: server umount lustre-MDT0001 complete [ 892.981632] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 893.986855] Key type lgssc unregistered [ 894.108347] LNet: 48877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.112175] LNetError: 48877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.123394] LNet: Removed LNI 192.168.202.160@tcp [ 894.448160] Key type .llcrypt unregistered [ 894.449336] Key type ._llcrypt unregistered [ 903.258075] Key type ._llcrypt registered [ 903.260119] Key type .llcrypt registered [ 903.300895] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 903.726827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 903.737567] alg: No test for adler32 (adler32-zlib) [ 904.593764] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 904.683318] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 906.264143] Key type lgssc registered [ 906.641044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 909.251403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 909.258276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 910.409433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 911.707859] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 914.475248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 914.591134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 915.920412] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 918.242717] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 919.992561] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 922.487022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 922.611700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 924.535617] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 926.773879] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 935.910973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 939.517394] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 940.599491] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 951.256553] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0811h44m03s [ 951.727607] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 952.862575] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.60@tcp (stopping) [ 953.879660] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.60@tcp (stopping) [ 955.873079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 955.875456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 955.879698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 958.616607] Lustre: server umount lustre-MDT0000 complete [ 960.023302] LustreError: 50270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 960.028047] LustreError: 50270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 960.349319] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 962.018150] LustreError: 52766:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 962.494060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 962.535110] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 962.625989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 962.651698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 963.688942] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 963.937565] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 965.955524] Lustre: DEBUG MARKER: Wait for df (36626) ... [ 966.446148] Lustre: DEBUG MARKER: done [ 967.649781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 967.652991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 967.656770] Lustre: Skipped 2 previous similar messages [ 967.658839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 967.660369] Lustre: Skipped 5 previous similar messages [ 969.188507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 969.421872] Lustre: server umount lustre-OST0000 complete [ 974.306834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 974.307686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 974.314579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 974.316423] Lustre: Skipped 3 previous similar messages [ 976.725249] Lustre: server umount lustre-MDT0000 complete [ 977.980339] LustreError: 50258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882270 with bad export cookie 15683114425646892210 [ 977.983181] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 977.984952] LustreError: 50258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 978.085435] Lustre: server umount lustre-MDT0001 complete [ 981.710416] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 982.786535] Key type lgssc unregistered [ 982.920334] LNet: 55210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.923460] LNetError: 55210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.933275] LNet: Removed LNI 192.168.202.160@tcp [ 983.280995] Key type .llcrypt unregistered [ 983.282531] Key type ._llcrypt unregistered [ 997.713440] Key type ._llcrypt registered [ 997.714456] Key type .llcrypt registered [ 997.755785] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 998.242931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 998.257646] alg: No test for adler32 (adler32-zlib) [ 999.117923] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 999.218877] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 1000.808109] Key type lgssc registered [ 1001.245887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1003.844085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1003.849994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1004.930845] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1004.938824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1004.991019] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1006.205793] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1008.679662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1008.710479] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1008.722186] Lustre: 57393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1008.818921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1010.229580] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1012.561532] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1014.329611] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1016.501010] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1016.569262] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1016.633511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1018.544062] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1018.772274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1020.973186] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1024.345411] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0811h45m16s [ 1024.875638] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1027.232492] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1028.068604] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1028.106744] LustreError: 57398:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1048.611428] Lustre: lustre-MDT0000: Client 6218da7a-18a0-4ea0-a144-591c21662ee1 (at 192.168.202.60@tcp) reconnecting [ 1068.152085] LustreError: 57398:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1068.859836] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1069.672872] Lustre: DEBUG MARKER: done [ 1074.146188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1074.148548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.152488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1075.170495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1075.171028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1075.173702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1077.338603] Lustre: server umount lustre-OST0000 complete [ 1080.289410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1080.290121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1080.292981] Lustre: Skipped 1 previous similar message [ 1080.297712] Lustre: Skipped 2 previous similar messages [ 1084.756487] Lustre: server umount lustre-MDT0000 complete [ 1085.896180] LustreError: 57383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882377 with bad export cookie 8031434928160948251 [ 1085.898667] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1085.900157] LustreError: 57383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1086.006374] Lustre: server umount lustre-MDT0001 complete [ 1089.283114] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 1090.206442] Key type lgssc unregistered [ 1090.320347] LNet: 61708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1090.322862] LNetError: 61708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1090.329234] LNet: Removed LNI 192.168.202.160@tcp [ 1090.617395] Key type .llcrypt unregistered [ 1090.618595] Key type ._llcrypt unregistered [ 1105.248295] Key type ._llcrypt registered [ 1105.249563] Key type .llcrypt registered [ 1105.288507] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 1105.615653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1105.642207] alg: No test for adler32 (adler32-zlib) [ 1106.484904] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 1106.568828] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 1108.152168] Key type lgssc registered [ 1108.502699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1109.596886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1109.602614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1110.688514] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1110.697228] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1110.760779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1111.110294] Lustre: Failing over lustre-MDT0000 [ 1111.207156] Lustre: server umount lustre-MDT0000 complete [ 1113.314319] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 11:46:45 (1788882405) [ 1115.584264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1115.733028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1117.188621] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1120.107918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1120.144291] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1120.155026] Lustre: 64476:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1120.259847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1121.728425] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1124.200821] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1126.037893] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1128.262626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1128.327788] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1128.393844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1128.658406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1130.282751] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1132.662355] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1135.712341] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1139.170953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1139.172323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1139.174905] Lustre: Skipped 2 previous similar messages [ 1139.176524] Lustre: Skipped 1 previous similar message [ 1143.519249] Lustre: server umount lustre-MDT0000 complete [ 1144.798398] LustreError: 66338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882436 with bad export cookie 5581461186916864170 [ 1144.801234] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1144.803224] LustreError: 66338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1144.917253] Lustre: server umount lustre-MDT0001 complete [ 1146.069680] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1147.641968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1150.502400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1154.528740] LustreError: 63398:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a00c334d x1875779149396736/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1154.632524] LustreError: 66322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1154.640417] LustreError: 66322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1154.661358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1154.689574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1155.951475] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1158.709063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1158.808848] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1160.201128] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1162.656715] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1164.261380] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1164.489683] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1165.792147] Lustre: 63399:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788882441/real 1788882441] req@0000000067a91167 x1875779149396480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788882457 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1169.313595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1169.315849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1169.321647] Lustre: Skipped 1 previous similar message [ 1170.336121] Lustre: 63400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788882446/real 1788882446] req@0000000008b4c750 x1875779149396608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788882462 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1172.375673] Lustre: server umount lustre-MDT0000 complete [ 1173.581391] LustreError: 65211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882465 with bad export cookie 5581461186916866438 [ 1173.584278] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1173.585219] LustreError: 65211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1173.697127] Lustre: server umount lustre-MDT0001 complete [ 1175.813576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1178.994568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1183.648550] LustreError: 63398:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000848f4c44 x1875779149414656/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1183.754274] LustreError: 66322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1183.776414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1183.778622] Lustre: Skipped 1 previous similar message [ 1183.804832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1185.155723] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1188.011309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1188.105848] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1189.526767] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1190.177105] Lustre: 63401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788882466/real 1788882466] req@0000000016119468 x1875779149414272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788882482 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1192.114274] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1193.446208] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1194.010455] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1194.976147] Lustre: 63402:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788882470/real 1788882470] req@000000000e3cc519 x1875779149414400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788882486 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1198.563108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1198.564057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1198.568970] Lustre: Skipped 1 previous similar message [ 1198.569532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1198.571149] Lustre: Skipped 3 previous similar messages [ 1199.584257] Lustre: 63400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788882475/real 1788882475] req@00000000f3629df8 x1875779149414784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788882491 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1199.595180] Lustre: 63400:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1201.880857] Lustre: server umount lustre-MDT0000 complete [ 1203.199318] LustreError: 66338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882495 with bad export cookie 5581461186916867810 [ 1203.201950] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1203.203337] LustreError: 66338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1203.321099] Lustre: server umount lustre-MDT0001 complete [ 1205.539613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1206.701021] Lustre: DEBUG MARKER: files compared the same [ 1217.697157] Lustre: server umount lustre-OST0000 complete [ 1223.110203] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 1224.132844] Key type lgssc unregistered [ 1224.268357] LNet: 74187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1224.270672] LNetError: 74187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1224.278306] LNet: Removed LNI 192.168.202.160@tcp [ 1224.585620] Key type .llcrypt unregistered [ 1224.586776] Key type ._llcrypt unregistered [ 1233.492782] Key type ._llcrypt registered [ 1233.494054] Key type .llcrypt registered [ 1233.536816] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 1233.995320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1234.052899] alg: No test for adler32 (adler32-zlib) [ 1234.914837] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 1235.010066] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 1236.600267] Key type lgssc registered [ 1237.000475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1239.592090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1239.597399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1240.740042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1242.196308] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug +malloc all [ 1245.212190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1245.309162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1246.727381] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug +malloc all [ 1249.123073] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1251.090696] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1253.424059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1253.521120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1255.588322] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug +malloc all [ 1258.068475] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1265.827689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1276.384952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1276.385136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1276.387168] LustreError: Skipped 1 previous similar message [ 1276.388092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1276.390421] Lustre: Skipped 1 previous similar message [ 1277.453461] Lustre: server umount lustre-OST0000 complete [ 1281.504704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1281.504727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1281.509408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1281.514351] Lustre: Skipped 2 previous similar messages [ 1284.752842] Lustre: server umount lustre-MDT0000 complete [ 1285.855049] LustreError: 75557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882577 with bad export cookie 1617757690119919607 [ 1285.857543] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1285.858084] LustreError: 75557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1285.955108] Lustre: server umount lustre-MDT0001 complete [ 1289.069399] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 1290.013024] Key type lgssc unregistered [ 1290.148356] LNet: 79126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1290.151907] LNetError: 79126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1290.162306] LNet: Removed LNI 192.168.202.160@tcp [ 1290.441284] Key type .llcrypt unregistered [ 1290.442853] Key type ._llcrypt unregistered [ 1298.599232] Key type ._llcrypt registered [ 1298.600758] Key type .llcrypt registered [ 1298.634672] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 1298.915986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1298.927735] alg: No test for adler32 (adler32-zlib) [ 1299.774484] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 1299.852434] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 1301.424099] Key type lgssc registered [ 1301.732885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1304.126281] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1304.132574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1319.584392] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1324.704392] LustreError: 80491:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.160@tcp: failed processing log, type 4: rc = -110 [ 1351.328226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1353.234978] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1355.590478] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1358.064643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1358.105428] Lustre: *** cfs_fail_loc=706, val=0*** [ 1358.234472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1358.256862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1359.519388] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1362.286663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1362.409787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1363.822203] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1366.191547] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1367.951185] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1371.620313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1371.621178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1371.623930] Lustre: Skipped 1 previous similar message [ 1374.744434] Lustre: server umount lustre-OST0000 complete [ 1376.737445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1376.738147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1376.740740] Lustre: Skipped 1 previous similar message [ 1376.742206] Lustre: Skipped 1 previous similar message [ 1381.857170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1381.859163] Lustre: Skipped 2 previous similar messages [ 1382.170678] Lustre: server umount lustre-MDT0000 complete [ 1383.419208] LustreError: 80498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882675 with bad export cookie 4088050741400554567 [ 1383.421852] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1383.424076] LustreError: 80498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1383.540516] Lustre: server umount lustre-MDT0001 complete [ 1387.060861] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 1388.078713] Key type lgssc unregistered [ 1388.212392] LNet: 84100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1388.214577] LNetError: 84100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.223302] LNet: Removed LNI 192.168.202.160@tcp [ 1388.523460] Key type .llcrypt unregistered [ 1388.524590] Key type ._llcrypt unregistered [ 1397.988575] Key type ._llcrypt registered [ 1397.989712] Key type .llcrypt registered [ 1398.030498] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 1398.406926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1398.431089] alg: No test for adler32 (adler32-zlib) [ 1399.281975] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 1399.383948] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 1400.968203] Key type lgssc registered [ 1401.342761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1403.885255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1403.891851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1406.298062] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1408.756924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1408.941083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1410.224256] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1413.057310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1413.187881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1415.111711] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1417.956826] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1417.962392] Lustre: MGS: Client 81cda74d-d0b0-43cd-ab8e-6867a39e05ba (at 0@lo) reconnecting [ 1418.054439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1418.084056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1419.417476] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1428.449340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1428.451346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1428.454754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1429.473905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1429.476612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1429.476661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1429.483023] Lustre: Skipped 1 previous similar message [ 1433.237981] Lustre: server umount lustre-OST0000 complete [ 1434.594292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1434.594526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1434.594930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1434.597527] Lustre: Skipped 1 previous similar message [ 1439.713305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1439.716787] Lustre: Skipped 2 previous similar messages [ 1440.577851] Lustre: server umount lustre-MDT0000 complete [ 1441.870771] Lustre: server umount lustre-MDT0001 complete [ 1443.072490] Lustre: server umount MGS complete [ 1446.757325] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:52:18 (1788882738) [ 1456.313389] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 1459.670849] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 1471.479930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1472.720618] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1476.248566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1476.332116] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1476.349352] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1476.387502] Lustre: lustre-MDT0001: new disk, initializing [ 1476.409552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1476.417598] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1476.419591] Lustre: 93335:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1476.421884] LustreError: 93335:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1477.612794] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1481.429842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1481.519231] Lustre: lustre-OST0000: new disk, initializing [ 1481.520808] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1481.523273] Lustre: 94101:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1481.542764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1483.215334] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1485.856746] Lustre: 94571:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000005f59a1f4 x1875779456631552/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788882793 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1486.824874] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1489.005909] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1489.010389] Lustre: MGS: Client 8be39706-f22b-42f6-ac7d-0e130f83e870 (at 0@lo) reconnecting [ 1489.024923] Lustre: 95147:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1489.031765] LustreError: 93319:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1489.035716] LustreError: 93319:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1489.041561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1489.075619] Lustre: lustre-MDT0000: new disk, initializing [ 1489.124723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1490.144777] Lustre: 95171:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000129300c1 x1875779456637056/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788882798 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1490.208519] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1490.355271] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1490.916208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1490.920480] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1492.194832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1492.198120] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1492.262570] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1509.857289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1509.859538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1509.863191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1521.633311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1521.635266] Lustre: Skipped 5 previous similar messages [ 1523.168102] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1523.216455] Lustre: server umount lustre-OST0000 complete [ 1526.753810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1526.754185] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1526.757599] LustreError: Skipped 1 previous similar message [ 1526.761451] Lustre: Skipped 1 previous similar message [ 1530.630806] Lustre: server umount lustre-MDT0000 complete [ 1546.208167] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1546.286125] Lustre: server umount lustre-MDT0001 complete [ 1547.518522] Lustre: server umount MGS complete [ 1551.243810] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:54:03 (1788882843) [ 1557.098494] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 1558.159192] Key type lgssc unregistered [ 1558.289424] LNet: 97664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1558.293260] LNetError: 97664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1558.306333] LNet: Removed LNI 192.168.202.160@tcp [ 1558.612893] Key type .llcrypt unregistered [ 1558.614422] Key type ._llcrypt unregistered [ 1565.351136] Key type ._llcrypt registered [ 1565.352415] Key type .llcrypt registered [ 1565.392894] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 1565.712731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1565.746112] alg: No test for adler32 (adler32-zlib) [ 1566.599362] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 1566.694825] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 1568.272221] Key type lgssc registered [ 1568.686545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1571.285924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1571.291273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1572.438201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1573.726284] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1576.493167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1576.610808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1577.923293] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1580.221189] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1582.150172] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1584.352251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1584.464396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1586.321369] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1588.658715] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1598.830597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1602.017257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1602.021131] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1602.027324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1604.066895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1604.067724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1604.068334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1607.631158] Lustre: server umount lustre-OST0000 complete [ 1609.186062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1609.186703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1609.189280] Lustre: Skipped 1 previous similar message [ 1609.194315] Lustre: Skipped 2 previous similar messages [ 1614.305186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1614.306979] Lustre: Skipped 1 previous similar message [ 1614.995364] Lustre: server umount lustre-MDT0000 complete [ 1616.233121] LustreError: 98842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882908 with bad export cookie 526067242889932733 [ 1616.236476] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1616.236544] LustreError: 98842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1616.357278] Lustre: server umount lustre-MDT0001 complete [ 1619.757707] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 1620.760255] Key type lgssc unregistered [ 1620.891408] LNet: 102410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1620.893663] LNetError: 102410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1620.901289] LNet: Removed LNI 192.168.202.160@tcp [ 1621.212311] Key type .llcrypt unregistered [ 1621.213415] Key type ._llcrypt unregistered [ 1627.691027] Key type ._llcrypt registered [ 1627.692034] Key type .llcrypt registered [ 1627.729289] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 1628.061039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1628.068602] alg: No test for adler32 (adler32-zlib) [ 1628.913081] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 1629.000815] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 1630.584125] Key type lgssc registered [ 1630.934588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1632.996473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1632.998095] LustreError: 103468:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 1632.998639] Lustre: Skipped 1 previous similar message [ 1633.002018] LustreError: 103467:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 1633.004280] LustreError: 103468:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1633.006343] LustreError: 103468:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1633.009235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1633.010538] LustreError: 103468:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1634.151269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1636.677162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1636.765121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1638.040530] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1639.647541] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing lsmod [ 1640.572245] LustreError: 104672:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 1640.579061] LustreError: 104675:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 1640.583036] LustreError: 104672:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4991 [ 1640.586168] LustreError: 104672:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1640.589620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1640.590132] LustreError: 104672:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1640.704363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1641.595953] LustreError: 103534:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1641.598630] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1641.599193] LustreError: 103534:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1641.599720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1641.602932] Lustre: Skipped 2 previous similar messages [ 1641.608269] LustreError: 103534:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 1646.755056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1646.757887] Lustre: Skipped 3 previous similar messages [ 1647.774546] Lustre: server umount lustre-MDT0000 complete [ 1648.972159] LustreError: 103944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882941 with bad export cookie 7206696941603557345 [ 1648.975214] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1648.975549] LustreError: 103944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1650.848264] LustreError: 103935:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006e31a50a x1875779697784960/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1650.891000] Lustre: server umount lustre-MDT0001 complete [ 1662.494396] Lustre: server umount lustre-OST0000 complete [ 1665.145452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1665.286122] LustreError: 105737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1665.294601] LustreError: 105737:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1665.311273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1666.590594] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1669.453582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1669.584212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1670.884437] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1673.220134] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1675.132446] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1677.366871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1679.411879] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1681.659890] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1690.789711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1695.202832] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1695.204952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1695.208653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1696.230829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1696.230936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1699.591141] Lustre: server umount lustre-OST0000 complete [ 1701.345779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1701.346507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1701.349846] Lustre: Skipped 1 previous similar message [ 1701.351689] Lustre: Skipped 2 previous similar messages [ 1706.466381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1706.470909] Lustre: Skipped 2 previous similar messages [ 1715.168092] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1715.227747] Lustre: server umount lustre-MDT0000 complete [ 1716.705410] LustreError: 106467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1716.710135] LustreError: 106467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1716.773831] LustreError: 105718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883008 with bad export cookie 7206696941603557989 [ 1716.780775] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1716.911452] Lustre: server umount lustre-MDT0001 complete [ 1720.815748] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 1721.907357] Key type lgssc unregistered [ 1722.028359] LNet: 109285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.030439] LNetError: 109285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1722.041426] LNet: Removed LNI 192.168.202.160@tcp [ 1722.351593] Key type .llcrypt unregistered [ 1722.353456] Key type ._llcrypt unregistered [ 1730.985156] Key type ._llcrypt registered [ 1730.986186] Key type .llcrypt registered [ 1731.020551] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 1731.354716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1731.405845] alg: No test for adler32 (adler32-zlib) [ 1732.260968] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 1732.349457] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 1733.936123] Key type lgssc registered [ 1734.347896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1736.822713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1736.830246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1737.966163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1739.445379] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1742.331311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1742.456248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1743.941064] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1746.291180] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1747.986705] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1750.171164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1750.288346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1752.229040] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1754.579773] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1764.647712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1766.848320] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1767.642213] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1769.954615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1769.955562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1769.958024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1769.960803] Lustre: Skipped 1 previous similar message [ 1769.963614] Lustre: Skipped 1 previous similar message [ 1770.766664] Lustre: server umount lustre-OST0000 complete [ 1773.537177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1773.540683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1773.546596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1775.074697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1775.074713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1775.079736] Lustre: Skipped 1 previous similar message [ 1778.145164] Lustre: server umount lustre-MDT0000 complete [ 1779.434624] LustreError: 110654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883071 with bad export cookie 17016880344201082208 [ 1779.438135] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1779.443260] LustreError: 110654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1779.570575] Lustre: server umount lustre-MDT0001 complete [ 1783.270872] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 1784.330843] Key type lgssc unregistered [ 1784.466494] LNet: 114323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1784.468832] LNetError: 114323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1784.478354] LNet: Removed LNI 192.168.202.160@tcp [ 1784.789767] Key type .llcrypt unregistered [ 1784.791271] Key type ._llcrypt unregistered [ 1791.889936] Key type ._llcrypt registered [ 1791.891045] Key type .llcrypt registered [ 1791.933042] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 1792.263160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1792.289981] alg: No test for adler32 (adler32-zlib) [ 1793.145473] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 1793.235923] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 1794.816140] Key type lgssc registered [ 1795.180949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1797.864658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1797.872330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1799.023221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1800.418823] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1803.236102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1803.359491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1804.676764] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1807.080905] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1808.992565] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1811.326910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1811.457535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1813.490361] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1815.942101] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1824.807820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1829.860571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1829.864076] Lustre: Skipped 1 previous similar message [ 1829.864148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1829.865531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1829.868511] Lustre: Skipped 1 previous similar message [ 1831.015268] Lustre: server umount lustre-OST0000 complete [ 1833.952910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1833.955770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1833.960317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1834.394913] Lustre: server umount lustre-MDT0000 complete [ 1834.978422] LustreError: 115518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1834.978769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1834.985098] LustreError: 115518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1835.646404] LustreError: 115498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883127 with bad export cookie 17573538417221636324 [ 1835.649568] LustreError: 115498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1835.650490] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1835.774235] Lustre: server umount lustre-MDT0001 complete [ 1839.321937] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 1840.253660] Key type lgssc unregistered [ 1840.377366] LNet: 119066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1840.381170] LNetError: 119066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1840.389351] LNet: Removed LNI 192.168.202.160@tcp [ 1840.730450] Key type .llcrypt unregistered [ 1840.732190] Key type ._llcrypt unregistered [ 1850.751627] Key type ._llcrypt registered [ 1850.752828] Key type .llcrypt registered [ 1850.792291] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 1851.210256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1851.224534] alg: No test for adler32 (adler32-zlib) [ 1852.082610] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 1852.170220] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 1853.760186] Key type lgssc registered [ 1854.158697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1856.797483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1856.803211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1857.943149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1859.177570] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1861.872183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1862.003114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1863.207407] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1865.493764] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1867.267427] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1869.644413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1869.752652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1871.522928] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1873.838403] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1883.048211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1886.591992] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1887.792373] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1887.794316] Lustre: Skipped 2 previous similar messages [ 1889.157874] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1889.160196] Lustre: Skipped 2 previous similar messages [ 1895.934643] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1895.949806] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1899.182472] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.60@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1899.187900] Lustre: Skipped 2 previous similar messages [ 1906.167285] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.60@tcp 192.168.0.[2,10]@tcp [ 1910.570715] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.60/24@tcp in log lustre-MDT0000 [ 1910.575892] Lustre: Skipped 2 previous similar messages [ 1914.617534] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.60/24@tcp [ 1914.619199] Lustre: Skipped 1 previous similar message [ 1918.946990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1918.947656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1918.948659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1918.952396] Lustre: Skipped 1 previous similar message [ 1918.954704] LustreError: Skipped 1 previous similar message [ 1924.054571] Lustre: server umount lustre-OST0000 complete [ 1928.673721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1928.678766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1928.688301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1928.693418] Lustre: Skipped 1 previous similar message [ 1931.415565] Lustre: server umount lustre-MDT0000 complete [ 1932.642585] LustreError: 120437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883224 with bad export cookie 9895425389277232836 [ 1932.647271] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1932.651391] LustreError: 120437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1932.768250] Lustre: server umount lustre-MDT0001 complete [ 1936.544147] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 1937.703574] Key type lgssc unregistered [ 1937.821675] LNet: 125745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1937.825288] LNetError: 125745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1937.837464] LNet: Removed LNI 192.168.202.160@tcp [ 1938.193973] Key type .llcrypt unregistered [ 1938.195167] Key type ._llcrypt unregistered [ 1948.592256] Key type ._llcrypt registered [ 1948.593342] Key type .llcrypt registered [ 1948.630133] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 1948.952238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1949.001055] alg: No test for adler32 (adler32-zlib) [ 1949.845200] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 1949.930941] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 1951.504163] Key type lgssc registered [ 1951.848772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1954.403839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1954.409716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1955.556577] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1955.561355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1955.566257] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.60/24@tcp [ 1956.951870] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1959.761594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1959.870815] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1959.878619] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.60/24@tcp [ 1959.894799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1961.205112] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1963.591513] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1965.361527] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1967.689632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1967.805705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1970.010217] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 1972.660991] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1980.137905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 1990.626195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1990.628649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1990.629660] LustreError: Skipped 1 previous similar message [ 1990.633984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1990.636703] Lustre: Skipped 1 previous similar message [ 1992.151723] Lustre: server umount lustre-OST0000 complete [ 1995.746089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1995.746698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1995.753110] Lustre: Skipped 1 previous similar message [ 1995.754884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1995.757353] Lustre: Skipped 1 previous similar message [ 1999.642950] Lustre: server umount lustre-MDT0000 complete [ 2000.854179] LustreError: 127260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883292 with bad export cookie 3462502583362073358 [ 2000.857509] LustreError: 127260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2000.858447] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2000.976339] Lustre: server umount lustre-MDT0001 complete [ 2004.897727] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 2005.990216] Key type lgssc unregistered [ 2006.143468] LNet: 130897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2006.146142] LNetError: 130897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2006.155522] LNet: Removed LNI 192.168.202.160@tcp [ 2006.517527] Key type .llcrypt unregistered [ 2006.518862] Key type ._llcrypt unregistered [ 2014.817096] Key type ._llcrypt registered [ 2014.818126] Key type .llcrypt registered [ 2014.853054] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 2021.611866] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 2021.925365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2021.932879] alg: No test for adler32 (adler32-zlib) [ 2022.793883] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 2022.886298] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 2024.472226] Key type lgssc registered [ 2024.953602] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2037.592798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2037.598051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.721148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2038.734433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2038.780403] Lustre: lustre-MDT0000: new disk, initializing [ 2038.809406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2038.818067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2040.411951] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2045.156515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2045.205154] Lustre: 134262:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2045.237633] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2045.239810] Lustre: Skipped 1 previous similar message [ 2045.278181] Lustre: lustre-MDT0001: new disk, initializing [ 2045.317291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2045.330704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2045.333831] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2046.843771] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2051.589866] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2053.645766] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2056.335502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2056.481249] Lustre: lustre-OST0000: new disk, initializing [ 2056.483493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2056.486741] Lustre: 136233:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2056.521139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2058.478516] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2062.337926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2062.342459] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2062.380930] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2062.468953] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2064.026350] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2064.118136] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2065.514700] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2065.590349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2067.428360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2067.431262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2067.432991] LustreError: Skipped 1 previous similar message [ 2067.440831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2067.446270] Lustre: Skipped 1 previous similar message [ 2068.320163] Lustre: server umount lustre-OST0000 complete [ 2071.009431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2071.012819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.017101] Lustre: Skipped 1 previous similar message [ 2071.019154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2072.546417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2072.547841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2077.666068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2077.670456] Lustre: Skipped 2 previous similar messages [ 2082.785455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2082.787706] Lustre: Skipped 1 previous similar message [ 2084.320104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2084.405071] Lustre: server umount lustre-MDT0000 complete [ 2085.689139] LustreError: 134255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883377 with bad export cookie 7341660200600591873 [ 2085.694697] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2085.696695] LustreError: 134255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2085.843870] Lustre: server umount lustre-MDT0001 complete [ 2088.763126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2088.966659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2090.185221] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2093.217181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2093.338743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2094.616282] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2097.082746] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2098.901549] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2101.248436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2101.358655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2103.587838] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2106.897499] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2108.896414] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2115.202876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 2116.936987] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2117.026072] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2121.215270] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2121.293438] Lustre: lustre-OST0001: new disk, initializing [ 2121.295814] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2121.299415] Lustre: 141859:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2121.343563] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2123.980900] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2127.790637] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2128.913241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2128.919205] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2128.972943] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2129.939209] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2131.389156] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2131.463572] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2133.137131] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa03506d6c000.ost_server_uuid 50 [ 2133.596919] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa03506d6c000.ost_server_uuid in FULL state after 0 sec [ 2139.108904] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2139.111465] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2139.116415] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2139.119826] Lustre: Skipped 1 previous similar message [ 2141.374529] Lustre: server umount lustre-OST0001 complete [ 2144.226361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2144.227875] LustreError: 140268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2144.231309] Lustre: Skipped 2 previous similar messages [ 2144.236042] LustreError: 140268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2148.688153] Lustre: server umount lustre-OST0000 complete [ 2154.465620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2154.469111] Lustre: Skipped 1 previous similar message [ 2159.585422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2159.586983] Lustre: Skipped 5 previous similar messages [ 2164.192131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2164.251952] Lustre: server umount lustre-MDT0000 complete [ 2164.704876] LustreError: 138447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2164.709549] LustreError: 138447:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2165.424066] LustreError: 140290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883457 with bad export cookie 7341660200600592741 [ 2165.426262] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2165.427363] LustreError: 140290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2165.534842] Lustre: server umount lustre-MDT0001 complete [ 2171.649871] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 2172.604824] Key type lgssc unregistered [ 2172.730322] LNet: 144869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2172.732708] LNetError: 144869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2172.744319] LNet: Removed LNI 192.168.202.160@tcp [ 2173.026774] Key type .llcrypt unregistered [ 2173.027972] Key type ._llcrypt unregistered [ 2184.933106] Key type ._llcrypt registered [ 2184.934084] Key type .llcrypt registered [ 2184.967858] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 2190.838321] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 2191.252735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2191.265751] alg: No test for adler32 (adler32-zlib) [ 2192.110744] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 2192.192990] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 2193.768136] Key type lgssc registered [ 2194.074714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2204.559578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2204.564031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2205.640961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2205.649975] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2205.682855] Lustre: lustre-MDT0000: new disk, initializing [ 2205.701969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2205.707448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2206.925903] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2210.836723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2210.878620] Lustre: 148991:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2210.901640] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2210.904134] Lustre: Skipped 1 previous similar message [ 2210.938473] Lustre: lustre-MDT0001: new disk, initializing [ 2210.973814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2210.984083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2210.986870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2212.270431] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2215.887058] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2217.645462] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2219.779461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2219.875215] Lustre: lustre-OST0000: new disk, initializing [ 2219.877342] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2219.880087] Lustre: 150959:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2219.911016] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2220.439803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2220.442640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2220.475202] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2221.712176] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2225.287965] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2227.191132] Lustre: Failing over lustre-OST0000 [ 2227.247266] Lustre: server umount lustre-OST0000 complete [ 2239.981016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2240.083082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2240.087310] Lustre: Skipped 1 previous similar message [ 2240.091878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2240.097952] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2241.925762] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2242.081654] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2243.211638] Lustre: Failing over lustre-MDT0000 [ 2243.307937] Lustre: server umount lustre-MDT0000 complete [ 2245.602272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2245.606119] Lustre: Skipped 2 previous similar messages [ 2246.624151] Lustre: 147218:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788883522/real 1788883522] req@0000000031247c5d x1875780288145792/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788883538 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2250.721446] LustreError: 149938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.725903] LustreError: 149938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2251.744163] Lustre: 147219:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788883527/real 1788883527] req@000000000ffe35bc x1875780288147584/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788883543 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2252.311727] LustreError: 148999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.841203] LustreError: 149003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.846900] LustreError: 149003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2256.170641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.211754] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2256.215589] Lustre: 148994:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.160@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2256.280310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2256.295331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2256.323283] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2257.431591] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2257.538930] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2261.477742] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2261.485055] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2261.488915] Lustre: Skipped 1 previous similar message [ 2261.508178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2266.594046] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2266.595091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2266.595859] LustreError: Skipped 1 previous similar message [ 2266.599458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2266.601640] Lustre: Skipped 1 previous similar message [ 2268.628076] Lustre: server umount lustre-OST0000 complete [ 2271.713502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2271.713832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2271.714341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2271.720478] Lustre: Skipped 2 previous similar messages [ 2275.918802] Lustre: server umount lustre-MDT0000 complete [ 2276.832981] LustreError: 149003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2276.839652] LustreError: 149003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2277.096931] LustreError: 149786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883569 with bad export cookie 3186203269169234161 [ 2277.099889] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2277.102022] LustreError: 149786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2277.202971] Lustre: server umount lustre-MDT0001 complete [ 2280.726937] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 2281.735058] Key type lgssc unregistered [ 2281.861363] LNet: 154605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2281.863566] LNetError: 154605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2281.873328] LNet: Removed LNI 192.168.202.160@tcp [ 2282.173321] Key type .llcrypt unregistered [ 2282.174701] Key type ._llcrypt unregistered [ 2301.881560] Key type ._llcrypt registered [ 2301.882520] Key type .llcrypt registered [ 2301.915075] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 2302.222347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2302.229568] alg: No test for adler32 (adler32-zlib) [ 2303.071467] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 2303.154548] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 2304.736174] Key type lgssc registered [ 2305.086994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2307.572983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2307.579903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.718972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2310.059170] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2313.654430] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2315.579534] LustreError: 156147:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2315.582615] LustreError: 156147:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2315.585211] LustreError: 156147:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2315.705392] Lustre: server umount lustre-MDT0000 complete [ 2320.417495] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 2323.478785] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 2334.534635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2334.626567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2334.636498] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2334.674698] Lustre: lustre-MDT0000: new disk, initializing [ 2334.694993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2334.701335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2335.979518] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2340.026661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2340.068156] Lustre: 159441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2340.091619] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2340.094161] Lustre: Skipped 1 previous similar message [ 2340.131271] Lustre: lustre-MDT0001: new disk, initializing [ 2340.170905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2340.183418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2340.187644] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2341.418739] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2344.902902] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2346.594915] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2348.734153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2348.829532] Lustre: lustre-OST0000: new disk, initializing [ 2348.831645] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2348.834333] Lustre: 161412:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2348.859687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2350.664633] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2354.275386] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2355.706842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2355.709853] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2355.738115] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2361.467973] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2362.015426] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 12:07:34 (1788883654) [ 2422.240749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2422.240755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2422.240808] Lustre: Skipped 1 previous similar message [ 2422.246771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2422.253476] Lustre: Skipped 2 previous similar messages [ 2427.360454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2427.362294] Lustre: Skipped 2 previous similar messages [ 2427.754052] Lustre: server umount lustre-MDT0000 complete [ 2428.973425] LustreError: 159435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883721 with bad export cookie 14870554306820789932 [ 2428.974557] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2428.976734] LustreError: 159435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2432.480434] LustreError: 162084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2432.480506] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2432.486169] LustreError: 162084:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2432.489896] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2433.504448] LustreError: 159449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2438.624315] LustreError: 159450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2438.624376] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2438.633347] Lustre: Skipped 1 previous similar message [ 2443.232123] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2443.303299] Lustre: server umount lustre-MDT0001 complete [ 2450.631366] Lustre: server umount lustre-OST0000 complete [ 2453.731384] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 2456.983444] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 2468.673748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2468.750923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2468.759951] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2468.795569] Lustre: lustre-MDT0000: new disk, initializing [ 2468.816188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2468.822327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2470.162897] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2474.205533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2474.247350] Lustre: 166040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2474.280438] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2474.281798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2474.282426] Lustre: Skipped 1 previous similar message [ 2474.284119] Lustre: Skipped 1 previous similar message [ 2474.318976] Lustre: lustre-MDT0001: new disk, initializing [ 2474.361032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2474.364648] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2475.659696] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2479.297718] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2481.082293] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2483.265089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2483.361165] Lustre: lustre-OST0000: new disk, initializing [ 2483.363281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2483.366201] Lustre: 168005:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2483.395680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2483.399060] Lustre: Skipped 1 previous similar message [ 2483.462701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2483.467908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2483.504803] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2485.436528] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2489.209220] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2490.758421] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2490.845961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2492.260235] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2492.338710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2493.923455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2493.924166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2493.927733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2493.928985] LustreError: Skipped 1 previous similar message [ 2493.934988] Lustre: Skipped 1 previous similar message [ 2507.232161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2507.292026] Lustre: server umount lustre-OST0000 complete [ 2509.281704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.285370] Lustre: Skipped 2 previous similar messages [ 2514.401672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2514.405117] Lustre: Skipped 7 previous similar messages [ 2514.650216] Lustre: server umount lustre-MDT0000 complete [ 2515.925466] LustreError: 166032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883808 with bad export cookie 14870554306820948762 [ 2515.929076] LustreError: 166032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2515.929706] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2516.044277] Lustre: server umount lustre-MDT0001 complete [ 2518.882150] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:10:11 (1788883811) [ 2524.197197] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 2527.483516] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 2539.172198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2539.258545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2539.269681] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2539.312233] Lustre: lustre-MDT0000: new disk, initializing [ 2539.334289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2539.340672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2540.762537] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2544.941760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2544.985114] Lustre: 172754:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2545.086647] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2546.388723] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2550.164967] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2551.897158] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2553.918444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2554.007465] Lustre: 174734:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2555.388273] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2555.430279] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2555.842015] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2559.536994] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2563.412591] Lustre: server umount lustre-OST0000 complete [ 2565.602039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2565.602279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2565.602933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2565.607535] Lustre: Skipped 1 previous similar message [ 2574.845086] Lustre: server umount lustre-MDT0000 complete [ 2575.841036] LustreError: 172766:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.845854] LustreError: 172766:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2576.081363] LustreError: 172748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883868 with bad export cookie 14870554306820949616 [ 2576.085118] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2576.085809] LustreError: 172748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2586.642397] Lustre: server umount lustre-MDT0001 complete [ 2589.470496] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:11:21 (1788883881) [ 2594.567514] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 2597.637098] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 2609.125865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2609.206575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2609.216356] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2609.217902] Lustre: Skipped 3 previous similar messages [ 2609.255497] Lustre: lustre-MDT0000: new disk, initializing [ 2609.256925] Lustre: Skipped 2 previous similar messages [ 2609.278468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2609.280805] Lustre: Skipped 2 previous similar messages [ 2609.285926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2609.288521] Lustre: Skipped 2 previous similar messages [ 2610.689140] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2614.864060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2614.910550] Lustre: 179046:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2615.015310] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2616.342599] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2620.068429] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2621.936436] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2624.174538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2624.280448] Lustre: 181024:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2626.234551] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2630.002111] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2630.684517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2640.866460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2640.867273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2640.868907] LustreError: Skipped 1 previous similar message [ 2640.869325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2640.872667] Lustre: Skipped 1 previous similar message [ 2640.874126] Lustre: Skipped 3 previous similar messages [ 2643.149891] Lustre: server umount lustre-OST0000 complete [ 2645.986095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2650.522763] Lustre: server umount lustre-MDT0000 complete [ 2651.105440] LustreError: 180879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2651.110621] LustreError: 180879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2651.805269] LustreError: 179847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883943 with bad export cookie 14870554306820950687 [ 2651.808105] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2651.810641] LustreError: 179847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2651.935698] Lustre: server umount lustre-MDT0001 complete [ 2655.615313] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 2656.628891] Key type lgssc unregistered [ 2656.758406] LNet: 182877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2656.761845] LNetError: 182877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2656.774325] LNet: Removed LNI 192.168.202.160@tcp [ 2657.105937] Key type .llcrypt unregistered [ 2657.107241] Key type ._llcrypt unregistered [ 2666.298716] Key type ._llcrypt registered [ 2666.299946] Key type .llcrypt registered [ 2666.340428] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 2666.711294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2666.750945] alg: No test for adler32 (adler32-zlib) [ 2667.597807] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 2667.678290] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 2669.256143] Key type lgssc registered [ 2669.575432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2671.942469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2671.947646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2673.075064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2674.296191] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2676.906052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2677.020263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2678.243741] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2680.404166] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2682.099984] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2684.198917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2684.319649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2686.053381] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2688.231602] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2698.664531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2713.057403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2713.059897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2713.064258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2714.083345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2714.084387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2714.084455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2714.086559] Lustre: Skipped 1 previous similar message [ 2716.430219] Lustre: server umount lustre-OST0000 complete [ 2718.176964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2718.179753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2718.184739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2723.861552] Lustre: server umount lustre-MDT0000 complete [ 2724.321068] LustreError: 184266:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2724.325838] LustreError: 184266:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2725.071680] LustreError: 184246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884017 with bad export cookie 3485980600178689283 [ 2725.074791] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2725.075073] LustreError: 184246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2725.184927] Lustre: server umount lustre-MDT0001 complete [ 2728.883554] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 2729.868474] Key type lgssc unregistered [ 2729.994433] LNet: 187815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2729.997399] LNetError: 187815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2730.007317] LNet: Removed LNI 192.168.202.160@tcp [ 2730.309923] Key type .llcrypt unregistered [ 2730.311328] Key type ._llcrypt unregistered [ 2739.303537] Key type ._llcrypt registered [ 2739.304538] Key type .llcrypt registered [ 2739.341178] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 2739.664441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2739.701919] alg: No test for adler32 (adler32-zlib) [ 2740.551685] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 2740.639672] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 2742.216192] Key type lgssc registered [ 2742.581538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2745.145179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2745.152083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2746.297743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2747.650281] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2750.515548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2750.662816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2752.029956] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2754.449443] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2756.354328] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2758.634689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2758.750541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2760.697077] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2763.152148] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2772.071273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2776.545488] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2776.547686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2776.551489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2777.570962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2777.571289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2777.572209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2778.659303] Lustre: lustre-OST0000: Not available for connect from 192.168.202.60@tcp (stopping) [ 2778.662447] Lustre: Skipped 1 previous similar message [ 2782.689933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2782.691952] Lustre: Skipped 1 previous similar message [ 2787.810598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2787.813400] Lustre: Skipped 2 previous similar messages [ 2789.856146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2789.906201] Lustre: server umount lustre-OST0000 complete [ 2792.380662] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffffa03506446000.ost_server_uuid 50 [ 2792.938220] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffffa03506446000.ost_server_uuid in DISCONN state after 0 sec [ 2793.508350] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2807.264965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2807.267191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2807.271405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2807.272993] Lustre: Skipped 1 previous similar message [ 2809.690264] Lustre: server umount lustre-MDT0000 complete [ 2810.971157] LustreError: 189186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884103 with bad export cookie 2756249246849265114 [ 2810.974137] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2810.975344] LustreError: 189186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2811.101986] Lustre: server umount lustre-MDT0001 complete [ 2814.010868] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 12:15:06 (1788884106) [ 2816.254975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2816.411367] LustreError: 193167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2816.417273] LustreError: 193167:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2816.436850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2817.808912] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2820.600117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2820.721493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2822.013931] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2824.349467] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2826.208397] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2828.439200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2830.456974] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2832.850561] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2835.125385] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2835.203790] Lustre: lustre-OST0001: new disk, initializing [ 2835.205487] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2835.207932] Lustre: 196002:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2835.243544] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2835.245542] Lustre: Skipped 1 previous similar message [ 2837.198885] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2840.975732] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2842.114175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2842.117333] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2842.149086] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2843.177240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2848.229505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2848.230920] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2848.231041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2848.231057] Lustre: Skipped 2 previous similar messages [ 2848.235073] Lustre: Skipped 2 previous similar messages [ 2853.052583] Lustre: server umount lustre-OST0000 complete [ 2853.348558] LustreError: 194987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2853.354677] LustreError: 194987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2855.104813] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2855.176409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2856.474506] LustreError: 194987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2856.595079] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2856.668993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2861.593338] LustreError: 194987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2861.602105] LustreError: 194987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2867.681201] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2867.681303] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2867.684325] LustreError: Skipped 1 previous similar message [ 2867.689447] Lustre: Skipped 1 previous similar message [ 2868.707465] LustreError: 194987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2868.714339] LustreError: 194987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2872.855498] Lustre: server umount lustre-OST0001 complete [ 2876.896743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2876.899058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2880.147609] Lustre: server umount lustre-MDT0000 complete [ 2881.246072] LustreError: 193149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884173 with bad export cookie 2756249246849266171 [ 2881.249054] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2881.249357] LustreError: 193149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2881.351378] Lustre: server umount lustre-MDT0001 complete [ 2888.478761] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 12:16:20 (1788884180) [ 2890.556042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.655611] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2890.668241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2890.755345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2891.984107] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2894.600926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2894.632743] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2894.644423] Lustre: 199421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2896.115873] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2898.286632] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2900.069066] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2902.319037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2902.393205] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2903.443051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 2904.431437] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2906.758161] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2908.839291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2908.861550] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2910.683832] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2912.831084] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2913.761134] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2913.761179] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2913.763372] LustreError: Skipped 1 previous similar message [ 2913.766826] Lustre: Skipped 1 previous similar message [ 2913.769515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2913.772767] Lustre: Skipped 10 previous similar messages [ 2919.740399] Lustre: server umount lustre-OST0000 complete [ 2920.931302] LustreError: 201269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2920.935882] LustreError: 201269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2936.266755] Lustre: server umount lustre-OST0001 complete [ 2943.509917] Lustre: server umount lustre-MDT0000 complete [ 2944.640017] LustreError: 203177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884236 with bad export cookie 2756249246849267557 [ 2944.643763] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2944.643878] LustreError: 203177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2944.750101] Lustre: server umount lustre-MDT0001 complete [ 2951.852774] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 12:17:24 (1788884244) [ 2956.803447] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 2959.779933] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 2970.480694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2970.555479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2970.565205] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2970.599394] Lustre: lustre-MDT0000: new disk, initializing [ 2970.618715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2970.621105] Lustre: Skipped 3 previous similar messages [ 2970.626347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2971.824833] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2975.777704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.818396] Lustre: 207480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2975.849681] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2975.851194] Lustre: Skipped 1 previous similar message [ 2975.888693] Lustre: lustre-MDT0001: new disk, initializing [ 2975.934550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2975.938192] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2977.238945] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2980.732709] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2982.426604] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2984.544233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2984.635310] Lustre: lustre-OST0000: new disk, initializing [ 2984.637754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2984.641300] Lustre: 209449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2986.462687] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 2989.969052] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2990.581854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2990.585610] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2990.610373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2991.256992] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2991.318082] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2992.542875] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2992.605233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2995.685209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2995.685710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2995.686282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2995.686288] Lustre: Skipped 9 previous similar messages [ 2995.688220] Lustre: Skipped 5 previous similar messages [ 2995.690420] LustreError: Skipped 3 previous similar messages [ 2999.310823] Lustre: server umount lustre-OST0000 complete [ 3006.554028] Lustre: server umount lustre-MDT0000 complete [ 3007.727285] LustreError: 207475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884299 with bad export cookie 2756249246849269104 [ 3007.730535] LustreError: 207475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3007.730907] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3007.841293] Lustre: server umount lustre-MDT0001 complete [ 3010.547975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3010.689596] LustreError: 211663:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3010.695901] LustreError: 211663:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3011.961636] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3014.598899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3015.931089] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3018.166708] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3019.824868] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3021.902453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3023.728789] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3025.902876] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3027.174988] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3035.389794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 3036.756483] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3036.824401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3043.535385] Lustre: server umount lustre-OST0000 complete [ 3045.595436] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3045.673227] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3047.080569] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3070.773613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3070.906153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3070.910316] Lustre: Skipped 5 previous similar messages [ 3072.788426] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3075.010561] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3075.516960] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3078.536104] Lustre: server umount lustre-OST0000 complete [ 3079.684694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3079.687814] Lustre: Skipped 5 previous similar messages [ 3085.845594] Lustre: server umount lustre-MDT0000 complete [ 3087.068122] LustreError: 211645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884379 with bad export cookie 2756249246849269986 [ 3087.070595] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3087.071522] LustreError: 211645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3087.178907] Lustre: server umount lustre-MDT0001 complete [ 3089.765304] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 12:19:41 (1788884381) [ 3091.943626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3092.086144] LustreError: 217691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3093.385040] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3095.965668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3097.327937] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3099.481289] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3101.201771] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3103.286487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3105.123836] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3107.431981] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3108.796466] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3118.042598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 3119.376271] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3119.442178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3121.520800] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3121.586512] Lustre: lustre-OST0001: new disk, initializing [ 3121.587961] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3121.590302] Lustre: 221120:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3123.368918] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3126.906360] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3128.244884] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3130.879718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3130.882561] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3130.905136] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3131.360500] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3132.609650] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3132.671494] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3135.971900] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3135.973860] LustreError: Skipped 2 previous similar messages [ 3135.975379] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3135.977717] Lustre: Skipped 13 previous similar messages [ 3139.322352] Lustre: server umount lustre-OST0001 complete [ 3141.060872] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3141.134976] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3142.552705] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3142.630447] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3156.453503] LustreError: 222093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3156.459740] LustreError: 222093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 3166.244701] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3168.281477] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3170.785284] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3171.327386] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3171.818431] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3171.823058] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3171.823153] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3174.076288] Lustre: server umount lustre-OST0001 complete [ 3192.207650] Lustre: server umount lustre-OST0000 complete [ 3199.642958] Lustre: server umount lustre-MDT0000 complete [ 3200.792912] LustreError: 217671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884492 with bad export cookie 2756249246849270917 [ 3200.795641] LustreError: 217671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3200.796305] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3215.328123] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3215.403760] Lustre: server umount lustre-MDT0001 complete [ 3222.537574] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 12:21:54 (1788884514) [ 3224.634587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3224.732109] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3224.742266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3224.817298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3224.819248] Lustre: Skipped 5 previous similar messages [ 3226.043580] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3228.674907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3228.705896] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3228.715311] Lustre: 226377:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3230.031753] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3232.246059] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3233.951204] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3236.031929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3236.093514] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3237.911446] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3240.132414] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3249.083278] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3249.113299] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3250.999455] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3253.475113] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3257.073206] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 3258.487251] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3258.569253] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3260.002500] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3260.074915] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3261.889577] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa03502cf7800.ost_server_uuid 50 [ 3262.413545] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa03502cf7800.ost_server_uuid in FULL state after 0 sec [ 3262.755283] Lustre: Permanently deactivating lustre-OST0001 [ 3262.760859] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3263.577145] Lustre: Permanently reactivating lustre-OST0001 [ 3268.577533] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3268.579550] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3268.580736] LustreError: Skipped 4 previous similar messages [ 3268.584984] Lustre: Skipped 8 previous similar messages [ 3270.655184] Lustre: server umount lustre-OST0001 complete [ 3278.037570] Lustre: server umount lustre-OST0000 complete [ 3285.467832] Lustre: server umount lustre-MDT0000 complete [ 3286.751038] LustreError: 226369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884578 with bad export cookie 2756249246849272191 [ 3286.754427] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3286.754885] LustreError: 226369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3286.864554] Lustre: server umount lustre-MDT0001 complete [ 3294.919071] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 12:23:07 (1788884587) [ 3297.537303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3297.648557] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3297.660891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3297.663805] Lustre: Skipped 5 previous similar messages [ 3299.189823] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3302.114766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3302.147947] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3302.157420] Lustre: 233050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3303.649965] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3306.151483] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3308.117782] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3310.456492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3310.542255] Lustre: Permanently deactivating lustre-OST0000 [ 3310.547781] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3311.734049] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3311.737827] LustreError: 235099:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 3311.743812] LustreError: 235099:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 3312.624032] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3315.044702] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3317.161522] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3319.072807] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3319.707084] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3321.455430] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3322.037174] Lustre: Permanently reactivating lustre-OST0000 [ 3325.689826] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3325.692401] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3325.696183] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3325.697796] Lustre: Skipped 1 previous similar message [ 3325.707755] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3325.713081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3338.110582] Lustre: server umount lustre-OST0001 complete [ 3340.260077] LustreError: 234897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3340.263300] LustreError: 234897:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 3345.435172] Lustre: server umount lustre-OST0000 complete [ 3346.815618] Lustre: server umount lustre-MDT0000 complete [ 3347.962211] LustreError: 234918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884640 with bad export cookie 2756249246849273794 [ 3347.966371] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3354.163594] Lustre: server umount lustre-MDT0001 complete [ 3357.948928] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 3358.998369] Key type lgssc unregistered [ 3359.114350] LNet: 237903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3359.118127] LNetError: 237903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3359.125339] LNet: Removed LNI 192.168.202.160@tcp [ 3359.452507] Key type .llcrypt unregistered [ 3359.453772] Key type ._llcrypt unregistered [ 3368.203530] Key type ._llcrypt registered [ 3368.204598] Key type .llcrypt registered [ 3368.248773] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 3368.626801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3368.667052] alg: No test for adler32 (adler32-zlib) [ 3369.544084] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 3369.655247] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 3371.248127] Key type lgssc registered [ 3371.583909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3375.010122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3375.016405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3376.165465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3377.623464] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3380.537238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3380.576257] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3380.587427] Lustre: 239333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3380.592115] Lustre: Permanently deactivating lustre-MDT0001 [ 3380.597456] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3380.708293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3382.098511] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3382.127672] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3382.130701] LustreError: 239374:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3382.134741] LustreError: 239374:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3382.138806] LustreError: 239374:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 3384.614081] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3386.480421] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3388.733529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3388.846694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3390.758532] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3390.883662] LustreError: 241565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3390.886754] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3390.889321] LustreError: 241565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3392.930905] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3395.124755] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3395.192312] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3397.130255] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3399.549644] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3400.678973] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3401.704859] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3401.705046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3405.265561] Lustre: Permanently reactivating lustre-MDT0001 [ 3405.271376] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3405.274579] Lustre: Skipped 2 previous similar messages [ 3406.302129] Lustre: lustre-MDT0001: Client 51be56ea-90e1-4f5b-8e35-96130f5c642c (at 192.168.202.60@tcp) reconnecting [ 3408.087588] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3408.598335] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3410.338264] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3410.415893] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3411.785740] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3414.912321] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 3 sec [ 3415.713718] Lustre: Permanently deactivating lustre-MDT0001 [ 3415.718755] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3415.720533] Lustre: Skipped 2 previous similar messages [ 3422.725175] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3425.762790] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3425.765477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3425.770129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3427.297428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3427.299108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.300785] Lustre: Skipped 1 previous similar message [ 3427.306088] Lustre: Skipped 1 previous similar message [ 3430.484926] Lustre: server umount lustre-MDT0000 complete [ 3431.682155] LustreError: 243917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884723 with bad export cookie 3446564792453587023 [ 3431.684731] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3431.685350] LustreError: 243917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3431.798669] Lustre: server umount lustre-MDT0001 complete [ 3439.105898] Lustre: server umount lustre-OST0000 complete [ 3442.806473] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:25:34 (1788884734) [ 3446.255945] Lustre: server umount lustre-OST0001 complete [ 3448.398526] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 3451.623662] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 3463.287223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3463.367645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3463.372433] Lustre: Skipped 2 previous similar messages [ 3463.380132] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3463.417851] Lustre: lustre-MDT0000: new disk, initializing [ 3463.439349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3463.444650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3464.766445] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3468.873469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3468.917414] Lustre: 247613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3468.940716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3468.942463] Lustre: Skipped 1 previous similar message [ 3468.979126] Lustre: lustre-MDT0001: new disk, initializing [ 3469.024020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3469.026792] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3470.401085] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3474.121965] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3475.937140] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3478.246379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3478.354391] Lustre: lustre-OST0000: new disk, initializing [ 3478.356248] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3478.359182] Lustre: 249579:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3478.391024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3478.393314] Lustre: Skipped 1 previous similar message [ 3479.608532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3479.611751] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3479.644572] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3480.290579] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3484.053925] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3484.717356] LustreError: 249587:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3488.842392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3488.901143] Lustre: lustre-OST0001: new disk, initializing [ 3488.902502] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3488.904684] Lustre: 250695:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3490.827790] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3494.409755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3494.414460] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3494.453626] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3494.513420] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3494.808091] LustreError: 249587:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3494.815053] LustreError: 247621:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3495.049188] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3504.904136] LustreError: 247621:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3508.255136] Lustre: lustre-OST0001: Not available for connect from 192.168.202.60@tcp (stopping) [ 3508.258250] Lustre: Skipped 2 previous similar messages [ 3509.732157] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3509.732894] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3509.734258] LustreError: Skipped 1 previous similar message [ 3509.737345] Lustre: Skipped 1 previous similar message [ 3511.610114] Lustre: server umount lustre-OST0001 complete [ 3514.852647] LustreError: 249565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3514.858309] LustreError: 249565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3519.973272] LustreError: 251464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3519.977325] LustreError: 251464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3525.090902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3525.092781] LustreError: 251464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3525.093397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3525.093401] Lustre: Skipped 2 previous similar messages [ 3525.094609] Lustre: Skipped 1 previous similar message [ 3525.101073] LustreError: 251464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3529.682884] Lustre: server umount lustre-OST0000 complete [ 3535.330593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3535.331525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3535.334807] Lustre: Skipped 1 previous similar message [ 3535.338345] Lustre: Skipped 2 previous similar messages [ 3537.126220] Lustre: server umount lustre-MDT0000 complete [ 3538.351177] LustreError: 247606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884830 with bad export cookie 3446564792453589837 [ 3538.354227] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3538.355931] LustreError: 247606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3544.501378] Lustre: server umount lustre-MDT0001 complete [ 3548.298054] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 3549.324728] Key type lgssc unregistered [ 3549.450392] LNet: 252726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3549.452582] LNetError: 252726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3549.463376] LNet: Removed LNI 192.168.202.160@tcp [ 3549.817333] Key type .llcrypt unregistered [ 3549.818407] Key type ._llcrypt unregistered [ 3563.762818] Key type ._llcrypt registered [ 3563.763817] Key type .llcrypt registered [ 3563.801197] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 3564.214729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3564.225708] alg: No test for adler32 (adler32-zlib) [ 3565.072988] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 3565.161466] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 3566.736128] Key type lgssc registered [ 3567.084664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3569.556900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3569.562767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3570.644819] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3570.651724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3570.711908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3572.051357] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3574.876363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3574.910126] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3574.923362] Lustre: 254905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3575.040193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3576.406629] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3578.844380] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3580.746293] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3583.160619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3583.227794] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3583.299725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3585.249138] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3587.156157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3587.753946] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3628.003678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3628.004526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3628.008394] LustreError: Skipped 1 previous similar message [ 3628.014795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3628.018694] Lustre: Skipped 1 previous similar message [ 3631.011169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3631.013577] Lustre: Skipped 1 previous similar message [ 3632.922573] Lustre: server umount lustre-OST0000 complete [ 3634.390301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3638.677781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3638.793446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3640.848925] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3643.397122] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3656.164804] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3656.168980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3656.169138] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3656.175399] Lustre: Skipped 1 previous similar message [ 3676.642923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3676.643186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3676.643967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3676.647133] Lustre: Skipped 2 previous similar messages [ 3676.652731] LustreError: Skipped 1 previous similar message [ 3681.763089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3681.765151] Lustre: Skipped 2 previous similar messages [ 3682.461947] Lustre: server umount lustre-OST0000 complete [ 3686.881828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3686.882671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3686.887375] Lustre: Skipped 1 previous similar message [ 3695.009661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3695.011660] Lustre: Skipped 4 previous similar messages [ 3698.144148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3698.211069] Lustre: server umount lustre-MDT0000 complete [ 3699.470439] LustreError: 254941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884991 with bad export cookie 10251993496550437847 [ 3699.473681] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3699.482735] LustreError: 254941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3699.604502] Lustre: server umount lustre-MDT0001 complete [ 3703.453077] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 3704.562579] Key type lgssc unregistered [ 3704.692383] LNet: 260316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3704.694880] LNetError: 260316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3704.704360] LNet: Removed LNI 192.168.202.160@tcp [ 3705.024238] Key type .llcrypt unregistered [ 3705.025525] Key type ._llcrypt unregistered [ 3714.103562] Key type ._llcrypt registered [ 3714.104622] Key type .llcrypt registered [ 3714.146993] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 3714.636604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3714.646386] alg: No test for adler32 (adler32-zlib) [ 3715.518123] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 3715.611915] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 3717.192145] Key type lgssc registered [ 3717.500390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3719.990853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3719.997309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3721.136077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3722.497177] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3725.448083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3725.578476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3727.019316] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3729.443226] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3731.372475] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3733.664442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3733.786181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3735.773364] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3738.340389] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3747.115306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3756.513498] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3756.517640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3756.523915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3757.538874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3757.540410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3757.540542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3762.577399] Lustre: server umount lustre-OST0000 complete [ 3766.752818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3766.755489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3766.759906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3766.762081] Lustre: Skipped 1 previous similar message [ 3769.945382] Lustre: server umount lustre-MDT0000 complete [ 3771.241273] LustreError: 261686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788885063 with bad export cookie 14593469001882756580 [ 3771.244617] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3771.246067] LustreError: 261686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3771.359812] Lustre: server umount lustre-MDT0001 complete [ 3775.088527] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 3776.135542] Key type lgssc unregistered [ 3776.278458] LNet: 266026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3776.280947] LNetError: 266026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3776.295324] LNet: Removed LNI 192.168.202.160@tcp [ 3776.630594] Key type .llcrypt unregistered [ 3776.631714] Key type ._llcrypt unregistered [ 3784.004375] Key type ._llcrypt registered [ 3784.005649] Key type .llcrypt registered [ 3784.047084] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 3784.406264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3784.474909] alg: No test for adler32 (adler32-zlib) [ 3785.332324] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 3785.423075] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 3787.008209] Key type lgssc registered [ 3787.396236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3790.131890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3790.137849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3791.289892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3792.656782] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3795.584081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3795.714977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3797.085632] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3799.417395] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3801.253803] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3803.489271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3803.604213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3805.555508] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3807.997797] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3816.938130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3822.049818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3822.050265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3822.052822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3822.056845] Lustre: Skipped 1 previous similar message [ 3824.017587] Lustre: server umount lustre-OST0000 complete [ 3826.656861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3826.659044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3826.662542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3826.664289] Lustre: Skipped 1 previous similar message [ 3827.354067] Lustre: server umount lustre-MDT0000 complete [ 3828.628186] LustreError: 267202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788885120 with bad export cookie 14389643785477181951 [ 3828.631263] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3828.633909] LustreError: 267202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3828.750332] Lustre: server umount lustre-MDT0001 complete [ 3832.626072] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 3833.653553] Key type lgssc unregistered [ 3833.780368] LNet: 270917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3833.783023] LNetError: 270917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3833.791325] LNet: Removed LNI 192.168.202.160@tcp [ 3834.102382] Key type .llcrypt unregistered [ 3834.103480] Key type ._llcrypt unregistered [ 3843.116274] Key type ._llcrypt registered [ 3843.117227] Key type .llcrypt registered [ 3843.155928] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 3843.503561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3843.563655] alg: No test for adler32 (adler32-zlib) [ 3844.422662] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 3844.511688] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 3846.088215] Key type lgssc registered [ 3846.454304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3848.994959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3849.001283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3850.142052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3851.517427] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3854.464544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3854.600480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3855.991437] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3858.510523] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3860.390541] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3862.654517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3862.770215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3864.663125] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3867.053346] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3876.076242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3885.537280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3885.539611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3885.543856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3886.562562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3886.564161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3886.570395] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3890.962978] Lustre: server umount lustre-OST0000 complete [ 3895.777156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3895.779452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3895.783402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3895.784891] Lustre: Skipped 1 previous similar message [ 3898.328092] Lustre: server umount lustre-MDT0000 complete [ 3899.607111] LustreError: 272285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788885191 with bad export cookie 15769797661991868631 [ 3899.609082] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3899.611898] LustreError: 272285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3899.725656] Lustre: server umount lustre-MDT0001 complete [ 3903.562960] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 3904.607186] Key type lgssc unregistered [ 3904.753433] LNet: 276627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3904.756135] LNetError: 276627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3904.767322] LNet: Removed LNI 192.168.202.160@tcp [ 3905.078355] Key type .llcrypt unregistered [ 3905.079574] Key type ._llcrypt unregistered [ 3912.364934] Key type ._llcrypt registered [ 3912.366007] Key type .llcrypt registered [ 3912.405588] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 3912.782796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3912.821686] alg: No test for adler32 (adler32-zlib) [ 3913.688961] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 3913.782881] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 3915.360171] Key type lgssc registered [ 3915.718075] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3918.238908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3918.245059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3919.405428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3920.729527] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3923.559038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3923.706728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3925.260967] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3927.693839] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3929.584039] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3931.785225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3931.895305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3933.870036] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 3936.307271] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3945.191762] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3954.657264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3954.659596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3954.663302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3955.682287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3955.682736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3955.684405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3956.685036] Lustre: server umount lustre-OST0000 complete [ 3959.776914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3959.779254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3959.782804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3959.784330] Lustre: Skipped 1 previous similar message [ 3964.057625] Lustre: server umount lustre-MDT0000 complete [ 3965.244117] LustreError: 277802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788885257 with bad export cookie 14106369704233831282 [ 3965.247436] LustreError: 277802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3965.247918] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3965.365803] Lustre: server umount lustre-MDT0001 complete [ 3968.896365] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 3969.886275] Key type lgssc unregistered [ 3970.010353] LNet: 281560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3970.012524] LNetError: 281560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3970.020312] LNet: Removed LNI 192.168.202.160@tcp [ 3970.331735] Key type .llcrypt unregistered [ 3970.332797] Key type ._llcrypt unregistered [ 3973.962604] Key type ._llcrypt registered [ 3973.963636] Key type .llcrypt registered [ 3974.002478] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 3979.567120] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 3985.777506] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 3986.223435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3986.240760] alg: No test for adler32 (adler32-zlib) [ 3987.113707] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 3987.210728] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 3988.792224] Key type lgssc registered [ 3989.190828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4000.891054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4000.896580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4001.981949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4001.992990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4002.030552] Lustre: lustre-MDT0000: new disk, initializing [ 4002.051250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4002.056944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4003.377457] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4007.474803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4007.517200] Lustre: 285077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4007.540610] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4007.542504] Lustre: Skipped 1 previous similar message [ 4007.578459] Lustre: lustre-MDT0001: new disk, initializing [ 4007.609281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4007.622378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4007.625716] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4009.038791] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4012.789856] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4014.606671] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4016.795772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4016.893622] Lustre: lustre-OST0000: new disk, initializing [ 4016.895544] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4016.898312] Lustre: 287046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4016.924120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4018.804477] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4018.877709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4018.881387] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4018.919391] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4022.488055] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4023.856726] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4023.925463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4025.246223] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4025.313305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4028.044450] Lustre: server umount lustre-OST0000 complete [ 4029.409624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4029.410134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4029.413385] Lustre: Skipped 1 previous similar message [ 4034.529224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4034.531972] Lustre: Skipped 2 previous similar messages [ 4039.650630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4039.652805] Lustre: Skipped 1 previous similar message [ 4039.683669] Lustre: server umount lustre-MDT0000 complete [ 4040.948562] LustreError: 285071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788885333 with bad export cookie 10639238680744401104 [ 4040.952426] LustreError: 285071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4040.954120] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4051.477923] Lustre: server umount lustre-MDT0001 complete [ 4054.346030] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 12:35:46 (1788885346) [ 4056.598240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4056.797706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4058.207725] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4061.093735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4061.223885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4062.594372] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4065.043330] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4066.966750] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4069.229382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4069.339224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4071.298470] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4073.803415] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4087.778898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4087.779347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4087.780158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4087.787358] Lustre: Skipped 1 previous similar message [ 4092.898214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4092.900235] Lustre: Skipped 2 previous similar messages [ 4093.268276] Lustre: server umount lustre-OST0000 complete [ 4096.992819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4096.995372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4100.760374] Lustre: server umount lustre-MDT0000 complete [ 4102.022326] LustreError: 289435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788885394 with bad export cookie 10639238680744401972 [ 4102.025685] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4102.027624] LustreError: 289435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4102.154524] Lustre: server umount lustre-MDT0001 complete [ 4105.941231] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 4107.041201] Key type lgssc unregistered [ 4107.169423] LNet: 293010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4107.171797] LNetError: 293010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4107.178384] LNet: Removed LNI 192.168.202.160@tcp [ 4107.503706] Key type .llcrypt unregistered [ 4107.504831] Key type ._llcrypt unregistered [ 4116.143958] Key type ._llcrypt registered [ 4116.145142] Key type .llcrypt registered [ 4116.186177] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 4122.003890] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 4122.318708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4122.341924] alg: No test for adler32 (adler32-zlib) [ 4123.189208] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 4123.272252] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 4124.848124] Key type lgssc registered [ 4125.195679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4140.380901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4140.388379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4141.473962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4141.485709] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4141.522181] Lustre: lustre-MDT0000: new disk, initializing [ 4141.543637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4141.549596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4142.986499] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4147.089842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4147.130047] Lustre: 296800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4147.152738] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4147.154425] Lustre: Skipped 1 previous similar message [ 4147.192845] Lustre: lustre-MDT0001: new disk, initializing [ 4147.222224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4147.232692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4147.235731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4148.561221] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4152.334507] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4154.193819] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4156.469584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4156.577695] Lustre: lustre-OST2710: new disk, initializing [ 4156.579333] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4156.582127] Lustre: 298770:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4156.609680] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4157.370106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4157.373642] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4157.411980] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4158.495410] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4162.205044] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4164.473381] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4164.544092] Lustre: lustre-OST03e8: new disk, initializing [ 4164.546252] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4164.549398] Lustre: 299832:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4164.588644] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4166.565366] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4169.152117] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4171.916206] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4172.005519] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4173.436542] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4173.522124] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4174.339066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4174.343911] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4174.375689] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4174.961088] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4175.044530] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4176.513066] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4176.598495] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4177.889759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4177.893189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4177.899102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4179.426968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4179.427115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4179.431446] Lustre: Skipped 2 previous similar messages [ 4179.433116] Lustre: Skipped 2 previous similar messages [ 4183.850088] Lustre: server umount lustre-MDT0000 complete [ 4185.122099] LustreError: 296795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788885477 with bad export cookie 6297876353921448142 [ 4185.125625] LustreError: 296795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4185.125779] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4185.237416] Lustre: server umount lustre-MDT0001 complete [ 4196.881677] Lustre: server umount lustre-OST2710 complete [ 4208.482364] Lustre: server umount lustre-OST03e8 complete [ 4214.732979] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 4218.256082] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 4230.453613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4230.559836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4230.570423] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4230.611947] Lustre: lustre-MDT0000: new disk, initializing [ 4230.635723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4230.643495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4232.016098] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4236.228452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4236.266576] Lustre: 305069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4236.357923] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4237.629991] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4241.415861] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4243.293534] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4245.572101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4245.671881] Lustre: lustre-OST0000: new disk, initializing [ 4245.674158] Lustre: Skipped 1 previous similar message [ 4245.675792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4245.678990] Lustre: Skipped 2 previous similar messages [ 4245.681680] Lustre: 307037:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4245.707821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4245.710784] Lustre: Skipped 1 previous similar message [ 4247.641518] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4251.459461] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4252.893969] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4253.687823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4253.691710] Lustre: Skipped 1 previous similar message [ 4253.694238] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4253.722481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4253.979318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4255.314289] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4255.385905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4258.788988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4258.789632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4258.790707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4258.793811] Lustre: Skipped 1 previous similar message [ 4258.803252] LustreError: Skipped 1 previous similar message [ 4262.162616] Lustre: server umount lustre-OST0000 complete [ 4263.905338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4263.905977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4263.908966] Lustre: Skipped 1 previous similar message [ 4263.914326] Lustre: Skipped 2 previous similar messages [ 4269.025524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4269.028212] Lustre: Skipped 1 previous similar message [ 4269.596765] Lustre: server umount lustre-MDT0000 complete [ 4270.865197] LustreError: 305061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788885562 with bad export cookie 6297876353921449801 [ 4270.868482] LustreError: 305061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4270.868609] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4271.001691] Lustre: server umount lustre-MDT0001 complete [ 4273.778363] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 12:39:25 (1788885565) [ 4274.329721] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4274.959976] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 12:39:27 (1788885567) [ 4276.000952] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 4284.828855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4284.912797] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4284.919329] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4284.976498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4286.320505] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4289.110636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4289.142766] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4289.151640] Lustre: 310896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4290.649366] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4306.531324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4306.607699] Lustre: Denying initial registration attempt from nid 192.168.202.160@tcp, specified as failover [ 4306.610736] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4306.615274] LustreError: 312348:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 4306.618922] LustreError: 312348:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 4306.621296] LustreError: 312348:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4306.642890] Lustre: server umount lustre-OST0000 complete [ 4306.645103] LustreError: 312348:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4309.984901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4309.988437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4309.994408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4323.296143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4323.362813] Lustre: server umount lustre-MDT0000 complete [ 4323.809102] LustreError: 310906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4323.814370] LustreError: 310906:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4324.561170] LustreError: 310886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788885616 with bad export cookie 6297876353921450669 [ 4324.564259] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4324.565041] LustreError: 310886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4324.676273] Lustre: server umount lustre-MDT0001 complete [ 4328.243578] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 4329.270667] Key type lgssc unregistered [ 4329.393442] LNet: 313467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4329.395967] LNetError: 313467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4329.405367] LNet: Removed LNI 192.168.202.160@tcp [ 4329.716700] Key type .llcrypt unregistered [ 4329.717828] Key type ._llcrypt unregistered [ 4333.270690] Key type ._llcrypt registered [ 4333.271731] Key type .llcrypt registered [ 4333.315642] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 4333.681341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4333.693246] alg: No test for adler32 (adler32-zlib) [ 4334.531979] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 4334.613723] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 4336.192173] Key type lgssc registered [ 4336.547270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4360.620370] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 4363.480779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4363.489827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4364.574696] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4364.581328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4364.644495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4365.951116] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4368.662181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4368.701316] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4368.712619] Lustre: 316200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4368.817169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4370.102265] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4372.487520] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4374.364669] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4389.730808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4389.796320] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4389.867279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4391.830268] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4394.320536] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4395.064399] Lustre: server umount lustre-OST0000 complete [ 4399.585052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4399.587271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4399.591454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4401.121118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4401.121764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4402.452522] Lustre: server umount lustre-MDT0000 complete [ 4403.697360] LustreError: 316192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788885695 with bad export cookie 18084678405463736936 [ 4403.699692] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4403.702889] LustreError: 316192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4403.826774] Lustre: server umount lustre-MDT0001 complete [ 4407.685913] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 4408.755346] Key type lgssc unregistered [ 4408.890369] LNet: 319766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4408.893993] LNetError: 319766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4408.901297] LNet: Removed LNI 192.168.202.160@tcp [ 4409.226314] Key type .llcrypt unregistered [ 4409.227508] Key type ._llcrypt unregistered [ 4418.633255] Key type ._llcrypt registered [ 4418.634218] Key type .llcrypt registered [ 4418.672499] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 4419.037969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4419.074242] alg: No test for adler32 (adler32-zlib) [ 4419.933175] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 4420.021343] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 4421.600207] Key type lgssc registered [ 4421.958332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4424.553702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4424.559119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4425.698888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4427.038457] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4429.838538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4429.963288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4431.259701] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4433.670873] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4435.568631] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4439.971553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4440.104559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4441.996802] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4444.379475] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4460.062382] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.60@tcp (stopping) [ 4460.517174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4460.520604] Lustre: Skipped 2 previous similar messages [ 4463.069102] Lustre: server umount lustre-MDT0000 complete [ 4464.273258] LustreError: 321135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788885756 with bad export cookie 3812324614595421639 [ 4464.276092] LustreError: 321135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4464.277118] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4465.176291] LustreError: 321149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4465.182110] LustreError: 321149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4465.182382] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.60@tcp (stopping) [ 4465.191307] Lustre: Skipped 3 previous similar messages [ 4465.633463] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4469.665092] LustreError: 321149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4469.665305] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4469.671251] LustreError: 321149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4469.673188] Lustre: Skipped 1 previous similar message [ 4470.441301] Lustre: server umount lustre-MDT0001 complete [ 4472.696404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4475.521930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4489.184951] LustreError: 320465:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000059cef168 x1875782624370816/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4489.296159] LustreError: 322982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4489.301510] LustreError: 322982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4489.318751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4489.347344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4490.649704] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4493.489680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4493.622581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4494.631317] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4494.943924] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4497.374822] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4499.751089] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4501.991441] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 4503.775517] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4504.283945] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4509.153364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4509.156112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4509.160408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4509.162605] Lustre: Skipped 1 previous similar message [ 4509.665027] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4511.641597] Lustre: server umount lustre-OST0000 complete [ 4514.272866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4514.275116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4514.278731] Lustre: Skipped 1 previous similar message [ 4514.280600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4514.282443] Lustre: Skipped 3 previous similar messages [ 4519.126847] Lustre: server umount lustre-MDT0000 complete [ 4520.378069] LustreError: 321179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788885812 with bad export cookie 3812324614595428793 [ 4520.381308] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4520.381682] LustreError: 321179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4520.499312] Lustre: server umount lustre-MDT0001 complete [ 4524.487679] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 4525.572451] Key type lgssc unregistered [ 4525.715385] LNet: 327434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4525.719046] LNetError: 327434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4525.727339] LNet: Removed LNI 192.168.202.160@tcp [ 4526.051155] Key type .llcrypt unregistered [ 4526.052491] Key type ._llcrypt unregistered [ 4535.636311] Key type ._llcrypt registered [ 4535.638270] Key type .llcrypt registered [ 4535.681044] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 4536.034708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4536.081373] alg: No test for adler32 (adler32-zlib) [ 4536.947293] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 4537.048992] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 4538.640194] Key type lgssc registered [ 4539.051197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4541.773549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4541.780525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4542.908279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4544.186731] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4547.064333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4547.178591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4548.630588] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4551.078355] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4552.779423] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4557.281227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4557.284735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4557.290877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4557.793629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4557.798166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4558.818209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4558.820059] Lustre: Skipped 1 previous similar message [ 4563.937210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4563.939368] Lustre: Skipped 1 previous similar message [ 4568.032104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4568.085373] Lustre: server umount lustre-MDT0000 complete [ 4569.057219] LustreError: 329549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4569.062090] LustreError: 329549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4569.317141] LustreError: 328802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788885861 with bad export cookie 672778869340754925 [ 4569.320764] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4569.321234] LustreError: 328802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4569.431378] Lustre: server umount lustre-MDT0001 complete [ 4575.441931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4575.594222] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4575.611440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4575.727063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4578.467439] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4584.563333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4584.625065] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4584.645874] Lustre: 331409:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4584.777170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4587.319109] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4591.430670] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4595.203952] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4601.830567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4601.933960] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4601.940563] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4601.945629] LustreError: 333289:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 4601.949609] LustreError: 333289:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 4601.952085] LustreError: 333289:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4601.993828] Lustre: server umount lustre-OST0000 complete [ 4601.996527] LustreError: 333289:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4608.755313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4608.966775] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4609.130782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4614.853748] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4619.416737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4621.927514] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4628.568609] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4628.678674] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4628.735887] Lustre: lustre-OST0001: new disk, initializing [ 4628.740212] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4628.749128] Lustre: 334829:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4628.855498] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4634.325397] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4634.650687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4634.657926] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4634.719131] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4640.692225] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4644.847393] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4644.849269] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4644.859283] LustreError: Skipped 1 previous similar message [ 4644.869892] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4644.881127] Lustre: Skipped 1 previous similar message [ 4649.435357] Lustre: server umount lustre-OST0001 complete [ 4649.966203] LustreError: 333758:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4649.976540] LustreError: 333758:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4655.077544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4655.078616] LustreError: 333756:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4655.078652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4655.082814] Lustre: Skipped 2 previous similar messages [ 4655.088952] LustreError: 333756:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4655.095973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4660.196918] LustreError: 333757:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4660.204024] LustreError: 333757:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4665.318715] LustreError: 335211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4665.337970] LustreError: 335211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4665.824121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4665.899361] Lustre: server umount lustre-OST0000 complete [ 4670.438222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4675.553582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4675.555886] Lustre: Skipped 7 previous similar messages [ 4682.208528] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4682.312537] Lustre: server umount lustre-MDT0000 complete [ 4684.392704] LustreError: 331401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788885976 with bad export cookie 672778869340755464 [ 4684.394929] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4684.404921] LustreError: 331401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4684.586686] Lustre: server umount lustre-MDT0001 complete [ 4690.356782] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 4691.673629] Key type lgssc unregistered [ 4691.888442] LNet: 336785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4691.893089] LNetError: 336785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4691.902350] LNet: Removed LNI 192.168.202.160@tcp [ 4692.381092] Key type .llcrypt unregistered [ 4692.382941] Key type ._llcrypt unregistered [ 4731.806590] Key type ._llcrypt registered [ 4731.808215] Key type .llcrypt registered [ 4731.850903] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 4739.576160] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 4740.199832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4740.225253] alg: No test for adler32 (adler32-zlib) [ 4741.131201] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 4741.281177] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 4742.896172] Key type lgssc registered [ 4743.422317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4760.828553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4760.835868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4761.944170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4761.968429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4762.010871] Lustre: lustre-MDT0000: new disk, initializing [ 4762.042617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4762.049802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4763.740238] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4769.025466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4769.086442] Lustre: 341687:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4769.117577] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4769.119880] Lustre: Skipped 1 previous similar message [ 4769.159131] Lustre: lustre-MDT0001: new disk, initializing [ 4769.226946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4769.246635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4769.251542] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4771.060262] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4776.436660] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4779.247973] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4782.481217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4782.617872] Lustre: lustre-OST0000: new disk, initializing [ 4782.620539] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4782.626828] Lustre: 343657:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4782.677975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4785.666851] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4789.774840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4789.781224] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4789.838792] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4791.187831] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4793.696862] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4793.810828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4795.847329] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4795.954877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4799.974326] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4799.980167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4799.981973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4799.992875] Lustre: Skipped 1 previous similar message [ 4800.003555] Lustre: Skipped 1 previous similar message [ 4805.091985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4805.094273] Lustre: Skipped 1 previous similar message [ 4810.212477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4810.216449] Lustre: Skipped 1 previous similar message [ 4811.232125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4811.358241] Lustre: server umount lustre-OST0000 complete [ 4815.330062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4815.331604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4815.332061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4815.335032] Lustre: Skipped 1 previous similar message [ 4815.338381] LustreError: Skipped 1 previous similar message [ 4819.871886] Lustre: server umount lustre-MDT0000 complete [ 4822.049333] LustreError: 341679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788886114 with bad export cookie 10845943851030222993 [ 4822.051826] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4822.055385] LustreError: 341679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4822.249096] Lustre: server umount lustre-MDT0001 complete [ 4826.529367] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:48:38 (1788886118) [ 4830.165586] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:48:42 (1788886122) [ 4835.313319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4835.588755] LustreError: 346411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4835.601905] LustreError: 346411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4835.640080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4838.139566] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4840.933339] LustreError: 346412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4843.196278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4843.386098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4845.802629] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4849.734355] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4852.590808] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4856.134550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4856.348844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4859.504178] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4863.463501] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4872.301487] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4873.103312] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4873.833649] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4874.622974] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4875.535243] Lustre: Failing over lustre-MDT0000 [ 4875.786618] Lustre: server umount lustre-MDT0000 complete [ 4878.818598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4878.819689] LustreError: 347337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4878.824856] Lustre: Skipped 2 previous similar messages [ 4878.831185] LustreError: 347337:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4883.939490] LustreError: 347138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4883.946377] LustreError: 347138:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4889.059217] LustreError: 346411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4889.065540] LustreError: 346411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4890.157220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4890.221756] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4890.371898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4890.400183] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4890.651395] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4892.531908] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4895.728776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4895.748189] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4895.775524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4898.344770] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4899.018632] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4901.192979] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4902.074295] Lustre: Failing over lustre-MDT0000 [ 4902.246467] Lustre: server umount lustre-MDT0000 complete [ 4905.955429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4905.955788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4905.956434] LustreError: 348901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4905.961024] Lustre: Skipped 1 previous similar message [ 4916.617322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4916.687807] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4916.841477] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4918.841749] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4921.369503] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4921.831832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4921.835205] Lustre: Skipped 2 previous similar messages [ 4921.852550] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4921.873064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4924.244872] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4924.930477] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4932.065488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4932.066034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4932.069988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4932.069996] Lustre: Skipped 1 previous similar message [ 4932.084792] Lustre: Skipped 3 previous similar messages [ 4933.736234] Lustre: server umount lustre-MDT0000 complete [ 4935.755192] LustreError: 346391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788886227 with bad export cookie 10845943851030225688 [ 4935.756776] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4935.764112] LustreError: 346391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4935.956152] Lustre: server umount lustre-MDT0001 complete [ 4948.207732] Lustre: server umount lustre-OST0000 complete [ 4955.984740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4956.237958] LustreError: 352738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4956.246279] LustreError: 352738:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 4956.269460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4956.272367] Lustre: Skipped 1 previous similar message [ 4958.502217] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4963.108540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4965.589324] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4969.124996] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4971.648786] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4974.981141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4978.183210] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 4981.353951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4981.726723] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4997.093462] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4997.099463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4997.106157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4997.108611] Lustre: Skipped 2 previous similar messages [ 5008.352148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5008.436951] Lustre: server umount lustre-OST0000 complete [ 5014.499040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5014.504699] Lustre: Skipped 1 previous similar message [ 5014.507505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5014.510257] Lustre: Skipped 5 previous similar messages [ 5016.226800] Lustre: server umount lustre-MDT0000 complete [ 5018.117499] LustreError: 352718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788886310 with bad export cookie 10845943851030226409 [ 5018.121187] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5018.128040] LustreError: 352718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5018.270524] Lustre: server umount lustre-MDT0001 complete [ 5024.114520] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 5025.451707] Key type lgssc unregistered [ 5025.639566] LNet: 356287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5025.649908] LNetError: 356287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5025.662164] LNet: Removed LNI 192.168.202.160@tcp [ 5026.136550] Key type .llcrypt unregistered [ 5026.138402] Key type ._llcrypt unregistered [ 5037.935374] Key type ._llcrypt registered [ 5037.936966] Key type .llcrypt registered [ 5037.991339] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 5038.455486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5038.505149] alg: No test for adler32 (adler32-zlib) [ 5039.373975] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 5039.487212] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 5041.080200] Key type lgssc registered [ 5041.590156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5044.929897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5044.938453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5046.134538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5048.091980] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5052.056443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5052.235426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5053.940113] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5056.872855] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5059.135508] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5062.015191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5062.261585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5065.117348] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5068.824954] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5071.473134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5082.081760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5082.084846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5082.086822] Lustre: Skipped 2 previous similar messages [ 5082.089576] Lustre: Skipped 1 previous similar message [ 5084.725134] Lustre: server umount lustre-MDT0000 complete [ 5086.377627] LustreError: 358511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788886378 with bad export cookie 12574147309407695550 [ 5086.381114] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5086.383172] LustreError: 358511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5086.507119] Lustre: server umount lustre-MDT0001 complete [ 5098.354224] Lustre: server umount lustre-OST0000 complete [ 5104.493190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5104.690875] LustreError: 361484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5104.699176] LustreError: 361484:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5104.721185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5106.522670] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5109.729313] LustreError: 361485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5109.954775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5110.099773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5111.762502] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5114.728555] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5117.036163] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5119.781165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5119.966870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5122.187234] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5124.797569] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5129.190326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5139.428551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5139.429355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5139.433661] Lustre: Skipped 2 previous similar messages [ 5139.436147] Lustre: Skipped 2 previous similar messages [ 5143.385131] Lustre: server umount lustre-MDT0000 complete [ 5144.546637] LustreError: 361484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5144.552401] LustreError: 361484:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5144.847772] LustreError: 361465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788886436 with bad export cookie 12574147309407696677 [ 5144.853246] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5144.853704] LustreError: 361465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5144.972553] Lustre: server umount lustre-MDT0001 complete [ 5156.848337] Lustre: server umount lustre-OST0000 complete [ 5163.184542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5163.407716] LustreError: 365392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5163.436838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5165.006167] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5168.405941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5170.019240] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5172.859912] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5174.910904] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5177.431606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5180.073829] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5183.403421] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5188.840389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5199.330082] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5199.331693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5199.336811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5199.338362] Lustre: Skipped 1 previous similar message [ 5201.170337] Lustre: server umount lustre-OST0000 complete [ 5204.450174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5204.450434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5204.450684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5204.455300] Lustre: Skipped 2 previous similar messages [ 5204.461542] LustreError: Skipped 1 previous similar message [ 5208.600171] Lustre: server umount lustre-MDT0000 complete [ 5209.569299] LustreError: 365392:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5209.575607] LustreError: 365392:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5209.899346] LustreError: 367235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788886501 with bad export cookie 12574147309407697685 [ 5209.902404] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5209.902772] LustreError: 367235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5210.011070] Lustre: server umount lustre-MDT0001 complete [ 5213.920343] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 5214.993215] Key type lgssc unregistered [ 5215.119418] LNet: 368941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5215.122715] LNetError: 368941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5215.130345] LNet: Removed LNI 192.168.202.160@tcp [ 5215.447627] Key type .llcrypt unregistered [ 5215.448874] Key type ._llcrypt unregistered [ 5225.331975] Key type ._llcrypt registered [ 5225.333457] Key type .llcrypt registered [ 5225.370755] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 5225.692503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5225.738510] alg: No test for adler32 (adler32-zlib) [ 5226.597635] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 5226.685038] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 5228.264176] Key type lgssc registered [ 5228.652682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5231.429149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5231.437579] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5231.441699] LustreError: 370355:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5231.451452] LustreError: 370355:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5231.454577] LustreError: 370355:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5231.457863] LustreError: 370355:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5231.461598] LustreError: 370355:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5234.327224] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5234.331188] LustreError: 370751:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5234.341341] LustreError: 370751:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5234.344126] LustreError: 370751:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5234.346913] LustreError: 370751:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5234.349605] LustreError: 370751:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5241.312849] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 5242.496942] Key type lgssc unregistered [ 5242.631405] LNet: 371751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5242.634345] LNetError: 371751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5242.642337] LNet: Removed LNI 192.168.202.160@tcp [ 5242.982333] Key type .llcrypt unregistered [ 5242.983798] Key type ._llcrypt unregistered [ 5249.404091] Key type ._llcrypt registered [ 5249.405232] Key type .llcrypt registered [ 5249.446546] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 5256.410244] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 5256.755221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5256.800187] alg: No test for adler32 (adler32-zlib) [ 5257.663963] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 5257.763070] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 5259.344166] Key type lgssc registered [ 5259.748491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5271.953076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5271.966941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5273.058206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5273.071625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5273.111199] Lustre: lustre-MDT0000: new disk, initializing [ 5273.144314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5273.153778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5274.727704] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5279.343425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5279.394951] Lustre: 374875:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5279.424774] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5279.426648] Lustre: Skipped 1 previous similar message [ 5279.465673] Lustre: lustre-MDT0001: new disk, initializing [ 5279.506603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5279.529594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5279.533635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5281.284335] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5285.645469] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5287.961797] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5290.876862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5291.006867] Lustre: lustre-OST0000: new disk, initializing [ 5291.009081] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5291.012234] Lustre: 376846:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5291.047726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5293.535537] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5296.647385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5296.652219] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5296.700093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5297.934155] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5299.716974] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5299.812113] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5301.536290] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5301.626086] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5302.462308] Lustre: server umount lustre-OST0000 complete [ 5305.313405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5305.316086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5305.320991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5306.850368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5306.850423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5311.971542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5311.973854] Lustre: Skipped 2 previous similar messages [ 5314.050374] Lustre: server umount lustre-MDT0000 complete [ 5315.487536] LustreError: 374867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788886607 with bad export cookie 2840989132251826592 [ 5315.490753] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5315.490829] LustreError: 374867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5325.858704] Lustre: server umount lustre-MDT0001 complete [ 5328.922908] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5329.535623] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:57:01 (1788886621) [ 5331.840389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5332.035664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5333.433408] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5336.451423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5336.588403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5338.074585] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5340.782408] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5342.889725] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5345.652624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5345.813074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5348.063090] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5350.929207] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5353.373855] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5353.447412] Lustre: lustre-OST0001: new disk, initializing [ 5353.449275] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5353.451975] Lustre: 382141:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5355.701715] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5359.614563] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5361.153269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5361.156406] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5361.193633] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5366.244861] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5366.244931] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5366.247246] LustreError: Skipped 1 previous similar message [ 5366.252111] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5366.254619] Lustre: Skipped 1 previous similar message [ 5367.621634] Lustre: server umount lustre-OST0001 complete [ 5371.364263] LustreError: 381126:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5371.368824] LustreError: 381126:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5376.485175] LustreError: 382625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5376.490607] LustreError: 382625:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5381.603182] LustreError: 381123:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5381.604030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5381.604849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5381.604854] Lustre: Skipped 1 previous similar message [ 5381.608744] LustreError: 381123:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5381.612116] Lustre: Skipped 2 previous similar messages [ 5385.936436] Lustre: server umount lustre-OST0000 complete [ 5387.744636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5387.747133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5391.841146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5391.843111] Lustre: Skipped 3 previous similar messages [ 5393.303459] Lustre: server umount lustre-MDT0000 complete [ 5394.590409] LustreError: 380038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788886686 with bad export cookie 2840989132251827460 [ 5394.593806] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5394.594290] LustreError: 380038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5400.767759] Lustre: server umount lustre-MDT0001 complete [ 5404.326750] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 5405.338427] Key type lgssc unregistered [ 5405.472455] LNet: 384121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5405.475914] LNetError: 384121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5405.484334] LNet: Removed LNI 192.168.202.160@tcp [ 5405.832425] Key type .llcrypt unregistered [ 5405.833592] Key type ._llcrypt unregistered [ 5425.115163] Key type ._llcrypt registered [ 5425.116754] Key type .llcrypt registered [ 5425.165217] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 5425.542974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5425.598964] alg: No test for adler32 (adler32-zlib) [ 5426.471339] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 5426.578787] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 5428.168157] Key type lgssc registered [ 5428.578958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5431.592868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5431.603481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5432.687277] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5432.694446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5432.755318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5434.231099] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5437.227897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5437.270397] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5437.281363] Lustre: 386893:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5437.381767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5438.820681] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5441.542128] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5443.507028] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5447.651311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5447.654721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5447.661638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5448.161769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5448.167261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5448.169293] Lustre: Skipped 1 previous similar message [ 5450.399267] Lustre: server umount lustre-MDT0000 complete [ 5451.817448] LustreError: 386883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788886743 with bad export cookie 11372324735777571106 [ 5451.820585] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5451.821136] LustreError: 386883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5451.931596] Lustre: server umount lustre-MDT0001 complete [ 5455.430041] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:59:07 (1788886747) [ 5457.881644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5458.090908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5459.551943] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5462.695308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5462.819795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5464.413091] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5467.214072] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5469.330376] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5471.947155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5472.026298] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5472.093056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5474.086054] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5476.612729] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5480.312715] Lustre: Permanently deactivating lustre-OST0000 [ 5480.317891] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5481.972066] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5483.495460] LustreError: 392286:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5483.498923] LustreError: 392286:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 5483.501995] LustreError: 392286:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5484.099762] LustreError: 392334:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 5484.102324] LustreError: 392334:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5490.837785] Lustre: server umount lustre-OST0000 complete [ 5491.763138] LustreError: 392584:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 5491.766760] LustreError: 392584:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5493.217355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5493.217927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5493.222364] Lustre: Skipped 1 previous similar message [ 5498.337018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5498.339011] Lustre: Skipped 2 previous similar messages [ 5502.458960] Lustre: server umount lustre-MDT0000 complete [ 5503.456926] LustreError: 389398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5503.461841] LustreError: 389398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5503.732097] LustreError: 391230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788886795 with bad export cookie 11372324735777571673 [ 5503.735362] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5503.736101] LustreError: 391230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5513.242445] Lustre: server umount lustre-MDT0001 complete [ 5516.572565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5518.123319] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5519.076318] LustreError: 393742:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5521.387846] Lustre: 394080:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.160@tcp [ 5522.195351] Lustre: server umount MGS complete [ 5526.623684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5526.839136] LustreError: 394803:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5526.845851] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5526.851115] Lustre: Skipped 1 previous similar message [ 5526.872684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5528.486346] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5531.656485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5531.758842] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5533.262929] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5535.766725] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5537.659454] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5538.339222] Lustre: Permanently reactivating lustre-OST0000 [ 5538.345705] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5538.348308] Lustre: Skipped 2 previous similar messages [ 5560.288395] LustreError: 396486:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5563.506828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5563.685535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5563.688055] Lustre: Skipped 1 previous similar message [ 5565.776259] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5568.298842] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5580.257464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5580.258921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5580.260273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5580.263530] Lustre: Skipped 1 previous similar message [ 5586.187334] Lustre: server umount lustre-OST0000 complete [ 5588.448780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5588.451699] LustreError: Skipped 1 previous similar message [ 5588.452938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5588.456931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5588.458715] Lustre: Skipped 3 previous similar messages [ 5593.553965] Lustre: server umount lustre-MDT0000 complete [ 5594.804970] LustreError: 394784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788886886 with bad export cookie 11372324735777572695 [ 5594.807284] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5594.809256] LustreError: 394784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5594.919269] Lustre: server umount lustre-MDT0001 complete [ 5598.788013] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 5599.813437] Key type lgssc unregistered [ 5599.948438] LNet: 399337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5599.951397] LNetError: 399337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5599.963352] LNet: Removed LNI 192.168.202.160@tcp [ 5600.281366] Key type .llcrypt unregistered [ 5600.282402] Key type ._llcrypt unregistered [ 5606.799753] Key type ._llcrypt registered [ 5606.800797] Key type .llcrypt registered [ 5606.839672] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 5613.269740] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 5613.574018] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5613.581249] alg: No test for adler32 (adler32-zlib) [ 5614.430686] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 5614.522029] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 5616.104188] Key type lgssc registered [ 5616.502738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5628.199485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5628.205422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5629.292520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5629.302067] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5629.339036] Lustre: lustre-MDT0000: new disk, initializing [ 5629.360374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5629.366853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5630.732506] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5635.038414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5635.080800] Lustre: 402461:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5635.106546] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5635.108989] Lustre: Skipped 1 previous similar message [ 5635.147255] Lustre: lustre-MDT0001: new disk, initializing [ 5635.183511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5635.195738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5635.199168] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5636.661419] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5640.570519] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5642.615402] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5645.044774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5645.154756] Lustre: lustre-OST0000: new disk, initializing [ 5645.157106] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5645.160443] Lustre: 404428:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5645.192093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5647.115190] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5650.892365] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5652.409259] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5655.032405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5655.036291] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5655.073838] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5655.552915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5657.155097] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5657.226500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5660.131114] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5660.131318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5660.131925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5660.133446] LustreError: Skipped 1 previous similar message [ 5660.139460] Lustre: Skipped 1 previous similar message [ 5664.033894] Lustre: server umount lustre-OST0000 complete [ 5666.273168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5666.275642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5666.280191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5666.283017] Lustre: Skipped 1 previous similar message [ 5670.369125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5670.369658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5671.579090] Lustre: server umount lustre-MDT0000 complete [ 5672.845598] LustreError: 402499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788886964 with bad export cookie 15164790065078628674 [ 5672.850302] LustreError: 402499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5672.851199] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5672.962545] Lustre: server umount lustre-MDT0001 complete [ 5675.586441] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5676.168350] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:02:48 (1788886968) [ 5678.386157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5678.548523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5679.869651] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5682.748075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5682.881809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5684.364264] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5686.901253] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5688.585486] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5690.739949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5692.762215] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5695.316603] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5695.940333] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5706.847663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5713.377195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5713.378139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5713.380253] LustreError: Skipped 1 previous similar message [ 5713.386022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5713.389171] Lustre: Skipped 2 previous similar messages [ 5714.445871] Lustre: server umount lustre-OST0000 complete [ 5718.497507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5718.498301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5718.501338] Lustre: Skipped 2 previous similar messages [ 5718.506011] Lustre: Skipped 1 previous similar message [ 5721.751850] Lustre: server umount lustre-MDT0000 complete [ 5722.885980] LustreError: 406885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788887014 with bad export cookie 15164790065078629542 [ 5722.888555] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5722.890089] LustreError: 406885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5722.995803] Lustre: server umount lustre-MDT0001 complete [ 5726.368689] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 5727.325276] Key type lgssc unregistered [ 5727.452426] LNet: 410649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5727.454478] LNetError: 410649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5727.461280] LNet: Removed LNI 192.168.202.160@tcp [ 5727.763923] Key type .llcrypt unregistered [ 5727.765149] Key type ._llcrypt unregistered [ 5735.348741] Key type ._llcrypt registered [ 5735.349938] Key type .llcrypt registered [ 5735.382316] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 5735.472344] Key type .llcrypt unregistered [ 5735.473621] Key type ._llcrypt unregistered [ 5741.813194] Key type ._llcrypt registered [ 5741.814146] Key type .llcrypt registered [ 5741.844018] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 5742.153861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5742.185396] alg: No test for adler32 (adler32-zlib) [ 5743.023810] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 5743.108400] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 5744.680130] Key type lgssc registered [ 5745.014748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5747.675762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5747.683163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5748.817615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5750.220179] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5752.823882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5752.923669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5754.706603] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5757.196178] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5757.921363] LustreError: 412830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5757.926649] LustreError: 412830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5758.945973] LustreError: 412829:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5758.949348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5758.954129] LustreError: 412829:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5759.434782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5759.570410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5760.826499] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5769.184844] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5769.187374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5769.192638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5769.698183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5769.700308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5769.700983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5773.966525] Lustre: server umount lustre-OST0000 complete [ 5778.914286] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5778.916118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5778.919441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5781.266444] Lustre: server umount lustre-MDT0000 complete [ 5782.416065] LustreError: 414937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788887074 with bad export cookie 14956436889149807735 [ 5782.418795] LustreError: 414937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5782.419163] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5782.522566] Lustre: server umount lustre-MDT0001 complete [ 5785.968754] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 5786.960905] Key type lgssc unregistered [ 5787.076447] LNet: 415975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5787.079756] LNetError: 415975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5787.087266] LNet: Removed LNI 192.168.202.160@tcp [ 5787.382548] Key type .llcrypt unregistered [ 5787.383548] Key type ._llcrypt unregistered [ 5796.285358] Key type ._llcrypt registered [ 5796.286687] Key type .llcrypt registered [ 5796.321605] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 5796.802466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5796.814252] alg: No test for adler32 (adler32-zlib) [ 5797.656449] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 5797.733667] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 5799.312145] Key type lgssc registered [ 5799.644479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5802.098138] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5802.103208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5817.504288] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5822.624221] LustreError: 417338:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.160@tcp: failed processing log, type 4: rc = -110 [ 5849.248190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5851.215901] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5853.665170] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5855.875034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5856.040649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5856.066426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5857.409906] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5860.659979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5860.832913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5862.327797] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5864.770324] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5866.526297] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5869.028063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5869.028516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5869.032855] Lustre: Skipped 1 previous similar message [ 5874.146270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5874.149728] Lustre: Skipped 2 previous similar messages [ 5874.829129] Lustre: server umount lustre-OST0000 complete [ 5876.192945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5876.195357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5876.200915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5879.265288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5879.265978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5882.263103] Lustre: server umount lustre-MDT0000 complete [ 5883.548249] LustreError: 417346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788887175 with bad export cookie 15860129373700638021 [ 5883.552058] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5883.553923] LustreError: 417346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5883.674861] Lustre: server umount lustre-MDT0001 complete [ 5887.366397] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 5888.404544] Key type lgssc unregistered [ 5888.535373] LNet: 420904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5888.537693] LNetError: 420904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5888.543344] LNet: Removed LNI 192.168.202.160@tcp [ 5888.868924] Key type .llcrypt unregistered [ 5888.869955] Key type ._llcrypt unregistered [ 5898.209994] Key type ._llcrypt registered [ 5898.211324] Key type .llcrypt registered [ 5898.252592] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 5898.605607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5898.665745] alg: No test for adler32 (adler32-zlib) [ 5899.537036] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 5899.634474] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 5901.224236] Key type lgssc registered [ 5901.631314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5904.294503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5904.299953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5905.452841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5906.815427] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5909.638722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5909.767462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5911.184567] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5913.597731] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5915.379480] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5917.576269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5917.691851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5919.646605] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5922.164125] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5929.963963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5935.073856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5935.076423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5935.079414] Lustre: Skipped 2 previous similar messages [ 5935.081172] Lustre: Skipped 1 previous similar message [ 5937.695088] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.60@tcp (stopping) [ 5937.697334] Lustre: Skipped 1 previous similar message [ 5939.540985] Lustre: server umount lustre-MDT0000 complete [ 5940.194039] LustreError: 424780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5940.198859] LustreError: 424780:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5940.278892] LustreError: 424133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.60@tcp arrived at 1788887232 with bad export cookie 11093426604778075473 [ 5940.283736] LustreError: 424133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5940.893148] LustreError: 422273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788887232 with bad export cookie 11093426604778074822 [ 5940.896838] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5940.898553] LustreError: 422273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5945.313117] LustreError: 422294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5945.313576] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5945.313803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5945.314537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5945.319330] LustreError: 422294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5947.024547] Lustre: server umount lustre-OST0000 complete [ 5949.340168] Lustre: server umount lustre-MDT0001 complete [ 5952.955658] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 5954.035678] Key type lgssc unregistered [ 5954.161404] LNet: 425994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5954.164839] LNetError: 425994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5954.177420] LNet: Removed LNI 192.168.202.160@tcp [ 5954.491206] Key type .llcrypt unregistered [ 5954.492634] Key type ._llcrypt unregistered [ 5964.249049] Key type ._llcrypt registered [ 5964.250077] Key type .llcrypt registered [ 5964.293531] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 5964.722572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5964.762641] alg: No test for adler32 (adler32-zlib) [ 5965.633499] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 5965.731648] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 5967.321688] Key type lgssc registered [ 5967.812234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5971.740535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5971.759135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5973.042961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5975.746461] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5980.368634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5980.562586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5982.772472] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5987.283559] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5989.480977] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5991.981017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5992.104042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5994.148579] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 5996.771225] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5998.247211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6004.193574] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6004.196673] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6004.200914] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6008.344274] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.60@tcp (stopping) [ 6009.315807] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6010.068942] Lustre: server umount lustre-MDT0001 complete [ 6013.920879] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6013.923496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6013.928182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6013.930675] Lustre: Skipped 2 previous similar messages [ 6014.433124] LustreError: 427386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6014.439488] LustreError: 427386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6017.820142] Lustre: server umount lustre-OST0000 complete [ 6019.330882] Lustre: server umount lustre-MDT0000 complete [ 6024.174257] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 6025.273414] Key type lgssc unregistered [ 6025.411724] LNet: 431069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6025.414577] LNetError: 431069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6026.470511] LNet: Removed LNI 192.168.202.160@tcp [ 6026.814329] Key type .llcrypt unregistered [ 6026.815870] Key type ._llcrypt unregistered [ 6035.892588] Key type ._llcrypt registered [ 6035.894605] Key type .llcrypt registered [ 6035.938451] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 6042.729449] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 6043.113485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6043.137039] alg: No test for adler32 (adler32-zlib) [ 6044.046215] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 6044.178401] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 6045.784270] Key type lgssc registered [ 6046.199374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6060.392995] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 6065.800227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6065.807095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6066.890192] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6066.902207] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6066.938706] Lustre: lustre-MDT0000: new disk, initializing [ 6066.961212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6066.967181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6068.428324] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6074.207160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6074.311961] Lustre: lustre-OST0000: new disk, initializing [ 6074.313733] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6074.315514] Lustre: Skipped 1 previous similar message [ 6074.317671] Lustre: 436069:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6074.344403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6076.386435] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6080.086072] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6082.051478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6082.055868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6082.065818] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6091.124654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6091.165292] Lustre: 435053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6091.192533] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6091.230116] Lustre: lustre-MDT0001: new disk, initializing [ 6091.275423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6091.287058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6091.290631] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6092.707114] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6096.583313] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6096.661105] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6098.127119] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6098.199441] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6100.001208] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.60@tcp (stopping) [ 6102.498099] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6102.500772] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6102.502204] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6102.504840] Lustre: Skipped 1 previous similar message [ 6102.510105] Lustre: Skipped 1 previous similar message [ 6105.112437] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.60@tcp (stopping) [ 6107.618706] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6107.620453] Lustre: Skipped 1 previous similar message [ 6112.740840] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6112.743608] Lustre: Skipped 2 previous similar messages [ 6114.272144] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6114.335412] Lustre: server umount lustre-MDT0001 complete [ 6122.981667] LustreError: 435059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6122.988019] LustreError: 435059:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6128.098910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6128.109263] LustreError: 437398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6128.109561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6128.110986] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6132.946515] Lustre: server umount lustre-OST0000 complete [ 6133.217979] LustreError: 435063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6140.547361] Lustre: server umount lustre-MDT0000 complete [ 6145.666307] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 6146.809591] Key type lgssc unregistered [ 6146.940363] LNet: 439814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6146.943160] LNetError: 439814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6146.949291] LNet: Removed LNI 192.168.202.160@tcp [ 6147.237585] Key type .llcrypt unregistered [ 6147.238568] Key type ._llcrypt unregistered [ 6160.273980] Key type ._llcrypt registered [ 6160.275439] Key type .llcrypt registered [ 6160.313044] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 6160.423243] Key type .llcrypt unregistered [ 6160.424320] Key type ._llcrypt unregistered [ 6167.303825] Key type ._llcrypt registered [ 6167.304825] Key type .llcrypt registered [ 6167.342955] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 6167.673435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6167.712746] alg: No test for adler32 (adler32-zlib) [ 6168.572644] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 6168.658063] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 6170.240178] Key type lgssc registered [ 6170.605779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6179.067850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6179.073669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6180.148756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6180.159491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6180.194827] Lustre: lustre-MDT0000: new disk, initializing [ 6180.215762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6180.223450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6181.716771] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6185.794186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6185.835552] Lustre: 443154:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6185.857635] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6185.859895] Lustre: Skipped 1 previous similar message [ 6185.897647] Lustre: lustre-MDT0001: new disk, initializing [ 6185.932138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6185.944295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6185.947779] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6187.277267] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6190.875395] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6192.642142] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6194.765969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6194.868226] Lustre: lustre-OST0000: new disk, initializing [ 6194.869787] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6194.872682] Lustre: 445120:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6194.908062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6195.833305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6195.836022] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6195.868159] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6196.758139] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6200.474142] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6206.435552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6206.438708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6206.439312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6206.439987] Lustre: Skipped 2 previous similar messages [ 6208.925482] Lustre: server umount lustre-MDT0000 complete [ 6210.196482] LustreError: 443145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788887502 with bad export cookie 3548578973185019076 [ 6210.199808] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6210.204422] LustreError: 443145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6210.301404] Lustre: server umount lustre-MDT0001 complete [ 6221.920434] Lustre: server umount lustre-OST0000 complete [ 6225.404722] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 13:11:57 (1788887517) [ 6227.790010] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6227.980071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6229.201264] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6232.026589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6232.147395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6233.442154] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6235.758970] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6237.540905] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6252.899098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6252.963412] Lustre: Found index 0 for lustre-OST0000, updating log [ 6253.024257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6254.056023] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6254.907895] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6257.390750] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6259.169097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6259.172375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6259.178243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6259.180884] Lustre: Skipped 2 previous similar messages [ 6262.242609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6262.244032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6262.247606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6264.596704] Lustre: server umount lustre-OST0000 complete [ 6278.113031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6278.116644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6278.122594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6278.125023] Lustre: Skipped 1 previous similar message [ 6283.290219] Lustre: server umount lustre-MDT0000 complete [ 6283.744944] LustreError: 447165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6283.750276] LustreError: 447165:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6284.603685] LustreError: 447126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788887576 with bad export cookie 3548578973185020455 [ 6284.605600] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6284.607447] LustreError: 447126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6284.723731] Lustre: server umount lustre-MDT0001 complete [ 6287.805357] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 13:12:59 (1788887579) [ 6317.152244] LNet: 1 local NIs in recovery (showing 1): 192.168.202.160@tcp [ 6867.056683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6867.209991] LustreError: 453549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6867.230575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6868.527702] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6871.192119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6872.640247] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6875.022340] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6876.858023] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6892.132107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6892.238446] Lustre: Found index 0 for lustre-OST0000, updating log [ 6894.229671] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 6894.375061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 6896.672622] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6898.401918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6898.406500] Lustre: Skipped 1 previous similar message [ 6898.409174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6898.411955] Lustre: Skipped 1 previous similar message [ 6903.503125] Lustre: server umount lustre-OST0000 complete [ 6905.761914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6905.763496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6905.768032] Lustre: Skipped 1 previous similar message [ 6906.785531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6906.788525] Lustre: Skipped 2 previous similar messages [ 6908.833239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6908.834960] Lustre: Skipped 3 previous similar messages [ 6912.929392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6912.931858] Lustre: Skipped 7 previous similar messages [ 6918.240212] LNet: 1 local NIs in recovery (showing 1): 192.168.202.160@tcp [ 6919.136105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6919.193833] Lustre: server umount lustre-MDT0000 complete [ 6920.097383] LustreError: 453550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6920.102068] LustreError: 453550:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6920.404480] LustreError: 453531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788888212 with bad export cookie 3548578973185021526 [ 6920.407465] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6920.407766] LustreError: 453531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6920.529861] Lustre: server umount lustre-MDT0001 complete [ 6937.596341] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 6941.150121] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 6952.976083] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 13:24:05 (1788888245) [ 6958.995505] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 6960.107135] Key type lgssc unregistered [ 6968.480382] LNet: 460201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6968.482711] LNetError: 460201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7021.734321] LNet: Removed LNI 192.168.202.160@tcp [ 7022.032041] Key type .llcrypt unregistered [ 7022.032996] Key type ._llcrypt unregistered [ 7029.081485] Key type ._llcrypt registered [ 7029.083255] Key type .llcrypt registered [ 7029.127311] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 7029.599026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7029.613259] alg: No test for adler32 (adler32-zlib) [ 7030.469087] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 7030.549877] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 7032.128207] Key type lgssc registered [ 7032.540452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7034.037961] LNet: Added LNI 192.168.202.160@tcp100 [8/256/0/180] [ 7034.355309] LNet: Added LNI 192.168.202.160@tcp101 [8/256/0/180] [ 7034.681291] LNet: Added LNI 192.168.202.160@tcp102 [8/256/0/180] [ 7034.987302] LNet: Added LNI 192.168.202.160@tcp103 [8/256/0/180] [ 7035.283844] LNet: Added LNI 192.168.202.160@tcp104 [8/256/0/180] [ 7035.569039] LNet: Added LNI 192.168.202.160@tcp105 [8/256/0/180] [ 7035.862780] LNet: Added LNI 192.168.202.160@tcp106 [8/256/0/180] [ 7036.154059] LNet: Added LNI 192.168.202.160@tcp107 [8/256/0/180] [ 7036.452608] LNet: Added LNI 192.168.202.160@tcp108 [8/256/0/180] [ 7036.764736] LNet: Added LNI 192.168.202.160@tcp109 [8/256/0/180] [ 7037.066056] LNet: Added LNI 192.168.202.160@tcp110 [8/256/0/180] [ 7037.368742] LNet: Added LNI 192.168.202.160@tcp111 [8/256/0/180] [ 7037.658227] LNet: Added LNI 192.168.202.160@tcp112 [8/256/0/180] [ 7037.965880] LNet: Added LNI 192.168.202.160@tcp113 [8/256/0/180] [ 7038.275528] LNet: Added LNI 192.168.202.160@tcp114 [8/256/0/180] [ 7038.566755] LNet: Added LNI 192.168.202.160@tcp115 [8/256/0/180] [ 7038.885202] LNet: Added LNI 192.168.202.160@tcp116 [8/256/0/180] [ 7039.171061] LNet: Added LNI 192.168.202.160@tcp117 [8/256/0/180] [ 7039.457868] LNet: Added LNI 192.168.202.160@tcp118 [8/256/0/180] [ 7039.738539] LNet: Added LNI 192.168.202.160@tcp119 [8/256/0/180] [ 7040.028279] LNet: Added LNI 192.168.202.160@tcp120 [8/256/0/180] [ 7040.320270] LNet: Added LNI 192.168.202.160@tcp121 [8/256/0/180] [ 7040.616980] LNet: Added LNI 192.168.202.160@tcp122 [8/256/0/180] [ 7040.903924] LNet: Added LNI 192.168.202.160@tcp123 [8/256/0/180] [ 7041.193298] LNet: Added LNI 192.168.202.160@tcp124 [8/256/0/180] [ 7041.466943] LNet: Added LNI 192.168.202.160@tcp125 [8/256/0/180] [ 7041.751367] LNet: Added LNI 192.168.202.160@tcp126 [8/256/0/180] [ 7042.048042] LNet: Added LNI 192.168.202.160@tcp127 [8/256/0/180] [ 7042.358430] LNet: Added LNI 192.168.202.160@tcp128 [8/256/0/180] [ 7042.664213] LNet: Added LNI 192.168.202.160@tcp129 [8/256/0/180] [ 7042.949047] LNet: Added LNI 192.168.202.160@tcp130 [8/256/0/180] [ 7043.230507] LNet: Added LNI 192.168.202.160@tcp131 [8/256/0/180] [ 7043.518997] LNet: Added LNI 192.168.202.160@tcp132 [8/256/0/180] [ 7043.807179] LNet: Added LNI 192.168.202.160@tcp133 [8/256/0/180] [ 7044.095901] LNet: Added LNI 192.168.202.160@tcp134 [8/256/0/180] [ 7044.383550] LNet: Added LNI 192.168.202.160@tcp135 [8/256/0/180] [ 7046.441121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7046.446882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7047.478303] Lustre: 463195:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.160@tcp: can't add NID 192.168.202.160@tcp130: rc = -75 [ 7047.502354] LNetError: 463215:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.160 [ 7047.505285] LNetError: Protocol error connecting to 192.168.202.160@tcp119 at host 192.168.202.160:988: is it running a compatible version of Lustre? [ 7047.508175] LNetError: Skipped 4 previous similar messages [ 7047.549540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7047.558058] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7047.589235] Lustre: lustre-MDT0000: new disk, initializing [ 7047.607701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7047.613794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7047.968085] LNet: 6 local NIs in recovery (showing 6): 192.168.202.160@tcp125, 192.168.202.160@tcp127, 192.168.202.160@tcp128, 192.168.202.160@tcp126, 192.168.202.160@tcp129, 192.168.202.160@tcp130 [ 7048.546930] LNetError: 463218:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.160 [ 7048.550084] LNetError: 463218:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [ 7048.552344] LNetError: Protocol error connecting to 192.168.202.160@tcp101 at host 192.168.202.160:988: is it running a compatible version of Lustre? [ 7048.555383] LNetError: Skipped 6 previous similar messages [ 7048.814557] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7050.019211] LNetError: 463230:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.160 [ 7050.024320] LNetError: 463230:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [ 7050.027716] LNetError: Protocol error connecting to 192.168.202.160@tcp125 at host 192.168.202.160:988: is it running a compatible version of Lustre? [ 7050.030727] LNetError: Skipped 6 previous similar messages [ 7052.977070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7053.033102] Lustre: 463212:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7053.065259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7053.068095] Lustre: Skipped 1 previous similar message [ 7053.112714] Lustre: lustre-MDT0001: new disk, initializing [ 7053.123138] LNetError: 460657:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.160 [ 7053.126101] LNetError: Protocol error connecting to 192.168.202.160@tcp131 at host 192.168.202.160:988: is it running a compatible version of Lustre? [ 7053.127851] Lustre: 464016:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.160@tcp: can't add NID 192.168.202.160@tcp130: rc = -75 [ 7053.131432] Lustre: 464016:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7053.181306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7053.192730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7053.196369] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7054.560850] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7058.391159] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7060.258380] LNetError: 463216:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.160 [ 7060.263145] LNetError: 463216:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 13 previous similar messages [ 7060.269027] LNetError: Protocol error connecting to 192.168.202.160@tcp133 at host 192.168.202.160:988: is it running a compatible version of Lustre? [ 7060.274537] LNetError: Skipped 13 previous similar messages [ 7060.340184] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7062.654610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7062.772926] Lustre: lustre-OST0000: new disk, initializing [ 7062.774582] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7062.776997] Lustre: 465187:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7062.818625] Lustre: 465192:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.160@tcp: can't add NID 192.168.202.160@tcp130: rc = -75 [ 7062.821260] Lustre: 465192:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [ 7062.841219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7064.757720] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7068.408095] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7068.450216] LNetError: 463215:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.160 [ 7068.453234] LNetError: 463215:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [ 7068.457272] LNetError: Protocol error connecting to 192.168.202.160@tcp133 at host 192.168.202.160:988: is it running a compatible version of Lustre? [ 7068.462690] LNetError: Skipped 1 previous similar message [ 7070.830443] Lustre: server umount lustre-OST0000 complete [ 7071.230798] Lustre: 466195:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.160@tcp: can't add NID 192.168.202.160@tcp130: rc = -75 [ 7071.233383] Lustre: 466195:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7073.761064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7073.763308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7073.768900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7076.321274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7076.321728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7078.240953] Lustre: server umount lustre-MDT0000 complete [ 7079.496389] LustreError: 463257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788888371 with bad export cookie 4937370957216469065 [ 7079.499067] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7079.500499] LustreError: 463257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7079.620501] Lustre: server umount lustre-MDT0001 complete [ 7083.242230] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 7084.403455] Key type lgssc unregistered [ 7084.578401] LNet: 467093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7084.582776] LNetError: 467093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7084.586296] LNet: Removed LNI 192.168.202.160@tcp [ 7085.600151] LNet: Removed LNI 192.168.202.160@tcp100 [ 7085.601717] LNetError: 467093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7085.603775] LNetError: 467093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7085.606129] LNet: Removed LNI 192.168.202.160@tcp101 [ 7086.624152] LNet: Removed LNI 192.168.202.160@tcp102 [ 7086.625340] LNetError: 467093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7086.627148] LNetError: 467093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7086.629164] LNet: Removed LNI 192.168.202.160@tcp103 [ 7087.648213] LNet: Removed LNI 192.168.202.160@tcp104 [ 7088.672112] LNet: Removed LNI 192.168.202.160@tcp105 [ 7088.673349] LNetError: 467093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7088.676033] LNetError: 467093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7088.679418] LNet: Removed LNI 192.168.202.160@tcp106 [ 7089.696132] LNet: Removed LNI 192.168.202.160@tcp107 [ 7090.720100] LNet: Removed LNI 192.168.202.160@tcp108 [ 7091.744132] LNet: Removed LNI 192.168.202.160@tcp109 [ 7092.768112] LNet: Removed LNI 192.168.202.160@tcp110 [ 7092.769992] LNetError: 467093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7092.773042] LNetError: 467093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7092.775976] LNet: Removed LNI 192.168.202.160@tcp111 [ 7093.792117] LNet: Removed LNI 192.168.202.160@tcp112 [ 7094.816133] LNet: Removed LNI 192.168.202.160@tcp113 [ 7095.840110] LNet: Removed LNI 192.168.202.160@tcp114 [ 7096.864193] LNet: Removed LNI 192.168.202.160@tcp115 [ 7097.888172] LNet: Removed LNI 192.168.202.160@tcp116 [ 7098.912193] LNet: Removed LNI 192.168.202.160@tcp117 [ 7099.936193] LNet: Removed LNI 192.168.202.160@tcp118 [ 7100.960130] LNet: Removed LNI 192.168.202.160@tcp119 [ 7100.962171] LNetError: 467093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7100.965107] LNetError: 467093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7100.968874] LNet: Removed LNI 192.168.202.160@tcp120 [ 7101.984170] LNet: Removed LNI 192.168.202.160@tcp121 [ 7103.008151] LNet: Removed LNI 192.168.202.160@tcp122 [ 7104.032133] LNet: Removed LNI 192.168.202.160@tcp123 [ 7105.056168] LNet: Removed LNI 192.168.202.160@tcp124 [ 7105.057583] LNet: Removed LNI 192.168.202.160@tcp125 [ 7106.080169] LNet: Removed LNI 192.168.202.160@tcp126 [ 7107.104153] LNet: Removed LNI 192.168.202.160@tcp127 [ 7108.128144] LNet: Removed LNI 192.168.202.160@tcp128 [ 7109.152175] LNet: Removed LNI 192.168.202.160@tcp129 [ 7110.176131] LNet: Removed LNI 192.168.202.160@tcp130 [ 7111.200148] LNet: Removed LNI 192.168.202.160@tcp131 [ 7112.224152] LNet: Removed LNI 192.168.202.160@tcp132 [ 7112.225349] LNet: Removed LNI 192.168.202.160@tcp133 [ 7113.248149] LNet: Removed LNI 192.168.202.160@tcp134 [ 7114.278561] LNet: Removed LNI 192.168.202.160@tcp135 [ 7114.583289] Key type .llcrypt unregistered [ 7114.585213] Key type ._llcrypt unregistered [ 7121.711960] Key type ._llcrypt registered [ 7121.713021] Key type .llcrypt registered [ 7121.748392] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 7127.990593] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 7128.516950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7128.567712] alg: No test for adler32 (adler32-zlib) [ 7129.428253] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 7129.519717] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 7131.104207] Key type lgssc registered [ 7131.494802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7143.447800] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 13:27:15 (1788888435) [ 7149.613426] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 7150.694404] Key type lgssc unregistered [ 7150.838466] LNet: 471147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7150.841885] LNetError: 471147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7150.849411] LNet: Removed LNI 192.168.202.160@tcp [ 7151.163385] Key type .llcrypt unregistered [ 7151.165135] Key type ._llcrypt unregistered [ 7158.421382] Key type ._llcrypt registered [ 7158.422670] Key type .llcrypt registered [ 7158.465825] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 7158.833165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7158.881647] alg: No test for adler32 (adler32-zlib) [ 7159.754857] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 7159.863238] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 7161.448180] Key type lgssc registered [ 7161.865532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7165.245856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7165.250545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7166.329233] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7166.340179] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7166.379570] Lustre: lustre-MDT0000: new disk, initializing [ 7166.400340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7166.406074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7167.834254] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7171.791593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7171.827827] Lustre: 472332:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7171.847888] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7171.849349] Lustre: Skipped 1 previous similar message [ 7171.880462] Lustre: lustre-MDT0001: new disk, initializing [ 7171.907863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7171.918522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7171.922363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7173.236257] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7177.009745] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7178.867611] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7181.187083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7181.294322] Lustre: lustre-OST0000: new disk, initializing [ 7181.296080] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7181.299522] Lustre: 474301:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7181.329303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7183.342337] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7187.252103] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7188.481192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7188.483835] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7188.531688] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7188.636736] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing lnet_if_list [ 7190.440283] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7190.454777] LNet: Added LNI 192.168.202.160@tcp10 [8/256/0/180] [ 7192.508780] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7192.521589] LNet: Added LNI 192.168.202.160@tcp11 [8/256/0/180] [ 7193.979437] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7193.996300] LNet: Added LNI 192.168.202.160@tcp12 [8/256/0/180] [ 7195.444482] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7195.457504] LNet: Added LNI 192.168.202.160@tcp13 [8/256/0/180] [ 7196.980401] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7196.992087] LNet: Added LNI 192.168.202.160@tcp14 [8/256/0/180] [ 7198.540545] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7198.553862] LNet: Added LNI 192.168.202.160@tcp15 [8/256/0/180] [ 7200.103986] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7200.117384] LNet: Added LNI 192.168.202.160@tcp16 [8/256/0/180] [ 7201.712894] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7201.724095] LNet: Added LNI 192.168.202.160@tcp17 [8/256/0/180] [ 7203.410313] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7203.423544] LNet: Added LNI 192.168.202.160@tcp18 [8/256/0/180] [ 7205.177740] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7205.190449] LNet: Added LNI 192.168.202.160@tcp19 [8/256/0/180] [ 7206.860874] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7206.872558] LNet: Added LNI 192.168.202.160@tcp20 [8/256/0/180] [ 7208.588443] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7208.599686] LNet: Added LNI 192.168.202.160@tcp21 [8/256/0/180] [ 7210.447070] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7210.462468] LNet: Added LNI 192.168.202.160@tcp22 [8/256/0/180] [ 7212.286514] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7212.301097] LNet: Added LNI 192.168.202.160@tcp23 [8/256/0/180] [ 7214.212176] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7214.227558] LNet: Added LNI 192.168.202.160@tcp24 [8/256/0/180] [ 7216.150308] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7216.162296] LNet: Added LNI 192.168.202.160@tcp25 [8/256/0/180] [ 7218.157277] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7218.169611] LNet: Added LNI 192.168.202.160@tcp26 [8/256/0/180] [ 7220.106205] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7220.119722] LNet: Added LNI 192.168.202.160@tcp27 [8/256/0/180] [ 7222.174657] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7222.187673] LNet: Added LNI 192.168.202.160@tcp28 [8/256/0/180] [ 7224.275800] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7224.289316] LNet: Added LNI 192.168.202.160@tcp29 [8/256/0/180] [ 7226.504656] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7226.516636] LNet: Added LNI 192.168.202.160@tcp30 [8/256/0/180] [ 7228.763120] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7228.778739] LNet: Added LNI 192.168.202.160@tcp31 [8/256/0/180] [ 7231.060883] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7231.073192] LNet: Added LNI 192.168.202.160@tcp32 [8/256/0/180] [ 7233.333499] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7233.345703] LNet: Added LNI 192.168.202.160@tcp33 [8/256/0/180] [ 7235.716919] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7235.730481] LNet: Added LNI 192.168.202.160@tcp34 [8/256/0/180] [ 7238.236767] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7238.248690] LNet: Added LNI 192.168.202.160@tcp35 [8/256/0/180] [ 7240.724281] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7240.736605] LNet: Added LNI 192.168.202.160@tcp36 [8/256/0/180] [ 7243.222915] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7243.236211] LNet: Added LNI 192.168.202.160@tcp37 [8/256/0/180] [ 7245.602375] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7245.614388] LNet: Added LNI 192.168.202.160@tcp38 [8/256/0/180] [ 7247.835207] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7247.845802] LNet: Added LNI 192.168.202.160@tcp39 [8/256/0/180] [ 7250.144298] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7250.155861] LNet: Added LNI 192.168.202.160@tcp40 [8/256/0/180] [ 7252.421335] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7252.432849] LNet: Added LNI 192.168.202.160@tcp41 [8/256/0/180] [ 7254.718367] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7254.732553] LNet: Added LNI 192.168.202.160@tcp42 [8/256/0/180] [ 7257.171351] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7257.181220] LNet: Added LNI 192.168.202.160@tcp43 [8/256/0/180] [ 7259.538771] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7259.549359] LNet: Added LNI 192.168.202.160@tcp44 [8/256/0/180] [ 7261.959710] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7261.971153] LNet: Added LNI 192.168.202.160@tcp45 [8/256/0/180] [ 7264.322599] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7264.331933] LNet: Added LNI 192.168.202.160@tcp46 [8/256/0/180] [ 7266.638175] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7266.649675] LNet: Added LNI 192.168.202.160@tcp47 [8/256/0/180] [ 7269.270119] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7269.282536] LNet: Added LNI 192.168.202.160@tcp48 [8/256/0/180] [ 7272.020691] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7272.031233] LNet: Added LNI 192.168.202.160@tcp49 [8/256/0/180] [ 7274.790960] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7274.802027] LNet: Added LNI 192.168.202.160@tcp50 [8/256/0/180] [ 7284.704840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7284.708145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7284.714825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7285.730152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7285.731929] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7285.735979] Lustre: Skipped 1 previous similar message [ 7288.473643] Lustre: server umount lustre-MDT0000 complete [ 7289.666131] LustreError: 472369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788888581 with bad export cookie 5878704125068535343 [ 7289.669210] LustreError: 472369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7289.670134] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7289.785869] Lustre: server umount lustre-MDT0001 complete [ 7297.161696] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 7300.014527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7306.208166] Lustre: 471605:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788888582/real 1788888582] req@000000003e299721 x1875785496480512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788888598 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7306.216208] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7310.240135] Lustre: 471606:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788888586/real 1788888586] req@000000002a6ccf34 x1875785496480640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788888602 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7312.352124] Lustre: 471607:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788888588/real 1788888588] req@0000000034bbcba1 x1875785496480768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788888604 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7314.400524] LustreError: 471604:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cf2f4327 x1875785496481664/t0(0) o250->MGC192.168.202.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7314.548042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7315.552101] Lustre: 471607:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788888591/real 1788888591] req@00000000e59f2b72 x1875785496481024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788888607 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7315.884393] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7318.641414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7318.777167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7320.220834] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7321.056159] Lustre: 471605:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788888596/real 1788888596] req@00000000fb0bce1d x1875785496481152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788888612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7322.687608] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7324.133226] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7324.599986] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7325.159051] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7334.370084] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7334.372729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7334.377565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7334.379507] Lustre: Skipped 2 previous similar messages [ 7335.396622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7338.208911] Lustre: server umount lustre-OST0000 complete [ 7339.488756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7339.490980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7339.495143] Lustre: Skipped 1 previous similar message [ 7339.496885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7339.498814] Lustre: Skipped 2 previous similar messages [ 7345.620108] Lustre: server umount lustre-MDT0000 complete [ 7345.633032] LustreError: 498489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7345.638206] LustreError: 498489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7346.882106] LustreError: 472369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788888638 with bad export cookie 5878704125068537380 [ 7346.885698] LustreError: 472369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7346.885904] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7347.005378] Lustre: server umount lustre-MDT0001 complete [ 7350.748755] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 7351.820862] Key type lgssc unregistered [ 7351.989230] LNet: 501262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7351.993234] LNetError: 501262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7351.995376] LNet: Removed LNI 192.168.202.160@tcp [ 7351.996690] LNet: Removed LNI 192.168.202.160@tcp10 [ 7351.997705] LNet: Removed LNI 192.168.202.160@tcp11 [ 7351.998767] LNet: Removed LNI 192.168.202.160@tcp12 [ 7351.999914] LNet: Removed LNI 192.168.202.160@tcp13 [ 7352.001327] LNet: Removed LNI 192.168.202.160@tcp14 [ 7352.002978] LNet: Removed LNI 192.168.202.160@tcp15 [ 7352.004116] LNet: Removed LNI 192.168.202.160@tcp16 [ 7352.005171] LNet: Removed LNI 192.168.202.160@tcp17 [ 7352.006291] LNet: Removed LNI 192.168.202.160@tcp18 [ 7352.007416] LNet: Removed LNI 192.168.202.160@tcp19 [ 7352.008579] LNet: Removed LNI 192.168.202.160@tcp20 [ 7352.009703] LNet: Removed LNI 192.168.202.160@tcp21 [ 7352.010862] LNet: Removed LNI 192.168.202.160@tcp22 [ 7352.011965] LNet: Removed LNI 192.168.202.160@tcp23 [ 7352.013091] LNet: Removed LNI 192.168.202.160@tcp24 [ 7352.014197] LNet: Removed LNI 192.168.202.160@tcp25 [ 7352.015281] LNet: Removed LNI 192.168.202.160@tcp26 [ 7352.016319] LNet: Removed LNI 192.168.202.160@tcp27 [ 7352.017534] LNet: Removed LNI 192.168.202.160@tcp28 [ 7352.018599] LNet: Removed LNI 192.168.202.160@tcp29 [ 7352.019716] LNet: Removed LNI 192.168.202.160@tcp30 [ 7352.021053] LNet: Removed LNI 192.168.202.160@tcp31 [ 7352.023165] LNet: Removed LNI 192.168.202.160@tcp32 [ 7352.025309] LNet: Removed LNI 192.168.202.160@tcp33 [ 7352.026426] LNet: Removed LNI 192.168.202.160@tcp34 [ 7352.027911] LNet: Removed LNI 192.168.202.160@tcp35 [ 7352.029197] LNet: Removed LNI 192.168.202.160@tcp36 [ 7352.031069] LNet: Removed LNI 192.168.202.160@tcp37 [ 7352.032201] LNet: Removed LNI 192.168.202.160@tcp38 [ 7352.033620] LNet: Removed LNI 192.168.202.160@tcp39 [ 7352.034735] LNet: Removed LNI 192.168.202.160@tcp40 [ 7352.036950] LNet: Removed LNI 192.168.202.160@tcp41 [ 7352.038131] LNet: Removed LNI 192.168.202.160@tcp42 [ 7352.039755] LNet: Removed LNI 192.168.202.160@tcp43 [ 7352.041911] LNet: Removed LNI 192.168.202.160@tcp44 [ 7352.043616] LNet: Removed LNI 192.168.202.160@tcp45 [ 7352.044733] LNet: Removed LNI 192.168.202.160@tcp46 [ 7352.045840] LNet: Removed LNI 192.168.202.160@tcp47 [ 7352.046910] LNet: Removed LNI 192.168.202.160@tcp48 [ 7352.048118] LNet: Removed LNI 192.168.202.160@tcp49 [ 7352.055630] LNet: Removed LNI 192.168.202.160@tcp50 [ 7352.346860] Key type .llcrypt unregistered [ 7352.347962] Key type ._llcrypt unregistered [ 7358.717036] Key type ._llcrypt registered [ 7358.718823] Key type .llcrypt registered [ 7358.758524] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 7365.144459] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 7365.491642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7365.557243] alg: No test for adler32 (adler32-zlib) [ 7366.400972] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 7366.483792] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 7368.056148] Key type lgssc registered [ 7368.391369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7379.846495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7379.851893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7380.938915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7380.948705] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7380.991764] Lustre: lustre-MDT0000: new disk, initializing [ 7381.014226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7381.020388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7382.465808] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7386.645345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7386.687217] Lustre: 504427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7386.711691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7386.714136] Lustre: Skipped 1 previous similar message [ 7386.749461] Lustre: lustre-MDT0001: new disk, initializing [ 7386.785208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7386.798688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7386.803135] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7388.209041] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7391.973993] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7393.842728] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7396.412083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7396.490261] Lustre: *** cfs_fail_loc=90f, val=0*** [ 7396.521140] Lustre: lustre-OST0000: new disk, initializing [ 7396.522834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7396.525352] Lustre: 506447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7396.554434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7398.438449] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7401.978570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7401.982737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7402.009185] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7402.192704] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7407.074618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7407.075926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7407.076715] LustreError: Skipped 1 previous similar message [ 7407.083568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7407.087213] Lustre: Skipped 1 previous similar message [ 7410.774754] Lustre: server umount lustre-OST0000 complete [ 7412.193533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7412.194056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7412.199063] Lustre: Skipped 2 previous similar messages [ 7417.313331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7417.315147] Lustre: Skipped 2 previous similar messages [ 7418.138947] Lustre: server umount lustre-MDT0000 complete [ 7419.346193] LustreError: 504420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788888711 with bad export cookie 11664674230246464014 [ 7419.348598] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7419.349650] LustreError: 504420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7419.464436] Lustre: server umount lustre-MDT0001 complete [ 7423.195785] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 7424.257027] Key type lgssc unregistered [ 7424.390458] LNet: 508347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7424.393709] LNetError: 508347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7424.402291] LNet: Removed LNI 192.168.202.160@tcp [ 7424.733732] Key type .llcrypt unregistered [ 7424.734940] Key type ._llcrypt unregistered [ 7432.430974] Key type ._llcrypt registered [ 7432.432561] Key type .llcrypt registered [ 7432.480173] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 7433.026294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7433.039803] alg: No test for adler32 (adler32-zlib) [ 7433.895907] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 7433.992276] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 7435.576138] Key type lgssc registered [ 7435.963636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7438.693363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7438.700596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7439.882216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7441.273685] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7444.217270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7444.346442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7445.660955] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7448.134798] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7449.991914] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7452.214583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7452.329551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7454.286398] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7456.667392] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7470.699280] loop0: detected capacity change from 0 to 204800000 [ 7470.756092] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7470.761565] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7470.766145] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7472.784174] loop0: detected capacity change from 0 to 204800000 [ 7472.838773] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7472.842371] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7472.847380] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7474.583501] loop0: detected capacity change from 0 to 204800000 [ 7476.411523] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7476.464390] Lustre: fs15246-OST0000: new disk, initializing [ 7476.467220] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7476.470388] Lustre: 513201:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7476.488192] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7478.314155] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7482.099299] loop1: detected capacity change from 0 to 204800000 [ 7483.900462] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7483.935054] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7483.954905] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7483.992390] Lustre: fs15246-MDT0000: new disk, initializing [ 7484.039610] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7484.047483] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7484.091056] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7485.152738] Lustre: 514323:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000066920476 x1875785784785664/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788888793 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7485.307267] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7494.240162] LustreError: 514323:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 7494.243045] LustreError: 514323:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 7494.245728] LustreError: 514323:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 7496.164556] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7496.171378] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7498.907200] Lustre: server umount fs15246-MDT0000 complete [ 7501.286065] LustreError: 509585:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7501.290459] LustreError: 509585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7502.054444] Lustre: server umount fs15246-OST0000 complete [ 7505.890025] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7505.890433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7505.892641] LustreError: Skipped 1 previous similar message [ 7505.896673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7505.899375] Lustre: Skipped 1 previous similar message [ 7511.522593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7511.524730] Lustre: Skipped 3 previous similar messages [ 7511.580240] Lustre: server umount lustre-OST0000 complete [ 7516.129444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7516.132161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7516.136025] Lustre: Skipped 1 previous similar message [ 7516.137890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7521.761156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7521.763053] Lustre: Skipped 2 previous similar messages [ 7527.392063] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7527.451695] Lustre: server umount lustre-MDT0000 complete [ 7528.647109] LustreError: 509569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788888820 with bad export cookie 1696374724069930895 [ 7528.649332] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7528.651160] LustreError: 509569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7528.760300] Lustre: server umount lustre-MDT0001 complete [ 7532.578108] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 7533.555581] Key type lgssc unregistered [ 7533.672389] LNet: 517415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7533.675198] LNetError: 517415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7533.682290] LNet: Removed LNI 192.168.202.160@tcp [ 7533.988392] Key type .llcrypt unregistered [ 7533.989524] Key type ._llcrypt unregistered [ 7541.645688] Key type ._llcrypt registered [ 7541.646735] Key type .llcrypt registered [ 7541.680092] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 7542.054371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7542.079210] alg: No test for adler32 (adler32-zlib) [ 7542.927478] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 7543.016378] LNet: Added LNI 192.168.202.160@tcp [8/256/0/180] [ 7544.600104] Key type lgssc registered [ 7544.948639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7557.166530] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 7560.501191] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 7572.180866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7572.185553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7573.267357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7573.279274] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7573.318477] Lustre: lustre-MDT0000: new disk, initializing [ 7573.342122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7573.348190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7574.747100] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7578.873800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7578.919458] Lustre: 521845:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7578.943340] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7578.945346] Lustre: Skipped 1 previous similar message [ 7578.984688] Lustre: lustre-MDT0001: new disk, initializing [ 7579.021041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7579.033961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7579.037192] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7580.365315] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7584.080675] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7585.910990] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7588.131037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7588.229297] Lustre: lustre-OST0000: new disk, initializing [ 7588.230890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7588.233178] Lustre: 523813:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7588.260416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7590.165283] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_default_debug -1 all [ 7590.265600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7590.268670] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7590.300436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7593.905641] Lustre: DEBUG MARKER: oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7595.360498] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7595.442789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7596.887528] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7596.973198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7600.611735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7600.611905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7600.614673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7600.617699] Lustre: Skipped 1 previous similar message [ 7600.621133] Lustre: Skipped 1 previous similar message [ 7603.734654] Lustre: server umount lustre-OST0000 complete [ 7605.730051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7605.730638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7605.733174] Lustre: Skipped 1 previous similar message [ 7610.850769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7610.855310] Lustre: Skipped 2 previous similar messages [ 7611.105771] Lustre: server umount lustre-MDT0000 complete [ 7612.301142] LustreError: 521837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788888904 with bad export cookie 2908006602025324466 [ 7612.304425] LustreError: MGC192.168.202.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7612.304889] LustreError: 521837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7612.426381] Lustre: server umount lustre-MDT0001 complete [ 7620.269586] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing set_hostid [ 7623.573424] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing load_modules_local [ 7633.923723] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7383 sec ========== 13:35:26 (1788888926) [ 7634.491524] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:35:26 (1788888926) === [ 7635.754124] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:35:27 (1788888927) === [ 7648.989879] Lustre: DEBUG MARKER: oleg260-server.virtnet: executing unload_modules_local [ 7650.103853] Key type lgssc unregistered [ 7650.243465] LNet: 530442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7650.245949] LNetError: 530442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7650.255342] LNet: Removed LNI 192.168.202.160@tcp [ 7650.583926] Key type .llcrypt unregistered [ 7650.585520] Key type ._llcrypt unregistered